Data asset redundancy determination method and device, electronic equipment and storage medium

Abstract

The application provides a data asset redundancy judgment method, a data asset redundancy judgment device, an electronic device and a computer readable storage medium. The method comprises: standardizing data assets to form a plurality of data tables; calculating the unilateral redundancy of each two data tables in the plurality of data tables relative to each other; calculating the overall redundancy of the two data tables according to the unilateral redundancy; and determining that the two data tables have redundancy if the overall redundancy exceeds a redundancy threshold. According to the application, the redundancy of the data assets can be calculated on the basis of standardization, which is beneficial to improving the calculation efficiency of the redundancy, and the redundancy is judged from the two angles of unilateral and overall, which is beneficial to improving the judgment accuracy of the redundancy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data asset redundancy judgment method, a data asset redundancy judgment device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Data assets often contain redundancy, leading to a waste of computing and storage resources. Therefore, it is necessary to identify and process redundant data within data assets to improve their value and utilization efficiency. Techniques for redundancy assessment of data assets are known. However, most existing methods for assessing data asset redundancy rely on manual methods, failing to achieve standardized redundancy assessments or efficient and accurate assessments. Therefore, there is an urgent need in this field for a data asset redundancy assessment technology that can assess redundancy using a unified standard and with high accuracy. Summary of the Invention

[0003] Therefore, this application aims to provide a data asset redundancy judgment method, a data asset redundancy judgment device, an electronic device, and a computer-readable storage medium, which can achieve efficient, accurate, and standardized data asset redundancy judgment.

[0004] In one aspect, this application provides a method for determining data asset redundancy, comprising: standardizing the data asset to form multiple data tables; calculating the one-sided redundancy of each pair of data tables relative to each other; calculating the overall redundancy of the two data tables based on the one-sided redundancy; and determining that the two data tables are redundant if the overall redundancy exceeds a redundancy threshold.

[0005] According to a particular embodiment of this application, data assets are standardized to form multiple data tables, including associating data fields in the data assets with data fields in the data standard.

[0006] According to a specific embodiment of this application, associating fields in data assets with fields in data standards includes: associating dimension fields with dimension fields in data standards based on the lineage, name, and data similarity of dimension fields in data assets; and associating indicator fields with indicator fields in data standards based on the lineage and name of indicator fields in data assets.

[0007] According to a particular embodiment of this application, the two data tables include a first table and a second table. Calculating the one-sided redundancy of the two data tables relative to each other for each pair of data tables includes: calculating a first redundancy of the first table relative to the second table based on the number of data fields in the first table that can be calculated from data fields in the second table; and calculating a second redundancy of the second table relative to the first table based on the number of data fields in the second table that can be calculated from data fields in the first table.

[0008] According to a particular embodiment of this application, the overall redundancy of two data tables is calculated based on the unilateral redundancy, including: calculating the overall redundancy of the first and second tables based on the first and second redundancy and the degree of kinship between the first and second tables.

[0009] According to a particular embodiment of this application, when the first and second tables share a common kinship source table, the degree of kinship is determined based on the kinship hierarchy of the first and second tables relative to the kinship source table; when the first and second tables do not share a common kinship source table, the degree of kinship is a fixed value.

[0010] According to a specific embodiment of this application, the data asset redundancy determination method further includes: if the overall redundancy does not exceed the redundancy threshold, then for each data table in the multiple data tables, determining whether the time during which the data table has not been used exceeds a first time threshold, and / or determining whether the time during which the data table has not been updated exceeds a second time threshold, and / or determining whether the time during which the data quality in the data table does not meet the standard exceeds a third time threshold; if the time during which the data table has not been used exceeds the first time threshold, and / or the time during which the data table has not been updated exceeds the second time threshold, and / or the time during which the data quality in the data table does not meet the standard exceeds the third time threshold, then the data table is determined to be redundant.

[0011] On the other hand, this application provides a data asset redundancy determination device, comprising: a standardization module for standardizing data assets to form multiple data tables; a first calculation module for calculating the one-sided redundancy of two data tables relative to each other for every two data tables in the multiple data tables; a second calculation module for calculating the overall redundancy of the two data tables based on the one-sided redundancy; and a determination module for determining that the two data tables are redundant when the overall redundancy exceeds a redundancy threshold.

[0012] On the other hand, this application provides an electronic device, including: a processor; a memory; and an application program, the application program being stored in the memory and configured to be executed by the processor, the application program including instructions for executing any of the data asset redundancy determination methods provided in this application.

[0013] On the other hand, this application provides a computer-readable storage medium storing a computer program for executing any of the data asset redundancy determination methods provided in this application.

[0014] According to the data asset redundancy judgment method, data asset redundancy judgment device, electronic device and computer-readable storage medium of this application, by first standardizing data assets, a basis is provided for subsequent intelligent automatic judgment of whether redundancy exists. Furthermore, by combining judgment methods from a single perspective and a holistic perspective, it is beneficial to achieve more accurate judgment of data asset redundancy. Attached Figure Description

[0015] The specific embodiments of this application are described in detail below with reference to the accompanying drawings, wherein:

[0016] Figure 1 A flowchart illustrating a data asset redundancy determination method according to an embodiment of this application is shown.

[0017] Figure 2 A flowchart illustrating a data asset redundancy determination method according to another embodiment of this application is shown;

[0018] Figure 3 This is a schematic diagram of the structure of a data asset redundancy determination device according to an embodiment of this application;

[0019] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0020] To enable those skilled in the art to more clearly understand the concepts and ideas of this application, the application is described in detail below with reference to specific embodiments. It should be understood that the embodiments given herein are only a part of all possible embodiments of this application. After reading the specification of this application, those skilled in the art are capable of making improvements, modifications, or substitutions to parts or the entirety of the following embodiments, and such improvements, modifications, or substitutions are also included within the scope of protection claimed in this application.

[0021] In this document, the terms "one," "an," and other similar words are not intended to indicate that only one of the described things exists, but rather that the description refers only to one of the described things, which may have one or more. In this document, the terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" means that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meanings of the terms "comprising," "including," and other similar words should be considered open-ended, not closed-ended. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.

[0022] In this document, the terms "first," "second," and other similar terms are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. In this document, the terms "embodiment," "this embodiment," "an embodiment," or "an example" do not indicate that the description applies only to one specific embodiment, but rather that such description may also be applicable to one or more other embodiments. Those skilled in the art will understand that any description made herein with respect to one embodiment can be substituted, combined, or otherwise combined with the descriptions in one or more other embodiments, and the new embodiments resulting from such substitutions, combinations, or other combinations are readily conceived by those skilled in the art and fall within the scope of protection of this application.

[0023] In various embodiments of this application, data assets can refer to assets or rights formed from data that have economic value. For example, data assets can refer to data resources recorded in a physical or electronic manner that are owned or controlled by an individual or enterprise and can bring future economic benefits to the enterprise. For example, data assets can refer to datasets in cyberspace that have data ownership (exploration rights, usage rights, ownership), are valuable, measurable, and readable.

[0024] In the various embodiments of this application, data asset redundancy can refer to the duplication between data, or the phenomenon of the same data being stored in different data files. Increasing data independence and reducing data redundancy are prerequisites for the success of information resource management and large-scale information systems. Data redundancy can hinder the integrity of data in the database and also cause a waste of storage space. Minimizing data redundancy is one of the main goals of database design.

[0025] In the field of data asset governance, intelligent data asset redundancy analysis methods are needed to determine whether data assets are redundant and whether they require governance to ensure their health. Currently, there are no specific redundancy analysis standards, and data asset platforms lack redundancy-specific governance mechanisms, relying primarily on manual judgment.

[0026] Some embodiments of this application provide a method for intelligent data asset redundancy analysis, which enables the data asset platform to manage redundancy models, reduce system resource consumption, lower operation and maintenance costs, reduce user choice problems, and prevent users from having multiple data assets to choose from with similar effects, but rather ensuring that each need has an optimal data asset to choose from.

[0027] Some embodiments of this application include the following three aspects: 1) Dimensional standardization, using AI algorithms and rules to standardize and correlate historical data with dimensions and indicators; 2) Redundancy analysis from a modeling perspective, defining redundancy expressions for data assets from a modeling perspective, calculating redundancy for each pair of data assets, and outputting whether the data assets are redundant based on the redundancy threshold; 3) Redundancy analysis from a value perspective, performing redundancy analysis from the perspective of the value generated by the data assets, determining whether the data assets generate value, and performing analysis on data assets that have been unused for a long time, have no data for a long time, and have data quality that has been substandard for a long time.

[0028] The key to some embodiments of this application lies in using AI algorithms and rules to standardize the dimensions and indicators of data assets, defining a data asset redundancy calculation formula from a modeling perspective, calculating the redundancy of each pair of data assets based on the data asset standardization, and defining data asset redundancy rules from a value perspective to confirm the redundancy judgment of data assets from the perspective of generating value.

[0029] The technical advantages of some embodiments of this application are as follows: 1) The dimensions and indicators of data assets are standardized by combining AI algorithms and rules; 2) A data asset redundancy calculation expression is defined from the modeling perspective, and redundancy calculation is performed on each pair of data assets; 3) Data asset redundancy rules are defined from the perspective of the value of data assets, thereby eliminating data assets that do not generate value.

[0030] Figure 1 A flowchart illustrating a data asset redundancy determination method according to an embodiment of this application is shown.

[0031] According to this embodiment, the data asset redundancy judgment method includes steps S110 to S140, and each step is described in detail below.

[0032] S110. Standardize the data assets to form multiple data tables.

[0033] In this embodiment, standardization can refer to an operation performed on data assets, the purpose of which is to give data assets a unified and standardized name, format and / or form, so as to facilitate the determination of whether data, fields and data tables are redundant.

[0034] In this embodiment, a data table can refer to a data table that already exists in the data asset or is formed after standardization, used to record and display data. Data can exist in various forms within a data asset, and the data table format is commonly used as one such form, possessing broad applicability.

[0035] As an example, a specific implementation of S110 could be to associate data fields in data assets with data fields in data standards.

[0036] In this embodiment, data fields can refer to fields in a data table (usually data represented by a column), including dimension fields and metric fields. In this embodiment, a dimension can refer to the perspective from which something is described, used to illustrate the data, i.e., the descriptive attributes or characteristics of an object. For example, year is a time dimension, Beijing is a geographical dimension, and monthly active users are a business dimension. A metric can refer to a numerical value or proportion, obtained through some calculation, used to measure business performance, such as positive review count, positive review rate, or growth rate.

[0037] In this embodiment, a data standard can refer to a reference object needed for standardizing and regulating data. In some embodiments, data standards include dimensional data standards and indicator data standards. Dimensional data standards can be the specific content of some pre-defined dimensional fields. For example, for the dimension of region, the dimensional data standard can include the names of the 34 provincial-level administrative regions in China. Indicator data standards can be pre-defined rules, formats, fixed forms, and other standard specifications for indicator data, used to standardize indicator data.

[0038] Specifically, through data standard management, the standards for the dimensions and indicators of all data assets are managed, and conversion rules between convertible dimensions are defined (e.g., the mobile phone number dimension can be converted into a mobile phone number segment dimension by truncating the first 7). Calculation rules between related indicators are defined, and then a master data management platform and an indicator management platform are established to standardize the dimensions and indicators of data assets, and to construct table-level and field-level lineage relationships for data assets.

[0039] As an example, S110 can be implemented by associating dimension fields with dimension fields in the data standard based on the lineage, name, and data similarity of dimension fields in the data asset.

[0040] Specifically, by using lineage, AI algorithms, and rules, the dimensional fields of data assets are linked to the master data (i.e., the dimensional data standards in the data standards), thus standardizing the dimensions of data assets. First, it is determined whether the lineage field B of data dimension field A exists and is already linked to master data C. If it exists, it is directly linked to master data C if direct pass-through is desired. If a transformation rule exists, the master data is searched for the master data D corresponding to the transformation rule of master data C, and then dimension field A is linked to master data D. If the corresponding master data cannot be found through lineage, then the master data that meets the naming rules (such as regular expressions, prefixes, suffixes, inclusion rules, etc.) is searched for. Then, the dimensional data of the data assets (such as data of a certain field: [a1, a2, a3, ...]) and these matching master data (such as data of the community master data: [community a, community b, ...]) are extracted to form a one-dimensional matrix. The similarity of the field content is analyzed by AI algorithms such as cosine similarity and edit distance. The similarity between the data asset dimension and the master data (range 0%-100%) is output. If the similarity is greater than the threshold (such as greater than >90%), then the data asset dimension and the master data are associated.

[0041] As an example, S110 can be implemented by associating indicator fields with indicator fields in data standards based on the lineage and name of indicator fields in data assets.

[0042] Specifically, by linking the indicator fields and indicators (i.e., indicator data standards in data standards) of data assets through lineage and rules, the indicators of data assets are standardized. First, it is determined whether the lineage field of the data indicator field has been associated with an indicator. If it has, then according to the calculation rules, the field is associated with the indicator corresponding to the calculation method of the lineage field indicator. If it has not, then according to the rules for the indicator names configured in the data indicator standard (such as regular expressions, prefixes, suffixes, inclusion rules, etc.), the indicator fields of the data assets are matched with the naming rules to find the matching indicators. Then, the matching indicators are verified according to the indicator range. If multiple matches still exist, then manual association is performed to ultimately achieve the goal of indicator standardization.

[0043] For example, a standardized user traffic table has now been developed, which includes dimensions TIME (hourly time), PHONE (phone number), and AREA_ID (city ID), and metrics UL_BYTES (uplink traffic) and DL_BYTES (downlink traffic). When accessing the user traffic table, data standards are constructed on the data standard platform for mobile phone numbers (names containing PHONE, length=11) and cities (names containing AREA), as well as number segments (name is SEGMENT, conversion rule substr(PHONE, 0, 7), data specification length=7). Original indicator data standards for uplink traffic (name is UL_BYTES, >=0) and downlink traffic (name is DL_BYTES, data specification >=0) are constructed, along with a derived indicator for total traffic (name is TOTAL_BYTES, algorithm UL_BYTES+DL_BYTES, data specification >=0). On the master data platform, data from each dimension of the user traffic table is extracted and deduplicated to generate master data for mobile phone numbers, number segments, and cities. On the indicator platform, original indicators such as uplink traffic, downlink traffic, and online duration are defined, and a derived indicator for total traffic is also defined.

[0044] The current standardized number segment traffic table includes TIME, SEGMENT, AREA_ID, and indicators UL_BYTES, DL_BYTES, and TOTAL_BYTES.

[0045] Assuming the number segment traffic table has a lineage originating from the user traffic table, the following derivations are made based on this lineage: SEGMENT originates from the mobile phone numbers in the user traffic table, and the conversion rule is substr(PHONE, 0, 7), so SEGMENT is associated with the number segments in the data standard; AREA_ID directly originates from the city in the user traffic table, so it is associated with the city in the data standard; UL_BYTES and DL_BYTES are derived from the SUM calculation of uplink and downlink traffic in the user traffic table, so they are directly associated with the uplink and downlink traffic in the data standard; TOTAL_BYTES is calculated by adding UL_BYTES and DL_BYTES, and is associated with the total traffic in the data standard based on the calculation expression. This completes the standardization of the table.

[0046] Assuming the number segment traffic table has no lineage, matching is performed by name in the data standard. The SEGMENT field searches for data standard objects that conform to the naming conventions within the data standard. If a SEGMENT conforms to the number segment naming conventions, it verifies whether the SEGMENT data conforms to the number segment data specifications (length=7). If it does, since the number segment is a dimension, similarity calculation is performed with the number segment data in the master data. The SEGMENT data in the number segment traffic table is extracted and deduplicated, and its similarity to the data in the master data number segment is calculated using AI algorithms such as cosine similarity and edit distance. If the similarity between the two datasets is 95.7%, the SEGMENT field is associated with the number segment in the data standard. The AREA_ID is associated with the city / region in the data standard, following the same standardization method as the SEGMENT. The UL_BYTES field searches for data standard objects that conform to the naming convention within the data standard. It then retrieves UL_BYTES objects that conform to the uplink traffic naming convention and checks if the UL_BYTES data conforms to the uplink traffic data specification (>=0). If it does, since uplink traffic is an indicator, UL_BYTES is considered associated with the uplink traffic of the data standard. DL_BYTES is associated with the downlink traffic of the data standard using the same standardization method as UL_BYTES. TOTAL_BYTES is calculated by adding UL_BYTES and DL_BYTES, and then associated with the total traffic of the data standard based on a calculation expression. If it cannot be associated with the object management within the data standard, manual intervention is required.

[0047] S120. For each pair of data tables in multiple data tables, calculate the one-sided redundancy of the two data tables relative to each other.

[0048] In this embodiment, one-sided redundancy refers to the redundancy calculated for one of two data tables, with the purpose of determining the redundancy of that data table relative to the other data table. One-sided redundancy is calculated only for one of the two data tables, and the calculated one-sided redundancy for each of the two data tables is generally different.

[0049] In this embodiment, when the data in two tables has a time granularity, the two tables used to calculate redundancy can have the same time granularity. For example, a large amount of data used in communication services involves time concepts, such as user usage time, call connection rate, and complaint rate, which are updated and calculated according to certain time intervals. The update time interval is the time granularity, such as 5 minutes, 10 minutes, 1 hour, 1 day, 1 month, 1 year, etc. When comparing redundancy, calculations and comparisons can be performed between data tables with the same time granularity, for example, between a data table with a 5-minute time granularity and another data table with the same 5-minute time granularity. Redundancy may not be calculated between two data tables with different time granularities.

[0050] As an example, the two data tables may include a first table and a second table. The specific implementation of S120 may be to calculate the first redundancy of the first table relative to the second table based on the number of data fields in the first table that can be calculated from the data fields in the second table, and to calculate the second redundancy of the second table relative to the first table based on the number of data fields in the second table that can be calculated from the data fields in the first table.

[0051] Specifically, based on the standardization of the dimensions and indicators of data assets, we conduct a redundancy analysis from a modeling perspective. We define the two models for analysis as A and B, and then A and B, based on the standardization of dimensions and indicators, respectively, calculate the following indicators.

[0052] dimSomeCount: The number of identical dimensions. The number of identical dimensions between A and B.

[0053] dimTransformCount: The number of convertible dimensions. For example, to calculate the number of convertible dimensions of A, it is the number of dimensions that A can be transformed by the dimensions of B, except for the same dimensions.

[0054] dimOtherCount: The number of other dimensions, excluding the number of identical dimensions and the number of convertible dimensions.

[0055] indexSomeCount: The number of identical metrics, the number of identical metrics between A and B.

[0056] indexCalcCount: The number of calculable indicators. For example, to calculate the number of calculable indicators of A, it is the number of indicators of A that can be calculated by indicators of B, other than the same indicators.

[0057] indexOtherCount: The number of other metrics, excluding the number of identical metrics and the number of computable metrics.

[0058] Based on the above indicators, the one-sided redundancy can be calculated using the following formula (e.g., table A versus table B):

[0059]

[0060] Wherein, coefficient 'a' represents the weight coefficient of the convertible dimension / indicator with respect to the same dimension / indicator, ranging from 0.0 to 1.0, and can be >= 0.8; coefficient 'b' represents the influence coefficient of other dimensions / indicators on the same dimension / indicator, with a value >= 1.0, and can be >= 10.0; coefficient 'c' represents the weight coefficient of the indicator with respect to the dimension, ranging from 0.0 to 1.0, and can be <= 0.2. The weight coefficients and influence coefficients can be set based on experience, or adjusted and optimized according to actual circumstances.

[0061] S130. Calculate the overall redundancy of the two data tables based on the unilateral redundancy.

[0062] In one embodiment, overall redundancy can refer to the degree of redundancy between the two data tables as a whole. Overall redundancy is relative to both data tables. In other words, the value of the overall redundancy between the two data tables is generally unique.

[0063] As an example, S130 can be implemented by calculating the overall redundancy of the first and second tables based on the first and second redundancy and the lineage degree between the first and second tables.

[0064] Calculate the one-sided redundancy (aRedundancy) and bRedundancy of tables A and B respectively, and then calculate the overall redundancy by combining the kinship degree (boolRel):

[0065]

[0066] The coefficient d represents the weighting coefficient of blood relation for redundancy, with a value >1.0 and can be >=10.

[0067] As an example, when there is a common kinship table between the first and second tables, the degree of kinship is determined according to the kinship hierarchy of the first and second tables relative to the kinship table; when there is no common kinship table between the first and second tables, the degree of kinship is a fixed value.

[0068] Specifically, `boolRel` represents the degree of kinship between A and B. If A and B share a common kinship source table K, then let the kinship level from table A to table K be BR1 (e.g., K->A is 1, K->C->A is 2, and so on), and the kinship level from table B to this kinship source table be BR2 (hypothetical value). Then, `boolRel = max(BR1, BR2)`. If there is no common kinship source table, then `boolRel = Integer.MAX_VALUE`. In one embodiment, if there is no common kinship source table, then `boolRel` is a fixed value of 2147483647. The purpose of setting this value is to reduce a certain degree of redundancy in the calculation, because the redundancy is higher when there is a kinship than when there is no kinship.

[0069] S140. If the overall redundancy exceeds the redundancy threshold, then it is determined that there is redundancy in the two data tables.

[0070] Specifically, after calculating the redundancy between two data assets using the above expression, if the redundancy is greater than or equal to the redundancy threshold (e.g., 90%), we consider these two data assets to be redundant. Based on the higher redundancy on one side, the data asset with the higher redundancy is taken offline, and subsequently deleted or merged with its corresponding data asset. If the redundancy threshold is set high (>80%), redundancy analysis can be performed only on pairs of data assets with the same data lineage.

[0071] As an example, S140 further includes: if the overall redundancy does not exceed the redundancy threshold, then for each of the multiple data tables, it is determined whether the time during which the data table has not been used exceeds a first time threshold, and / or whether the time during which the data table has not been updated exceeds a second time threshold, and / or whether the time during which the data quality in the data table is substandard exceeds a third time threshold; if the time during which the data table has not been used exceeds the first time threshold, and / or the time during which the data table has not been updated exceeds the second time threshold, and / or the time during which the data quality in the data table is substandard exceeds the third time threshold, then the data table is determined to be redundant.

[0072] Specifically, data asset redundancy can be judged from a value perspective. If a data asset has not been used for a long period of time (such as 3 months), it is considered a redundant data asset that does not generate value and should be taken offline. If a data asset has no data or the data quality is substandard for a long period of time (such as 1 month), it is also considered a redundant asset and needs to be taken offline. Once there is data or the data quality meets the standards, it can be republished as a data asset.

[0073] Figure 2 A flowchart illustrating a data asset redundancy determination method according to another embodiment of this application is shown.

[0074] See Figure 2 First, create or import data assets, and then standardize the data assets.

[0075] Then, from a modeling perspective, the redundancy of the data assets is calculated to determine whether the data assets are duplicates. If so, the data asset is taken offline, and then the data assets are merged or modeled.

[0076] If not, further determine whether it is a duplicate data asset in terms of dimension. If so, remove the data asset from the database, and then merge or model the data assets.

[0077] If not, then launch the data asset and use it.

[0078] During use, the redundancy of the data asset can be calculated from a value perspective to determine whether it is redundant in terms of value. If so, the data asset is taken offline and then rectified. After rectification, the data asset is brought back online.

[0079] If not, then end the entire process.

[0080] Figure 3 A schematic diagram of a data asset redundancy determination device according to an embodiment of this application is shown.

[0081] In this embodiment, the data asset redundancy determination device 300 includes:

[0082] Standardization module 310 is used to standardize data assets and form multiple data tables;

[0083] The first calculation module 320 is used to calculate the one-sided redundancy of two data tables relative to each other for every two data tables in multiple data tables.

[0084] The second calculation module 330 is used to calculate the overall redundancy of the two data tables based on the unilateral redundancy.

[0085] The first determination module 340 is used to determine that two data tables have redundancy when the overall redundancy exceeds the redundancy threshold.

[0086] In one embodiment, the standardization module 310 is further configured to:

[0087] Associate data fields in data assets with data fields in data standards.

[0088] In one embodiment, the standardization module 310 is further configured to:

[0089] Based on the lineage, name, and data similarity of the dimension fields in the data assets, the dimension fields are associated with the dimension fields in the data standards.

[0090] Based on the lineage and name of the indicator fields in the data assets, the indicator fields are associated with the indicator fields in the data standards.

[0091] In one embodiment, the two data tables include first and second tables, and the first calculation module 320 is further configured to:

[0092] Calculate the first redundancy of the first table relative to the second table based on the number of data fields in the first table that can be obtained from the data fields in the second table.

[0093] The second redundancy of the second table relative to the first table is calculated based on the number of data fields in the second table that can be obtained from the data fields in the first table.

[0094] In one embodiment, the second computing module 330 is further configured to:

[0095] The overall redundancy of the first and second tables is calculated based on the first and second redundancy and the degree of kinship between the first and second tables.

[0096] In one embodiment, when there is a common kinship source table between the first and second tables, the degree of kinship is determined according to the kinship hierarchy of the first and second tables relative to the kinship source table; when there is no common kinship source table between the first and second tables, the degree of kinship is a fixed value.

[0097] In one embodiment, the data asset redundancy determination device 300 further includes:

[0098] The judgment module is used to determine, for each data table in multiple data tables, whether the time the data table has not been used exceeds the first time threshold, and / or whether the time the data table has not been updated exceeds the second time threshold, and / or whether the time the data quality in the data table has not met the standard exceeds the third time threshold when the overall redundancy does not exceed the redundancy threshold.

[0099] The second determining module is used to determine that a data table is redundant when the time a data table has not been used exceeds a first time threshold, and / or the time a data table has not been updated exceeds a second time threshold, and / or the time a data table has not met the data quality standards exceeds a third time threshold.

[0100] The following combination Figure 4 This application describes an electronic device according to an embodiment of the present application.

[0101] like Figure 4 As shown, the electronic device 400 includes one or more processors 410 and memory 420.

[0102] The processor 410 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0103] The memory 420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 410 may execute the program instructions to implement the data asset redundancy determination method of the various embodiments of this application described above, and / or other desired functions.

[0104] In one example, the electronic device 400 may also include an input device 430 and an output device 440, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0105] For example, the input device 430 may be a microphone or microphone array for capturing voice input signals; it may be a communication network connector for receiving the collected input signals from the cloud or other devices; and it may also include, for example, a keyboard, mouse, etc.

[0106] The output device 440 can output various information to the outside, including determined distance information, direction information, etc. The output device 440 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0107] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 400 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 400 may include any other suitable components depending on the specific application.

[0108] Embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the data asset redundancy determination method according to various embodiments of this application described above.

[0109] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0110] The concepts, principles, and ideas of this application have been described in detail above with reference to specific embodiments (including examples and instances). Those skilled in the art should understand that the embodiments of this application are not limited to those given above. After reading this application, those skilled in the art can make any possible improvements, substitutions, and equivalents to the steps, methods, apparatus, and components in the above embodiments, and such improvements, substitutions, and equivalents should be considered to fall within the scope of this application. The scope of protection of this application is limited to the claims.

Claims

1. A method for determining data asset redundancy, comprising: The data assets are standardized to form multiple data tables; wherein, the standardization operation includes associating data fields in the data assets with data fields in the data standard; the association operation includes: Based on the lineage, name, and data similarity of the dimension fields in the data assets, the dimension fields are associated with the dimension fields in the data standard; Based on the lineage and name of the indicator fields in the data assets, the indicator fields are associated with the indicator fields in the data standards; For every two data tables in the plurality of data tables, calculate the one-sided redundancy of the two data tables relative to each other; wherein, the two data tables include a first table and a second table, and calculating the one-sided redundancy of the two data tables relative to each other includes: Calculate the first redundancy of the first table relative to the second table based on the number of data fields in the first table that can be obtained from the data fields in the second table. Calculate the second redundancy of the second table relative to the first table based on the number of data fields in the second table that can be obtained from the data fields in the first table. The overall redundancy of the two data tables is calculated based on the unilateral redundancy and the lineage relationship between the two data tables; wherein, when the first and second tables have a common lineage source table, the lineage relationship is determined according to the lineage hierarchy of the first and second tables relative to the lineage source table; when the first and second tables do not have a common lineage source table, the lineage relationship is a fixed value. If the overall redundancy exceeds the redundancy threshold, then the two data tables are determined to be redundant.

2. The data asset redundancy judgment method according to claim 1 further includes: If the overall redundancy does not exceed the redundancy threshold, then for each of the multiple data tables, it is determined whether the time during which the data table has not been used exceeds a first time threshold, and / or whether the time during which the data table has not been updated exceeds a second time threshold, and / or whether the time during which the data quality in the data table does not meet the standard exceeds a third time threshold. If the data table is not used for more than the first time threshold, and / or the data table is not updated for more than the second time threshold, and / or the data quality in the data table is substandard for more than the third time threshold, then the data table is determined to be redundant.

3. A data asset redundancy determination device, comprising: A standardization module is used to perform standardization operations on the data assets to form multiple data tables; wherein, the standardization operation includes associating data fields in the data assets with data fields in the data standard; the association operation includes: Based on the lineage, name, and data similarity of the dimension fields in the data assets, the dimension fields are associated with the dimension fields in the data standard; Based on the lineage and name of the indicator fields in the data assets, the indicator fields are associated with the indicator fields in the data standards; A first calculation module is used to calculate the one-sided redundancy of two data tables relative to each other for every two data tables in the plurality of data tables; wherein, the two data tables include a first table and a second table, and calculating the one-sided redundancy of the two data tables relative to each other includes: Calculate the first redundancy of the first table relative to the second table based on the number of data fields in the first table that can be obtained from the data fields in the second table. The second redundancy of the second table relative to the first table is calculated based on the number of data fields in the second table that can be calculated from the data fields in the first table. The second calculation module is used to calculate the overall redundancy of the two data tables based on the unilateral redundancy and the lineage relationship between the two data tables. Wherein, when the first and second tables have a common lineage source table, the lineage relationship is determined according to the lineage hierarchy of the first and second tables relative to the lineage source table; when the first and second tables do not have a common lineage source table, the lineage relationship is a fixed value. The determination module is used to determine that the two data tables have redundancy when the overall redundancy exceeds the redundancy threshold.

4. An electronic device, comprising: processor; Memory; An application, stored in the memory and configured to be executed by the processor, the application including instructions for performing the data asset redundancy determination method according to any one of claims 1 to 2.

5. A computer-readable storage medium storing a computer program for executing the data asset redundancy determination method according to any one of claims 1 to 2.

Images