Data redundancy identification method and device, electronic equipment and storage medium

Abstract

This application provides a data redundancy identification method, apparatus, electronic device, and storage medium. The data redundancy identification method includes: obtaining fields to be identified from at least two data tables in a preset database; obtaining the support level of the fields to be identified; and identifying reasonable redundant fields among the fields to be identified based on the support level of the fields to be identified, wherein reasonable redundant fields refer to fields whose frequency of occurrence is higher than a preset value. This application implements the identification of reasonable redundant fields.

Description

Technical Field

[0001] This application belongs to the field of mobile communication technology, and in particular relates to a data redundancy identification method, device, electronic device and storage medium. Background Technology

[0002] A data warehouse is a strategic collection of all types of data that supports business development decisions. It is a single data store created for analytical reporting and decision support purposes. Data warehouses are characterized by large data volumes and diverse data types. Especially since the beginning of the 21st century, data volume has grown exponentially. Due to this increase in data volume, the cost of data management for enterprises has also increased rapidly. For enterprises, effectively managing data in a data warehouse, identifying metadata redundancy, and selectively backing up or deleting redundant tables to free up data warehouse storage resources are key means to ensure the rational use of data warehouse resources.

[0003] One approach to identifying redundant metadata in a data warehouse is to assess table redundancy based on the data manager's familiarity with the metadata. Another approach is to screen and compare all tables in the warehouse to check for redundancy. However, the first method requires data managers to be extremely familiar with all tables in the warehouse, placing very high demands on personnel. The second method involves iteratively comparing all tables in the warehouse, judging redundancy by comparing field similarities. However, this process can incorrectly classify highly similar but otherwise valid fields as redundant, leading to an inaccurate redundancy assessment and low redundancy identification accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a data redundancy identification method, apparatus, electronic device, and storage medium to solve the problem of low accuracy in data redundancy identification in related technologies.

[0005] In a first aspect, embodiments of this application provide a data redundancy identification method, the data redundancy identification method comprising:

[0006] Retrieve the fields to be identified from at least two data tables within a preset database;

[0007] Obtain the support level of the field to be identified;

[0008] Based on the support of the field to be identified, reasonable redundant fields in the field to be identified are identified, wherein the reasonable redundant fields refer to fields that appear more frequently than a preset value.

[0009] Secondly, embodiments of this application provide a data redundancy identification device, the redundancy identification device comprising:

[0010] The first acquisition module is used to acquire the fields to be identified from at least two data tables in a preset database;

[0011] The second acquisition module is used to acquire the support level of the field to be identified;

[0012] The redundancy identification module is used to identify reasonable redundant fields in the field to be identified based on the support of the field to be identified, wherein the reasonable redundant fields refer to fields that appear more frequently than a preset value.

[0013] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0016] In this application, the field to be identified is obtained from at least two data tables in a preset database, and the support of the field to be identified is obtained. Then, based on the support of the field to be identified, reasonable redundant fields in the field to be identified are identified. The reasonable redundant fields refer to fields that appear more frequently than a preset value. This realizes the identification of redundant fields that are highly similar but are reasonable in themselves, so that the identified redundancy is more in line with reality and the accuracy of redundancy identification is improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the data redundancy identification method in the embodiments of this application;

[0019] Figure 2 This is a schematic diagram of the module composition of the data redundancy device in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more.

[0023] The data redundancy identification method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0024] like Figure 1 The diagram shown is a flowchart of the data redundancy identification method provided in this application embodiment. The execution entity of this method can be a server, which can be an independent server or a server cluster composed of multiple servers. Furthermore, the server can be a server capable of program operation processing, such as a server for data redundancy identification. The method includes:

[0025] Step 101: Obtain the fields to be identified from at least two data tables in the preset database.

[0026] In this step, specifically, the fields to be identified can be obtained from at least two data tables in a preset database.

[0027] It should be noted that the default database can be a data warehouse; at least two data tables can be all the metadata tables in the default database.

[0028] The field to be identified can be field information from a data table.

[0029] Step 102: Obtain the support level of the field to be identified.

[0030] Support is a commonly used metric in data analysis or data mining to calculate the frequency of two related terms appearing together in a dataset. Support represents the number of times multiple potentially related fields appear in a dataset and their proportion of the total dataset, or the probability of several fields appearing together. If there are two fields A, B, C, and D whose relationship needs to be analyzed, the corresponding support formula is:

[0031] S(A,B,C,D)=P{ABCD}=P(ABCD) / P(Samples)

[0032] By obtaining the support level of the field to be identified, we can determine the probability of the field appearing.

[0033] Step 103: Based on the support of the field to be identified, identify the reasonable redundant fields in the field to be identified.

[0034] Among them, field redundancy refers to the frequency of a field appearing in various data tables; reasonable redundancy refers to fields that appear more frequently than a preset value.

[0035] Specifically, since support reflects the probability of a field appearing, the frequency of a field's appearance can be determined by its support, thus enabling the identification of reasonably redundant fields from the field to be identified.

[0036] Furthermore, if a field exists in most data tables and is not a common field, then that field is suspected of being redundant. This embodiment identifies reasonably redundant fields, enabling the filtering of potentially redundant fields based on these identified fields. This avoids incorrectly classifying highly similar but valid fields as redundant.

[0037] In this way, the embodiments of this application obtain the fields to be identified from at least two data tables in a preset database, obtain the support of the fields to be identified, and then identify reasonable redundant fields in the fields to be identified based on the support of the fields to be identified. The reasonable redundant fields refer to fields that appear more frequently than a preset value. This realizes the identification of redundant fields that are highly similar but are reasonable in themselves, thereby making the identified redundancy more in line with reality and improving the accuracy of redundancy identification.

[0038] In one implementation, when identifying reasonable redundant fields in the field to be identified based on the support of the field to be identified, target fields with support higher than a preset support threshold can be selected based on the support of the field to be identified; any two of the target fields are combined into a common field set, and the support of the common field set is calculated to select a target common field set with support higher than the preset support threshold; the fields in the target common field set are determined as the reasonable redundant fields.

[0039] By filtering through the above steps, reasonable redundant fields are obtained, ensuring that the frequency of occurrence of reasonable redundant fields is higher than the preset value.

[0040] Furthermore, in one implementation, after identifying reasonable redundant fields in the fields to be identified based on the support of the fields to be identified, it is also possible to obtain the fields to be calculated for redundancy other than the reasonable redundant fields in the fields to be identified; then, based on the fields to be calculated for redundancy in each data table and a preset reference index, the data redundancy between the first data table and the second data table is calculated, wherein the first data table and the second data table are any two of the at least two data tables.

[0041] Optionally, the reference indicators include at least one of the following: non-public field overlap, table domain overlap, storage cycle overlap, table layer overlap, source table overlap, and public field overlap; wherein the non-public fields corresponding to the non-public field overlap and the public fields corresponding to the public field overlap are selected from the fields to be redundantly calculated.

[0042] Optionally, when calculating the data redundancy between the first and second data tables based on the redundancy calculation fields in each data table and preset reference indicators, the weight value corresponding to each reference indicator can be calculated; then, the product of each reference indicator and its corresponding weight value is calculated, and the sum of all calculated products is determined as the data redundancy.

[0043] Specifically, the weight value corresponding to each reference indicator is the ratio of the weight score of the reference indicator to the weight scores of all reference indicators. The weight score of the reference indicator is calculated based on the importance of the reference indicator among all reference indicators.

[0044] For example, when calculating the weight score of reference indicators Ai and Aj, if reference indicator Ai is more important than reference indicator Aj, then reference indicator Ai gets 1 point; if they are equally important, reference indicator Ai gets 0.5 points; if reference indicator Aj is more important than reference indicator Ai, then reference indicator Ai gets 0 points. In this manner, the weight score of each reference indicator can be calculated, thus obtaining the weight value corresponding to each reference indicator.

[0045] In this way, based on the fields to be calculated for redundancy in each data table and the preset reference indicators, the data redundancy between the first data table and the second data table is calculated. This realizes that redundancy calculation is performed after removing reasonably redundant fields, which improves the accuracy of redundancy identification.

[0046] The following is a detailed description of this embodiment.

[0047] Optionally, the data redundancy identification process includes the following steps:

[0048] Step 1: Start the data redundancy identification task.

[0049] Step 2: Read all metadata table information in the preset database (e.g., data warehouse) and identify reasonable redundant fields.

[0050] Specifically, identifying reasonable redundant fields can avoid incorrectly classifying fields with high similarity but which are reasonable as redundant fields, thereby avoiding the problem of the identified redundancy not matching reality and the redundancy identification being inaccurate.

[0051] For example, databases typically contain a large number of common attributes, such as city, district / county, user, ID card number, location, and grid attributes. These common attributes usually exist in every data table, and a large number of common attributes can lead to redundant identification results that do not match reality. For example, as shown in the table below:

[0052]

[0053]

[0054] As shown in the table above, Tables 1-3 all contain the fields user_id and bill_id. Based on field overlap calculations, all three tables are redundant. However, in reality, these three tables store data from different business processes and should not be identified as redundant data. The correct identification result should be that only Table 4 is truly redundant.

[0055] In addition, the steps to identify reasonably redundant fields may include:

[0056] 1. Obtain the fields to be identified and their corresponding support levels in each data table.

[0057] 2. Determine the support threshold required to filter reasonably redundant fields.

[0058] 3. Delete fields with support below the support threshold to obtain target fields with support above the preset support threshold in the fields to be identified. Combine any two of the target fields into a common field set and calculate the support of the common field set. Filter to obtain the target common field set with support above the preset support threshold.

[0059] 4. If an itemset is a common set of fields, then all its subsets are also common fields. That is, if {a, b, c} is a common set of fields, then {a, b} and {a, c} must also be common sets of fields. Continue this process iteratively until all field sets are iteratively filtered, until an N-itemset of fields is generated, making it impossible to generate an (N+1)-itemset that satisfies the minimum support requirement, such as... Figure 2 As shown.

[0060] 5. Identify the fields in the target common fields as reasonably redundant fields.

[0061] In addition, specifically, the support of subsets in the N-field set can be generated through the above calculations to obtain the correlation between fields, thereby finding reasonable redundant fields and avoiding misjudgment of redundant fields.

[0062] Step 3: Based on the results of Step 2, calculate the redundancy of fields other than those with reasonable redundancy. Details are as follows:

[0063] In the process of redundancy calculation, the redundancy of the data table is mainly determined by the overlap of various reference indicators and the weight value corresponding to each reference indicator. In this embodiment, six types of reference indicators for calculating redundancy are provided, along with a method for calculating indicator weights based on element information.

[0064] Specifically, the reference metrics may include at least one of the following: non-common field overlap, table domain overlap, storage cycle overlap, table layer overlap, source table overlap, and common field overlap. The following explanation uses an example where all of the above are included as reference metrics.

[0065] Specifically, for reference indicators Ai and Aj, when calculating the weight score of the reference indicators, if reference indicator Ai is more important than reference indicator Aj, then reference indicator Ai receives 1 point; if they are equally important, reference indicator Ai receives 0.5 points; if reference indicator Aj is more important than reference indicator Ai, then reference indicator Ai receives 0 points. This process is repeated to calculate the weight score of each reference indicator. The ratio of the weight score of a reference indicator to the total weight scores of all reference indicators is then used to determine the weight value of that reference indicator.

[0066] For example, the weight values ​​for the above six reference indicators are shown in the table below:

[0067]

[0068]

[0069] For example, taking the overlap of non-public fields as an example, the weight value of the overlap of non-public fields is calculated as follows: the weight value of the overlap of non-public fields is equal to the sum of the weight scores of the overlap of non-public fields (0.5+1+1+1+1+1=5.5) divided by the sum of all weight scores in the table above (19). The weight value of the overlap of non-public fields is calculated to be 0.2895.

[0070] Then, the product of each reference indicator and its corresponding weight value can be calculated, and the sum of all the calculated products is determined as the data redundancy:

[0071] For example, the data redundancy between the first data table and the second data table = non-common field overlap × corresponding weight value + table domain overlap × corresponding weight value + storage cycle overlap × corresponding weight value + table layer overlap × corresponding weight value + source table overlap × corresponding weight value + common field overlap × corresponding weight value.

[0072] In addition, the values ​​of each reference indicator are calculated as follows:

[0073] Non-common field overlap = (number of redundant fields between the first and second data tables) / total number of fields in the first data table.

[0074] The overlap of the two tables refers to whether they are in the same domain. The overlap is a binary result; where 1 indicates that they are in the same domain and 0 indicates that they are not in the same domain.

[0075] Storage cycle overlap refers to whether the two tables being compared belong to the same storage cycle. The overlap is a binary result; where 1 indicates that they belong to the same storage cycle and 0 indicates that they belong to the same storage cycle.

[0076] The overlap of the two tables is determined as follows: between the two tables being compared, it is determined whether they are in the same model level. The overlap is a binary result; where 1 indicates that they are in the same model level and 0 indicates that they are in the same model level.

[0077] The overlap of the source tables is determined as follows: between two tables being compared, it is determined whether both are extracted from the same source table. The overlap is a binary result; where 1 indicates that they are extracted from the same source table, and 0 indicates that they are not extracted from the same source table.

[0078] The overlap of common fields is calculated between the two tables being compared, with a maximum overlap of 1.

[0079] For example, suppose the values ​​of the reference indicators are shown in the table below:

[0080]

[0081] Based on the weight values ​​of each reference indicator calculated above, the data redundancy between the first data table and the second data table is 0.5×0.2895+0×0.2368+0×0.1842+0.5×0.1842+1×0.0789+0×0.0263=0.31.

[0082] In this way, this embodiment can identify reasonable redundant fields in the data table, and perform redundancy calculation after removing reasonable redundant fields, thereby improving the accuracy of redundancy identification. In addition, redundancy calculation is performed by referring to indicators, which further improves the accuracy of redundancy identification.

[0083] It should be noted that the data redundancy identification method provided in this application embodiment can be executed by a data redundancy identification device, or a control module within that data redundancy identification device for executing the data redundancy identification method. This application embodiment uses the execution of the data redundancy identification method by a data redundancy identification device as an example to illustrate the data redundancy identification device provided in this application embodiment.

[0084] like Figure 2 As shown, the device includes:

[0085] The first acquisition module 201 is used to acquire the fields to be identified from at least two data tables in a preset database.

[0086] The second acquisition module 202 is used to acquire the support level of the field to be identified;

[0087] The redundancy identification module 203 is used to identify reasonable redundant fields in the field to be identified based on the support of the field to be identified, wherein the reasonable redundant fields refer to fields that appear more frequently than a preset value.

[0088] In one implementation, the redundancy identification module is used to: filter target fields in the field to be identified whose support is higher than a preset support threshold based on the support of the field to be identified; form a common field set by any two of the target fields, calculate the support of the common field set, filter target common field sets in the common field set whose support is higher than the preset support threshold; and determine the fields in the target common field set as the reasonable redundant fields.

[0089] In one implementation, the data redundancy identification device further includes: a redundancy calculation module, used to obtain the fields to be calculated for redundancy in the fields to be identified, excluding the reasonable redundancy fields; and to calculate the data redundancy between the first data table and the second data table based on the fields to be calculated for redundancy in each data table and a preset reference index, wherein the first data table and the second data table are any two of the at least two data tables.

[0090] In one implementation, the reference indicators include at least one of the following: non-public field overlap, table domain overlap, storage cycle overlap, table layer overlap, source table overlap, and public field overlap; wherein the non-public field corresponding to the non-public field overlap and the public field corresponding to the public field overlap are selected from the fields to be redundantly calculated.

[0091] In one implementation, the redundancy calculation module is used to calculate the weight value corresponding to each reference indicator, wherein the weight value corresponding to each reference indicator is the ratio of the weight score of the reference indicator to the weight scores of all reference indicators, and the weight score of the reference indicator is calculated based on the importance of the reference indicator among all reference indicators; calculate the product of each reference indicator and its corresponding weight value, and determine the sum of all calculated products as the data redundancy.

[0092] The data redundancy identification device provided in this embodiment obtains the fields to be identified from at least two data tables in a preset database through a first acquisition module, and obtains the support of the fields to be identified through a second acquisition module. Then, based on the support of the fields to be identified, the redundancy identification module identifies reasonable redundant fields in the fields to be identified. The reasonable redundant fields refer to fields that appear more frequently than a preset value. This realizes the identification of fields with high similarity but which are reasonable in themselves, thereby making the identified redundancy more in line with reality and improving the accuracy of redundancy identification.

[0093] It should be noted that the data redundancy identification device provided in the above embodiments can realize all the method steps and beneficial effects of the above data redundancy identification method embodiments. In order to avoid repetition, the same method steps and beneficial effects as those in the above method embodiments will not be described again here.

[0094] Corresponding to the data redundancy identification method provided in the above embodiments, based on the same technical concept, this application also provides an electronic device for performing the above data redundancy identification method. Figure 3 This is a schematic diagram of the structure of an electronic device to implement various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call a computer program stored in the memory 330 and executable on the processor 310 to perform the following steps:

[0095] Retrieve the fields to be identified from at least two data tables within a preset database;

[0096] Obtain the support level of the field to be identified;

[0097] Based on the support of the field to be identified, reasonable redundant fields in the field to be identified are identified, wherein the reasonable redundant fields refer to fields that appear more frequently than a preset value.

[0098] In one implementation, identifying reasonable redundant fields in the field to be identified based on the support of the field to be identified includes: filtering target fields in the field to be identified whose support is higher than a preset support threshold based on the support of the field to be identified; forming a common field set from any two of the target fields, calculating the support of the common field set, filtering target common field sets whose support is higher than the preset support threshold; and determining the fields in the target common field set as the reasonable redundant fields.

[0099] In one implementation, after identifying reasonable redundant fields in the field to be identified based on the support of the field to be identified, the method further includes: obtaining redundant calculation fields in the field to be identified other than the reasonable redundant fields; calculating the data redundancy between the first data table and the second data table based on the redundant calculation fields in each data table and a preset reference index, wherein the first data table and the second data table are any two of the at least two data tables.

[0100] In one implementation, the reference indicators include at least one of the following: non-public field overlap, table domain overlap, storage cycle overlap, table layer overlap, source table overlap, and public field overlap; wherein the non-public field corresponding to the non-public field overlap and the public field corresponding to the public field overlap are selected from the fields to be redundantly calculated.

[0101] In one implementation, calculating the data redundancy between the first data table and the second data table based on the redundancy calculation field in each data table and a preset reference indicator includes: calculating the weight value corresponding to each reference indicator, wherein the weight value corresponding to each reference indicator is the ratio of the weight score of the reference indicator to the weight scores of all reference indicators, and the weight score of the reference indicator is calculated based on the importance of the reference indicator among all reference indicators; calculating the product of each reference indicator and its corresponding weight value, and determining the sum of all calculated products as the data redundancy.

[0102] This application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the various processes of the above-described data redundancy identification method embodiments and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0103] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0104] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described data redundancy identification method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0105] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0106] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0108] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for identifying data redundancy, characterized in that, The data redundancy identification method includes: Retrieve the fields to be identified from at least two data tables within a preset database; Obtain the support level of the field to be identified; Based on the support of the field to be identified, reasonable redundant fields in the field to be identified are identified, wherein the reasonable redundant fields refer to fields that appear more frequently than a preset value. Obtain the fields to be calculated for redundancy from the fields to be identified, excluding the reasonable redundancy fields; Based on the redundancy calculation field in each data table and the preset reference index, the data redundancy between the first data table and the second data table is calculated, wherein the first data table and the second data table are any two of the at least two data tables. The process of identifying reasonable redundant fields in the field to be identified based on the support of the field to be identified includes: Based on the support of the field to be identified, target fields with support higher than a preset support threshold are selected from the field to be identified. Take any two of the target fields to form a common field set, calculate the support of the common field set, and filter to obtain a target common field set whose support is higher than the preset support threshold. The fields in the target public field set are identified as the reasonable redundant fields.

2. The data redundancy identification method according to claim 1, characterized in that, The reference metrics include at least one of the following: non-public field overlap, table domain overlap, storage cycle overlap, table layer overlap, source table overlap, and public field overlap. The non-public field corresponding to the non-public field overlap degree and the public field corresponding to the public field overlap degree are selected from the fields to be calculated for redundancy.

3. The data redundancy identification method according to claim 1 or 2, characterized in that, The calculation of data redundancy between the first and second data tables, based on the redundancy calculation fields in each data table and preset reference indicators, includes: The weight value corresponding to each reference indicator is calculated, wherein the weight value corresponding to each reference indicator is the ratio of the weight score of the reference indicator to the weight scores of all reference indicators, and the weight score of the reference indicator is calculated based on the importance of the reference indicator among all reference indicators; Calculate the product of each reference indicator and its corresponding weight value, and determine the sum of all calculated products as the data redundancy.

4. A data redundancy identification device, characterized in that, The redundancy identification device includes: The first acquisition module is used to acquire the fields to be identified from at least two data tables in a preset database; The second acquisition module is used to acquire the support level of the field to be identified; The redundancy identification module is used to identify reasonable redundant fields in the field to be identified based on the support of the field to be identified, wherein the reasonable redundant fields refer to fields whose frequency of occurrence is higher than a preset value; The redundancy identification module is used to: filter target fields in the field to be identified whose support is higher than a preset support threshold based on the support of the field to be identified; form a common field set by any two of the target fields, calculate the support of the common field set, filter target common field sets in the common field set whose support is higher than the preset support threshold; and determine the fields in the target common field set as the reasonable redundant fields. The data redundancy identification device further includes: A redundancy calculation module is used to obtain the fields to be redundant calculated in the fields to be identified, excluding the reasonable redundancy fields; and to calculate the data redundancy between the first data table and the second data table based on the fields to be redundant calculated in each data table and a preset reference index, wherein the first data table and the second data table are any two of the at least two data tables.

5. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data redundancy identification method as described in any one of claims 1 to 3.

6. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data redundancy identification method as described in any one of claims 1 to 3.

Images