Method, device and terminal for determining unique column combination from consistent set

Abstract

The present invention provides a method, device and terminal for determining a unique column combination from a consistent set. The method includes: obtaining an attribute set and a consistent set corresponding to the data to be processed; calculating the combination key existing in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adding the combination key to the minimum combination key set; updating the attributes in the attribute set according to the combination key to obtain an updated attribute set; generating a new attribute set according to the updated attribute set, and jumping to the step of "calculating the combination key existing in the attribute set according to the elements in the attribute set and the elements in the consistent set" to continue executing until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated, and the process is terminated, and the obtained minimum combination key set is output as a unique column combination. The present invention can find a unique column combination from a consistent set, simplify calculations, and improve the efficiency of finding a unique column combination.

Description

Technical Field

[0001] The present invention relates to the technical field of database table processing, and in particular to a method, device and terminal for determining a unique column combination from a consistent set. Background Art

[0002] In the data age, many industries will generate massive amounts of data. When recording these data, many rows and columns are used. To ensure data quality and query performance, it is crucial to actually query massive amounts of data. However, direct query will exceed our current data processing capabilities, so data analysis is used. Data analysis is an activity to discover and understand the relevant attributes of a data set. An important task of data analysis is to discover unique column combinations (UCC) and non-unique column combinations, where UCC is a column set of a relational database table that satisfies the uniqueness constraint and is a column set that contains only unique values.

[0003] Currently, unique column combinations can be discovered using either row-based techniques or column-based techniques. Gordian is an example of a row-based technique that pre-organizes the table's data in the form of a prefix tree and traverses the prefix tree to find the maximum non-uniqueness, then calculates the minimum uniqueness based on the maximum non-uniqueness. However, the main disadvantage of Gordian is that it requires the prefix tree to be in main memory, which results in slow computation when the prefix tree is as large as the input table. In addition, when the number of maximum non-uniqueness is large, generating minimum uniqueness from maximum non-uniqueness can be a severe computational bottleneck. Column-based approaches perform top-down and hybrid runs on the power set lattice of relational database attributes. Their technique is based on the a priori intuition that the superset of discovered uniqueness and the subset of discovered non-uniqueness can be pruned through further analysis, but the computation is complex and inefficient. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and terminal for determining a unique column combination from a consistent set, so as to solve the problems of complex calculation and low efficiency in calculating the unique column combination in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining a unique column combination from a consistent set, comprising:

[0006] Acquire an attribute set and a consistent set corresponding to the data to be processed, wherein the data to be processed is data in a data table format including row data and column data;

[0007] Calculate the composite key in the attribute set according to the elements in the attribute set and the elements in the consistent set, and add the composite key to the minimum composite key set;

[0008] Update the attributes in the attribute set according to the combination key to obtain an updated attribute set;

[0009] Generate a new attribute set based on the updated attribute set, and jump to the step of "calculating the composite key existing in the attribute set based on the elements in the attribute set and the elements in the consistent set" to continue execution until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated, then end the process and output the obtained minimum composite key set as a unique column combination.

[0010] In a possible implementation, calculating the combination key in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adding the combination key to the minimum combination key set includes:

[0011] Detecting whether a current element in the attribute set is a subset of the elements in the consistent set;

[0012] When the current element is not a subset of the elements in the consistent set, determining that the current element is a composite key; when the current element is a subset of the elements in the consistent set, determining that the current element is not a composite key;

[0013] Get the next element in the attribute set, and check whether the next element exists;

[0014] When the next element exists, jump to the step of "detecting whether the current element in the attribute set is a subset of the elements in the consistent set" and continue to execute;

[0015] When the next element does not exist, all elements are detected, all key combinations corresponding to the attribute set are obtained, and all key combinations are added to the minimum key combination set.

[0016] In a possible implementation, calculating the combination key in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adding the combination key to the minimum combination key set includes:

[0017] Detecting whether a current element in the attribute set is a subset of the elements in the consistent set;

[0018] When all elements in the attribute set are subsets of the elements in the consistent set, it is determined that no combination key exists in the attribute set, and the minimum combination key set remains unchanged.

[0019] In a possible implementation, updating the attributes in the attribute set according to the combination key to obtain an updated attribute set includes:

[0020] All key combinations determined in the attributes of the attribute set are deleted, and all remaining attributes constitute an updated attribute set.

[0021] In a possible implementation manner, generating a new attribute set according to the updated attribute set includes:

[0022] When each attribute in the updated attribute set is a single attribute, the union of any two attributes is determined as an element in the new attribute set.

[0023] In a possible implementation manner, generating a new attribute set according to the updated attribute set includes:

[0024] When the attributes in the updated attribute set are combined attributes and the lengths of the attributes are the same, a union of two combined attributes with the same leading block is determined as an element in the new attribute set.

[0025] In a possible implementation, when the elements in the attribute set are composite attributes, the composite attributes are recorded in the same order of attributes.

[0026] In a second aspect, an embodiment of the present invention provides a device for determining a unique column combination from a consistent set, including:

[0027] An acquisition module, used to acquire an attribute set and a consistent set corresponding to the data to be processed, wherein the data to be processed is data in a data table format including row data and column data;

[0028] A calculation module, used for calculating the combination key existing in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adding the combination key to the minimum combination key set;

[0029] An updating module, used for updating the attributes in the attribute set according to the combination key set to obtain an updated attribute set;

[0030] A generation module is used to generate a new attribute set according to the updated attribute set; and then jump to the calculation module to execute until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated, and the process ends;

[0031] The output module is used to output the obtained minimum composite key set as a unique column combination.

[0032] In a third aspect, an embodiment of the present invention provides a terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a method for determining a unique column combination from a consistent set as described in the first aspect or any possible implementation of the first aspect are implemented.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for determining a unique column combination from a consistent set as described in the first aspect or any possible implementation of the first aspect are implemented.

[0034] The method, device and terminal for determining a unique column combination from a consistent set provided by the embodiments of the present invention obtain the attribute set and the consistent set of the data to be processed, determine whether there is a combination key in the attribute set based on the comparison between the elements of the attribute set and the elements in the consistent set, and update the attribute set based on the obtained combination key, so that a new attribute set can be generated based on the updated attribute set to continue calculating the combination key in the new attribute set. When the updated attribute set includes only one element or the remaining attribute set elements do not have a common leading block and the next level candidate attribute set cannot be generated, the process is terminated, and the set of all the combination keys obtained is the unique column combination, so that by extending the application of the consistent set, the unique column combination can be found from the consistent set, the calculation is simplified, and the efficiency of finding the unique column combination is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 is a flow chart of a method for determining a unique column combination from a consistent set provided by an embodiment of the present invention;

[0037] Figure 2 This is a data representation diagram provided by an embodiment of the present invention;

[0038] Figure 3 is a flowchart for implementing the calculation of the combination keys present in the attribute set provided by an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of a power set lattice provided by an embodiment of the present invention;

[0040] Figure 5is a flow chart of a method for determining a unique column combination from a consistent set provided by another embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the structure of an apparatus for determining a unique column combination from a consistent set provided by an embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0045] A unique column combination is a set of columns that only contain unique values. Discovering such a combination can be used in many fields such as anomaly detection, data integration, data modeling, duplicate detection, indexing, and query optimization. Consistent sets are an important method for discovering functional dependencies. Functional dependencies are dependencies between database table attributes. Discovering dependencies helps repair missing or inaccurate data in the database. It should be noted that the data targeted by consistent sets and unique column combinations is table data, that is, it includes both row data and column data, and each column of data is a data attribute. In this application, unique column combinations are mainly discovered from consistent sets.

[0046] Figure 1 A flowchart of a method for determining a unique column combination from a consistent set provided by an embodiment of the present invention is described in detail as follows:

[0047] Step 101: Obtain the attribute set and consistent set corresponding to the data to be processed.

[0048] The data to be processed is data in the form of a data table including row data and column data. The data to be processed may be industrial data, such as power data. The data to be processed may be data in a relational database, that is, including both row data and column data, and each column data is a data attribute. For example, in a data table, each row is a data record, and each column is an attribute of the recorded data. Figure 2The data shown is an employee department information table, which includes 7 data items. Each data item includes 5 attributes, namely: employee number, department number, year of birth, department name and manager. For simplicity, we replace the above 5 attributes with A, B, C, D and E respectively.

[0049] For a consistent set, given a relational database schema R and an instance r of schema R, let t i and t j is a tuple, i.e., row data in the relational database R, and X is an attribute set. i [X] = t j [X] is called tuple t i and t j Consistent on X. t i and t j The consistent set of is defined as follows: For instance r, ag(r)=ag(t i ,t j )|t i ,t j ∈r,t i ≠t j}.

[0050] Theorem 1 Given a relational database schema R and an instance r of schema R, let P(r) denote the power set of all attributes, K denote the unique column combination, and ag(r) denote the consistent set, then

[0051]

[0052] See also Figure 2 , we calculate Figure 2 The consistent set in the data table shown. Compare the data distribution of each row in the data table with the data of other rows for the same attributes. When the attribute value of a certain attribute is the same, it means that there is a consistent set. When the attribute value of each attribute is different, it means that there is no consistent set between the two rows of data. Then, ag(t1, t2) = {A}, that is, the attribute value of attribute A between the first row of data and the second row of data is 1, and the attribute value is the same, indicating that there is a consistent set between the first row of data and the second row of data, and the element of the consistent set is A. Similarly, This means that there is no consistent set between the first row of data and the second row of data, and the set is empty. After the consistent set calculations between the first row of data and other data are completed, the consistent set calculations between the second row of data and the third row of data, the fourth row of data, and the seventh row of data are calculated. At this time, the consistent set between the second row of data and the first row of data is no longer calculated to prevent duplication. In summary, Figure 2 The consistent set in the data table shown is

[0053] Step 102: Calculate the composite key in the attribute set according to the elements in the attribute set and the elements in the consistent set, and add the composite key to the minimum composite key set.

[0054] The elements in the determined attribute set are A, B, C, D, and E. At this time, the elements in the attribute set are basic attributes and do not include combined attributes. We can call it the first-level attribute set, denoted by L1={A,B,C,D,E}.

[0055] The consistent set is

[0056] Before computing unique column combinations based on the consistent set, we first define the uniqueness of the column combination K.

[0057] Definition 1: Given a i The relational database schema of each column is R = {C1, C2, ... C m} and examples Column combination is unique if and only if

[0058] Unique column combinations is minimal if no proper subset of K is unique. Its mathematical definition is as follows:

[0059] Definition 2: Unique column combination is the smallest if and only if

[0060] According to the definitions of unique column combinations and consistent sets, any unique column combination X must be unique across all tuples, so it cannot belong to any consistent set, and vice versa.

[0061] When calculating the composite keys present in the attribute set, you can use the following steps, such as Figure 3 As shown:

[0062] Check whether the current element in the attribute set is a subset of the elements in the consistent set;

[0063] When the current element is not a subset of the elements in the consistent set, it is determined that the current element is a composite key,

[0064] When the current element is a subset of the elements in the consistent set, it is determined that the current element is not a composite key;

[0065] Get the next element in the attribute set and check whether the next element exists;

[0066] When the next element exists, jump to the step of "checking whether the current element in the attribute set is a subset of the elements in the consistent set" to continue execution;

[0067] When the next element does not exist, all elements are detected, all key combinations corresponding to the attribute set are obtained, and all key combinations are added to the minimum key combination set.

[0068] For example, if the current element is A, then A is a subset of element A in the consistent set, so continue with the next element B until all elements are detected.

[0069] It should be noted that it is detected whether the current element in the attribute set is a subset of the elements in the consistent set; when all elements in the attribute set are subsets of the elements in the consistent set, it is determined that there is no composite key in the attribute set, that is, the composite key set corresponding to the first-level attribute set is an empty set, and the composite key set corresponding to the first-level attribute set is recorded as K1. At this time, there is no need to add new elements to the minimum key combination set, and the minimum key combination set remains unchanged. Here, the minimum key combination set is the unique column combination to be determined, denoted by K min .

[0070] Step 103: Update the attributes in the attribute set according to the combination key to obtain an updated attribute set.

[0071] This step updates the attribute set to obtain an updated attribute set, and then calculates the next layer of attribute sets based on the updated attribute set.

[0072] Optionally, when updating the attribute set in this step, the step may include: deleting all combination keys determined in the attributes of the attribute set, and all remaining attributes constitute the updated attribute set.

[0073] It can be understood that the determined combination key in the attribute set is deleted, and the remaining attributes are the updated attribute set.

[0074] It should be noted that when there is no composite key in the attribute set, the updated attribute set is itself, that is, if the composite key set of the current layer attribute set is an empty set, there is no need to update the current layer attribute set. Then update the current layer attribute set L1 = {A, B, C, D, E}.

[0075] Step 104, generate a new attribute set based on the updated attribute set, and jump to the step of "calculating the composite key existing in the attribute set based on the elements in the attribute set and the elements in the consistent set" to continue execution until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated. The process ends and the obtained minimum composite key set is output as a unique column combination.

[0076] Optionally, before this step, it is checked whether a new attribute set can be generated based on the updated attribute set. When a new attribute set can be generated, a new attribute set, i.e., a next-level candidate attribute set, is generated according to the specific description of step 104. When a new attribute set cannot be generated, the process is terminated and the obtained minimum combination key set is output as a unique column combination.

[0077] It should be noted that when we describe the hierarchy of attribute sets, the concept used is based on the power set lattice. The power set lattice is composed of the combination of all attributes, such as Figure 4 The schematic diagram of the power set lattice is shown as follows. For R = {A, B, C, D, E}, the first layer of the power set lattice L1 = {A, B, C, D, E}, and the second layer is formed according to the union of the attributes of the first layer, L2 = {AB, AC, AD, AE, BC, BD, BE, CD, CE, DE}, similarly L3 = {ABC, ABD, ABE, ACD, ACE, ADE, BCE, BDE, CDE}, L4 = {ABCD, ABCE, ABDE, ACDE, BCDE}, L5 = {ABCDE}. The elements in each layer of the attribute set have the same length. Here, the length of the element or the attribute length refers to the same number of attributes that constitute the combined attribute. When it is a single attribute, all elements in the attribute set are single attributes.

[0078] In this step, when a new attribute set is generated based on the updated attribute set, it is different from the calculation of the next layer i+1 from the current layer i in the power set grid, as described in detail below. Figure 2 ,The calculation of attribute sets at each level in the power set lattice can be divided into two categories, one is the calculation of attribute sets of single attributes, and the other is the calculation of attribute sets of combined attributes except single attributes.

[0079] Optionally, when each attribute in the attribute set of the current layer is a single attribute, the union of any two attributes is determined as an element in the attribute set of the next layer.

[0080] The attribute of the first layer is a single attribute, L1 = {A, B, C, D, E}. When calculating L2 based on L1, just calculate the union of any two attributes, and get L2 = {AB, AC, AD, AE, BC, BD, BE, CD, CE, DE}.

[0081] It is understandable that the union of A and B is the same as the union of B and A. Therefore, in order to prevent duplicate calculations, we can sort the attributes and calculate the union of the current attribute and the next attribute in turn, without calculating the union of the current attribute and the previous attribute in turn. Here, the previous attribute refers to the attribute before the current attribute, for example, the previous attribute of the B attribute is A, and the next attribute refers to the attribute after the current attribute, for example, the next attributes of the A attribute include: B, C, D, E.

[0082] Optionally, when each attribute in the attribute set of the current layer is a combined attribute, a union of two combined attributes having the same leading block is determined as an element in the attribute set of the next layer.

[0083] The elements included in the attribute sets above the second level are all composite attributes. When calculating the attribute set of the next level, we only need to calculate the union of the composite attributes with the same leading block. The same leading block refers to the same attributes or the same composite attributes with the same length except for the last different attribute in two composite attributes. That is, two sets X, Y∈L i They belong to the same leading block if they have a common leading block of length i-1, that is, they only differ in the last attribute. For example, the combined attributes AB and AC have the same leading block A; the combined attributes ABD and ABE have the same leading block AB.

[0084] The element in the next attribute set of the combined attributes AB and AC is ABC, and the element in the next attribute set of the combined attributes ABD and ABE is ABDE.

[0085] Optionally, when the elements in the attribute set are composite attributes, the composite attributes are recorded in the order of the same attributes, so that it is easy to determine whether two composite attributes have the same leading block and to calculate the next layer of attribute sets.

[0086] After obtaining a new attribute set, that is, an attribute set of a next layer of the current layer, the process continues with step 102 to calculate a composite key from the attribute set of the next layer.

[0087] See also Figure 5 , when calculating the unique column combination according to the consistent set, let L i Denotes the layer with i attributes in the power set lattice. Calculate L2 from L1 = {{A}|A∈R}, calculate L3 from L2, and so on, and calculate layer by layer. The algorithm is pruned according to the calculation results. The process of finding unique column combinations from consistent sets can also be as follows:

[0088] Get the attribute set and consistent set of the data to be processed; calculate the composite key of the i-layer attribute set; detect whether the composite key exists in the i-layer attribute set; if it exists, add the composite key corresponding to the i-layer attribute set to the minimum composite set, update the i-layer attribute set, and detect whether the updated i-layer attribute set includes only one element; if it does not exist, directly detect whether the updated i-layer attribute set includes only one element; when the updated i-layer attribute set includes only one element, end the process, and the obtained composite key is the minimum composite key set, which is output as a unique column combination.

[0089] When the updated i-layer attribute set includes more than one element, check whether there is the same leading block in the updated i-layer attribute set; when the updated i-layer attribute set does not have the same leading block, end the process, and the obtained combination key is the minimum combination key set, which is output as a unique column combination. That is, the termination condition of the process is that when there is only one element in the updated attribute set or the remaining attribute set elements do not have a common leading block, a new attribute set cannot be generated, and the process ends. At this time, K min Output as unique column combinations.

[0090] When the updated attribute set does not include only one element, a new attribute set is generated according to the updated attribute set, that is, an i+1 layer attribute set is generated, and then the combination key of the i layer attribute set is calculated again, and i=i+1 at this time.

[0091] When calculating unique column combinations based on a consistent set, the algorithm involved is as follows:

[0092] Input: attribute set R, consistent set ag(r)

[0093] Output: Key combination K min

[0094]

[0095] Among them, the subroutine COMPUTE_KEYS(L i ) is used to calculate L i The key combination in .

[0096] Subroutine PRUNE(L i ) by deleting the i The key combinations found in prune the search space. The idea of ​​pruning is that the superset of the key combinations is also a key combination. Since we want to find the minimum key combination, we delete the key combinations that have been found to achieve our goal.

[0097] Subroutine GENERATE_NEXT_LEVEL(L i ) is used to generate the key combination of the next layer from the current layer.

[0098] Each subroutine is described in detail below.

[0099] For the subroutine COMPUTE_KEYS(L i ), calculate L i The key combination in K i .

[0100]

[0101]

[0102] For the subroutine Prune(L i ), delete the files already in L i Key combinations found in .

[0103] 1 do while

[0104] 2 for each X∈K i do

[0105] 3 delete X from L i

[0106] For the subroutine GENERATE_NEXT_LEVEL(L i ), generate the composite key of the next layer based on the following formula. Formula:

[0107]

[0108] The subroutine PREFIX BLOCKS (L i ) i Divide into different non-overlapping blocks, let the set X∈L i is an ordered list of attributes. Two sets X, Y∈L i Belong to the same leading block if they have a common leading block of length i-1, i.e. they differ only in the last attribute. All leading blocks constitute L i The prefix blocks are therefore easy to sort according to the lexicographic order of L i For example, the sets {AD} and {AE} have a common leading block {A}, and they form the next level set {ADE}.

[0109] Next we will Figure 2 Taking the data table shown in the figure as an example, the execution process of the above algorithm is described by example.

[0110] Input consistent set

[0111] To calculate the key combination, first call the subroutine COMPUTE_KEYS(L1={A,B,C,D,E}) based on L1={A,B,C,D,E}. i ), since A, B, C, D, E are consistent sets A subset of , so no composite keys are generated at this stage, return

[0112] Call the subroutine Prune(L i ), since no composite key is generated, there is no need to update L1.

[0113] Call the subroutine GENERATE_NEXT_LEVEL(L i ), generate L2 = {AB,AC,AD,AE,BC,BD,BE,CD,CE,DE} according to L1 = {A,B,C,D,E}.

[0114] Call subroutine COMPUTE_KEYS(L i ), find the key combination in L2, return K2 = {AB, AC, AD, AE, BC, CD}, and other elements in L2 are subsets of an element in the consistent set. For example, BD is a subset of BDE in the consistent set.

[0115] Add K2 to K min In, K min =K min +K2={AB,AC,AD,AE,BC,CD}.

[0116] Call the subroutine Prune(L i ) is updated to obtain L2=L2-K2={BD,BE,CE,DE}.

[0117] Call the subroutine GENERATE_NEXT_LEVEL(L i ), generating L3 = {BDE}, because only BD and BE have a common leading block, and they constitute the next layer attribute set {BDE}.

[0118] Call subroutine COMPUTE_KEYS(L i ), the key combination is not found in L3, return This is because {BDE} is in the consistent set.

[0119] Call the subroutine Prune(L i ), since no composite key is generated, there is no need to update L1.

[0120] Call the subroutine GENERATE_NEXT_LEVEL(L i ), because there is only one element in L3, there is no need to generate the next level attribute set, and the algorithm ends. Output K min ={AB,AC,AD,AE,BC,CD}, this is the minimum unique column combination we need.

[0121] We can apply the unique column combination. For example, we can verify other newly added data in the database table based on the output unique column combination to see whether there are data anomalies or data duplication in the newly added data.

[0122] For example, if a company adds a new employee, the corresponding Figure 2 In the data table shown, multiple rows of new data are added. The determined unique column combination also applies to the newly added data, that is, the data in the unique column combination cannot be repeated. If the unique column combination data is repeated, it means that the newly added data is repeated. Therefore, the unique column combination can be used to detect duplicate data. The other columns corresponding to the unique column combination in the data table are also determined. If the data in the same row corresponding to the unique column combination is inconsistent, it means that the data is abnormal. Therefore, the unique column combination can be used to detect data anomalies. In addition, when querying a list with a determined unique column combination, the query or optimization can be based on the unique column combination.

[0123] The following is an explanation of a simple administrative data list, as shown in Table 1. The data in Table 1 is only part of the data for illustration, not the real and complete data. For example, the unique key in Table 1 is the attribute of the administrative code column. If there is new data and it is desired to detect whether the new data has data duplication or data anomaly, the administrative code column is used as the detection basis for detection. When there is data consistent with the administrative code column data, the new data can be directly determined to be duplicate data. For example, the administrative data in the new data 1 is the same as the third data in the administrative code column data, and the new data 1 is repeated. It should be clear that when the unique key is the administrative code column, it means that only the data corresponding to the administrative code in each row of data is different from the data corresponding to the administrative code in other data. Therefore, when there is data consistent with a certain administrative code in the new data, the other data in the new data should also be the same as the data in the corresponding column. When the other data is different from the data in the corresponding column, there is a data anomaly. For example, in Table 1, the new data 2 is the same as the second data in the administrative code column data. When the country column in the new data 2 is different from the country data in the second row of the administrative code column data, it indicates that the new data 2 is abnormal.

[0124] Table 1

[0125]

[0126]

[0127] When the data table is data from the power industry or other industries, the method provided by the embodiment of the present invention can also be used to determine the unique column combination, and then the new data can be detected to see if there is duplication or anomaly through the unique column combination. For example, a data table for purchasing equipment in the power industry can include attributes such as purchase date, equipment name, equipment quantity, application department, responsible maintenance department, maintenance personnel, and maintenance date.

[0128] The embodiment of the present invention obtains the attribute set and consistent set corresponding to the data to be processed, obtains the combination key existing in the attribute set by comparing the elements in the attribute set with the elements in the consistent set, and adds the combination key to the combination key set; updates the attributes in the attribute set according to the combination key to obtain the updated attribute set; generates a new attribute set according to the updated attribute set, and calculates the combination key existing in the attribute set according to the elements in the new attribute set and the elements in the consistent set, until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated, and the process is terminated, and the obtained minimum combination key set is a unique column combination. The present invention calculates the minimum unique column combination according to the consistent set by expanding the application of the consistent set, and the calculation is very simple and the operation efficiency is high.

[0129] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0130] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0131] Figure 6 A schematic diagram of the structure of an apparatus for determining a unique column combination from a consistent set provided by an embodiment of the present invention is shown. For ease of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0132] like Figure 6 As shown, the apparatus for determining a unique column combination from a consistent set includes: an acquisition module 601 , a calculation module 602 , an update module 603 , a generation module 604 and an output module 605 .

[0133] An acquisition module 601 is used to acquire an attribute set and a consistent set corresponding to the data to be processed, wherein the data to be processed is data in a data table format including row data and column data;

[0134] A calculation module 602, configured to calculate the combination keys existing in the attribute set according to the elements in the attribute set and the elements in the consistent set, and add the combination keys to the minimum combination key set;

[0135] An updating module 603, used to update the attributes in the attribute set according to the combination key to obtain an updated attribute set;

[0136] The generation module 604 is used to generate a new attribute set according to the updated attribute set; then jump to the calculation module 602 for execution, and the process ends when the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block;

[0137] The output module 605 is used to output the obtained minimum combination key set as a unique column combination.

[0138] In a possible implementation, the calculation module 602 calculates the combination key in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adds the combination key to the minimum combination key set, for:

[0139] Check whether the current element in the attribute set is a subset of the elements in the consistent set;

[0140] When the current element is not a subset of the elements in the consistent set, it is determined that the current element is a composite key;

[0141] When the current element is a subset of the elements in the consistent set, determine that the current element is not a composite key, obtain the next element in the attribute set, and jump to the step of "detecting whether the current element in the attribute set is a subset of the elements in the consistent set" to continue executing until all elements in the attribute set are detected, all composite keys corresponding to the attribute set are obtained, and all composite keys are added to the minimum composite key set.

[0142] Optionally, when the calculation module 602 calculates the combination key in the attribute set according to the elements in the attribute set and the elements in the consistent set and adds the combination key to the minimum combination key set, it is used to:

[0143] Check whether the current element in the attribute set is a subset of the elements in the consistent set;

[0144] When all elements in the attribute set are subsets of the elements in the consistent set, it is determined that there is no composite key in the attribute set, and the minimum composite key set remains unchanged.

[0145] Optionally, the updating module 603 updates the attributes in the attribute set according to the combination key, and when obtaining the updated attribute set, is used to:

[0146] All key combinations identified in the attributes of the attribute set are deleted, and all remaining attributes constitute the updated attribute set.

[0147] Optionally, when generating a new attribute set according to the updated attribute set, the generating module 604 is used to:

[0148] When each attribute in the updated attribute set is a single attribute, the union of any two attributes is determined as an element in the new attribute set.

[0149] Optionally, when generating a new attribute set according to the updated attribute set, the generating module 604 is used to:

[0150] When the attributes in the updated attribute set are combined attributes and the lengths of the attributes are the same, the union of two combined attributes with the same leading block is determined as an element in the new attribute set.

[0151] Optionally, when the elements in the attribute set are composite attributes, the composite attributes are recorded in the same order.

[0152] The above-mentioned device for determining a unique column combination from a consistent set obtains the attribute set and the consistent set corresponding to the data to be processed through the acquisition module, obtains the combination key existing in the attribute set by comparing the elements in the attribute set with the elements in the consistent set through the calculation module, and adds the combination key to the minimum combination key set; the update module updates the attributes in the attribute set according to the combination key to obtain an updated attribute set; based on the updated attribute set, the generation module generates a new attribute set, and calculates the combination key existing in the attribute set based on the elements in the new attribute set and the elements in the consistent set, until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the process ends, and the obtained minimum combination key set is a unique column combination. The present invention calculates the minimum unique column combination based on the consistent set by expanding the application of the consistent set, and the calculation is very simple and the operation efficiency is high.

[0153] Figure 7 is a schematic diagram of a terminal provided by an embodiment of the present invention. Figure 7 As shown, the terminal 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned method embodiments for determining a unique column combination from a consistent set are implemented, for example Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 6 Functionality of modules / units 601 to 605 is shown.

[0154] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 72 in the terminal 7. For example, the computer program 72 may be divided into Figure 6 Modules / units 601 to 605 are shown.

[0155] The terminal 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of terminal 7 and does not constitute a limitation on terminal 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0156] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0157] The memory 71 may be an internal storage unit of the terminal 7, such as a hard disk or memory of the terminal 7. The memory 71 may also be an external storage device of the terminal 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 7. Further, the memory 71 may also include both an internal storage unit of the terminal 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the terminal. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0158] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0161] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments for determining a unique column combination from a consistent set can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0165] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for determining a unique column combination from a consistent set, characterized in that: include: Acquire an attribute set and a consistent set corresponding to the data to be processed, wherein the data to be processed is data in a data table format including row data and column data; Calculate the composite key in the attribute set according to the elements in the attribute set and the elements in the consistent set, and add the composite key to the minimum composite key set; Update the attributes in the attribute set according to the combination key to obtain an updated attribute set; Generate a new attribute set according to the updated attribute set, and jump to the step of "calculating the composite key in the attribute set according to the elements in the attribute set and the elements in the consistent set" to continue the execution until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated, and the process ends, and the obtained minimum composite key set is output as a unique column combination; The calculating, according to the elements in the attribute set and the elements in the consistent set, the composite key existing in the attribute set and adding the composite key to the minimum composite key set comprises: detecting whether the current element in the attribute set is a subset of the elements in the consistent set, when the current element is not a subset of the elements in the consistent set, determining that the current element is a composite key, and when the current element is a subset of the elements in the consistent set, determining that the current element is not a composite key; Get the next element in the attribute set, and check whether the next element exists; When the next element exists, jump to the step of "detecting whether the current element in the attribute set is a subset of the elements in the consistent set" and continue to execute; When the next element does not exist, all elements are detected, all key combinations corresponding to the attribute set are obtained, and all key combinations are added to the minimum key combination set.

2. The method for determining a unique column combination from a consistent set according to claim 1, characterized in that: The step of calculating the composite key existing in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adding the composite key to the minimum composite key set includes: Detecting whether a current element in the attribute set is a subset of the elements in the consistent set; When all elements in the attribute set are subsets of the elements in the consistent set, it is determined that no combination key exists in the attribute set, and the minimum combination key set remains unchanged.

3. The method for determining a unique column combination from a consistent set according to claim 1 or 2, characterized in that: The updating of the attributes in the attribute set according to the combination key to obtain an updated attribute set includes: All key combinations determined in the attributes of the attribute set are deleted, and all remaining attributes constitute an updated attribute set.

4. The method for determining a unique column combination from a consistent set according to claim 1 or 2, characterized in that: Generating a new attribute set according to the updated attribute set includes: When each attribute in the updated attribute set is a single attribute, the union of any two attributes is determined as an element in the new attribute set.

5. The method for determining a unique column combination from a consistent set according to claim 1 or 2, characterized in that: Generating a new attribute set according to the updated attribute set includes: When the attributes in the updated attribute set are combined attributes and the lengths of the attributes are the same, a union of two combined attributes with the same leading block is determined as an element in the new attribute set.

6. The method for determining a unique column combination from a consistent set according to claim 5, characterized in that: When the elements in the attribute set are composite attributes, the composite attributes are recorded in the order of the same attributes.

7. A device for determining a unique column combination from a consistent set, characterized in that: include: An acquisition module, used to acquire an attribute set and a consistent set corresponding to the data to be processed, wherein the data to be processed is data in a data table format including row data and column data; A calculation module, used for calculating the combination key existing in the attribute set according to the elements in the attribute set and the elements in the consistent set, and adding the combination key to the minimum combination key set; An updating module, used for updating the attributes in the attribute set according to the combination key set to obtain an updated attribute set; A generation module is used to generate a new attribute set according to the updated attribute set; and then jump to the calculation module to execute until the updated attribute set has only one element or the remaining attribute set elements do not have a common leading block, and the next level candidate attribute set cannot be generated, and the process ends; An output module, used for outputting the obtained minimum combination key set as a unique column combination; The calculation module is specifically used to: detect whether the current element in the attribute set is a subset of the elements in the consistent set, and when the current element is not a subset of the elements in the consistent set, determine that the current element is a composite key; when the current element is a subset of the elements in the consistent set, determine that the current element is not a composite key; Get the next element in the attribute set, and check whether the next element exists; When the next element exists, jump to the step of "detecting whether the current element in the attribute set is a subset of the elements in the consistent set" and continue to execute; When the next element does not exist, all elements are detected, all key combinations corresponding to the attribute set are obtained, and all key combinations are added to the minimum key combination set.

8. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images