Primary key lookup method and device of data model, electronic equipment and storage medium
By performing primary key encoding and AND operations on the dimension fields of the data model, combined with an active learning algorithm, the problem of low primary key efficiency in data warehouses is solved, enabling fast and accurate primary key lookup and improving computational performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD
- Filing Date
- 2022-07-18
- Publication Date
- 2026-06-09
AI Technical Summary
In the process of big data development and data mining, the existing technology for determining the primary key of the data model is inefficient. Especially in the Hive data warehouse tool, the primary key information depends on the modeling habits of the developers, which leads to manual judgment of the primary key and seriously slows down the development progress.
By performing primary key encoding calculations on the dimensional fields of the same digital model, primary key encoding results are generated. The dimensional fields are sorted according to information entropy to determine the target dimensional field. The joint primary key is output through AND calculation. The dimensional fields of the data model are extracted using an active learning algorithm, thereby improving the efficiency of primary key lookup.
It enables fast and accurate querying of primary keys in the data model of the data warehouse, reduces memory space, and improves computing performance and search efficiency.
Smart Images
Figure CN117493467B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of big data technology, and in particular to a method, apparatus, electronic device, and computer storage medium for finding primary keys in a data model. Background Technology
[0002] In big data development, data modeling, and data mining, numerous table join operations are involved, which require primary keys. However, in data warehouses, such as Hive tables, primary keys are not explicitly defined. Primary key information relies heavily on the modeling habits of data developers, typically being specified in the table description dictionary. However, each data developer creates and maintains hundreds or even thousands of data models, many of which lack primary keys. This necessitates manual determination of primary keys when using these models later, significantly slowing down the development, modeling, and mining processes.
[0003] Currently, the COUNT DISTINCT algorithm, which counts distinct values, is commonly used to determine single primary keys. However, in data warehouses, the primary key of a typical data model usually consists of more than one field; it is often composed of multiple fields and is not unique. The identification of dynamic composite primary keys relies more on the developers' understanding of the data model, combined with the COUNT DISTINCT method. However, both determining single primary keys and dynamic composite primary keys suffer from low efficiency.
[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] This disclosure provides a method, apparatus, electronic device, and computer storage medium for finding primary keys in a data model, which at least to some extent overcomes the problem of low efficiency in querying primary keys in related technologies.
[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.
[0007] According to one aspect of this disclosure, a primary key lookup method for a data model is provided, comprising:
[0008] The primary key encoding is calculated for the dimension fields of the same digital model to generate the primary key encoding result;
[0009] Based on the primary key encoding result, the target dimension field is determined, wherein the target dimension field is the dimension field whose primary key encoding result is not equal to 0;
[0010] Perform an AND operation on the element encodings of the target dimension field to output the composite primary key.
[0011] In one embodiment of this disclosure, the primary key encoding calculation includes the following method:
[0012] Each element in the dimension field is binary encoded to generate an element code;
[0013] Calculate the number of target elements, where the primary key encoding result is the number of target elements;
[0014] Wherein, the number of 1s in the element code of the target element is greater than 1.
[0015] In one embodiment of this disclosure, it further includes:
[0016] Calculate the information entropy of the dimension fields in the digital model;
[0017] Sort the dimension fields according to their information entropy;
[0018] Based on the sorting results, the primary key encoding calculation is performed on the dimension field.
[0019] In one embodiment of this disclosure, performing a bitwise AND operation on the element encodings of the target dimension field to output a composite primary key includes:
[0020] The loop process is executed until a preset condition is met, and the loop process includes:
[0021] Get the element code of the new dimension field, where the new dimension field is the latest target dimension field entered according to the sorting result;
[0022] The element code of the target dimension field or the filter element code of the joint field is ANDed with the element code of the new dimension field to generate the element code of the joint field.
[0023] Obtain the element codes of the joint field whose number of 1s is greater than 1, and generate the filter element codes of the joint field;
[0024] The preset condition is that the number of the filtered element codes is 0;
[0025] When the preset conditions are met, the composite field becomes the composite primary key, and the composite primary key is output.
[0026] In one embodiment of this disclosure, it further includes:
[0027] When the primary key encoding result of a dimension field is 0, the dimension field is the single primary key of the digital model.
[0028] In one embodiment of this disclosure, it further includes:
[0029] Obtain the data model of the entire data warehouse;
[0030] The dimensional fields of the data model are obtained based on the active learning algorithm.
[0031] In one embodiment of this disclosure, obtaining the dimension fields of the data model based on an active learning algorithm includes:
[0032] Select unlabeled samples, label them, add them to the training set, and build a classification model;
[0033] According to the query criteria, unlabeled samples are selected, labeled, added to the training set, and the classification model is retrained.
[0034] Based on the classification model, obtain the dimension fields of the data model.
[0035] According to another aspect of this disclosure, a primary key lookup device for a data model is also provided, comprising:
[0036] The primary key encoding calculation module performs primary key encoding calculations on the dimension fields of the same digital model and generates primary key encoding results.
[0037] The target dimension field module determines the target dimension field based on the primary key encoding result, wherein the target dimension field is the dimension field whose primary key encoding result is 0;
[0038] The joint field output module performs an AND operation on the element codes of the target dimension field and outputs the joint primary key.
[0039] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform a primary key lookup method of any of the data models described above by executing the executable instructions.
[0040] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the primary key lookup method of the data model described in any of the preceding claims.
[0041] The embodiments of this disclosure provide a method, apparatus, electronic device, and computer storage medium for finding the primary key of a data model. This method calculates the information entropy of dimension fields in a digital model, sorts the dimension fields based on their information entropy, calculates the primary key encoding for dimension fields of the same digital model based on the sorting result, generates a primary key encoding result, determines the target dimension field based on the primary key encoding result (where the target dimension field is a dimension field whose primary key encoding result is not equal to 0), and iteratively performs AND calculations and filtering between the element encoding of the target dimension field and the new dimension field until preset conditions are met to output a combined primary key. This method enables quick and accurate retrieval of the primary key of a data model in a data warehouse.
[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0044] Figure 1 This diagram illustrates a primary key lookup method for a data model according to an embodiment of the present disclosure.
[0045] Figure 2 This diagram illustrates a flowchart of a dimension field sorting method according to an embodiment of the present disclosure;
[0046] Figure 3 This diagram illustrates a flowchart of a combined primary key lookup method according to an embodiment of the present disclosure;
[0047] Figure 4 This diagram illustrates a primary key lookup device for a data model according to an embodiment of the present disclosure.
[0048] Figure 5 This diagram illustrates a primary key lookup method for another data model in an embodiment of the present disclosure.
[0049] Figure 6 This diagram illustrates a primary key lookup method for another data model in an embodiment of the present disclosure.
[0050] Figure 7 This invention illustrates a system for extracting dimension fields from a data model according to an embodiment of the present disclosure;
[0051] Figure 8 This diagram illustrates a primary key lookup diagram of a data model according to an embodiment of the present disclosure; and
[0052] Figure 9 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0053] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0054] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0055] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.
[0056] This disclosure provides a primary key lookup method for a data model, which can be executed by any electronic device with computing capabilities.
[0057] Figure 1 This diagram illustrates a primary key lookup method for a data model according to an embodiment of the present disclosure, such as... Figure 1 As shown, the primary key lookup method for the data model provided in this embodiment includes the following steps:
[0058] S102, perform primary key encoding calculation on the dimension fields of the same digital model to generate primary key encoding results.
[0059] In one embodiment, the data model of the entire data warehouse is obtained; based on an active learning algorithm, the dimensional fields of the data model are obtained.
[0060] In one embodiment, obtaining the dimension fields of a data model based on an active learning algorithm includes the following steps:
[0061] Select unlabeled samples, label them, add them to the training set, and build a classification model; according to the query criteria, select unlabeled samples, label them, add them to the training set, and retrain the classification model; based on the classification model, obtain the dimension fields of the data model.
[0062] In one embodiment, primary key encoding calculation includes the following method:
[0063] Encode each element in the dimension field into binary to generate an element code; calculate the number of target elements, and the primary key encoding result is the number of target elements, where the number of 1s in the element code of the target element is greater than 1.
[0064] In one embodiment, the dimension fields of the same digital model are sorted, and the primary key encoding is calculated sequentially according to the sorting results to generate the primary key encoding results.
[0065] In one embodiment, the sorting method includes, but is not limited to, sorting based on the information entropy of the dimension field.
[0066] S104. Determine the target dimension field based on the primary key encoding result, wherein the target dimension field is the dimension field whose primary key encoding result is not equal to 0;
[0067] S106 performs a bitwise AND operation on the element codes of the target dimension field to output the composite primary key.
[0068] In one embodiment, when the primary key encoding result of the dimension field is 0, the dimension field is the single primary key of the digital model.
[0069] In one embodiment, when the primary key encoding result of the dimension field is not equal to 0, the dimension field is used as the target dimension field, and the target dimension field is ANDed with the next target dimension field to output the composite primary key.
[0070] In one embodiment, when the primary key encoding result of the dimension field is not equal to 0, the dimension field is used as the target dimension field. The target dimension field is then ANDed with the next dimension field to calculate the composite primary key.
[0071] In the above embodiments, the dimension fields in the data warehouse are extracted based on the active learning algorithm. The encoding order is determined by calculating the information entropy of each dimension field in each data model. The dimension fields are encoded as binary integers, and a bitwise AND operation is performed based on the binary integers. According to the encoding after the bitwise AND operation, it is determined whether the joint field satisfies the uniqueness principle, thereby reducing memory space, maximizing computational performance, and improving the lookup efficiency of primary keys for all data models in the entire data warehouse.
[0072] Figure 2 This diagram illustrates a flowchart of a dimension field sorting method according to an embodiment of the present disclosure, as follows: Figure 2 As shown, the dimension field sorting method provided in this embodiment includes the following steps:
[0073] S202, Calculate the information entropy of the dimension fields in the digital model.
[0074] In one embodiment, the formula for calculating information entropy is as follows:
[0075] H(X)=∑ i P(x i )I(x i )=-∑ i P(x i )log b P(x i (1)
[0076] Here, X refers to the dimension field;
[0077] xi refers to a specific element in the dimension field;
[0078] i represents the element number of the dimension field, with a value range of [1, n], where n is the field length.
[0079] S204, Sort the dimension fields according to their information entropy;
[0080] S206, Calculate the primary key encoding for the dimension fields based on the sorting results.
[0081] In one embodiment, the dimension field is sorted in descending order based on its information entropy, and the primary key encoding of each element in the dimension field is calculated sequentially based on the descending sort result.
[0082] In one embodiment, primary key encoding calculations can be performed sequentially on the elements of the dimension field of the same data model, and the next step of calculation can be performed after the calculation is completed. Alternatively, primary key encoding calculations can be performed when the dimension field is called, which can reduce computational costs.
[0083] In the above embodiments, the dimensional fields in the data warehouse are extracted based on the active learning algorithm combined with the classification algorithm. The encoding order is determined by calculating the information entropy of each dimensional field in each data model, which can improve the efficiency of primary key encoding calculation and improve the accuracy of primary key lookup.
[0084] Figure 3 This diagram illustrates a flowchart of a composite primary key lookup method according to an embodiment of the present disclosure, as follows: Figure 3 As shown, the composite primary key lookup method provided in this embodiment includes the following steps:
[0085] The loop process continues until a preset condition is met. The loop process includes:
[0086] S302, obtain the element code of the new dimension field, where the new dimension field is the latest input dimension field based on the sorting result.
[0087] In one embodiment, unused dimension fields are directly retrieved based on the order of dimension fields in the digital model.
[0088] In one embodiment, a target dimension field that has not been called is obtained based on the sorting of dimension fields in the digital model. The target dimension field is a dimension field whose primary key encoding result is not equal to 0.
[0089] S304, perform an AND operation between the element code of the target dimension field or the filter element code of the joint field and the element code of the new dimension field to generate the element code of the joint field;
[0090] In one embodiment, the element code of the target dimension field is ANDed with the element code of the new dimension field to generate the element code of the joint field.
[0091] In one embodiment, the filtered element code of the saved joint field is ANDed with the element code of the new dimension field to generate the element code of the joint field.
[0092] S306, Obtain the element codes of the combined field whose number of 1s is greater than 1, and generate the filter element codes of the combined field;
[0093] The preset condition is that the number of filtered element codes is 0;
[0094] When the preset conditions are met, the composite field becomes a composite primary key, and the composite primary key is output.
[0095] In the above embodiments, the encoding algorithm encodes the dimension field as a binary integer, performs an AND operation based on the binary integer, and determines whether the joint field satisfies the uniqueness principle by judging whether the number of 1s in the encoded data after the AND operation is greater than 1. This encoding algorithm maximizes computational performance by sacrificing very little memory space.
[0096] Based on the same inventive concept, this disclosure also provides a primary key lookup device for a data model, as shown in the following embodiment. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiment, the implementation of this device embodiment can refer to the implementation of the above-described method embodiment, and repeated details will not be elaborated further.
[0097] Figure 4 This diagram illustrates a primary key lookup device for a data model according to an embodiment of the present disclosure, such as... Figure 4 As shown, the primary key lookup device 4 of the data model includes: a primary key encoding calculation module 401, a target dimension field module 402, and a union field output module 403;
[0098] The primary key encoding calculation module 401 performs primary key encoding calculation on the dimension fields of the same digital model and generates primary key encoding results.
[0099] The target dimension field module 402 determines the target dimension field based on the primary key encoding result, wherein the target dimension field is the dimension field whose primary key encoding result is 0;
[0100] The joint field output module 403 performs an AND operation on the element codes of the target dimension field and outputs the joint primary key.
[0101] In the above embodiments, the dimension fields in the data warehouse are extracted based on the active learning algorithm. The encoding order is determined by calculating the information entropy of each dimension field in each data model. The dimension fields are encoded as binary integers, and a bitwise AND operation is performed based on the binary integers. According to the encoding after the bitwise AND operation, it is determined whether the joint field satisfies the uniqueness principle, thereby reducing memory space, maximizing computational performance, and improving the lookup efficiency of primary keys for all data models in the entire data warehouse.
[0102] Figure 5 This invention discloses a flowchart of a primary key lookup method for another data model in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the primary key lookup method for the data model provided in this embodiment includes the following steps:
[0103] S502, extract all dimension fields of all data models in the full data warehouse;
[0104] S504, calculate the information entropy of the dimension fields in each data model and sort them in descending order;
[0105] S506 uses the primary key encoding algorithm of the data model in the data warehouse to encode and calculate the input dimension fields in sequence, and outputs the single primary key and composite primary key through AND calculation.
[0106] In the above embodiments, the dimension fields in the data warehouse are extracted based on the active learning algorithm. The encoding order is determined by calculating the information entropy of each dimension field in each data model. The dimension fields are encoded as binary integers, and a bitwise AND operation is performed based on the binary integers. According to the encoding after the bitwise AND operation, it is determined whether the joint field satisfies the uniqueness principle, thereby reducing memory space, maximizing computational performance, and improving the lookup efficiency of primary keys for all data models in the entire data warehouse.
[0107] Figure 6 This invention discloses a flowchart of a primary key lookup method for another data model in an embodiment of the present disclosure, as shown below. Figure 6 As shown, the primary key lookup method for the data model provided in this embodiment includes the following steps:
[0108] S602, Input the data model of the full data warehouse;
[0109] S604, based on an active learning algorithm and combined with a classification model, extracts all dimensional fields of all data models in the entire data warehouse.
[0110] In one embodiment, Figure 7 This disclosure illustrates a system for extracting dimension fields from a data model, such as... Figure 7 As shown, the data model dimension field extraction system provided in this embodiment includes: labeled sample set 701, unlabeled sample set 702, and classification model 703;
[0111] Active learning algorithms mainly consist of two stages:
[0112] The first stage is the initialization stage. Experts randomly select a portion of the unlabeled sample set 702 as the training set and label it to generate the labeled sample set 701, which is used to build the classification model 703.
[0113] The second stage is the iterative query stage. Experts select a certain number of samples from the unlabeled sample set 702 according to a certain query standard, label them, add them to the training set, and retrain the classification model 703 until the training stops.
[0114] It should be noted that the active learning algorithm is an iterative process. The classification model 703 uses the samples fed back during the iteration to train and continuously improve classification efficiency.
[0115] In one embodiment, the classification model 703 may employ a supervised classification algorithm, such as Naive Bayes, Support Vector Machine, and various tree models.
[0116] In one embodiment, the tree model includes, but is not limited to: Random Forest (RF), XgboostLGB, and LightGBM.
[0117] It should be noted that Random Forest is an ensemble learning model that uses trees as the basic unit. Each tree produces a classification result, which are then voted on, and the forest ultimately selects the category with the most votes as the final result.
[0118] It's worth noting that LightGBM can process attribute data using feature names as input; it doesn't perform one-hot encoding on the data, making it much faster than one-hot encoding. LGBM uses a special algorithm to determine the segmentation values for attribute features.
[0119] It's important to note that XGBoost itself cannot handle categorical variables; like Random Forest, it only accepts numerical data. Therefore, before feeding categorical data into XGBoost, the data must be processed using various encoding methods, such as labeled encoding, mean encoding, or one-hot encoding.
[0120] S606, calculate the information entropy of each dimension field of each data model respectively;
[0121] S608 sorts the dimension fields in the same data model in descending order based on information entropy.
[0122] In one embodiment, the information entropy of each dimension field of each data model is calculated according to formula (1).
[0123] S610, based on the above sorting results, perform primary key encoding calculations on the dimension fields in sequence, and determine whether the primary key encoding result is 0;
[0124] In one embodiment, primary key encoding calculation includes the following steps:
[0125] Each element in the dimension field is encoded as a binary integer.
[0126] It should be noted that binary integers are integers composed of combinations of 0s and 1s.
[0127] Check sequentially whether the number of 1s in the binary integer encoded from all elements of this dimension field is greater than 1, as follows:
[0128] count(xi==1)>1 (2)
[0129] Where xi represents the binary integer encoded as the element of the dimension field.
[0130] The following formula is used to calculate the number of all elements in this dimension field whose binary integer representation contains more than one 1:
[0131] count(count(xi==1)>1) (3)
[0132] Where xi represents the binary integer encoded as the element of the dimension field.
[0133] S612, if the above primary key encoding result is equal to 0, then this dimension field is the single primary key of this data model;
[0134] S614, If the primary key encoding result above is not equal to 0, then the next dimension field sorted in S610 needs to be encoded.
[0135] S616, Perform an AND operation on the two dimension fields after the above encoding to obtain the binary encoding of all elements of the joint field;
[0136] S618, filter all elements in the binary code of the above composite field whose number of 1s is greater than 1. If the number of filtered elements is 0, it indicates that the composite field is the composite primary key of the data model; if the number of filtered elements is greater than 0, the filtered element code needs to be saved.
[0137] S620: Encode the next dimension field of the data model after sorting in S610, and perform an AND operation with the element code filtered in S618.
[0138] S622 executes the judgment in S118 that count(count(xi==1)>1)==0. If the condition is met, the composite primary key of the data model is output. If the condition is not met, S618, S620, and S622 are executed in a loop until the condition is met.
[0139] In the above embodiments, the dimension fields in the data warehouse are extracted based on the active learning algorithm. The encoding order is determined by calculating the information entropy of each dimension field in each data model. The dimension fields are encoded as binary integers, and an AND operation is performed based on the binary integers. By judging whether the number of 1s in the encoded data after the AND operation is greater than 1, it can be determined whether the joint field satisfies the uniqueness principle. This reduces memory space, maximizes computational performance, and improves the lookup efficiency of primary keys for all data models in the entire data warehouse.
[0140] Figure 8 This diagram illustrates a primary key lookup method for a data model according to an embodiment of the present disclosure, such as... Figure 8 As shown:
[0141] The original data 801 includes dimension field 1, dimension field 2, and dimension field 3;
[0142] Dimension field 1 is: a1, b1, b1, c1, a1, c1, d1, b1;
[0143] Dimension field 1 is: b2, b2, d2, a2, b2, a2, b2, b2;
[0144] Dimension field 3 is: b3, d3, a3, c3, b3, c3, b3, c3.
[0145] Based on the primary key encoding algorithm, dimension field 1, dimension field 2 and dimension field 3 are encoded respectively to obtain dimension field encoded data 802. That is, the dimension field encoded data includes: element encoding of dimension field 1, element encoding of dimension field 2 and element encoding of dimension field 3; Table 1 gives one possible dimension field encoded data.
[0146] Table 1: Encoded Data for Dimension Fields
[0147]
[0148] Perform a bitwise AND operation between the element codes of dimension field 1 and the element codes of dimension field 2 to obtain the first bitwise AND operation data 803, which is the binary code of all elements of the joint field.
[0149] Filter all elements in the binary code of this combined field that have a greater than 1 1, and get the filtered element data 804, that is, the filtered elements are: c1&a2, a1&b2, b1&d2. Table 2 shows the filtered element data 804.
[0150] Table 2 Filter Element Data
[0151] c1&a2 00010100 a1&b2 10001000 b1&d2 00100001
[0152] Perform a bitwise AND operation between the filter element code and the element code of dimension field 3 to obtain the second bitwise AND calculation data 805. Table 3 shows the second bitwise AND calculation data.
[0153] Table 3 Second Phase and Calculated Data
[0154] c1&a2&a3 00000000 c1&a2&b3 00000000 c1&a2&c3 00010100 c1&a2&d3 00000000 a1&b2&a3 00000000 a1&b2&b3 10001000 a1&b2&c3 00000000 a1&b2&d3 00000000 b1&d2&a3 00100000 b1&d2&b3 00000000 b1&d2&c3 00000001 b1&d2&d3 00000000
[0155] The second AND calculation data 805 is used to perform the count(count(xi==1)>1). The conditions are met for c1&a2&c3 and a1&b2&b3, meaning the result is 2 and not 0. Therefore, the above steps of encoding, AND calculation, and filtering elements with count(xi==1)>1 need to be repeated until the condition count(count(xi==1)>1)==0 is met. That is, c1&a2&c3 and a1&b2&b3 are ANDed with the element encoding of the new dimension field, and then the result of the AND calculation is checked to see if count(count(xi==1)>1)==0 is met. This process is repeated until count(count(xi==1)>1)==0 is met.
[0156] In the above embodiments, the dimension fields in the data warehouse are extracted based on the active learning algorithm. The encoding order is determined by calculating the information entropy of each dimension field in each data model. The dimension fields are encoded as binary integers, and an AND operation is performed based on the binary integers. By judging whether the number of 1s in the encoded data after the AND operation is greater than 1, it can be determined whether the joint field satisfies the uniqueness principle. This reduces memory space, maximizes computational performance, and improves the lookup efficiency of primary keys for all data models in the entire data warehouse.
[0157] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."
[0158] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0159] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).
[0160] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.
[0161] For example, the processing unit 910 can perform the following steps in the above method embodiment: perform primary key encoding calculation on the dimension fields of the same digital model to generate primary key encoding results; determine the target dimension field based on the primary key encoding results, wherein the target dimension field is the dimension field whose primary key encoding results are not equal to 0; perform AND calculation on the element encoding of the target dimension field to output the joint primary key.
[0162] For example, the processing unit 910 can perform the following steps in the above method embodiment: calculate the information entropy of the dimension fields in the digital model; sort the dimension fields according to the information entropy of the dimension fields; and calculate the primary key encoding of the dimension fields according to the sorting result.
[0163] For example, the processing unit 910 can execute the following steps of the above method embodiment: execute a loop process until a preset condition is met, the loop process includes: obtaining the element code of the new dimension field, wherein the new dimension field is the dimension field most recently input according to the sorting result; performing a bitwise AND operation between the element code of the target dimension field or the filter element code of the joint field and the element code of the new dimension field to generate the element code of the joint field; obtaining the element codes of the joint field whose number of 1s is greater than 1, and generating the filter element code of the joint field; wherein the preset condition is that the number of filter element codes is 0; when the preset condition is met, the joint field is a joint primary key, and the joint primary key is output.
[0164] For example, the processing unit 910 can perform the following steps in the above method embodiment:
[0165] Input the full data warehouse data model; based on the active learning algorithm and combined with the classification model, extract all dimensional fields of all data models in the full data warehouse; calculate the information entropy of each dimensional field of each data model, and sort the dimensional fields in the same data model in descending order according to the information entropy; perform primary key encoding calculation on the dimensional fields in sequence according to the above sorting results;
[0166] If the primary key encoding result above is equal to 0, then this dimension field is the single primary key of this data model; if the primary key encoding result above is not equal to 0, then the next dimension field after sorting needs to be encoded; perform an AND operation on the two dimension fields after the above encoding to obtain the binary encoding of all elements of the joint field;
[0167] Filter all elements in the above composite field whose binary codes contain more than 1 1s. If the number of filtered elements is 0, it indicates that the composite field is the composite primary key of the data model. If the number of filtered elements is greater than 0, the filtered element codes need to be saved. Encode the next dimension field of the sorted data model and perform an AND operation with the filtered element codes. Execute the check if count(count(xi==1)>1)==0. If the condition is met, output the composite primary key of the data model. If the condition is not met, repeat the filtering and AND operation until the condition is met.
[0168] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.
[0169] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0170] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0171] Electronic device 900 can also communicate with one or more external devices 940 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). Such communication can be performed through input / output (I / O) interface 950.
[0172] Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 960. As shown in the figure, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0173] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0174] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.
[0175] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0176] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.
[0177] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0178] In practical implementation, program code for performing the operations of this disclosure can be written using any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0179] In cases involving remote computing devices, the remote computing devices can be connected to user computing devices via any type of network, including local area networks (LANs) or wide area networks (WANs), or they can be connected to external computing devices (e.g., via the Internet using an Internet service provider).
[0180] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0181] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0182] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0183] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A primary key lookup method for a data model, characterized in that, include: Each element in the dimension field of the same digital model is binary encoded to generate an element code, and the number of target elements is calculated as the primary key encoding result, wherein the number of 1s in the element code of the target element is greater than 1. Based on the primary key encoding result, the target dimension field is determined, wherein the target dimension field is the dimension field whose primary key encoding result is not equal to 0; Perform an AND operation on the element codes of the target dimension field to output the composite primary key; The step of performing an AND operation on the element encodings of the target dimension field to output the composite primary key includes executing a loop process until a preset condition is met, the loop process including: Get the element code of the new dimension field, where the new dimension field is the latest target dimension field entered according to the sorting result; The element code of the target dimension field or the filter element code of the joint field is ANDed with the element code of the new dimension field to generate the element code of the joint field. Obtain the element codes of the joint field whose number of 1s is greater than 1, and generate the filter element codes of the joint field; The preset condition is that the number of the filtered element codes is 0; When the preset conditions are met, the composite field becomes the composite primary key, and the composite primary key is output.
2. The primary key lookup method for the data model according to claim 1, characterized in that, Also includes: Calculate the information entropy of the dimension fields in the digital model; Sort the dimension fields according to their information entropy; Based on the sorting results, the primary key encoding of the dimension field is calculated.
3. The primary key lookup method for the data model according to claim 1, characterized in that, Also includes: When the primary key encoding result of a dimension field is 0, the dimension field is the single primary key of the digital model.
4. The primary key lookup method for the data model according to claim 1, characterized in that, Also includes: Obtain the data model of the full data warehouse; The dimensional fields of the data model are obtained based on the active learning algorithm.
5. The primary key lookup method for the data model according to claim 4, characterized in that, The method for obtaining the dimension fields of the data model based on the active learning algorithm includes: Select unlabeled samples, label them, add them to the training set, and build a classification model; According to the query criteria, unlabeled samples are selected, labeled, added to the training set, and the classification model is retrained. Based on the classification model, obtain the dimension fields of the data model.
6. A primary key lookup device for a data model, characterized in that, include: The primary key encoding calculation module performs binary encoding on each element in the dimension field of the same digital model to generate an element encoding, and calculates the number of target elements as the primary key encoding result, wherein the number of 1s in the element encoding of the target element is greater than 1. The target dimension field module determines the target dimension field based on the primary key encoding result, wherein the target dimension field is a dimension field whose primary key encoding result is not equal to 0; The composite field output module performs a bitwise AND operation on the element codes of the target dimension field to output a composite primary key. This bitwise AND operation includes executing a loop until a preset condition is met. The loop includes: obtaining the element code of a new dimension field, where the new dimension field is the latest input target dimension field based on the sorting result; performing a bitwise AND operation on the element code of the target dimension field or the filter element code of the composite field with the element code of the new dimension field to generate the element code of the composite field; obtaining the element codes of the composite field where the number of 1s is greater than 1, and generating the filter element code of the composite field; wherein the preset condition is that the number of filter element codes is 0; when the preset condition is met, the composite field is a composite primary key, and the composite primary key is output.
7. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the primary key lookup method of the data model according to any one of claims 1 to 5 by executing the executable instructions.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the primary key lookup method of the data model according to any one of claims 1 to 5.