Data processing method and device

Abstract

The invention provides a data processing method and device, and the method comprises the steps: determining a first reference table combination corresponding to a query statement according to a query log of a data warehouse; a second reference table combination is determined from the first reference table combination, and the second reference table combination comprises a subset of the first reference table combination; determining a target reference table combination from the second reference table combination according to the query parameters of the second reference table combination; and updating a data public layer of the data warehouse according to the target reference table combination. According to the embodiment of the invention, repeated association calculation and redundant data storage of the data common layer of the data warehouse can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouse, and in particular to a data processing method and device. Background Art

[0002] A data warehouse is a subject-oriented, integrated, non-volatile, and time-variant data collection used to support management decisions. The core of a data warehouse is to build a common data model (CDM). When building a common data layer, developers manually analyze the relationships between data tables, and determine possible common relationships by reading a large amount of source code and relying on their own experience, and then build a common data layer based on the relationships. Summary of the invention

[0003] The present application provides a data processing method and device. The data processing method of the present application can not only improve the update efficiency of the data common layer and the coverage of the association relationship, but also reduce the repeated association calculations and redundant data storage of the data common layer.

[0004] In a first aspect, the present application provides a data processing method, which includes: determining a first reference table combination corresponding to a query statement based on a query log of a data warehouse; determining a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; determining a target reference table combination from the second reference table combination based on query parameters of the second reference table combination; and updating a data common layer of the data warehouse based on the target reference table combination.

[0005] In a second aspect, the present application provides a data processing device, which includes: a first determination module, used to determine a first reference table combination corresponding to a query statement based on a query log of a data warehouse; a second determination module, used to determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; a third determination module, used to determine a target reference table combination from the second reference table combination based on a query parameter of the second reference table combination; and an update module, used to update a data common layer of the data warehouse based on the target reference table combination.

[0006] In a third aspect, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can perform the above-mentioned data processing method.

[0007] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor.

[0008] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned data processing method when executed by a processor.

[0009] The embodiments provided by the present application can determine a first reference table combination corresponding to a query statement based on a query log of a data warehouse, and determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; then, based on query parameters of the second reference table combination, a target reference table combination that is frequently used by users and has an associated relationship is determined from the second reference table combination, and based on the target reference table combination, the data common layer of the data warehouse is updated by constructing a wide table or establishing an associated relationship, thereby not only improving the update efficiency of the data common layer and the coverage of the associated relationship, but also reducing repeated associated calculations and redundant data storage of the data common layer.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0012] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of a processing process of a data processing method provided in an embodiment of the present application;

[0014] Figure 3A block diagram of a data processing device provided in an embodiment of the present application;

[0015] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solution of the present application, the exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0017] In the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0018] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0019] The terms used herein are only used to describe specific embodiments and are not intended to limit the present application. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0021] As mentioned above, a data warehouse is a subject-oriented, integrated, relatively stable data set that reflects historical changes and is used to support management decisions. The core of a data warehouse is to build a data common layer. The data common layer includes a common detail layer (Data Warehouse Detail, DWD) and a common summary layer (Data Warehouse Summary, DWS). The common detail layer DWD is oriented to specific business processes and uses dimensional modeling to build the most fine-grained detail data. The common summary layer DWS is built on top of the common detail layer DWD layer, defines a unified indicator caliber (i.e., the calculation logic of the indicator), and performs summary calculations.

[0022] In the related art, the data common layer is constructed manually. Developers manually analyze the relationships between data tables, and determine possible common relationships by reading a large amount of source code and relying on their own experience, and then construct the data common layer based on the relationships. However, because it is a manual analysis, the construction efficiency of the data common layer is low, and because each developer has different experiences, it is difficult to find all the common relationships and sink them (i.e., build a data model based on the found relationships and store them in the data common layer). Therefore, constructing the data common layer manually is not only inefficient, but also has a low coverage rate of relationships, resulting in many repeated association calculations and redundant data storage in the data common layer.

[0023] In view of this, an embodiment of the present application provides a data processing method, which can determine a first reference table combination corresponding to a query statement based on a query log of a data warehouse, and determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; then determine a target reference table combination from the second reference table combination based on the query parameters of the second reference table combination; and then update the data common layer of the data warehouse based on the target reference table combination.

[0024] An embodiment of the present application can determine a first reference table combination corresponding to a query statement based on a query log of a data warehouse, and determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; then, based on query parameters of the second reference table combination, a target reference table combination that is frequently used by users and has an associated relationship is determined from the second reference table combination, and based on the target reference table combination, the data common layer of the data warehouse is updated by constructing a wide table or establishing an associated relationship, thereby not only improving the update efficiency of the data common layer and the coverage of the associated relationship, but also reducing repeated associated calculations and redundant data storage of the data common layer.

[0025] It should be noted that "updating the data common layer of the data warehouse" in the embodiments of the present application may refer to storing the association relationship determined according to the target reference table combination in the existing data common layer to optimize the data common layer; it may also refer to constructing the data common layer of the data warehouse for the first time according to the association relationship determined by the target reference table combination. Therefore, the description method of "constructing or building the data common layer of the data warehouse" may also be used. In the embodiments of the present application, they have the same meaning, and the following embodiments will use the description method of "updating the data common layer of the data warehouse" for explanation.

[0026] The data processing method of the embodiment of the present application can be used to update the data common layer of the data warehouse. The data common layer of the data warehouse can actively identify and mine the data tables that are commonly used by users and have association relationships, and pre-build corresponding detailed tables or summary tables based on them, so that users can directly access the constructed detailed tables or summary tables when using them. This not only eliminates the need to repeat the association calculation, but also improves the uniformity of the indicator caliber. Therefore, updating the data common layer can reduce repeated association calculations and reduce redundant data storage.

[0027] The update of the data common layer can be a periodic update (also known as a normalized update), and the update period can be set by a technician in this field according to actual needs. For example, the update period is 24 hours, 48 ​​hours, 7 days, 10 days, etc., and this application does not limit this. The update of the data common layer can also be a campaign update (i.e., an abnormal update), that is, an update of the data common layer triggered by external events. For example, an update performed when the data common layer is abnormal or needs to be upgraded can be regarded as a campaign update of the data common layer. Both the periodic update and the campaign update of the data common layer can use the data processing method of the embodiment of the present application. In practical applications, the periodic update and the campaign update of the data common layer can complement each other.

[0028] The data processing method of the embodiment of the present application can be executed by an electronic device, such as a server in a data warehouse. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The data processing method of the embodiment of the present application can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0029] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application. Figure 1 , the method comprising:

[0030] Step S11, determining a first reference table combination corresponding to the query statement according to the query log of the data warehouse.

[0031] The query log of the data warehouse refers to the log recorded by the background of the data warehouse when the user performs a query operation. The query statement is the statement in the query log that performs operations such as reading and writing on the data table of the data warehouse. For example, the query log can be the SQL (Structured Query Language) execution log of the data warehouse, and the query statement can be the SQL statement in the SQL execution log.

[0032] In some possible implementations, the SQL execution log of the data warehouse within a preset time period can be obtained and used as the query log of the data warehouse. The length of the preset time period can be set by those skilled in the art according to actual conditions, and this application does not limit this.

[0033] The query log of the data warehouse includes multiple query statements (such as SQL statements). The following is an example of a query statement:

[0034] select a.c1,a.c2,b.c1,c.c1

[0035] From table1 a left outer join

[0036] Table2 b on a.c3=b.c3

[0037] left outer join

[0038] Table5 c on a.c3=c.c3

[0039] The query log of the data warehouse can be analyzed to obtain the first reference table combination corresponding to each query statement in the query log. The first reference table combination corresponding to the query statement refers to the combination of data tables operated by the query statement. Each query statement corresponds to a first reference table combination. For example, the query statement in the above example operates on the three data tables table1, table2, and table5. Therefore, the first reference table combination corresponding to the above query statement is {table1, table2, table5}.

[0040] In some possible implementations, the query log of the data warehouse can be regarded as a natural language text, and the natural language text is segmented to extract the data table operated by each query statement in the query log, and then obtain the first reference table combination corresponding to each query statement. It should be noted that the reference table in the embodiment of the present application is the data table. Since the data table is referenced by the query statement, the embodiment of the present application uses the term "reference table" for description.

[0041] Step S12: determining a second reference table combination from the first reference table combination.

[0042] The second reference table combination includes a subset of the first reference table combination, and the second reference table combination includes at least two reference tables. That is, the second reference table combination is a subset of all subsets of the first reference table combination in which the number of reference tables is greater than or equal to 2. Since any set is a subset of itself, the second reference table combination also includes the first reference table combination.

[0043] For any first reference table combination, at least one second reference table combination can be determined. When the number of reference tables in the first reference table combination is 2, the first reference table combination can be directly determined as the second reference table combination. When the number of reference tables in the first reference table combination is greater than 2, a subset of the first reference table combination whose number of reference tables is greater than or equal to 2 can be determined as the second reference table combination.

[0044] For example, when the first reference table combination is {table1, table2, table5}, its subsets whose number of reference tables is greater than 2 are {table1, table2}, {table1, table5}, {table1, table2, table5}. Then, from the first reference table combination {table1, table2, table5}, the second reference table combination determined is {table1, table2}, {table1, table5}, {table1, table2, table5}.

[0045] According to the method described in the above example, the corresponding second reference table combination can be determined from each first reference table combination.

[0046] Step S13: determining a target reference table combination from the second reference table combination according to the query parameter of the second reference table combination.

[0047] The query parameter of the second reference table combination can be used to characterize the frequency of the second reference table combination being referenced, the association relationship of the reference tables in the second reference table combination, etc. The query parameter of the second reference table can be, for example, support, confidence, etc. Those skilled in the art can set the specific type of the query parameter according to actual conditions, and this application does not limit this.

[0048] The query parameters of each second reference table combination can be determined respectively, and according to the query parameters of each second reference table combination, a target reference table combination can be determined from all second reference table combinations. The target reference table combination can be regarded as a reference table combination that is frequently used by users and has an associated relationship among all second reference table combinations.

[0049] Step S14: updating the data common layer of the data warehouse according to the target reference table combination.

[0050] In some possible implementations, when updating the data common layer of the data warehouse according to the target reference table combination, wide tables corresponding to each target reference table combination can be established in the data common layer. For example, assuming there are three target reference table combinations, {table1, table2, table5}, {table1, table2}, {table1, table5}, then three wide tables can be established in the data common layer of the data warehouse, the first wide table TableA includes the contents of table1, table2 and table5, the second wide table TableB includes the contents of table1 and table2, and the third wide table TableC includes the contents of table1 and table5.

[0051] After establishing a wide table in the data common layer, query statements can be optimized based on the wide table. For example, after establishing the three wide tables TableA, TableB, and TableC described in the above example, the operations on table1 and table2 in the query statement can be optimized to operations on TableB; the operations on table1, table2, and table5 in the query statement can be optimized to operations on TableA; the operations on table1, table4, and table5 in the query statement can be optimized to operations on TableC and table4; the operations on tables table1, table2, table3, and table5 in the query statement can be optimized to operations on TableA and table3. In these four query statements, the calculation and storage of the wide table TableA are reused. In this way, query statements can be optimized based on wide tables, so that the optimized query statements can directly operate on wide tables without recalculating associations and storing data.

[0052] In some possible implementations, when the data common layer of the data warehouse is updated according to the target reference table combination, the association relationship between the reference tables can also be determined and established according to the target reference table combination, and then the calculation logic or data model corresponding to the association relationship with the reference table in the data common layer is updated to reduce repeated association calculations and redundant data storage in subsequent use.

[0053] According to an embodiment of the present application, it is possible to determine a first reference table combination corresponding to a query statement based on a query log of a data warehouse, and determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; then, based on the query parameters of the second reference table combination, determine a target reference table combination that is frequently used by users and has an association relationship from the second reference table combination, and based on the target reference table combination, update the data common layer of the data warehouse by constructing a wide table or establishing an association relationship, thereby not only improving the update efficiency of the data common layer and the coverage of the association relationship, but also reducing repeated association calculations and redundant data storage of the data common layer.

[0054] The data processing method according to the embodiment of the present application is described below.

[0055] As described above, in step S13, the target reference table combination may be determined from the second reference table combination according to the query parameter of the second reference table combination. In some possible implementations, the query parameter of the second reference table combination may include support, and the support may represent the probability that the reference tables in the second reference table combination co-appear in the first reference table combination. Step S13 may include: determining the support of the second reference table combination; and determining the second reference table combination as the target reference table combination when the support of the second reference table combination is greater than or equal to a support threshold.

[0056] The support of the second reference table combination refers to the probability that the reference tables in the second reference table combination appear simultaneously in the first reference table combination, which indicates how frequently the second reference table combination is referenced. The higher the support of the second reference table combination, the more frequently the second reference table combination is referenced by the query statement. For example, assuming that there are 50 first reference table combinations, the second reference table combination is {table1, table2}, and the number of times that reference tables table1 and table2 appear simultaneously in all first reference table combinations is 30, then the support of the second reference table combination {table1, table2} is 60%.

[0057] In the case where the query parameters include support, the support of each second reference table combination can be determined separately. For any second reference table combination, it can be determined whether the support of the second reference table combination is greater than or equal to a preset support threshold; when the support of the second reference table combination is greater than or equal to the support threshold, it is considered that the second reference table combination is often referenced by the query statement, and there is an association relationship between the reference tables of the second reference table combination, and the second reference table combination can be determined as the target reference table combination. Among them, the support threshold can be set by those skilled in the art according to actual conditions, and this application does not limit its specific value.

[0058] In an embodiment of the present application, when the query parameters include support, the support of the second reference table combination can be determined, and when the support of the second reference table combination is greater than or equal to a support threshold, the second reference table combination is determined as a target reference table combination, thereby being able to determine, from all second reference table combinations, a target reference table combination that is frequently used by users and has an associated relationship.

[0059] In some possible implementations, determining the support of the second reference table combination includes: determining the number of first occurrences of the second reference table combination in the first reference table combination; and determining the support of the second reference table combination as the ratio of the first number of occurrences to the total number of the first reference table combinations.

[0060] When determining the support of the second reference table combination, the first number of times the second reference table combination appears in the first reference table combination and the total number of the first reference table combinations can be determined respectively, and then the ratio of the first number to the total number of the first reference table combinations can be determined as the support of the second reference table combination.

[0061] For example, assuming that the total number of first reference table combinations is 20, and the number of times the second reference table combination appears in these 20 first reference table combinations is 16, then the support degree of the second reference table combination is 80% (16÷20×100%).

[0062] In some possible implementations, the support Support(X,Y) of the second reference table combination {X,Y} may be calculated by the following formula (1):

[0063]

[0064] In formula (1), X represents the first reference table in the second reference table combination {X, Y}, Y represents the other reference tables in the second reference table combination {X, Y}, Q(X∪Y) represents the number of times the second reference table combination {X, Y} appears in all first reference table combinations (i.e., the first number), and M represents the number of first reference table combinations.

[0065] In an embodiment of the present application, it is possible to determine the first number of times a second reference table combination appears in a first reference table combination, and to determine the support of the second reference table combination by taking the ratio of the first number to the total number of the first reference table combinations, thereby being able to quickly and accurately calculate the support of the second reference table combination, thereby improving processing efficiency.

[0066] In some possible implementations, the query parameter of the second reference table combination may further include a confidence level, and the confidence level may indicate the probability that the first reference table combination including one of the reference tables in the second reference table combination also includes other reference tables in the second reference table combination. Step S13 may further include: determining the confidence level of the second reference table combination; and determining the second reference table combination as the target reference table combination when the support level of the second reference table combination is greater than or equal to a support level threshold and the confidence level is greater than or equal to a confidence level threshold.

[0067] The confidence of the second reference table combination may represent the probability that the first reference table combination including one of the reference tables in the second reference table combination also includes other reference tables in the second reference table combination, that is, the probability that other reference tables appear at the same time when one of the reference tables in the second reference table combination appears. For example, assuming that the second reference table combination is {table2, table3}, the confidence of the second reference table combination may represent the probability that the first reference table combination including table2 also includes table3, that is, the probability that table3 appears at the same time when table2 appears in the second reference table combination. For another example, assuming that the second reference table combination is {table2, table3, table5}, the confidence of the second reference table combination may represent the probability that the first reference table combination including table2 also includes table3 and table5, that is, the probability that table3 and table5 appear at the same time when table2 appears in the second reference table combination. The higher the confidence of the second reference table combination, the higher the probability that other reference tables appear at the same time when one of the reference tables in the second reference table combination appears, and the stronger the association between the reference tables.

[0068] In the case where the query parameters of the second reference table combination also include confidence, the confidence of each second reference table combination can be determined separately. For any second reference table combination, it can be determined whether the support of the second reference table combination is greater than or equal to the support threshold, and whether the confidence of the second reference table combination is greater than or equal to the preset confidence threshold; when the support of the second reference table combination is greater than or equal to the support threshold and the confidence is greater than or equal to the confidence threshold, the second reference table combination is determined as the target reference table combination. Among them, if the confidence of the second reference table combination is greater than or equal to the confidence threshold, it can be considered that there is a strong correlation between the reference tables in the second reference table combination. It should be noted that the confidence threshold here can be set by those skilled in the art according to actual conditions, and the present application does not limit its specific value.

[0069] In an embodiment of the present application, when the query parameters also include confidence, the confidence of the second reference table combination can be determined, and when the support of the second reference table combination is greater than or equal to the support threshold and the confidence is greater than or equal to the confidence threshold, the second reference table combination is determined as the target reference table combination, thereby being able to determine, from all second reference table combinations, a target reference table combination that is frequently used by users and has a strong association.

[0070] In some possible implementations, determining the confidence of the second reference table combination includes: determining the first number of times the second reference table combination appears in the first reference table combination; determining the second number of times the first reference table in the second reference table combination appears in the first reference table combination; and determining the ratio of the first number to the second number as the confidence of the second reference table combination.

[0071] When determining the confidence of the second reference table combination, it can be calculated based on the first reference table in the second reference table combination. The first reference table can be any reference table in the second reference table combination. Then, the other reference tables in the second reference table combination except the first reference table can be regarded as the second reference table. In the embodiment of the present application, the first reference table in the second reference table set is regarded as the first reference table.

[0072] When calculating the confidence of the second reference table combination, we can first determine the first number of times the second reference table combination appears in the first reference table combination, and determine the second number of times the first reference table in the second reference table combination appears in the first reference table combination, and then determine the ratio of the first number to the second number as the confidence of the second reference table combination.

[0073] In some possible implementations, the first reference table in the second reference table set is regarded as the first reference table, and the confidence P(Y|X) of the second reference table combination {X, Y} can be calculated by the following formula (2):

[0074]

[0075] In formula (2), X represents the first reference table in the second reference table combination {X, Y} (i.e., the first reference table), Y represents the other reference tables in the second reference table combination {X, Y} (i.e., the second reference table), P(X∪Y) represents the number of times the second reference table combination {X, Y} appears in all first reference table combinations (i.e., the first number), and P(X) represents the number of times X appears in all first reference table combinations (i.e., the second number).

[0076] In an embodiment of the present application, when determining the confidence of the second reference table combination, first determine the first number of times the second reference table combination appears in all first reference table combinations, and the second number of times the first reference table in the second reference table combination appears in all first reference table combinations, and then determine the ratio of the first number to the second number as the confidence of the second reference table combination, thereby being able to quickly and accurately calculate the confidence of the second reference table combination, thereby improving processing efficiency.

[0077] In some possible implementations, when the query parameters include support and confidence, the target reference table combination can be determined from the second reference table combination by frequent item set mining. An item set is a collection of several items (reference tables in this embodiment), and a frequent item set refers to an item set that appears frequently in a data set. The support of a frequent item set is greater than or equal to a preset support threshold.

[0078] The step S13 may include: determining the reference tables in all the second reference table combinations as level 1 itemsets; pruning the level 1 itemsets according to the support and support threshold of the level 1 itemsets to obtain level 1 frequent itemsets; expanding the K-1 level frequent itemsets to obtain the K level itemsets, wherein the K level itemsets do not include pruned itemsets, K is an integer and 2≤K≤N-1, and N represents the number of reference tables in all the second reference table combinations; pruning the K level itemsets according to the support and support threshold of the K level itemsets to obtain the K level frequent itemsets; determining the 2 to K level frequent itemsets as the third reference table combination when the K level frequent itemsets meet the expansion end condition, wherein the expansion end condition is that the K+1 level itemsets obtained by expanding the K level frequent itemsets are empty; determining the third reference table combination whose confidence is greater than or equal to the confidence threshold as the target reference table combination.

[0079] When determining the target reference table combination from the second reference table combination by frequent item set mining, the reference tables in all the second reference table combinations can be first determined as level 1 itemsets. Level 1 itemsets are single item sets, that is, each level 1 itemset includes only one item (one reference table).

[0080] For example, assuming that all second reference table combinations are {table1, table2}, {table2, table3}, {table1, table2, table3, table4}, then the reference tables table1, table2, table3, table4 in all second reference table combinations can be determined as a first-level item set, that is, the first-level item set is {table1}, {table2}, {table3}, {table4}.

[0081] Then the support of each level 1 item set can be determined. The support of an item set can represent the probability that the reference table in the item set appears simultaneously in the first reference table combination. The support of each level 1 item set can be determined by the above formula (1). Then, according to the support of the level 1 item set and the support threshold, the level 1 item set is pruned to obtain the level 1 frequent item set.

[0082] For any level 1 itemset, it can be determined whether the support of the level 1 itemset is greater than or equal to the support threshold. If the support of the level 1 itemset is greater than or equal to the support threshold, the level 1 itemset can be determined to be a level 1 frequent itemset; if the support of the level 1 itemset is less than the support threshold, the level 1 itemset can be determined to be an infrequent itemset and pruned. The pruned itemsets can be recorded for use in subsequent processing.

[0083] For example, after calculation, in the above example, the support of the level 1 item set {table1} is 66.7%, the support of the level 1 item set {table2} is 100%, the support of the level 1 item set {table3} is 66.7%, and the support of the level 1 item set {table4} is 33.3%. Assuming that the support threshold is 50%, then, based on the support and support threshold of the level 1 item sets, the level 1 item sets {table1}, {table2}, and {table3} can be determined as level 1 frequent item sets, and the level 1 item set {table4} can be determined as an infrequent item set, and pruned, that is, the pruned item set is {table4}.

[0084] Then, the K-1 level frequent item set can be expanded to obtain the K-level item set, where K is an integer and 2≤K≤N-1, and N represents the number of reference tables in all second reference table combinations. Among them, the K-1 level frequent item set is a K-1 item set, that is, each K-1 level frequent item set includes K-1 items; the K-level item set is a K-item set, that is, each K-level item set includes K items. When expanding the K-1 level frequent item set, an item can be added to the K-1 level frequent item set to obtain the K-level item set, wherein the added item is different from the item in the K-1 level frequent item set, and the K-level item set does not include the pruned item set.

[0085] The support of a K-level item set can be determined by the above formula (1), and the K-level item set can be pruned according to the support and support threshold of the K-level item set to obtain a K-level frequent item set. For any K-level item set, the support of the K-level item set can be determined by the above formula (1), and then it can be determined whether the support of the K-level item set is greater than or equal to the preset support threshold. If the support of the K-level item set is greater than or equal to the support threshold, the K-level item set can be determined as a K-level frequent item set. If the support of the K-level item set is less than the support threshold, the K-level item set can be determined as an infrequent item set and pruned. The pruned item set can be recorded for use in subsequent processing.

[0086] Then, it can be determined whether the K-level frequent itemset meets the expansion end condition, and the expansion end condition is that the K+1-level itemset obtained by expanding the K-level frequent itemset is empty. In other words, when determining whether the K-level frequent itemset meets the expansion end condition, the K-level frequent itemset can be expanded first to obtain the K+1-level itemset, and then it can be determined whether the K+1-level itemset is empty; if the K+1-level itemset is not empty, it is considered that the K-level frequent itemset does not meet the expansion end condition; if the K+1-level itemset is empty, it is considered that the K-level frequent itemset meets the expansion end condition.

[0087] When the K-level frequent itemset does not meet the expansion end condition, the above method is used to continue to determine the K+1-level frequent itemset: the K-level frequent itemset is expanded to obtain the K+1-level itemset, and then the K+1-level itemset is pruned according to the support and support threshold of the K+1-level itemset to obtain the K+1-level frequent itemset.

[0088] When the K-level frequent item set meets the expansion end condition, the 2 to K-level frequent item sets are determined as the third reference table combination. That is, the third reference table combination includes the 2-level frequent item set, the 3-level frequent item set, ..., the K-level frequent item set.

[0089] For any third reference table combination, the confidence of the third reference table combination can be calculated by the above formula (2). When the confidence of the third reference table combination is greater than or equal to the confidence threshold, the third reference table combination is determined as the target reference table combination.

[0090] According to an embodiment of the present application, when determining a target reference table combination from a second reference table combination by frequent item set mining, firstly, all reference tables in the second reference table combination are determined as level 1 itemsets, and the level 1 itemsets are pruned according to the support and support threshold of the level 1 itemsets to obtain level 1 frequent itemsets; then, the K-1 level frequent itemsets are expanded to obtain level K itemsets, and the level K itemsets are pruned according to the support and support threshold of the level K itemsets to obtain level K frequent itemsets; when the level K frequent itemsets meet the expansion end condition, the level 2 to K frequent itemsets are determined as a third reference table combination; then, the third reference table combination whose confidence is greater than or equal to the confidence threshold is determined as the target reference table combination, so that the target reference table combination can be determined from the second reference table combination quickly and accurately based on the support and confidence, through level-by-level frequent item set mining and pruning of infrequent item sets, thereby improving processing efficiency.

[0091] In some possible implementations, the K-1 level frequent item set is expanded to obtain a K level item set, including: for any K-1 level frequent item set, adding an expansion table to the K-1 level frequent item set to obtain a K level candidate item set, wherein the expansion table is a reference table in all combinations of second reference tables and different from the reference table in the K-1 level frequent item set; when the K level candidate item set does not include a pruned item set, determining the K level candidate item set as the K level item set.

[0092] When expanding the K-1 level frequent itemsets, for any K-1 level frequent itemsets, the expansion table can be added to the K-1 level frequent itemsets to obtain the K-level candidate itemsets. The expansion table is a reference table in all the second reference table combinations and is different from the reference table in the K-1 level frequent itemsets. In other words, the expansion table is a reference table not included in the K-1 level frequent itemsets.

[0093] For example, assuming K = 2, all second reference table combinations are {table1, table2}, {table2, table3}, {table1, table2, table3, table4}, K-1 level frequent itemsets are 1 level frequent itemsets, and 1 level frequent itemsets are {table1}, {table2}, {table3}. When expanding the 1 level frequent itemset {table1}, the expanded tables are table2, table3, table4, and the 2 level candidate itemsets obtained after expansion are {table1, table2}, {table1, table3}, {table1, table4}; When expanding the 1 level frequent itemset {table2}, the expanded tables are table1, table3, table4, and the 2 level candidate itemsets obtained after expansion are {table2, table1}, {table2, table3}, {table2, table4}; When expanding the 1 level frequent itemset {table2}, the expanded tables are table1, table3, table4, and the 2 level candidate itemsets obtained after expansion are {table2, table1}, {table2, table3}, {table2, table4}; e3}, the expanded tables are table1, table2, and table4 respectively, and the second-level candidate item sets obtained after expansion are {table3, table1}, {table3, table2}, and {table3, table4} respectively; after deduplication, the second-level candidate item sets obtained after the expansion of the above-mentioned first-level frequent item sets are obtained, and the final second-level candidate item sets are {table1, table2}, {table1, table3}, {table1, table4}, {table2, table3}, {table2, table4}, and {table3, table4}.

[0094] Then it can be determined whether the K-level candidate itemset includes the pruned itemset. For any K-level candidate itemset, if the K-level candidate itemset includes the pruned itemset, according to the basic principle of frequent itemsets (if a subset is infrequent, then its superset must also be infrequent), the K-level candidate itemset is also an infrequent itemset and needs to be pruned. The meaning of a superset can be, for example: if every element in a set S2 is in a set S1, and set S1 may contain elements that are not in S2, then set S1 is a superset of S2, and conversely, S2 is a subset of S1. In the case that the K-level candidate itemset does not include the pruned itemset, the K-level candidate itemset can be determined as a K-level itemset.

[0095] For example, the 2nd-level candidate itemsets are {table1,table2}, {table1,table3}, {table1,table4}, {table2,table3}, {table2,table4}, {table3,table4}, and the pruned itemset is {table4}, then the 2nd-level candidate itemsets {table1,table2}, {table1,table3}, {table2,table3} excluding the pruned itemset {table4} can be determined as the 2nd-level itemset.

[0096] According to an embodiment of the present application, when a K-1 level frequent item set is expanded, for any K-1 level frequent item set, the expansion table is added to the K-1 level frequent item set to obtain a K-level candidate item set, and then when the K-level candidate item set does not include the pruned item set, the K-level candidate item set is determined as the K-level item set, so that when the K-1 level frequent item set is expanded, the pruned item set is not included, which not only improves the accuracy of the K-level item set but also reduces the amount of subsequent calculations.

[0097] The following will be combined with a specific example to illustrate the process of determining the target reference table combination from the second reference table combination by frequent item set mining. Assuming that the query log of the data warehouse includes 4 SQL statements (i.e., query statements), after analyzing the query log, the first reference table combination corresponding to each SQL statement is obtained. All the first reference table combinations are shown in the following Table 1:

[0098] Table 1 All first reference table combinations

[0099] SQLID Reference table collection SQL1 {table1,table2} SQL2 {table2,table3,table5} SQL3 {table1,table4,table5} SQL4 {table1,table2,table3,table5}

[0100] In Table 1, SQLID represents the ID of the SQL statement, which is used to identify the SQL statement. In Table 1, the first reference table combination corresponding to SQL1 is {table1, table2}, the first reference table combination corresponding to SQL2 is {table2, table3, table5}, the first reference table combination corresponding to SQL3 is {table1, table4, table5}, and the first reference table combination corresponding to SQL4 is {table1, table2, table3, table5}.

[0101] Then, from each first reference table combination, the second reference table combination is determined respectively: from the first reference table combination {table1, table2}, the second reference table combination determined is {table1, table2}; from the first reference table combination {table2, table3, table5}, the second reference table combination determined is {table2, table3}, {table2, table5}, {table3, table5}, {table2, table3, table5}; from the first reference table combination {table1, table4, table5}, the second reference table combination determined is {table1, table4}, {table1, table5}, {table4, From the first reference table combination {table1,table2,table3,table5}, the second reference table combination determined is {table1,table2}, {table1,table3}, {table1,table5}, {table2,table3}, {table2,table5}, {table3,table5}, {table1,table2,table3}, {table1,table2,table5}, {table2,table3,table5} and {table1,table2,table3,table5}

[0102] When mining frequent item sets for all the above second reference table combinations, the reference tables in all the second reference table combinations can be first determined as level 1 item sets, and the level 1 item sets include {table1}, {table2}, {table3}, {table4}, and {table5}. The first occurrence number of each level 1 item set in all the first reference table combinations is counted, and the support of the level 1 item set is calculated by the above formula (1). The support of the level 1 item set is shown in the following Table 2:

[0103] Table 2 Support of level 1 itemsets

[0104] Level 1 Itemset First count Support {table1} 3 75％ {table2} 3 75％ {table3} 2 50％ {table4} 1 25％ {table5} 3 75％

[0105] Assuming the support threshold is 50%, the level 1 itemsets are pruned according to the support and support threshold of the level 1 itemsets, and the level 1 itemsets {table4} are deleted to obtain the level 1 frequent itemsets. The level 1 frequent itemsets are {table1}, {table2}, {table3}, and {table5}.

[0106] The method described in the above embodiment can be used to expand the level 1 frequent item set to obtain the level 2 item set. The level 1 frequent item set can also be expanded by merging the level 1 frequent item sets in pairs and removing the duplicates to obtain the level 2 item set. The number of first occurrences of each level 2 item set in all first reference table combinations is counted, and the support of the level 2 item set is calculated by the above formula (1). The support of the level 2 item set is shown in the following Table 3:

[0107] Table 3 Support of level 2 itemsets

[0108] Level 2 Itemset First count Support {table1,table2} 2 50％ {table1,table3} 1 25％ {table1,table5} 2 50％ {table2,table3} 2 50％ {table2,table5} 2 50％ {table3,table5} 2 50％

[0109] According to the support and support threshold of the level 2 itemsets, the level 2 itemsets are pruned, and the level 2 itemsets {table1, table3} are deleted to obtain the level 2 frequent itemsets. The level 2 frequent itemsets are {table1, table2}, {table1, table5}, {table2, table3}, {table2, table5} and {table3, table5}.

[0110] The method described in the above embodiment can be used to expand the level 2 frequent item sets to obtain level 3 item sets. The number of first occurrences of each level 3 item set in all first reference table combinations is counted, and the support of the level 3 item set is calculated using the above formula (1). The support of the level 3 item set is shown in the following Table 4:

[0111] Table 4 Support of level 3 itemsets

[0112] Level 3 Itemset First count Support {table1,table2,table5} 1 25％ {table2,table3,table5} 2 50％

[0113] According to the support and support threshold of the level 3 itemset, the level 3 itemset is pruned, and the level 3 itemset {table1, table2, table5} is deleted to obtain the level 3 frequent itemset. The level 3 frequent itemset is {table2, table3, table5}.

[0114] The method described in the above embodiment can be used to expand the level 3 frequent item set to obtain the level 4 item set. For example, the level 3 frequent item set is expanded using the expansion tables table1 and table4 respectively to obtain two level 4 candidate item sets {table1, table2, table3, table5} and {table2, table3, table4, table5}. Since the level 4 candidate item set {table1, table2, table3, table5} includes the pruned item set {table1, table3}, and the level 4 candidate item set {table2, table3, table4, table5} includes the pruned item set {table4}, both level 4 candidate item sets are pruned, and the obtained level 4 item set is empty. It can be considered that the level 3 frequent item set meets the expansion end condition, and the expansion is stopped.

[0115] Then the level 2 frequent item sets and the level 3 frequent item sets are determined as the third reference table combination. The third reference table combination is {table1, table2}, {table1, table5}, {table2, table3}, {table2, table5}, {table3, table5} and {table2, table3, table5}.

[0116] For any third reference table combination, its confidence can be calculated by the above formula (2). The following Table 5 exemplarily shows the support and confidence of the third reference table combination:

[0117] Table 5 Support and confidence of the third reference table combination

[0118] Third reference table combination Support Confidence {table1,table2} 50％ 66.7％ {table1,table5} 50％ 66.7％ {table2,table3} 50％ 66.7％ {table2,table5} 50％ 66.7％ {table3,table5} 50％ 100％ {table2,table3,table5} 50％ 66.7％

[0119] Assuming that the support threshold is 50% and the confidence threshold is 60%, then the support of all third reference table combinations in Table 5 is equal to the support threshold, and the confidence is greater than the confidence threshold. All third reference table combinations can be determined as target reference table combinations.

[0120] Assuming that the support threshold is 50% and the confidence threshold is 70%, then among the 6 third reference table combinations in Table 5, only the third reference table combination {table3, table5} has a support equal to the support threshold and a confidence greater than the confidence threshold. The third reference table combination {table3, table5} can be determined as the target reference table combination.

[0121] It should be noted that the specific values ​​of the support threshold and the confidence threshold can be set by technical personnel in this field according to actual conditions, and this application does not impose any restrictions on this.

[0122] In some possible implementations, the step S14 may include: determining a maximum frequent itemset of the target reference table combination; and updating a data common layer of the data warehouse according to the maximum frequent itemset.

[0123] Since the maximum frequent itemset includes the data of the subset, the maximum frequent itemset can be used when updating the data common layer. Specifically, the maximum frequent itemset of the target reference table combination can be determined, and the data common layer of the data warehouse can be updated according to the maximum frequent itemset.

[0124] For example, assuming that the target reference table combination is {table1, table2}, {table1, table5}, {table2, table3}, {table2, table5}, {table3, table5} and {table2, table3, table5}, since {table2, table3, table5} includes {table2, table3}, {table2, table5} and {table3, table5}, {table2, table3, table5} can be regarded as a maximum frequent item set; since {table1, table2} and {table1, table5} are not subsets of any target reference table combination, {table1, table2} and {table1, table5} can be regarded as maximum frequent item sets. Therefore, the three maximum frequent item sets {table2, table3, table5}, {table1, table2} and {table1, table5} can be determined from the above target reference table combination, and then the data common layer of the data warehouse can be updated according to these three maximum frequent item sets.

[0125] In an embodiment of the present application, the maximum frequent item set of the target reference table combination can be determined, and the data common layer of the data warehouse can be updated according to the maximum frequent item set, so that the data common layer can be updated by the maximum frequent item set of the target reference table combination, further reducing repeated association calculations and redundant data storage of the data common layer.

[0126] In some possible implementations, determining the maximum frequent itemset of the target reference table combination includes: determining all target reference table combinations as a target set; and determining the target reference table combination whose superset is not in the target set as a maximum frequent itemset.

[0127] When determining the maximum frequent itemset of a target reference table combination, all target reference table combinations may be first determined as a target set, and then for any target reference table combination, it is determined whether there is a superset of the target reference table combination in the target set. When there is no superset of the target reference table combination in the target set, the target reference table combination is determined as the maximum frequent itemset.

[0128] For example, assuming that the target reference table combinations are {table1, table2}, {table1, table5}, {table2, table3}, {table2, table5}, {table3, table5} and {table2, table3, table5}, the target set includes these five target reference table combinations. For the target reference table combination {table1, table2}, there is no superset of {table1, table2} in the target set, so {table1, table2} is determined as the maximum frequent item set; for the target reference table combination {table1, table5}, there is no superset of {table1, table5} in the target set, so {table1, table5} is determined as the maximum frequent item set; For the target reference table combination {table2,table3}, there is a superset {table2,table3,table5} of {table2,table3} in the target set, so {table2,table3} is not the maximum frequent itemset; for the target reference table combination {table3,table5}, there is a superset {table2,table3,table5} of {table3,table5} in the target set, so {table3,table5} is not the maximum frequent itemset; for the target reference table combination {table2,table3,table5}, there is no superset {table2,table3,table5} in the target set, so {table2,table3,table5} is determined to be the maximum frequent itemset.

[0129] In an embodiment of the present application, when determining the maximum frequent item set of a target reference table combination, all target reference table combinations are determined as a target set, and then the target reference table combination whose superset is not in the target set is determined as the maximum frequent item set, which is fast and accurate, thereby improving the processing efficiency when determining the maximum frequent item set of a target reference table combination.

[0130] Figure 2 A schematic diagram of a processing process of a data processing method provided in an embodiment of the present application. Figure 2 , the data processing method of the embodiment of the present application includes:

[0131] Step S201, determining a first reference table combination corresponding to a query statement according to a query log of a data warehouse;

[0132] Step S202, determining a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination;

[0133] Step S203, determining the support of the second reference table combination;

[0134] Step S203, determining the confidence level of the second reference table combination;

[0135] Step S205, when the support of the second reference table combination is greater than or equal to the support threshold and the confidence is greater than or equal to the confidence threshold, determine the second reference table combination as the target reference table combination;

[0136] Step S206, determining the maximum frequent itemset of the target reference table combination;

[0137] Step S207: updating the data common layer of the data warehouse according to the maximum frequent itemset.

[0138] The data processing method of the embodiment of the present application mines out data tables that appear frequently and have association relationships from the query logs of the data warehouse, and then establishes corresponding wide tables in the data common layer, so that subsequent queries can be performed based on the wide tables, that is, users can directly use the wide tables without having to re-perform association calculations and data storage, thereby reducing repeated association calculations and redundant data storage in the data common layer.

[0139] It can be understood that the above-mentioned various method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, this application will not go into details. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined by its function and possible internal logic.

[0140] In addition, the present application also provides a data processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any data processing method provided by the present application. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0141] Figure 3 A block diagram of a data processing device provided in an embodiment of the present application.

[0142] Reference Figure 3 , the embodiment of the present application provides a data processing device, the data processing device 30 includes:

[0143] A first determination module 31, configured to determine a first reference table combination corresponding to a query statement according to a query log of a data warehouse;

[0144] A second determination module 32, configured to determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination;

[0145] A third determination module 33, configured to determine a target reference table combination from the second reference table combination according to a query parameter of the second reference table combination;

[0146] The updating module 34 is used to update the data common layer of the data warehouse according to the target reference table combination.

[0147] In some possible implementations, the query parameters include support, and the support represents the probability that the reference tables in the second reference table combination appear simultaneously in the first reference table combination; the third determination module 33 includes: a support determination submodule, used to determine the support of the second reference table combination; and a first determination submodule, used to determine the second reference table combination as the target reference table combination when the support of the second reference table combination is greater than or equal to a support threshold.

[0148] In some possible implementations, the support determination submodule is used to: determine the first number of times the second reference table combination appears in the first reference table combination; and determine the support of the second reference table combination as the ratio of the first number of times to the total number of the first reference table combinations.

[0149] In some possible implementations, the query parameters also include confidence, and the confidence represents the probability that the first reference table combination including one reference table in the second reference table combination also includes other reference tables in the second reference table combination; the third determination module 33 includes: a confidence determination submodule, used to determine the confidence of the second reference table combination; a second determination submodule, used to determine the second reference table combination as the target reference table combination when the support of the second reference table combination is greater than or equal to a support threshold and the confidence is greater than or equal to a confidence threshold.

[0150] In some possible implementations, the confidence determination submodule is used to: determine the first number of times the second reference table combination appears in the first reference table combination; determine the second number of times the first reference table in the second reference table combination appears in the first reference table combination; and determine the ratio of the first number to the second number as the confidence of the second reference table combination.

[0151] In some possible implementations, the updating module 34 includes: a third determining submodule, configured to determine a maximum frequent itemset of the target reference table combination; and an updating submodule, configured to update the data common layer of the data warehouse according to the maximum frequent itemset.

[0152] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application.

[0153] Reference Figure 4 An embodiment of the present application provides an electronic device, which includes: at least one processor 701; at least one memory 702, and one or more I / O interfaces 703, connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by at least one processor 701, and the one or more computer programs are executed by at least one processor 701 so that the at least one processor 701 can execute the above-mentioned data processing method.

[0154] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data processing method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0155] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0156] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0157] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0158] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0159] The computer program instructions for performing the operation of the present application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0160] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0161] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0162] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0163] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0164] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0165] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. A data processing method, characterized in that: include: Determine a first reference table combination corresponding to the query statement according to the query log of the data warehouse; Determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; Determining a target reference table combination from the second reference table combination according to a query parameter of the second reference table combination; The data common layer of the data warehouse is updated according to the target reference table combination.

2. The method according to claim 1, characterized in that The query parameter includes support, and the support represents the probability that the reference tables in the second reference table combination appear simultaneously in the first reference table combination; The step of determining a target reference table combination from the second reference table combination according to the query parameter of the second reference table combination includes: Determining the support of the second reference table combination; When the support degree of the second reference table combination is greater than or equal to a support degree threshold, the second reference table combination is determined as a target reference table combination.

3. The method according to claim 2, characterized in that The determining the support degree of the second reference table combination includes: Determine the first number of times the second reference table combination appears in the first reference table combination; The ratio of the first number of times to the total number of the first reference table combinations is determined as the support degree of the second reference table combination.

4. The method according to claim 2, characterized in that: The query parameter further includes a confidence level, wherein the confidence level indicates a probability that the first reference table combination including one reference table in the second reference table combination also includes other reference tables in the second reference table combination; The step of determining a target reference table combination from the second reference table combination according to the query parameter of the second reference table combination includes: determining a confidence level of the second reference table combination; When the support of the second reference table combination is greater than or equal to a support threshold, and the confidence is greater than or equal to a confidence threshold, the second reference table combination is determined as a target reference table combination.

5. The method according to claim 4, characterized in that Determining the confidence of the second reference table combination includes: Determine the first number of times the second reference table combination appears in the first reference table combination; determining a second number of times a first reference table in the second reference table combination appears in the first reference table combination; The ratio of the first number of times to the second number of times is determined as the confidence of the second reference table combination.

6. The method according to claim 1, characterized in that The updating of the data common layer of the data warehouse according to the target reference table combination includes: Determine the maximum frequent itemset of the target reference table combination; The data common layer of the data warehouse is updated according to the maximum frequent itemset.

7. The method according to claim 6, characterized in that The determining of the maximum frequent item set of the target reference table combination includes: Combine all target reference tables to determine the target set; The target reference tables whose supersets are not in the target set are combined to determine the maximum frequent itemset.

8. A data processing device, characterized in that: include: A first determination module, used to determine a first reference table combination corresponding to the query statement according to the query log of the data warehouse; A second determining module, configured to determine a second reference table combination from the first reference table combination, wherein the second reference table combination includes a subset of the first reference table combination; A third determination module, configured to determine a target reference table combination from the second reference table combination according to a query parameter of the second reference table combination; An updating module is used to update the data common layer of the data warehouse according to the target reference table combination.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images