Data processing method and device, computer device, storage medium and program product

Abstract

The application relates to a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring multiple rows of incremental data in a detail table; respectively for each row of incremental data, mapping multiple dimension field values in the incremental data into one value to obtain a detail data identifier of the incremental data; acquiring a summary table corresponding to the detail table; constructing a filtering condition according to the detail data identifiers of the multiple rows of incremental data; filtering at least one row of candidate summary data from the multiple rows of summary data according to the filtering condition and taking the summary data identifier as the basis; determining a summary mode of the incremental data based on the dimension field values contained in each row of incremental data and the dimension field values contained in the at least one row of candidate summary data; and according to the summary mode of each row of incremental data, summarizing the measurement field values of each row of incremental data into the summary table. The method can improve the data summarization efficiency and reduce the consumption of computing resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, a massive amount of data has been generated. To fully leverage the value of this data, multidimensional data analysis techniques have emerged. Multidimensional data analysis organizes data into complex, multidimensional data models to enable the viewing and understanding of the data. Multidimensional data analysis typically includes detail tables and summary tables. Detail tables record detailed data, such as transaction details and material details; while summary tables are reports that summarize and calculate based on the detailed data in the detail tables to extract the overall characteristics of the data. Both detail tables and summary tables include multiple dimension fields and at least one measure field. When summarizing detailed data from the detail tables into the summary table, the value of the measure field is typically summarized based on the common values ​​of the same fields under multiple dimension fields in both tables.

[0003] However, as business grows, the amount of data generated also increases. The amount of existing summary data in the summary table is usually large. When adding detailed data to the detail table, the traditional method of summarizing and calculating the data is inefficient and consumes a lot of computing resources because it is necessary to perform summary calculations based on the new detailed data and all the existing summary data in the summary table. Even if the amount of new detailed data is small, the traditional method of summarizing and calculating the data results in low data summarization efficiency and high computing resource consumption. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can solve the above-mentioned technical problems.

[0005] In a first aspect, this application provides a data processing method, including:

[0006] Retrieve multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​and at least one measure field value.

[0007] For each row of incremental data, the values ​​of the multiple dimension fields in the incremental data are mapped to a single value to obtain the detailed data identifier of the incremental data;

[0008] Obtain the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0009] Filtering conditions are constructed based on the detailed data identifiers of the multiple rows of incremental data. Using the filtering conditions, at least one row of candidate summary data is selected from the multiple rows of summary data based on the summary data identifier.

[0010] Based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate summary data, the aggregation method for the incremental data is determined.

[0011] According to the respective aggregation methods of the multiple rows of incremental data, the values ​​of the respective measurement fields of the multiple rows of incremental data are aggregated into the summary table.

[0012] Secondly, this application also provides a data processing apparatus, comprising:

[0013] The acquisition module is used to acquire multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​of specified multiple dimension fields and at least one measure field value.

[0014] The mapping module is used to map the values ​​of the multiple dimension fields in the incremental data to a single value for each row of incremental data, thereby obtaining the detailed data identifier of the incremental data.

[0015] The acquisition module is also used to acquire the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0016] The filtering module is used to construct filtering conditions based on the detailed data identifiers of the multiple rows of incremental data, and to use the filtering conditions to filter at least one row of candidate summary data from the multiple rows of summary data based on the summary data identifier.

[0017] The aggregation module is used to determine the aggregation method for the incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate aggregated data; and to aggregate the measurement field values ​​of the multiple rows of incremental data into the aggregation table according to the respective aggregation methods of the multiple rows of incremental data.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0019] Retrieve multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​and at least one measure field value.

[0020] For each row of incremental data, the values ​​of the multiple dimension fields in the incremental data are mapped to a single value to obtain the detailed data identifier of the incremental data;

[0021] Obtain the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0022] Filtering conditions are constructed based on the detailed data identifiers of the multiple rows of incremental data. Using the filtering conditions, at least one row of candidate summary data is selected from the multiple rows of summary data based on the summary data identifier.

[0023] Based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate summary data, the aggregation method for the incremental data is determined.

[0024] According to the respective aggregation methods of the multiple rows of incremental data, the values ​​of the respective measurement fields of the multiple rows of incremental data are aggregated into the summary table.

[0025] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0026] Retrieve multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​and at least one measure field value.

[0027] For each row of incremental data, the values ​​of the multiple dimension fields in the incremental data are mapped to a single value to obtain the detailed data identifier of the incremental data;

[0028] Obtain the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0029] Filtering conditions are constructed based on the detailed data identifiers of the multiple rows of incremental data. Using the filtering conditions, at least one row of candidate summary data is selected from the multiple rows of summary data based on the summary data identifier.

[0030] Based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate summary data, the aggregation method for the incremental data is determined.

[0031] According to the respective aggregation methods of the multiple rows of incremental data, the values ​​of the respective measurement fields of the multiple rows of incremental data are aggregated into the summary table.

[0032] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0033] Retrieve multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​and at least one measure field value.

[0034] For each row of incremental data, the values ​​of the multiple dimension fields in the incremental data are mapped to a single value to obtain the detailed data identifier of the incremental data;

[0035] Obtain the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0036] Filtering conditions are constructed based on the detailed data identifiers of the multiple rows of incremental data. Using the filtering conditions, at least one row of candidate summary data is selected from the multiple rows of summary data based on the summary data identifier.

[0037] Based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate summary data, the aggregation method for the incremental data is determined.

[0038] According to the respective aggregation methods of the multiple rows of incremental data, the values ​​of the respective measurement fields of the multiple rows of incremental data are aggregated into the summary table.

[0039] The aforementioned data processing methods, apparatus, computer equipment, computer-readable storage media, and computer program products, since detailed data identifiers are obtained by mapping multiple dimension field values ​​in incremental data to a single value, and summary data identifiers are obtained by mapping multiple dimension field values ​​in summary data to a single value, can quickly filter at least one row of candidate summary data from multiple rows of summary data by using filtering conditions constructed based on the detailed data identifiers of each row of incremental data and relying on the summary data identifiers. Since at least one row of candidate summary data is a partial summary of multiple rows of summary data, the summarization method for incremental data can be quickly determined based on the multiple dimension field values ​​of each row of incremental data and the multiple dimension field values ​​of at least one row of candidate summary data, thereby improving the data summarization efficiency of summarizing incremental data into a summary table. In addition, there is no need to perform summary processing based on the field values ​​of all summary data, reducing the consumption of computing resources. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0042] Figure 2 This is a schematic diagram illustrating the structure of the detail table and summary table in one embodiment;

[0043] Figure 3 This is a flowchart illustrating the data filtering steps in one embodiment;

[0044] Figure 4 This is a flowchart illustrating the parallel aggregation steps in one embodiment;

[0045] Figure 5 This is a structural block diagram of a data processing device in one embodiment;

[0046] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] In one embodiment, such as Figure 1 As shown, a data processing method is provided. This embodiment illustrates the method by applying it to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be a personal computer, a laptop, a smartphone, or a tablet. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps:

[0049] Step 102: Obtain multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​of specified multiple dimension fields and at least one measure field value.

[0050] A detail table is a spreadsheet that records detailed data. A detail table can be a sales detail table, a material detail table, or something else. A detail table can store multiple rows of detailed data. Each row of detailed data can contain a set of dimension field values ​​from a set of dimension fields, and a measure field value from at least one measure field. A dimension field set is the collection of all dimension fields contained in the detail table or summary table. A dimension field value set is the collection of the dimension field values ​​of each dimension field in the dimension field set. Multiple rows of incremental data are detailed data from the detail table that have not been summarized into the summary table.

[0051] Multiple dimension fields are specified dimension fields, which can be all dimension fields in the set of dimension fields, or a specified subset of dimension fields. In other words, the set of dimension fields can include only multiple dimension fields, or it can include multiple dimension fields along with other dimension fields. For example, the dimension fields in the set can be organization, period, subject, and product; multiple dimension fields can be organization, subject, and product, while other dimension fields can be period. Dimension fields are fields used to represent descriptive content of data. Dimension fields are used to categorize and define data. Dimension fields can be text, time, or other non-numeric fields, or they can be represented numerically.

[0052] At least one measure field can be one or more measure fields. A measure field is a numeric field. The value of a measure field can be used for calculations, such as summing, averaging, finding the maximum or minimum value, or other calculations. For example, at least one measure field can be the sales amount over a period of time, or it can be the beginning balance (e.g., the account balance when no transactions occurred this month), the current period's transactions (e.g., the transaction amount of a single transaction during the month), or the ending balance (e.g., the remaining account balance at the end of the month).

[0053] In one embodiment, the server can retrieve multiple rows of incremental data from a detail table in response to a data aggregation trigger event triggered on that detail table. The data aggregation trigger event is an event that triggers the aggregation of multiple rows of incremental data into a summary table. The data aggregation trigger event can be an automatically triggered event, for example, checking whether each row of detail data in the detail table has been aggregated into the summary table; or it can be an automatically triggered event that occurs at preset time intervals when detail data not yet aggregated into the summary table is found. The data aggregation trigger event can also be a manually triggered operation, for example, if there is a corresponding data aggregation trigger control in the detail table, the data aggregation trigger event can be an operation that triggers the data aggregation trigger control.

[0054] Step 104: For each row of incremental data, map the values ​​of multiple dimension fields in the incremental data to a single value to obtain the detailed data identifier of the incremental data.

[0055] The detail data identifier is used to identify the distinct values ​​of multiple dimension fields in each row of incremental data. The detail data identifier can be a number or a string. Different incremental data may contain different values ​​of multiple dimension fields, or they may contain the same values ​​of multiple dimension fields. When different incremental data contain the same values ​​of multiple dimension fields, these incremental data containing the same values ​​of multiple dimension fields can have the same detail data identifier.

[0056] For example, when multiple dimension fields are organization, subject, and product, the values ​​of multiple dimension fields in incremental data 1 can be department 1, 1001, and product A; the values ​​of multiple dimension fields in incremental data 2 can be department 2, 1001, and product B; and the values ​​of multiple dimension fields in incremental data 3 can be department 1, 1001, and product A. In this case, incremental data 1 and incremental data 3 have the same detailed data identifier, while the detailed data identifier of incremental data 2 can be different from the detailed data identifiers of incremental data 1 and incremental data 3.

[0057] In one embodiment, the server may store a pre-generated value mapping table. The value mapping table may contain multiple rows of value mapping information. Each row of value mapping information may include multiple dimension field values ​​and a corresponding numerical value. The multiple dimension field values ​​contained in the value mapping information of different rows are different, and they also correspond to different numerical values. In this embodiment, the server can obtain the pre-generated value mapping table and, for each row of incremental data, query a row of value mapping information containing multiple dimension field values ​​from the value mapping table. The multiple dimension field values ​​in the incremental data are mapped to the numerical value contained in the queried row of value mapping information, thus obtaining the detailed data identifier of the incremental data. For example, when the multiple dimension field values ​​are Department 1, 1001, and Product A, the corresponding numerical value could be 1; when the multiple dimension field values ​​are Department 1, 1001, and Product B, the corresponding numerical value could be 2.

[0058] In one embodiment, the server can use a hash function to map the values ​​of multiple dimension fields in each row of incremental data to a hash value, thereby obtaining the detailed data identifier of the incremental data. The hash function is a mathematical algorithm that converts data of any length into a fixed-length string. This fixed-length string is the hash value. The hash function can be either MD5 (Message-Digest Algorithm 5) or SHA (Secure Hash Algorithm).

[0059] Step 106: Obtain the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of multiple dimension fields, at least one measure field value, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0060] The summary table is a spreadsheet that records summary data. Summary data can be obtained by summarizing historical detailed data from the detail tables. The summary data identifier is used to identify the unique multi-dimensional field values ​​in each row of summary data. The summary data identifier is obtained by mapping multiple dimension field values ​​in a given row of summary data to a single value. When mapping the summary data identifier, the same hash function used to map the detail data identifier can be used; therefore, if multiple dimension field values ​​in a row of detail data are the same as those in a row of summary data, then the detail data identifier for that row of detail data is the same as the summary data identifier for that row of summary data; if the summary data identifier for a row of summary data is the same as the detail data identifier for a row of detail data, then the multiple dimension field values ​​for that row of summary data may be the same as those in that row of detail data; if the summary data identifier for a row of summary data is different from the detail data identifier for that row of detail data, then the multiple dimension field values ​​for that row of summary data are definitely different from those in that row of detail data.

[0061] Each row of summary data can contain a set of dimension field values ​​for a set of dimension fields and a set of measure field values ​​for a set of measure fields. In multi-row summary data, the set of dimension field values ​​differs for each row. Within each row of summary data, the set of dimension field values ​​for a set of dimension fields can include only the values ​​of multiple dimension fields, or it can include the values ​​of multiple dimension fields plus other dimension field values. Similarly, the set of measure field values ​​for a set of measure fields can include only the values ​​of at least one measure field, or it can include the values ​​of other measure fields. For example, at least one measure field could be the sales amount for a single period, while other measure fields could be the sales amount for multiple periods, such as a month, a quarter, or a year.

[0062] In one embodiment, the server can determine a first table identifier for the detail table and query a second table identifier that is bound to the first table identifier. If the table type corresponding to the second table identifier is a summary type, the server retrieves the data table corresponding to the second table identifier to obtain the summary table corresponding to the detail table. The first table identifier is used to identify the detail table and can be the name or number of the detail table. The second table identifier is used to identify the summary table and can be the name or number of the summary table.

[0063] Step 108: Construct filtering conditions based on the detailed data identifiers of each row of incremental data, and use the filtering conditions to filter at least one row of candidate summary data from the multiple rows of summary data based on the summary data identifier.

[0064] Among them, at least one candidate summary data row is the summary data selected from multiple summary data rows by filtering conditions.

[0065] In one embodiment, both the detailed data identifier and the summary data identifier can be numerical values. The detailed data identifiers of multiple rows of incremental data can be arranged according to their numerical values, and the summary data identifiers of multiple rows of summary data can also be arranged according to their numerical values. Therefore, the filtering condition can be the numerical range formed by the largest and smallest detailed data identifiers among the detailed data identifiers of the multiple rows of incremental data. In this embodiment, the server can filter out summary data whose summary data identifiers fall within the aforementioned numerical range from the multiple rows of summary data, and determine the filtered summary data as at least one row of candidate summary data. For example, if the numerical range formed by the detailed data identifiers of the multiple rows of incremental data can be 20 to 100, then the at least one row of candidate summary data is the summary data with summary data identifiers between 20 and 100.

[0066] In one embodiment, the detailed data identifier and the summary data identifier can be strings. In this embodiment, the server can construct a Bloom filter based on the detailed data identifiers of each of the multiple rows of incremental data, and filter at least one row of candidate summary data from the multiple rows of summary data based on the Bloom filter and the summary data identifiers of each of the multiple rows of summary data. The Bloom filter contains a binary vector and a series of mapping functions.

[0067] Step 110: Determine the aggregation method for the incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in at least one row of candidate summary data.

[0068] The dimension field values ​​contained in each row of incremental data are all the dimension field values ​​contained in that incremental data, that is, the set of dimension field values ​​under the dimension field set of the detail table for that incremental data. It can be understood that the dimension field values ​​contained in each row of incremental data can include only multiple dimension field values ​​of the incremental data, or it can include multiple dimension field values ​​of the incremental data plus other dimension field values ​​of the incremental data.

[0069] The dimension field values ​​contained in at least one row of candidate summary data are all the dimension field values ​​contained in each of the at least one row of candidate summary data; that is, the set of dimension field values ​​of each of the at least one row of candidate summary data under the dimension field set of the summary table. It can be understood that the dimension field values ​​contained in at least one row of candidate summary data may include only multiple dimension field values ​​from the at least one row of candidate summary data, or it may include multiple dimension field values ​​from the at least one row of candidate summary data, and other dimension field values ​​from the at least one row of candidate summary data. Detail tables and summary tables can have the same set of dimension fields. The summary method is used to indicate how to summarize multiple rows of incremental data into a summary table.

[0070] In one embodiment, the server can, for each row of incremental data, query target candidate summary data from at least one row of candidate summary data whose dimension field values ​​are the same as those of the incremental data, and obtain query results; when the query results indicate that target candidate summary data has been found, the aggregation method for the incremental data is determined to be the first aggregation method; when the query results indicate that target candidate summary data has not been found, the aggregation method for the incremental data is determined to be the second aggregation method.

[0071] In one embodiment, the server can determine, for each row of incremental data, whether there exists a row of candidate summary data in at least one row of candidate summary data whose summary data identifier is the same as the detail data identifier of the incremental data, and obtain a determination result; when the determination result indicates that there exists a row of candidate summary data whose summary data identifier is the same as the detail data identifier of the incremental data, and the dimension field value contained in the row of candidate summary data is the same as the dimension field value contained in the incremental data, then the aggregation method for the incremental data is determined to be the first aggregation method.

[0072] In one embodiment, when the judgment result indicates that the detailed data identifier of the incremental data is different from the summary data identifier of any summary data identifier in at least one row of candidate summary data, the server can determine that the summary method for the incremental data is the second summary method.

[0073] Step 112: According to the respective aggregation methods of the multiple rows of incremental data, aggregate the values ​​of the measurement fields of the multiple rows of incremental data into the summary table.

[0074] In one embodiment, the server may, for each row of incremental data in the first aggregation method, aggregate the measurement field value of the incremental data into a summary table based on the dimension field value contained in the incremental data.

[0075] In one embodiment, the server may summarize each of the incremental data in the second summary method for multiple rows of incremental data based on the measure field value of the incremental data and the measure field value of the target candidate summary data, so as to summarize the measure field value of the incremental data into a summary table.

[0076] In the above data processing, since the detailed data identifier is obtained by mapping multiple dimension field values ​​in the incremental data to a single value, and the summary data identifier is obtained by mapping multiple dimension field values ​​in the summary data to a single value, by using the filtering conditions constructed based on the detailed data identifiers of each row of incremental data, and using the summary data identifier as the basis, at least one row of candidate summary data can be quickly initially filtered from the multiple rows of summary data. Since at least one row of candidate summary data is a partial summary data in the multiple rows of summary data, based on the multiple dimension field values ​​of each row of incremental data and the multiple dimension field values ​​of at least one row of candidate summary data, the summarization method for the incremental data can be quickly determined, thereby improving the data summarization efficiency of summarizing incremental data into a summary table. In addition, there is no need to perform summary processing based on the field values ​​of all summary data, reducing the consumption of computing resources.

[0077] In one embodiment, step 110 includes: for each row of incremental data, querying target candidate summary data from at least one row of candidate summary data to obtain a query result; the dimension field values ​​contained in the target candidate summary data are the same as the dimension field values ​​contained in the incremental data; when the query result indicates that the target candidate summary data has been found, determining that the aggregation method for the incremental data is the first aggregation method; step 112 includes: aggregating the measurement field values ​​of the incremental data and the measurement field values ​​of the target candidate summary data to obtain a aggregation result; updating the measurement field values ​​of the target candidate summary data according to the aggregation result.

[0078] The summary result is obtained by summarizing the measure field values ​​of the incremental data and the measure field values ​​of the target candidate summary data according to the summary calculation logic of at least one measure field in the summary table. The summary calculation logic can be a calculation logic that performs summation, averaging, finding the maximum value, finding the minimum value, etc., on at least two values ​​to be summarized. For example, the measure field value of the incremental data can be summed with the measure field value of a row of candidate summary data, and the summation result can be the summary result.

[0079] In this embodiment, after filtering out at least one row of candidate summary data based on the detailed data identifier and the summary data identifier, the dimension field values ​​contained in the incremental data and the at least one row of candidate summary data are compared to accurately query whether the target candidate summary data exists. Then, when the target candidate summary data is found, the measurement field values ​​of the incremental data and the measurement field values ​​of the target candidate summary data are summarized to realize the summarization of the incremental data of the first summarization method into the summary table.

[0080] In one embodiment, the server may use the summary results to replace the measure field values ​​of the target candidate summary data in the summary table to update the measure field values ​​of the target candidate summary data.

[0081] In one embodiment, refer to the detailed representation example shown in Table 1 below. The detailed representation table may include multiple dimension fields such as organization, subject, and product, other dimension fields for the period, and a measurement field such as the sales amount for this month. Refer to the summary representation example shown in Table 2 below, which is obtained by summarizing the detailed representation example shown in Table 1. The summary table may include multiple dimension fields such as organization, subject, and product, other dimension fields for the period, a measurement field such as the sales amount for this month, and other measurement fields such as the cumulative sales amount for this quarter and the cumulative sales amount for this year.

[0082] Table 1. Example of detailed list

[0083]

[0084] Table 2 Summary Examples

[0085]

[0086] In one embodiment, after updating the measure field values ​​of the target candidate summary data, the server can query the first summary data from multiple rows of summary data. When the first summary data is found, according to the preset business rules for other dimension fields specified by other measure fields in the summary table, the server filters out the second summary data based on the other dimension field values ​​of the target candidate summary data and the other dimension field values ​​of the first summary data. The server then summarizes the measure field values ​​of the second summary data and the target candidate summary data, and updates the summarized result with the other measure field values ​​of the target candidate summary data.

[0087] In this context, multiple dimension field values ​​of the first summary data are identical to multiple dimension field values ​​of the target candidate summary data. For example, when summarizing incremental data number 5 in the details table shown in Table 1, the target candidate summary data could be the summary data number 4 in the summary table shown in Table 2. Therefore, the first summary data could be the summary data number 2 in the summary table shown in Table 2. Preset business rules are the conditions that other dimension field values ​​must meet when summarizing the field values ​​of other measurement fields. For example, for the other measurement field "cumulative sales amount for this quarter," the preset business rule could be all periods included in this quarter up to the period containing the target candidate summary data. Specifically, if the period containing the target candidate summary data is February, then the other dimension field values ​​that meet the preset business rule are January and February. Therefore, the first summary data is filtered. For example, if the summary data numbered 2 in the summary table shown in Table 2 is filtered, it is found that the other dimension field values ​​of the first summary data are January, which meets the preset business rules. Then the first summary data can be determined as the second summary data. The sales amount of the second summary data this month (100) and the sales amount of the target candidate summary data this month (100) are summed and summarized to obtain the sales amount of the target candidate summary data for the current quarter as 200.

[0088] In one embodiment, step 110 includes: when the query result indicates that no target candidate summary data was found, determining the summary method for the incremental data as the second summary method; step 112 includes: in the summary table, generating new summary data based on the dimension field values ​​contained in the incremental data, and summarizing the measurement field values ​​of the incremental data into the measurement field values ​​of the new summary data.

[0089] In this embodiment, when no target candidate summary data is found, it indicates that at least one row of candidate summary data does not contain candidate summary data whose dimension field values ​​are the same as those contained in the incremental data. In order to summarize the incremental data into the summary table, new summary data is generated in the summary table, thereby summarizing the measurement field values ​​of the incremental data into the measurement field values ​​of the new summary data, which can realize the summarization of incremental data in the second summary method.

[0090] In one embodiment, for incremental data in the second aggregation method that is not aggregated in the multi-row incremental data, the server can generate new aggregated data with the dimension field values ​​contained in the incremental data in the aggregation table; query other unaggregated incremental data from the multi-row incremental data whose dimension field values ​​are the same as those contained in the target incremental data; when other incremental data is found, aggregate the target incremental data and other incremental data according to the aggregation calculation logic of at least one metric field in the aggregation table, determine the aggregation result as the metric field value of the new aggregated data, and update the target incremental data and other incremental data to the aggregated state.

[0091] In one embodiment, when no other incremental data is found, the server can determine the metric field value of the incremental data as the metric field value of the newly added summary data, and update the incremental data to the summarized state.

[0092] In one embodiment, step 104 includes: for each row of incremental data, mapping the values ​​of multiple dimension fields in the incremental data to a hash value using a hash function; mapping the hash value to a string in a preset base, and extracting a substring arranged within a preset position range from the string, and determining the substring as the detailed data identifier of the incremental data.

[0093] The preset base is a pre-defined base, such as decimal or hexadecimal. A substring is formed by a portion of consecutive characters in the string obtained from the hash value mapping. The preset position range is a pre-defined range of positions. For example, the string obtained from the hash value mapping could be a 32-character string, and the preset position range could be from position 1 to position 16. When the preset base is hexadecimal, the substring could be, for example, "4844b44a4bb50dab".

[0094] In this embodiment, a hash function maps multiple dimension field values ​​in the incremental data to a hash value, and the hash value is mapped to a string in a preset base. Then, a substring is extracted. This eliminates the need for subsequent filtering based directly on the multiple dimension field values ​​themselves. Instead, it uses detailed data identifiers and summary data identifiers to quickly filter out at least one row of candidate summary data with less computational resource consumption. Furthermore, the substring is formed by characters arranged within a preset position range in the string mapped by the hash value, which standardizes the detailed data identifiers and is more conducive to quickly filtering out at least one row of candidate summary data, thereby improving data aggregation efficiency.

[0095] In one embodiment, the server can concatenate the values ​​of multiple dimension fields in each row of incremental data, and map the concatenated result to a hash value using a hash function, thus obtaining a hash value mapped from multiple dimension field values.

[0096] In one embodiment, at least one row of candidate summary data is multiple rows of candidate summary data. The data processing method further includes the following steps: dividing the multiple rows of incremental data into multiple groups of incremental data subsets according to the detailed data identifiers of each row of incremental data and according to a preset grouping method; dividing the multiple rows of candidate summary data into multiple groups of summary data subsets according to the summary data identifiers of each row of candidate summary data and according to a preset grouping method; establishing the association relationship between the incremental data subsets in the multiple groups of incremental data subsets and the summary data subsets in the multiple groups of summary data subsets according to the preset grouping method. Step 110 includes: summarizing each group of incremental data subsets in the multiple groups of incremental data subsets in parallel, and in the summarization of each group of incremental data subsets, for each row of incremental data in the incremental data subset, determining the summarization method of the incremental data based on the dimension field values ​​contained in the incremental data and the dimension field values ​​contained in the candidate summary data in the summary data subsets that have an association relationship with the incremental data subset.

[0097] The preset grouping method involves grouping multiple rows of data based on data identifiers. For example, multiple rows of incremental data can be grouped by detailed data identifiers, and multiple rows of candidate summary data can be grouped by summary data identifiers. For instance, the preset grouping method could be based on the first character of the data identifier, grouping incremental data rows where the first character of the detailed data identifier is the same into the same subset of summary data. When the detailed data identifier consists of hexadecimal characters, since hexadecimal characters can be any of the 10 Arabic numerals from 0 to 9 and the 5 English letters from A to E, the maximum number of subsets of summary data can be 16. Alternatively, the hexadecimal characters could be divided into 8 groups, and incremental data rows where the first character of the detailed data identifier is in the same group could be grouped into the same subset of summary data, resulting in a maximum of 8 subsets of summary data.

[0098] When multiple rows of incremental data and multiple rows of candidate summary data are each grouped according to the same preset grouping method, the preset grouping method indicates that these two types of subsets belong to the same group, and a relationship can be established between subsets that belong to the same group. For example, when the preset grouping method is based on the first character of the data identifier, a relationship can be established between the incremental data subsets in the multiple incremental data subsets that are grouped to the character 'a' and the summary data subsets in the multiple summary data subsets that are grouped to the character 'a'. When searching for data identifiers whose first character is a certain character, the LIKE statement in SQL (Structured Query Language) can be used (used to search for records with a specific pattern in the database). For example, the statement "like '0%'" can be used to search for data identifiers whose first character is the character '0', and the statement "like '0%' and like '1%'" can be used to search for data identifiers whose first character is either the character '0' or the character '1'. Parallel processing involves summarizing multiple sets of incremental data subsets simultaneously, and the summarization between different sets of incremental data subsets is independent.

[0099] In this embodiment, according to a preset grouping method, multiple rows of incremental data are grouped based on detailed data identifiers, and multiple rows of candidate summary data are grouped based on summary data identifiers. Furthermore, according to the preset grouping method, a relationship is established between incremental data subsets and their corresponding summary data subsets. Thus, during the summary process, each group of incremental data subsets within multiple groups is summarized in parallel. This transforms the summarization of multiple rows of incremental data into multiple parallel branch summarizations, where each branch summarization summarizes the incremental data within a set of incremental data subsets, thereby improving data summarization efficiency.

[0100] In one embodiment, step 108 includes: for each row of incremental data, mapping the detailed data identifier of the incremental data to first fingerprint information corresponding to the multiple preset mapping functions through multiple preset mapping functions; constructing a Bloom filter based on the first fingerprint information corresponding to each row of incremental data; for each row of summary data in the multiple rows of summary data, mapping the summary data identifier of the summary data to second fingerprint information corresponding to the multiple preset mapping functions through multiple preset mapping functions; and selecting at least one row of candidate summary data from the multiple rows of summary data based on the second fingerprint information corresponding to each row of summary data and the Bloom filter.

[0101] The multiple preset mapping functions are multiple pre-set mapping functions. These preset mapping functions can be multiple preset hash functions, for example, three preset hash functions. The first fingerprint information is fingerprint information used to identify the detailed data identifier, obtained by mapping the detailed data identifiers using the multiple preset mapping functions respectively. The first fingerprint information may include multiple function values ​​obtained by mapping the detailed data identifiers using the multiple preset mapping functions. The second fingerprint information is fingerprint information used to identify the summary data identifier, obtained by mapping the summary data identifiers using the multiple preset mapping functions respectively.

[0102] A Bloom filter can contain a binary vector. When constructing a Bloom filter, the values ​​at corresponding positions in the binary vector can be updated based on the first fingerprint information to store this first fingerprint information in the Bloom filter. For example, if the Bloom filter contains a 100-bit binary vector with an initial value of 0 (i.e., all 100 bits in the binary vector are 0), and the first fingerprint information is 2, 4, and 5, then the values ​​at positions 2, 4, and 5 in the binary vector are all set to 1. The summary data identifier of each of the at least one row of candidate summary data obtained through the Bloom filter may be the same as one of the detailed data identifiers of each of the multiple rows of incremental data; however, the summary data identifiers of the other summary data (excluding the at least one row of candidate summary data) are different from the detailed data identifiers of the multiple rows of incremental data. If both the detailed data identifier and the summary data identifier are obtained based on a hash function mapping, then the values ​​of multiple dimension fields of the other summary data are also different from the values ​​of multiple dimension fields of the multiple rows of incremental data.

[0103] In this embodiment, multiple preset mapping functions are used to map the detailed data identifiers of incremental data to first fingerprint information. Then, based on the first fingerprint information corresponding to each row of incremental data, a Bloom filter is constructed, which reduces the amount of data in the Bloom filter. Moreover, multiple rows of summary data are mapped to second fingerprint information respectively, eliminating the need to directly filter the multiple rows of summary data based on the detailed data identifiers and summary data identifiers. Based on the Bloom filter, at least one row of candidate summary data can be quickly filtered from the multiple rows of summary data, which, combined with subsequent steps, can improve the efficiency of data aggregation.

[0104] In one embodiment, the step of selecting at least one candidate summary data row from the multiple summary data rows based on the second fingerprint information corresponding to each row of summary data and the Bloom filter includes: obtaining a target batch identifier pre-generated for the multiple incremental data rows; querying the Bloom filter for the second fingerprint information corresponding to each row of summary data rows in the multiple summary data rows; when the second fingerprint information corresponding to the summary data rows is found in the Bloom filter, adding the target batch identifier to the summary data rows; and determining the summary data rows containing the target batch identifier as at least one candidate summary data row selected from the multiple summary data rows.

[0105] The target batch identifier is used to identify multiple rows of incremental data. For example, if the incremental data in the details table is summarized every hour, the multiple rows of incremental data generated between 9:00 and 10:00 can be considered as one batch of incremental data, while the multiple rows of incremental data generated between 10:00 and 11:00 are from different batches. The target batch identifier can be a target batch number, which can be the date and time period in which the multiple rows of incremental data were generated.

[0106] In this embodiment, since the preset mapping function used to map the second fingerprint information corresponding to the summary data is the same as the preset mapping function used to map the first fingerprint information, when the second fingerprint information is found in the Bloom filter, it is highly likely that the Bloom filter contains the same first fingerprint information as the second fingerprint information. It is also highly likely that the multiple dimension field values ​​of the summary data are the same as the multiple dimension field values ​​of the incremental data in the multiple rows of incremental data, which makes the at least one row of candidate summary data obtained by filtering beneficial to subsequent summarization. Moreover, by adding a target batch identifier, at least one row of candidate summary data can be quickly marked, thereby quickly filtering at least one row of candidate summary data.

[0107] In one embodiment, see as follows Figure 2The diagram illustrates the structure of the detail table and summary table. The detail table may include a set of dimension fields and at least one measure field, and the summary table also includes a set of dimension fields and at least one measure field. During summarization, duplicates of the dimension field values ​​for different rows of detail data in the detail table are removed and copied to the summary table, forming a set of dimension field values ​​for each row of summary data. Then, the measure field values ​​for the same dimension value sets in both the detail table and the summary table are summarized. The detail table may also include a data identifier field (dimension combination hash field) to record the detail data identifier for each row of detail data. The summary table may also include a data identifier field to record the summary data identifier for each row of summary data. The summary table may also include a summary batch number field, used to update the summary batch number field value of at least one candidate summary data row to the target batch number when summarizing multiple rows of incremental data. Based on this, the data processing steps specifically include the following steps.

[0108] The server can retrieve multiple rows of incremental data from the detail table; for each row of incremental data, a hash function is used to map the values ​​of multiple dimension fields in the incremental data to a hash value; the hash value is mapped to a string in a preset base, and a substring arranged within a preset position range in the string is extracted from the string, and the substring is determined as the detail data identifier of the incremental data.

[0109] The server can retrieve the summary table corresponding to the detailed table; see example... Figure 3 The flowchart shown illustrates the summary data filtering steps. For each row of incremental data, the server uses multiple preset mapping functions to map the detailed data identifier of the incremental data to the first fingerprint information corresponding to each preset mapping function. Based on the first fingerprint information corresponding to each row of incremental data, a Bloom filter is constructed. A target batch number is generated for the multiple rows of incremental data. For each row of summary data, using multiple preset mapping functions, the summary data identifier of the summary data is mapped to the second fingerprint information corresponding to each preset mapping function. For each row of summary data, the second fingerprint information corresponding to the summary data is queried in the Bloom filter. When the second fingerprint information corresponding to the summary data is found in the Bloom filter, the field value of the summary batch number of the summary data is updated to the target batch identifier. The summary data containing the target batch identifier is determined as at least one candidate summary data row selected from the multiple rows of summary data.

[0110] The server can divide multiple rows of incremental data into multiple subsets of incremental data according to their respective detailed data identifiers and a preset grouping method; divide multiple rows of candidate summary data into multiple subsets of summary data according to their respective summary data identifiers and a preset grouping method; and establish the association relationship between the incremental data subsets in the multiple subsets of incremental data and the summary data subsets in the multiple subsets of summary data according to the preset grouping method.

[0111] See also Figure 4 The flowchart shown illustrates the parallel aggregation steps. The server can aggregate multiple incremental data subsets in parallel. Within each incremental data subset aggregation, the incremental data subset (e.g., incremental data subset 1) and the aggregated data subset related to it (e.g., aggregated data subset 1) are used as input data. The SQL statement uses the GROUP clause to group the dimension field value sets of the incremental data in the incremental data subset and the dimension field value sets of the candidate aggregated data in the aggregated data subset. The dimension field value sets are divided into three types: the first type is where the same dimension field value set exists in both incremental and aggregated data; the second type is where the dimension field value set is... The first type has two sets of data types: segment value sets exist only in incremental data, and dimension field value sets exist only in summary data. For the first type, an update summary table SQL statement is generated to summarize the measure field values ​​of incremental data and candidate summary data that contain the same dimension field value set, obtain the summary result, and update the measure field value of the candidate summary data based on the summary result. For the second type, an insert summary table SQL statement is generated to generate new summary data in the summary table based on the dimension field value set, and summarize the measure field values ​​of the incremental data corresponding to the dimension field value set into the measure field values ​​of the new summary data. For the third type, since there is no incremental data, no summary is required.

[0112] In one embodiment, when performing parallel aggregation, the aggregation can also be performed according to the following steps:

[0113] The server can divide multiple rows of incremental data into multiple subsets of incremental data according to their respective detailed data identifiers and a preset grouping method; divide multiple rows of candidate summary data into multiple subsets of summary data according to their respective summary data identifiers and a preset grouping method; and establish the association relationship between the incremental data subsets in the multiple subsets of incremental data and the summary data subsets in the multiple subsets of summary data according to the preset grouping method.

[0114] The server can aggregate multiple incremental data subsets in parallel. Within the aggregation of each subset, for each row of incremental data, it queries the candidate aggregated data of the aggregated subset to obtain the target candidate aggregated data. If the query result indicates that the target candidate aggregated data is found, the aggregation method for the incremental data is determined to be the first aggregation method. The measure field values ​​of the incremental data and the measure field values ​​of the target candidate aggregated data are aggregated to obtain the aggregation result. Based on the aggregation result, the measure field values ​​of the target candidate aggregated data are updated. If the query result indicates that the target candidate aggregated data is not found, the aggregation method for the incremental data is determined to be the second aggregation method. In the aggregation table, new aggregated data is generated based on the dimension field values ​​contained in the incremental data, and the measure field values ​​of the incremental data are aggregated into the measure field values ​​of the new aggregated data.

[0115] In the above data processing steps, when the amount of data in the summary table is large and the amount of data in the incremental data is small, the data processing steps map multiple dimension field values ​​to data identifiers and perform queries based on Bloom filters. This allows for the rapid identification of whether target candidate summary data exists in the summary data, thus enabling aggregation and greatly improving data aggregation efficiency with low computational resource consumption. When the amount of data in the incremental data is fixed, the time required to aggregate the incremental data does not increase linearly with the amount of data in the summary table. When the amount of data in the incremental data is large, the above data processing steps also use grouping and parallel aggregation to horizontally expand computational resources and improve overall aggregation efficiency.

[0116] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0117] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0118] In one exemplary embodiment, such as Figure 5 As shown, a data processing apparatus 500 is provided, including: an acquisition module 510, a mapping module 520, a filtering module 530, and a summarizing module 540, wherein:

[0119] The acquisition module 510 is used to acquire multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​of specified multiple dimension fields and at least one measure field value.

[0120] The mapping module 520 is used to map the values ​​of multiple dimension fields in the incremental data to a single value for each row of incremental data, thereby obtaining the detailed data identifier of the incremental data.

[0121] The acquisition module 510 is also used to acquire the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of multiple dimension fields, at least one measure field value, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value.

[0122] The filtering module 530 is used to construct filtering conditions based on the detailed data identifiers of each of the multiple rows of incremental data, and to filter at least one row of candidate summary data from the multiple rows of summary data based on the summary data identifier.

[0123] The aggregation module 540 is used to determine the aggregation method for incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in at least one row of candidate aggregation data; and to aggregate the measurement field values ​​of each row of incremental data into a aggregation table according to the aggregation method of each row of incremental data.

[0124] In one embodiment, the aggregation module 540 is further configured to query target candidate aggregation data from at least one row of candidate aggregation data for each row of incremental data to obtain a query result; the dimension field values ​​contained in the target candidate aggregation data are the same as the dimension field values ​​contained in the incremental data; when the query result indicates that the target candidate aggregation data has been found, the aggregation method for the incremental data is determined to be the first aggregation method; the measurement field values ​​of the incremental data and the measurement field values ​​of the target candidate aggregation data are aggregated to obtain an aggregation result; and the measurement field values ​​of the target candidate aggregation data are updated according to the aggregation result.

[0125] In one embodiment, the aggregation module 540 is further configured to determine the aggregation method for incremental data as the second aggregation method when the query result indicates that no target candidate aggregation data was found; in the aggregation table, new aggregation data is generated based on the dimension field values ​​contained in the incremental data, and the measurement field values ​​of the incremental data are aggregated into the measurement field values ​​of the new aggregation data.

[0126] In one embodiment, the mapping module 520 is further configured to, for each row of incremental data, map the values ​​of multiple dimension fields in the incremental data to a hash value using a hash function; map the hash value to a string in a preset base; and extract a substring arranged within a preset position range from the string, and determine the substring as the detailed data identifier of the incremental data.

[0127] In one embodiment, at least one row of candidate summary data is multiple rows of candidate summary data. The data processing device 500 further includes a grouping module, which is used to divide the multiple rows of incremental data into multiple groups of incremental data subsets according to the detailed data identifiers of each row of incremental data and according to a preset grouping method; divide the multiple rows of candidate summary data into multiple groups of summary data subsets according to the summary data identifiers of each row of candidate summary data and according to a preset grouping method; and establish the association relationship between the incremental data subsets in the multiple groups of incremental data subsets and the summary data subsets in the multiple groups of summary data subsets according to the preset grouping method. The summarization module 540 is also used to summarize each group of incremental data subsets in the multiple groups of incremental data subsets in parallel, and in the summarization of each group of incremental data subsets, for each row of incremental data in the incremental data subset, determine the summarization method of the incremental data based on the dimension field values ​​contained in the incremental data and the dimension field values ​​contained in the candidate summary data in the summary data subsets that are associated with the incremental data subset.

[0128] In one embodiment, the filtering module 530 is further configured to: for each row of incremental data, map the detailed data identifier of the incremental data to first fingerprint information corresponding to the multiple preset mapping functions through multiple preset mapping functions; construct a Bloom filter based on the first fingerprint information corresponding to each row of incremental data; for each row of summary data in the multiple rows of summary data, map the summary data identifier of the summary data to second fingerprint information corresponding to the multiple preset mapping functions through multiple preset mapping functions; and filter out at least one row of candidate summary data from the multiple rows of summary data based on the second fingerprint information corresponding to each row of summary data and the Bloom filter.

[0129] In one embodiment, the filtering module 530 is further configured to obtain a target batch identifier pre-generated for the multi-row incremental data; for each row of summary data in the multi-row summary data, query the second fingerprint information corresponding to the summary data in the Bloom filter; when the second fingerprint information corresponding to the summary data is found in the Bloom filter, add the target batch identifier to the summary data; and determine the summary data containing the target batch identifier in the multi-row summary data as at least one row of candidate summary data filtered from the multi-row summary data.

[0130] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0131] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores the data required for executing the aforementioned data processing methods. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0132] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0134] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0135] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0137] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0138] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​and at least one measure field value. For each row of incremental data, the values ​​of the multiple dimension fields in the incremental data are mapped to a single value to obtain the detailed data identifier of the incremental data; Obtain the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value. For each row of incremental data, the detailed data identifier of the incremental data is mapped to the first fingerprint information of the incremental data corresponding to the multiple preset mapping functions through multiple preset mapping functions; A Bloom filter is constructed based on the first fingerprint information corresponding to each of the multiple rows of incremental data. For each row of summary data in the multiple rows of summary data, the summary data identifier of the summary data is mapped to the second fingerprint information of the summary data corresponding to the multiple preset mapping functions through the multiple preset mapping functions; Based on the second fingerprint information corresponding to each of the multiple rows of summary data and the Bloom filter, at least one row of candidate summary data is selected from the multiple rows of summary data; Based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate summary data, the aggregation method for the incremental data is determined. According to the respective aggregation methods of the multiple rows of incremental data, the values ​​of the respective measurement fields of the multiple rows of incremental data are aggregated into the summary table.

2. The method according to claim 1, characterized in that, The step of determining the aggregation method for the incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in at least one row of candidate aggregated data includes: For each row of incremental data, the target candidate summary data is queried from the at least one row of candidate summary data to obtain the query result; the dimension field values ​​contained in the target candidate summary data are the same as the dimension field values ​​contained in the incremental data. When the query result indicates that the target candidate summary data has been found, the aggregation method for the incremental data is determined to be the first aggregation method; The step of summarizing the measure field values ​​of the multiple rows of incremental data into the summary table according to their respective summarization methods includes: The measurement field values ​​of the incremental data and the measurement field values ​​of the target candidate summary data are summarized to obtain the summary result; Based on the summary results, update the measure field values ​​of the target candidate summary data.

3. The method according to claim 2, characterized in that, The step of determining the aggregation method for the incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in at least one row of candidate aggregated data includes: When the query result indicates that the target candidate summary data was not found, the aggregation method for the incremental data is determined to be the second aggregation method. The step of summarizing the measure field values ​​of the multiple rows of incremental data into the summary table according to their respective summarization methods includes: In the summary table, new summary data is generated based on the dimension field values ​​contained in the incremental data, and the measurement field values ​​of the incremental data are summarized into the measurement field values ​​of the new summary data.

4. The method according to claim 1, characterized in that, The step of mapping the values ​​of the multiple dimension fields in each row of incremental data to a single value to obtain the detailed data identifier of the incremental data includes: For each row of incremental data, a hash function is used to map the values ​​of the multiple dimension fields in the incremental data to a hash value. The hash value is mapped to a string in a preset base, and a substring arranged within a preset position range in the string is extracted from the string. The substring is then identified as the detailed data identifier of the incremental data.

5. The method according to claim 1, characterized in that, The at least one row of candidate summary data is multiple rows of candidate summary data, and the method further includes: Based on the detailed data identifiers of each of the multiple rows of incremental data, the multiple rows of incremental data are divided into multiple subsets of incremental data according to a preset grouping method; Based on the summary data identifiers of the multiple candidate summary data rows, the multiple candidate summary data rows are divided into multiple summary data subsets according to the preset grouping method; According to the preset grouping method, establish the association relationship between the incremental data subset in the multiple incremental data subsets and the summary data subset in the multiple summary data subsets; The step of determining the aggregation method for the incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in at least one row of candidate aggregated data includes: The incremental data subsets in the multiple sets of incremental data subsets are summarized in parallel. In the summary of each set of incremental data subsets, for each row of incremental data in the incremental data subset, the summary method of the incremental data is determined based on the dimension field values ​​contained in the incremental data and the dimension field values ​​contained in the candidate summary data in the summary data subset that is related to the incremental data subset.

6. The method according to claim 1, characterized in that, The step of filtering at least one candidate summary data row from the multiple summary data rows based on the second fingerprint information corresponding to each row of summary data and the Bloom filter includes: Obtain a target batch identifier pre-generated for the multiple rows of incremental data; For each row of summary data in the multi-row summary data, query the second fingerprint information corresponding to the summary data in the Bloom filter; When the second fingerprint information corresponding to the summary data is found in the Bloom filter, the target batch identifier is added to the summary data; The summary data containing the target batch identifier in the multi-row summary data is determined as at least one candidate summary data row selected from the multi-row summary data.

7. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire multiple rows of incremental data from the details table. Each row of incremental data contains multiple dimension field values ​​of specified multiple dimension fields and at least one measure field value. The mapping module is used to map the values ​​of the multiple dimension fields in the incremental data to a single value for each row of incremental data, thereby obtaining the detailed data identifier of the incremental data. The acquisition module is also used to acquire the summary table corresponding to the detail table. The summary table contains multiple rows of summary data. Each row of summary data contains multiple dimension field values ​​of the multiple dimension fields, the measurement field value of the at least one measurement field, and a summary data identifier obtained by mapping the multiple dimension field values ​​to a single value. The filtering module is used to, for each row of incremental data, map the detailed data identifier of the incremental data to first fingerprint information corresponding to the multiple preset mapping functions through multiple preset mapping functions; construct a Bloom filter based on the first fingerprint information corresponding to each row of incremental data; for each row of summary data in the multiple rows of summary data, map the summary data identifier of the summary data to second fingerprint information corresponding to the multiple preset mapping functions through the multiple preset mapping functions; and filter out at least one row of candidate summary data from the multiple rows of summary data based on the second fingerprint information corresponding to each row of summary data and the Bloom filter. The aggregation module is used to determine the aggregation method for the incremental data based on the dimension field values ​​contained in each row of incremental data and the dimension field values ​​contained in the at least one row of candidate aggregated data; and to aggregate the measurement field values ​​of the multiple rows of incremental data into the aggregation table according to the respective aggregation methods of the multiple rows of incremental data.

8. The apparatus according to claim 7, characterized in that, The aggregation module is also used for: For each row of incremental data, the target candidate summary data is queried from the at least one row of candidate summary data to obtain the query result; the dimension field values ​​contained in the target candidate summary data are the same as the dimension field values ​​contained in the incremental data. When the query result indicates that the target candidate summary data has been found, the aggregation method for the incremental data is determined to be the first aggregation method; The measurement field values ​​of the incremental data and the measurement field values ​​of the target candidate summary data are summarized to obtain the summary result; Based on the summary results, update the measure field values ​​of the target candidate summary data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images