Data processing method and device, computer equipment and computer readable storage medium

Abstract

The application relates to a data processing method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: determining a first dimension and a second dimension in a summary data table, the first dimension being a sender dimension of data allocation, and the second dimension being a receiver dimension of data allocation; determining first data corresponding to the first dimension from the summary data table; performing allocation processing on the first data of the first dimension according to the second dimension to obtain second data corresponding to the second dimension; and writing the second data corresponding to the second dimension into a detailed data table and the summary data table respectively to obtain an updated detailed data table and an updated summary data table. The method can improve resource utilization.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Allocation derivation is a commonly used data processing method in management accounting, which allocates indirect costs and expenses to different dimensions according to different proportions. Existing allocation derivation systems generally consist of two parts: allocation derivation and aggregation. Allocation derivation refers to the process of allocating a dataset to different dimensions according to certain principles and methods to generate a new dataset. The new dataset generated through allocation derivation is then stored in a detailed data table. The newly generated data in the detailed data table is then aggregated with the data in the summary data table for subsequent allocation derivation, other data processing, and financial data analysis.

[0003] However, in traditional methods, the summarization part can only be performed after the allocation derivation is completed and the data is written into the detailed data table, resulting in low utilization of computer resources. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can effectively improve the utilization rate of computer resources.

[0005] On the one hand, this application provides a data processing method, including:

[0006] Determine the first and second dimensions in the summary data table. The first dimension is the dimension of the data distribution sender, and the second dimension is the dimension of the data distribution receiver.

[0007] From the summarized data table, determine the first data corresponding to the first dimension;

[0008] According to the second dimension, the first data of the first dimension is distributed to obtain the second data corresponding to the second dimension;

[0009] Write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively to obtain the updated detailed data table and the updated summary data table.

[0010] On the other hand, this application also provides a data processing apparatus, including:

[0011] The dimension determination module is used to determine the first dimension and the second dimension in the summary data table. The first dimension is the data distribution sender dimension, and the second dimension is the data distribution receiver dimension.

[0012] The data determination module is used to determine the first data corresponding to the first dimension from the summary data table;

[0013] The allocation module is used to allocate the first data of the first dimension according to the second dimension to obtain the second data corresponding to the second dimension.

[0014] The writing module is used to write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, so as to obtain the updated detailed data table and the updated summary data table.

[0015] On the other hand, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0016] Determine the first and second dimensions in the summary data table. The first dimension is the dimension of the data distribution sender, and the second dimension is the dimension of the data distribution receiver.

[0017] From the summarized data table, determine the first data corresponding to the first dimension;

[0018] According to the second dimension, the first data of the first dimension is distributed to obtain the second data corresponding to the second dimension;

[0019] Write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively to obtain the updated detailed data table and the updated summary data table.

[0020] On the other hand, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0021] Determine the first and second dimensions in the summary data table. The first dimension is the dimension of the data distribution sender, and the second dimension is the dimension of the data distribution receiver.

[0022] From the summarized data table, determine the first data corresponding to the first dimension;

[0023] According to the second dimension, the first data of the first dimension is distributed to obtain the second data corresponding to the second dimension;

[0024] Write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively to obtain the updated detailed data table and the updated summary data table.

[0025] On the other hand, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0026] Determine the first and second dimensions in the summary data table. The first dimension is the dimension of the data distribution sender, and the second dimension is the dimension of the data distribution receiver.

[0027] From the summarized data table, determine the first data corresponding to the first dimension;

[0028] According to the second dimension, the first data of the first dimension is distributed to obtain the second data corresponding to the second dimension;

[0029] Write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively to obtain the updated detailed data table and the updated summary data table.

[0030] The aforementioned data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product determine the data allocation sender dimension and data allocation receiver dimension by determining a first dimension and a second dimension in a summary data table. From the summary data table, the first data corresponding to the first dimension is determined, thereby identifying the data to be allocated. The first data of the first dimension is allocated according to the second dimension, thus allocating the first data of the first dimension to the second dimension to obtain the second data corresponding to the second dimension. The second data corresponding to the second dimension is written into both a detailed data table and a summary data table, resulting in updated detailed data tables and updated summary data tables. This allows for simultaneous allocation and aggregation, improving data processing efficiency. Furthermore, it fully utilizes the generated second data, helping to save computer resources. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0033] Figure 2 This is a flowchart of a data processing method in one embodiment;

[0034] Figure 3 This is a comparative diagram of the data tables before and after the allocation in one embodiment;

[0035] Figure 4 This is an architecture diagram of a data processing method in one embodiment;

[0036] Figure 5 This is an architecture diagram of a data processing method in one embodiment;

[0037] Figure 6 This is a structural block diagram of a data processing device in one embodiment;

[0038] Figure 7 This is a block diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on a cloud or other network server. Both terminal 102 and server 104 can independently execute the data processing method provided in this embodiment. Terminal 102 and server 104 can also work together to execute the data processing method provided in this embodiment. When terminal 102 and server 104 work together to execute the data processing method provided in this embodiment, terminal 102 can determine a first dimension and a second dimension from the summary data table stored on server 104. The first dimension is the data allocation sender dimension, and the second dimension is the data allocation receiver dimension. Terminal 102 determines the first data corresponding to the first dimension from the summary data table. Terminal 102 allocates the first data of the first dimension according to the second dimension to obtain the second data corresponding to the second dimension. Terminal 102 writes the second data corresponding to the second dimension into a detailed data table and a summary data table, respectively, to obtain updated detailed data tables and updated summary data tables. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0041] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which is applied to a computer device (such as...) Figure 1 Taking a terminal or server as an example, the following steps are included:

[0042] Step 202: Determine the first dimension and the second dimension in the summary data table. The first dimension is the dimension of the data distribution sender, and the second dimension is the dimension of the data distribution receiver.

[0043] The first dimension is the dimension of the sender of the data allocation, and the second dimension is the dimension of the receiver of the data allocation.

[0044] The first and second dimensions can be dimensions of the same category or dimensions of different categories. For example, the first dimension A1 and the second dimension A2 both belong to dimension category A, or the first dimension A1 belongs to dimension category A and the second dimension B1 belongs to dimension category B.

[0045] Specifically, the computer equipment determines the first and second dimensions in the summary data table. The first dimension is the sender dimension of the data allocation, and the second dimension is the receiver dimension of the data allocation. The data corresponding to the sender dimension is the data to be allocated and needs to be allocated to the receiver dimension to form the data corresponding to the receiver dimension.

[0046] In this embodiment, the computer device can obtain the first dimension filtering condition and the second dimension filtering condition, filter out the first dimension from the summary data table according to the first dimension filtering condition, and filter out the second dimension from the summary data table according to the second dimension filtering condition.

[0047] Step 204: Determine the first data corresponding to the first dimension from the summary data table.

[0048] The first data is the data corresponding to the first dimension in the summary data table.

[0049] Specifically, the computer device reads the data corresponding to the first dimension from the summary data table as the first data.

[0050] Step 206: According to the second dimension, the first data of the first dimension is distributed to obtain the second data corresponding to the second dimension.

[0051] Specifically, the computer device distributes the first data of the first dimension according to the second dimension to obtain the second data corresponding to the second dimension.

[0052] The first dimension may include at least one sender dimension, and the second dimension may include at least one receiver dimension. The computer device may distribute the first data of each sender dimension according to each receiver dimension to obtain the second data of each receiver dimension.

[0053] In this embodiment, the computer device can determine the allocation ratio of at least two second dimensions, and allocate the first data according to the allocation ratio of at least two second dimensions to obtain the second data of at least two second dimensions.

[0054] Step 208: Write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively to obtain the updated detailed data table and the updated summary data table.

[0055] Specifically, after obtaining the second data corresponding to the second dimension, the computer device can execute the two parts of generating the detailed data table and summarizing the data in parallel. That is, the second data corresponding to the second dimension is written into the detailed data table and the summary data table respectively, so as to update the detailed data table and the summary data table respectively, and obtain the updated detailed data table and the updated summary data table.

[0056] In this embodiment, the computer device summarizes the second data of the second dimension with the original data corresponding to the second dimension in the summary data table to obtain an updated summary data table.

[0057] In this embodiment, the computer device can write the second data into the original detailed data table to update the detailed data table and obtain the updated detailed data table.

[0058] In the above data processing method, the sender and receiver dimensions of data allocation are determined by identifying the first and second dimensions in the summary data table. From the summary data table, the first data corresponding to the first dimension is determined, thus identifying the data to be allocated. The first data of the first dimension is allocated according to the second dimension, thereby distributing the first data of the first dimension to the second dimension to obtain the second data corresponding to the second dimension. The second data corresponding to the second dimension is written into the detail data table and the summary data table respectively, resulting in updated detail data tables and updated summary data tables. This allows for simultaneous allocation and aggregation, improving data processing efficiency. Furthermore, it fully utilizes the generated second data, helping to save computer resources.

[0059] In one embodiment, determining the first data corresponding to the first dimension from the summary data table includes:

[0060] Based on the first and second dimensions, read the data to be processed from the summary data table. The data to be processed includes at least the first data corresponding to the first dimension.

[0061] Write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, to obtain the updated detailed data table and the updated summary data table, including:

[0062] Based on the data to be processed and the second data corresponding to the second dimension, update the detailed data table to obtain the updated detailed data table; based on the data to be processed and the second data corresponding to the second dimension, generate summary data, write the summary data into the summary data table to obtain the updated summary data table.

[0063] The data to be processed refers to the data that changes during data allocation and aggregation.

[0064] Specifically, the computer device reads the data to be processed from the summary data table based on the first dimension and the second dimension. The data to be processed includes at least the first data corresponding to the first dimension.

[0065] The computer device updates the detailed data table based on the data to be processed and the second data corresponding to the second dimension, obtaining an updated detailed data table. Furthermore, based on the data to be processed and the second data corresponding to the second dimension, the computer device generates summary data corresponding to the second dimension, writes the summary data into a summary data table, obtaining an updated summary data table.

[0066] In this embodiment, the data to be processed may include first data corresponding to the first dimension and original data corresponding to the second dimension. Based on the first data and the second data corresponding to the second dimension in the data to be processed, the detailed data table is updated to obtain an updated detailed data table. Furthermore, the computer device generates summary data based on the original data and the second data corresponding to the second dimension in the data to be processed, and writes the summary data into a summary data table to obtain an updated summary data table.

[0067] In this embodiment, the summary data of the second dimension replaces the original data of the second dimension in the summary data table to obtain an updated summary data table.

[0068] In this embodiment, since the summary data table may contain a certain dimension without corresponding original data, when the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, the first data of the first dimension and the original data of the second dimension can be used as data to be processed.

[0069] In this embodiment, based on the first dimension and the second dimension, data to be processed is read from the summary data table. The data to be processed includes at least the first data corresponding to the first dimension. Based on the data to be processed and the second data corresponding to the second dimension, the detailed data table is updated to obtain an updated detailed data table. Based on the data to be processed and the second data corresponding to the second dimension, summary data is generated and written into the summary data table to obtain an updated summary data table. Thus, the detailed data table is updated and the summary data of the second dimension is generated in parallel, thereby realizing the update of the summary data table and improving the efficiency of data distribution and data aggregation.

[0070] In one embodiment, reading the data to be processed from the summary data table based on a first dimension and a second dimension includes:

[0071] When the summary data table contains the first data corresponding to the first dimension and the original data corresponding to the second dimension, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as the data to be processed; when the summary data table contains the first data and the original data, and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension and the first data.

[0072] Specifically, the computer device can determine whether the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension. If the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, the computer device determines whether the first data can cover the original data.

[0073] If the first data can cover the original data, it means that the first data of the first dimension contains the original data of the second dimension, and the first data can be used as the data to be processed.

[0074] If the first data cannot cover the original data, it means that the first data of the first dimension cannot fully contain the original data of the second dimension. Therefore, it is necessary to determine the data to be processed from the summary data table based on the first dimension, the second dimension, and the first data, so as to read the relevant data that can simultaneously cover the first and second dimensions from the summary data table.

[0075] In one embodiment, when the summary data table contains first data and original data, and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension, and the first data, including:

[0076] When the summary data table contains first data and original data, and the first data does not cover the original data, determine the type of the first dimension filter condition and the type of the second dimension filter condition; based on the type of the first dimension filter condition and the type of the second dimension filter condition, the first dimension, the second dimension, and the first data, read the data to be processed from the summary data table.

[0077] In this embodiment, based on the type of the first dimension filtering condition and the type of the second dimension filtering condition, the first dimension, the second dimension, and the first data, the data to be processed is read from the summary data table, including:

[0078] When both the first and second dimension filtering conditions belong to the first preset filtering type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as the data to be processed; when both the first and second dimension filtering conditions belong to the second preset filtering type, the dimensional intersection between the first and second dimensions is determined; from the summary data table, the first data of the first dimension in the dimensional intersection and the original data of the second dimension in the dimensional intersection are read together as the data to be processed.

[0079] In this embodiment, when the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as the data to be processed. When the summary data table contains first data and original data, and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension, and the first data, so as to expand the data range and read the data to be processed that simultaneously covers the data range of the first dimension and the data range of the second dimension.

[0080] In one embodiment, the method further includes:

[0081] If the summary data table contains the first data corresponding to the first dimension, but does not contain the original data corresponding to the second dimension, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0082] Specifically, when the computer device detects that there is first data corresponding to the first dimension in the summary data table, but no original data corresponding to the second dimension, it means that the data under the second dimension in the summary data table is empty. Then, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0083] In this embodiment, if the summary data table contains the first data corresponding to the first dimension, but does not contain the original data corresponding to the second dimension, it means that the original data under the second dimension in the summary data table is empty. In this case, the first data corresponding to the first dimension can be directly read from the summary data table as the data to be processed.

[0084] In one embodiment, determining the first dimension and the second dimension in the summary data table includes:

[0085] Obtain the first-dimensional filter condition and the second-dimensional filter condition; filter the first dimension from the summary data table according to the first-dimensional filter condition, and filter the second dimension from the summary data table according to the second-dimensional filter condition.

[0086] The first-dimensional filter condition is used to select the first dimension from the summary data table. The first-dimensional filter condition is a constraint on the selected first dimension. The second-dimensional filter condition is used to select the second dimension from the summary data table. The second-dimensional filter condition is a constraint on the selected second dimension.

[0087] Specifically, users can configure first-dimensional filtering conditions and second-dimensional filtering conditions. The computer device filters out the first dimension from the summary data table based on the user-configured first-dimensional filtering conditions, and filters out the second dimension from the summary data table based on the second-dimensional filtering conditions.

[0088] In one embodiment, the user can edit and input the first-dimensional filtering condition and the second-dimensional filtering condition in the condition configuration interface.

[0089] In this embodiment, the condition configuration interface can pre-set multiple filter condition types, as well as multiple dimension categories and dimensions under each dimension category. The user can select the type of filter condition. Based on the user's selected filter condition type, dimension category, and dimension, the computer device generates the first dimension filter condition and the second dimension filter condition, respectively.

[0090] The types of filter conditions include, for example, "in..." or "not in...".

[0091] In this embodiment, since there is a large amount of data in the summary data table, by obtaining the first dimension filtering condition and the second dimension filtering condition, the first dimension can be quickly filtered out from the large amount of data in the summary data table according to the first dimension filtering condition, and the second dimension can be quickly filtered out from the large amount of data in the summary data table according to the second dimension filtering condition, thereby improving data processing efficiency.

[0092] In one embodiment, reading the data to be processed from the summary data table based on a first dimension and a second dimension includes:

[0093] Determine the types of the first-dimensional and second-dimensional filter conditions; based on the types of the first-dimensional and second-dimensional filter conditions, and the first and second dimensions, read the data to be processed from the summary data table.

[0094] The filter criteria can be of two types: a reserved type and an exclusion type. A reserved type is, for example, "in...", and an exclusion type is, for example, "not in...".

[0095] Specifically, the computer device determines the type of the first-dimensional filtering condition and the type of the second-dimensional filtering condition, and reads the data to be processed from the summary data table based on the type of the first-dimensional filtering condition, the type of the second-dimensional filtering condition, and the first and second dimensions.

[0096] In this embodiment, the types of the first-dimensional filtering conditions and the second-dimensional filtering conditions are determined. Based on the types of the first-dimensional filtering conditions, the second-dimensional filtering conditions, the first dimension, and the second dimension, the data to be processed is read from the summary data table. This allows for the rapid filtering of all the data needed in the data allocation and summarization process based on the types of filtering conditions.

[0097] In one embodiment, reading the data to be processed from the summary data table based on the type of the first dimension filter condition, the type of the second dimension filter condition, the first dimension, and the second dimension includes:

[0098] When the types of the first-dimensional filtering conditions and the second-dimensional filtering conditions belong to the first preset filtering type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as data to be processed; when the types of the first-dimensional filtering conditions and the second-dimensional filtering conditions belong to the second preset filtering type, the dimensional intersection between the first dimension and the second dimension is determined; the first data corresponding to the first dimension in the dimensional intersection and the original data corresponding to the second dimension in the dimensional intersection are read from the summary data table as data to be processed.

[0099] The first preset filter type can be a retention type, and the second preset filter type can be an exclusion type. The first preset filter type can be a "in" type, and the second preset filter type can be a "notin" type.

[0100] Specifically, when the types of the first dimension filtering conditions and the second dimension filtering conditions belong to the first preset filtering type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as the data to be processed.

[0101] like Figure 3As shown, the first dimension filter condition is "Dimension A in (A1)", which means selecting dimension A1 under dimension category A from the summary data table as the first dimension, i.e., the first dimension is A=A1. The second dimension filter condition is "Dimension A in (A2, A3)", which means selecting dimensions A2 and A3 under dimension category A from the summary data table as the second dimensions, i.e., the second dimensions A=A2 and A=A3. Both the first dimension filter condition "Dimension A in (A1)" and the second dimension filter condition "Dimension A in (A2, A3)" belong to the first preset filter type, i.e., both belong to the "in" type of filter condition. Therefore, the data corresponding to the first dimension A1 and the data corresponding to the second dimensions A2 and A3 are read from the summary data table as the data to be processed. If the allocation ratio for both dimensions A2 and A3 is 50%, then the first data point "100" corresponding to dimension A1 is allocated to dimensions A2 and A3, resulting in the second data point "50" for both dimensions A2 and A3. This data is written into the explicit data table to obtain an updated explicit data table. Simultaneously, the second data point "50" of dimension A2 is summarized with the original data "100" of dimension A2 in the summary data table, and the second data point "50" of dimension A3 is summarized with the original data "100" of dimension A2 in the summary data table, resulting in an updated summary data table. In the updated summary data table, the data corresponding to dimension A2 is updated from "100" to "150", and the data corresponding to dimension A3 is updated from "100" to "150".

[0102] When the types of the first and second dimension filtering conditions belong to the second preset filtering type, the intersection of the first and second dimensions is determined. The computer device reads the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as the data to be processed.

[0103] In this embodiment, when the types of the first dimension filtering conditions and the second dimension filtering conditions belong to the first preset filtering type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as data to be processed; when the types of the first dimension filtering conditions and the second dimension filtering conditions belong to the second preset filtering type, the dimensional intersection between the first dimension and the second dimension is determined, and the first data corresponding to the first dimension and the original data corresponding to the second dimension in the dimensional intersection are read from the summary data table as data to be processed. This allows the data that needs to be changed in data allocation and data aggregation to be read from the summary data table, so as to perform data aggregation and improve the efficiency of data allocation and aggregation.

[0104] In one embodiment, the first data of the first dimension is distributed according to the second dimension to obtain the second data for each of the second dimensions, including:

[0105] The first dimension is combined with each dimension in the second dimension to obtain a dimension combination; the second data is then distributed according to the dimension combination to obtain the second data for each dimension combination.

[0106] Specifically, the computer device can combine the first dimension with each dimension in the second dimension to obtain dimension combinations. Based on the dimension combinations, the second data is then distributed to obtain the second data for each dimension combination.

[0107] For example, the first dimension A1 and the second dimensions A2 and A3 are combined to obtain dimension combination A1A2 and dimension combination A1A3 respectively. The first data of the first dimension A1 is distributed to dimension combination A1A2 and dimension combination A1A3 respectively to obtain the second data corresponding to dimension combination A1A2 and the second data corresponding to dimension combination A1A3.

[0108] In this embodiment, the dimension categories of the first dimension and the second dimension are determined. When the dimension category of the first dimension is different from that of the second dimension, the first dimension is combined with each dimension in the second dimension to obtain a dimension combination. The second data is then distributed according to the dimension combinations to obtain the second data for each dimension combination.

[0109] In this embodiment, when the dimension category of the first dimension is the same as that of the second dimension, the first data of the first dimension is distributed to the second dimension according to the second dimension to obtain the second data corresponding to the second dimension.

[0110] For example, if the first dimension A1 and the second dimensions A2 and A3 both belong to dimension category A, then the first data of the first dimension A1 is distributed to the second dimensions A2 and A3 to obtain the second data of the second dimension A2 and the second data of the second dimension A3.

[0111] Alternatively, the first dimension A1 belongs to dimension category A, and the second dimensions B1 and B2 belong to dimension category B. A computer device can combine the first dimension A1 with the second dimensions B1 and B2 respectively to obtain dimension combinations A1B1 and A1B2. The first data of the first dimension A1 is distributed among dimension combinations A1B1 and A1B2 to obtain the second data of dimension combination A1B1 and the second data of dimension combination A1B2.

[0112] In this embodiment, the first dimension is combined with each dimension in the second dimension to obtain dimension combinations. The second data is then distributed according to these dimension combinations to obtain the second data for each dimension combination. This allows for data distribution in multiple ways, providing flexibility in data distribution. Furthermore, different data distribution methods can be set according to different dimension categories, increasing the diversity of data distribution.

[0113] In one embodiment, a data processing method is provided, applied to a computer device, comprising:

[0114] Obtain the first-dimensional filtering conditions and the second-dimensional filtering conditions;

[0115] The first dimension is selected from the summary data table based on the first dimension filter condition, and the second dimension is selected from the summary data table based on the second dimension filter condition. The first dimension is the dimension of the data distribution sender, and the second dimension is the dimension of the data distribution receiver.

[0116] Determine whether the summary data table contains the original data corresponding to the first data of the first dimension and the original data of the second dimension;

[0117] When the summary data table contains the first data corresponding to the first dimension and the original data corresponding to the second dimension, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0118] When the summary data table contains first data and original data, and the first data does not cover the original data, determine the type of the first dimension filter condition and the type of the second dimension filter condition.

[0119] When the type of the first dimension filter condition and the type of the second dimension filter condition belong to the first preset filter type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as the data to be processed.

[0120] When the type of the first dimension filter condition and the type of the second dimension filter condition belong to the second preset filter type, determine the dimensional intersection between the first dimension and the second dimension.

[0121] From the summary data table, read the first data corresponding to the first dimension in the dimension intersection and the original data corresponding to the second dimension in the dimension intersection as the data to be processed;

[0122] If the summary data table contains the first data corresponding to the first dimension, but does not contain the original data corresponding to the second dimension, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0123] According to the second dimension, the first data of the first dimension is distributed to obtain the second data corresponding to the second dimension;

[0124] Based on the data to be processed and the second data corresponding to the second dimension, update the detailed data table to obtain the updated detailed data table;

[0125] Based on the data to be processed and the second data corresponding to the second dimension, generate summary data, write the summary data into the summary data table, and obtain the updated summary data table.

[0126] In this embodiment, as Figure 4 The diagram illustrates an architecture for a data processing method. The Filter, FlatMap, and CoGroup operations represent relevant operations within a typical distributed stream processing framework in a computer system. Filter filters the dataset; FlatMap transforms a single data point into multiple data points, suitable for handling amortization derivations; and CoGroup is suitable for scenarios requiring the processing of data from multiple sources and the desire to group elements with the same key together.

[0127] The detailed process steps of this solution are described below:

[0128] The first step involves a big data computing framework (using Flink as an example) retrieving all data from the database within the range of the sender and receiver datasets in the summary data table. This includes all datasets required for this allocation derivation and the range within which the generated data falls. The data in the sender dataset is the sender data, and the data in the receiver dataset is the receiver data.

[0129] Specifically, there are two ways to describe the filtering conditions for the range of the sender's and receiver's datasets: one is "dimension X is in...", and the other is "dimension X is not in...". This is used to determine the range of dimensional members of dimension X. The final dataset range is obtained by ANDing multiple dimensions. The data within this final dataset range is the data to be processed.

[0130] There are two scenarios: the range of the sender's dataset completely includes the range of the receiver's dataset; the range of the sender's dataset does not completely include the range of the receiver's dataset.

[0131] The corresponding approaches for the two scenarios are: directly taking the sender's dataset; and expanding the scope of the dataset until it can also include the receiver's dataset.

[0132] The detailed processing flow for database query-related dataset filtering conditions is as follows:

[0133] 1) If the sender has a dataset but the receiver does not, take the data from the sender's dataset as the data to be processed.

[0134] 2) If the sender has a dataset and the receiver has the dimensions of the dataset, process it as follows:

[0135] A) If all filtering conditions are in, then the data from the union of the sender dimension and the receiver dimension is taken as the data to be processed.

[0136] B) All filtering conditions are "not in": Data that is the intersection of the sender's dimension and the receiver's dimension is used as the data to be processed;

[0137] C) Other types of conditions: Ignore and do not process.

[0138] 3) If the sender has no dataset but the receiver has a dataset dimension, ignore it and do not process it.

[0139] 4) If the sender has no dataset and the receiver has no dataset, ignore this and do not process it.

[0140] By following the steps above, a filter condition can be obtained that includes both the sender's dataset range and the receiver's dataset range (i.e., all data within the range of "sender's dataset" and "receiver's dataset" to be read from the summary data table, which includes data corresponding to both the sender's dimension and the receiver's dimension).

[0141] The second step involves filtering out the sender's dataset and sending it to the FlatMap node for the amortization derivation operation. After the amortization derivation operation, the data generates receiver data and flows to the CoGroup node for subsequent aggregation. Additionally, non-sender datasets are filtered out and directly transferred to the CoGroup node.

[0142] The third step is to summarize all the data in CoGroup, and output the recipient data to the detailed data table and the summarized data to the summary data table through the corresponding Filter, and then end all operations.

[0143] In this embodiment, based on a big data computing platform, the "allocation derivation" and "summarization" parts are merged into a single computing task, requiring only one task submission. This fully utilizes the capabilities of the big data processing platform and distributed computing framework, reduces database read and write operations, and fully utilizes computer resources to enhance the system's data processing capabilities by pre-analyzing the dataset size and dynamically arranging the execution sequence of multiple computing tasks using intelligent algorithms.

[0144] Furthermore, the amortization derivation system based on the big data processing platform in this embodiment merges the amortization derivation and aggregation processes into a single big data task. This means that one task completes both the amortization derivation and aggregation, reducing reliance on the database and improving the utilization rate of the big data platform. Moreover, by pre-analyzing the range of the dataset, multiple amortization derivation big data processing tasks are merged into a single big data processing task, further enhancing the system's processing capacity and horizontal scalability. Not only does it achieve the merged execution of individual amortization derivation tasks, but it can also automatically merge multiple amortization derivation tasks with similar or identical processing ranges into a single task. Tasks with different processing ranges can be executed in parallel, further improving system processing efficiency in some scenarios.

[0145] On the other hand, during the aggregation, it is no longer necessary to read the detailed data table. As the detailed data table becomes larger and larger, the processing performance of the computer equipment in the traditional solution will decrease, while in this embodiment, the data processing capability of the computer equipment will not decrease significantly.

[0146] In one embodiment, such as Figure 5 As shown, an architectural diagram of another data processing method is provided.

[0147] To achieve the combined execution of allocation derivation and aggregation, in addition to the technical solution described in this invention, the big data processing platform can also merge the filtering operation into the allocation derivation operation. In the Flatmap operation, the Filter is directly called to determine the data. If the data matches the sender's data, the allocation derivation logic is executed to generate the receiver's data, and both the sender's and receiver's data are passed to the next aggregation operation. If the data does not match the sender's data, it is directly passed to the next aggregation operation. Then, the aggregation operation completes the aggregation of all data, and finally, the corresponding detailed data and aggregated data are written to the detailed data table or the aggregated data table, respectively.

[0148] In addition, the range of the dataset can be analyzed by algorithms, and the amortization derivation of different ranges can be placed in different big data processing tasks through program coordination, and started simultaneously to achieve parallel execution of multiple tasks.

[0149] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0151] In one embodiment, such as Figure 6 As shown, a data processing device 600 is provided, including: a dimension determination module 602, a data determination module 604, an allocation module 606, and a writing module 608, wherein:

[0152] The dimension determination module 602 is used to determine the first dimension and the second dimension in the summary data table. The first dimension is the data distribution sender dimension, and the second dimension is the data distribution receiver dimension.

[0153] The data determination module 604 is used to determine the first data corresponding to the first dimension from the summary data table.

[0154] The allocation module 606 is used to allocate the first data of the first dimension according to the second dimension to obtain the second data corresponding to the second dimension.

[0155] The write module 608 is used to write the second data corresponding to the second dimension into the detail data table and the summary data table respectively, so as to obtain the updated detail data table and the updated summary data table.

[0156] In one embodiment, the data determination module 604 is further configured to read data to be processed from the summary data table based on a first dimension and a second dimension, wherein the data to be processed includes at least the first data corresponding to the first dimension;

[0157] The writing module 608 is also used to update the detailed data table based on the data to be processed and the second data corresponding to the second dimension, and obtain the updated detailed data table; and to generate summary data based on the data to be processed and the second data corresponding to the second dimension, and write the summary data into the summary data table, and obtain the updated summary data table.

[0158] In one embodiment, the data determination module 604 is further configured to: when the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, and the first data covers the original data, read the first data corresponding to the first dimension from the summary data table as data to be processed; when the summary data table contains first data and original data, and the first data does not cover the original data, determine the data to be processed from the summary data table based on the first dimension, the second dimension and the first data.

[0159] In one embodiment, the data determination module 604 is further configured to read the first data corresponding to the first dimension from the summary data table as the data to be processed when there is first data corresponding to the first dimension in the summary data table and there is no original data corresponding to the second dimension.

[0160] In one embodiment, the data determination module 604 is further configured to obtain a first dimension filtering condition and a second dimension filtering condition; filter out the first dimension from the summary data table according to the first dimension filtering condition, and filter out the second dimension from the summary data table according to the second dimension filtering condition.

[0161] In one embodiment, the data determination module 604 is further configured to determine the type of the first dimension filtering condition and the type of the second dimension filtering condition; and to read the data to be processed from the summary data table based on the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension, and the second dimension.

[0162] In one embodiment, the data determination module 604 is further configured to: when the type of the first dimension filtering condition and the type of the second dimension filtering condition belong to a first preset filtering type, read the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as data to be processed; when the type of the first dimension filtering condition and the type of the second dimension filtering condition belong to a second preset filtering type, determine the dimensional intersection between the first dimension and the second dimension; and read the first data corresponding to the first dimension in the dimensional intersection and the original data corresponding to the second dimension in the dimensional intersection from the summary data table as data to be processed.

[0163] In one embodiment, the dimension determination module 602 is further configured to combine the first dimension with each dimension in the second dimension to obtain a dimension combination; and to distribute the second data according to the dimension combination to obtain the second data of each dimension combination.

[0164] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0165] In one embodiment, a computer device is provided, which may be a terminal or a server. Taking a server as an example, its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores distributed and aggregated data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0166] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of a data processing method.

[0168] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform a data processing method.

[0169] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0170] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: In the same computing task, the first dimension is filtered from the summary data table according to the first dimension filtering condition, and the second dimension is filtered from the summary data table according to the second dimension filtering condition. The first dimension is the data distribution sender dimension, and the second dimension is the data distribution receiver dimension. When the types of the first dimension filtering conditions and the second dimension filtering conditions belong to the first preset filtering type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as data to be processed; the first preset filtering type is a retention type. When the types of the first dimension filtering condition and the second dimension filtering condition both belong to the second preset filtering type, the dimensional intersection between the first dimension and the second dimension is determined; the second preset filtering type is an exclusion type. From the summary data table, read the first data corresponding to the first dimension in the dimension intersection and the original data corresponding to the second dimension in the dimension intersection as data to be processed; According to the second dimension, the first data in the data to be processed is distributed to obtain the second data corresponding to the second dimension; The second data corresponding to the second dimension is written into the detailed data table and the summary data table respectively, and the data to be processed is written into the summary data table to obtain the updated detailed data table and the updated summary data table.

2. The method according to claim 1, characterized in that, The step of writing the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, and writing the data to be processed into the summary data table to obtain updated detailed data tables and updated summary data tables includes: Based on the data to be processed and the second data corresponding to the second dimension, update the detailed data table to obtain the updated detailed data table; Based on the data to be processed and the second data corresponding to the second dimension, summary data is generated, and the summary data is written into the summary data table to obtain an updated summary data table.

3. The method according to claim 2, characterized in that, The method further includes: When the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as data to be processed. When the first data and the original data exist in the summary data table, and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension, and the first data.

4. The method according to claim 3, characterized in that, The method further includes: If the first data corresponding to the first dimension exists in the summary data table, but the original data corresponding to the second dimension does not exist, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

5. The method according to any one of claims 1 to 4, characterized in that, The step of distributing the first data of the first dimension according to the second dimension to obtain the second data of the second dimension includes: The first dimension is combined with each dimension in the second dimension to obtain a dimension combination; According to the said dimension combination, the second data is distributed to obtain the second data for each dimension combination.

6. A data processing apparatus, characterized in that, The device includes: The dimension determination module is used to filter out a first dimension from the summary data table based on a first dimension filtering condition and a second dimension from the summary data table based on a second dimension filtering condition in the same computing task. The first dimension is the data distribution sender dimension and the second dimension is the data distribution receiver dimension. The data determination module is used to: when the types of the first dimension filtering condition and the second dimension filtering condition belong to a first preset filtering type, read the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as data to be processed; when the types of the first dimension filtering condition and the second dimension filtering condition belong to a second preset filtering type, determine the dimensional intersection between the first dimension and the second dimension; and read the first data corresponding to the first dimension and the original data corresponding to the second dimension from the dimensional intersection from the summary data table as data to be processed; the first preset filtering type is a retention type, and the second preset filtering type is an exclusion type. The allocation module is used to allocate the first data in the data to be processed according to the second dimension to obtain the second data corresponding to the second dimension. The writing module is used to write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, and write the data to be processed into the summary data table to obtain an updated detailed data table and an updated summary data table.

7. The apparatus according to claim 6, characterized in that, The dimension determination module is further configured to combine the first dimension with each dimension in the second dimension to obtain a dimension combination; According to the said dimension combination, the second data is distributed to obtain the second data for each dimension combination.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images