Data processing method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to a data processing method and device, computer equipment and a computer readable storage medium. The method comprises the steps that a first dimension and a second dimension in a summarized data table are determined, the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation; determining first data corresponding to the first dimension from the summarized data table; according to the second dimension, performing apportionment processing on the first data of the first dimension to obtain second data corresponding to the second dimension; and respectively writing second data corresponding to the second dimension into a detail data table and the summarized data table to obtain an updated detail data table and an updated summarized data table. By adopting the method, the resource utilization rate can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Allocation derivation is a commonly used data processing method in the field of management accounting. It can allocate indirect expenses and indirect costs to different dimensions according to different proportions. The existing allocation derivation system generally consists of two parts: allocation derivation and aggregation. Allocation derivation refers to the process of allocating a data set to different dimensions according to certain principles and methods to generate a new data set. The new data set generated by allocation derivation will be stored in the detailed data table, and the newly generated data in the detailed data table will continue to be aggregated with the data in the summary data table for the next allocation derivation or other data processing and financial data analysis.

[0003] However, in the traditional method, the summary part must wait until the allocation derivation is completed and written into the detailed data table, resulting in low utilization of computer resources. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product, which can effectively improve computer resource utilization.

[0005] On the one hand, the present application provides a data processing method, comprising:

[0006] Determine a first dimension and a second dimension in the summary data table, wherein the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation;

[0007] Determining first data corresponding to the first dimension from the summary data table;

[0008] According to the second dimension, the first data of the first dimension is apportioned to obtain second data corresponding to the second dimension;

[0009] The second data corresponding to the second dimension are written into the detail data table and the summary data table respectively to obtain an updated detail data table and an updated summary data table.

[0010] On the other hand, the present application also provides a data processing device, including:

[0011] A dimension determination module, used to determine a first dimension and a second dimension in the summary data table, wherein the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation;

[0012] A data determination module, used to determine first data corresponding to the first dimension from the summary data table;

[0013] An allocating module, configured to allocate the first data of the first dimension according to the second dimension to obtain second data corresponding to the second dimension;

[0014] The writing module is used to write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, so as to obtain an updated detailed data table and an updated summary data table.

[0015] On the other hand, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Determine a first dimension and a second dimension in the summary data table, wherein the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation;

[0017] Determining first data corresponding to the first dimension from the summary data table;

[0018] According to the second dimension, the first data of the first dimension is apportioned to obtain second data corresponding to the second dimension;

[0019] The second data corresponding to the second dimension are written into the detail data table and the summary data table respectively to obtain an updated detail data table and an updated summary data table.

[0020] On the other hand, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] Determine a first dimension and a second dimension in the summary data table, wherein the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation;

[0022] Determining first data corresponding to the first dimension from the summary data table;

[0023] According to the second dimension, the first data of the first dimension is apportioned to obtain second data corresponding to the second dimension;

[0024] The second data corresponding to the second dimension are written into the detail data table and the summary data table respectively to obtain an updated detail data table and an updated summary data table.

[0025] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Determine a first dimension and a second dimension in the summary data table, wherein the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation;

[0027] Determining first data corresponding to the first dimension from the summary data table;

[0028] According to the second dimension, the first data of the first dimension is apportioned to obtain second data corresponding to the second dimension;

[0029] The second data corresponding to the second dimension are written into the detail data table and the summary data table respectively to obtain an updated detail data table and an updated summary data table.

[0030] The above-mentioned data processing method, device, computer equipment, computer-readable storage medium and computer program product determine the sender dimension of data allocation and the receiver dimension of data allocation by determining the first dimension and the second dimension in the summary data table. From the summary data table, the first data corresponding to the first dimension is determined, thereby determining the data to be allocated. According to the second dimension, the first data of the first dimension is allocated, so that the first data of the first dimension can be allocated to the second dimension to obtain the second data corresponding to the second dimension. The second data corresponding to the second dimension is written into the detailed data table and the summary data table respectively to obtain an updated detailed data table and an updated summary data table, so that the allocation and aggregation can be performed simultaneously, thereby improving the data processing efficiency. And the generated second data can be fully utilized, which helps to save computer resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 An application environment diagram of a data processing method in an embodiment;

[0033] Figure 2 is a flow chart of a data processing method in one embodiment;

[0034] Figure 3 A schematic diagram showing comparison of a data table before and after apportionment in an embodiment;

[0035] Figure 4 is an architecture diagram of a data processing method in one embodiment;

[0036] Figure 5 is an architecture diagram of a data processing method in one embodiment;

[0037] Figure 6 is a structural block diagram of a data processing device in one embodiment;

[0038] Figure 7 FIG. 4 is a block diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] The data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Both the terminal 102 and the server 104 can independently execute the data processing method provided in the embodiment of the present application. The terminal 102 and the server 104 can also be used in conjunction to execute the data processing method provided in the embodiment of the present application. When the terminal 102 and the server 104 are used in conjunction to execute the data processing method provided in the embodiment of the present application, the terminal 102 can determine the first dimension and the second dimension from the summary data table stored on the server 104, the first dimension is the sender dimension of the data allocation, and the second dimension is the receiver dimension of the data allocation. The terminal 102 determines the first data corresponding to the first dimension from the summary data table. The terminal 102 allocates the first data of the first dimension according to the second dimension to obtain the second data corresponding to the second dimension. The terminal 102 writes the second data corresponding to the second dimension into the detailed data table and the summary data table, respectively, to obtain an updated detailed data table and an updated summary data table. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0041] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to a computer device (computer device such as Figure 1 The following steps are taken as an example:

[0042] Step 202, determine the first dimension and the second dimension in the summary data table, the first dimension is the sender dimension of data allocation, and the second dimension is the receiver dimension of data allocation.

[0043] Among them, the first dimension is the sender dimension of data sharing, and the second dimension is the receiver dimension of data sharing.

[0044] The first dimension and the second dimension can be dimensions of the same category or dimensions of different categories. For example, the first dimension A1 and the second dimension A2 both belong to dimension category A, or the first dimension A1 belongs to dimension category A, and the second dimension B1 belongs to dimension category B.

[0045] Specifically, the computer device determines the first dimension and the second dimension in the summary data table, the first dimension is the sender dimension of data allocation, and the second dimension is the receiver dimension of data allocation. The data corresponding to the sender dimension is the data to be allocated, and needs to be allocated to the receiver dimension to form the data corresponding to the receiver dimension.

[0046] In this embodiment, the computer device can obtain the first dimension filtering condition and the second dimension filtering condition, filter out the first dimension from the summary data table according to the first dimension filtering condition, and filter out the second dimension from the summary data table according to the second dimension filtering condition.

[0047] Step 204: determine the first data corresponding to the first dimension from the summary data table.

[0048] The first data is the data corresponding to the first dimension in the summary data table.

[0049] Specifically, the computer device reads data corresponding to the first dimension from the summary data table as the first data.

[0050] Step 206: distribute the first data of the first dimension according to the second dimension to obtain second data corresponding to the second dimension.

[0051] Specifically, the computer device distributes the first data of the first dimension according to the second dimension to obtain second data corresponding to the second dimension.

[0052] The first dimension may include at least one sender dimension, and the second dimension may include at least one receiver dimension. The computer device may allocate the first data of each sender dimension according to each receiver dimension to obtain the second data of each receiver dimension.

[0053] In this embodiment, the computer device may determine the respective allocation ratios of at least two second dimensions, allocate the first data according to the respective allocation ratios of the at least two second dimensions, and obtain the respective second data of the at least two second dimensions.

[0054] Step 208, write the second data corresponding to the second dimension into the detail data table and the summary data table respectively, and obtain an updated detail data table and an updated summary data table.

[0055] Specifically, after obtaining the second data corresponding to the second dimension, the computer device can execute the two parts of detailed data table generation and data aggregation in parallel, that is, write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, so as to update the detailed data table and the summary data table respectively, and obtain an updated detailed data table and an updated summary data table.

[0056] In this embodiment, the computer device aggregates the second data of the second dimension with the original data corresponding to the second dimension in the aggregated data table to obtain an updated aggregated data table.

[0057] In this embodiment, the computer device may write the second data into the original detailed data table to update the detailed data table and obtain an updated detailed data table.

[0058] In the above data processing method, the sender dimension of data allocation and the receiver dimension of data allocation are determined by determining the first dimension and the second dimension in the summary data table. From the summary data table, the first data corresponding to the first dimension is determined, thereby determining the data to be allocated. According to the second dimension, the first data of the first dimension is allocated, so that the first data of the first dimension can be allocated to the second dimension to obtain the second data corresponding to the second dimension. The second data corresponding to the second dimension is written into the detailed data table and the summary data table respectively to obtain an updated detailed data table and an updated summary data table, so that the allocation and aggregation can be performed simultaneously to improve data processing efficiency. And the generated second data can be fully utilized, which helps to save computer resources.

[0059] In one embodiment, determining first data corresponding to the first dimension from the summary data table includes:

[0060] Based on the first dimension and the second dimension, read the data to be processed from the summary data table, where the data to be processed at least includes the first data corresponding to the first dimension;

[0061] The second data corresponding to the second dimension is written into the detail data table and the summary data table respectively, and an updated detail data table and an updated summary data table are obtained, including:

[0062] Based on the data to be processed and the second data corresponding to the second dimension, the detailed data table is updated to obtain an updated detailed data table; based on the data to be processed and the second data corresponding to the second dimension, summary data is generated, and the summary data is written into the summary data table to obtain an updated summary data table.

[0063] The data to be processed refers to the data that has changed during data allocation and aggregation.

[0064] Specifically, the computer device reads the data to be processed from the summary data table based on the first dimension and the second dimension, and the data to be processed at least includes the first data corresponding to the first dimension.

[0065] The computer device updates the detailed data table based on the data to be processed and the second data corresponding to the second dimension to obtain an updated detailed data table. Furthermore, the computer device generates summary data corresponding to the second dimension based on the data to be processed and the second data corresponding to the second dimension, writes the summary data into the summary data table, and obtains an updated summary data table.

[0066] In this embodiment, the data to be processed may include first data corresponding to the first dimension and original data corresponding to the second dimension. Based on the first data in the data to be processed and the second data corresponding to the second dimension, the detailed data table is updated to obtain an updated detailed data table. Furthermore, the computer device generates summary data based on the original data in the data to be processed and the second data corresponding to the second dimension, writes the summary data into the summary data table, and obtains an updated summary data table.

[0067] In this embodiment, the summary data of the second dimension replaces the original data of the second dimension in the summary data table to obtain an updated summary data table.

[0068] In this embodiment, since there may be a dimension in the summary data table without the original data corresponding to the dimension, when there are first data corresponding to the first dimension and original data corresponding to the second dimension in the summary data table, the first data of the first dimension and the original data of the second dimension can be used as the data to be processed.

[0069] In this embodiment, based on the first dimension and the second dimension, the data to be processed is read from the summary data table, and the data to be processed includes at least the first data corresponding to the first dimension. Based on the data to be processed and the second data corresponding to the second dimension, the detailed data table is updated to obtain an updated detailed data table; based on the data to be processed and the second data corresponding to the second dimension, the summary data is generated, and the summary data is written into the summary data table to obtain an updated summary data table, thereby updating the detailed data table and generating the summary data of the second dimension in parallel, thereby realizing the update of the summary data table and improving the efficiency of data allocation and data aggregation.

[0070] In one embodiment, based on the first dimension and the second dimension, reading the data to be processed from the summary data table includes:

[0071] When the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as the data to be processed; when the summary data table contains first data and original data, and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension and the first data.

[0072] Specifically, the computer device may determine whether there is first data corresponding to the first dimension and original data corresponding to the second dimension in the summary data table. When the first data corresponding to the first dimension and original data corresponding to the second dimension exist in the summary data table, the computer device determines whether the first data can cover the original data.

[0073] When the first data can cover the original data, it means that the first data of the first dimension contains the original data of the second dimension, and the first data can be used as the data to be processed.

[0074] When the first data cannot cover the original data, it means that the first data of the first dimension cannot completely contain the original data of the second dimension. It is necessary to determine the data to be processed from the summary data table based on the first dimension, the second dimension and the first data, so as to read the relevant data that can cover both the first dimension and the second dimension from the summary data table.

[0075] In one embodiment, when the first data and the original data exist in the summary data table, and the first data does not cover the original data, determining the data to be processed from the summary data table based on the first dimension, the second dimension, and the first data includes:

[0076] When the first data and the original data exist in the summary data table and the first data does not cover the original data, determine the type of the first dimension filtering condition and the type of the second dimension filtering condition; read the data to be processed from the summary data table according to the type of the first dimension filtering condition and the type of the second dimension filtering condition, the first dimension, the second dimension and the first data.

[0077] In this embodiment, according to the type of the first dimension filtering condition and the type of the second dimension filtering condition, the first dimension, the second dimension and the first data, the data to be processed is read from the summary data table, including:

[0078] When the type of the first dimension filter condition and the type of the second dimension filter condition both belong to the first preset filter type, read the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as the data to be processed; when the type of the first dimension filter condition and the type of the second dimension filter condition both belong to the second preset filter type, determine the dimensional intersection between the first dimension and the second dimension; read the first data of the first dimension in the dimensional intersection and the original data of the second dimension in the dimensional intersection from the summary data table together as the data to be processed.

[0079] In this embodiment, when there are first data corresponding to the first dimension and original data corresponding to the second dimension in the summary data table, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as the data to be processed; when there are first data and original data in the summary data table, and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension and the first data to expand the data range, so as to read out the data to be processed within the data range covering both the first dimension and the second dimension.

[0080] In one embodiment, the method further comprises:

[0081] When the first data corresponding to the first dimension exists in the summary data table and the original data corresponding to the second dimension does not exist, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0082] Specifically, when the computer device detects that there is first data corresponding to the first dimension in the summary data table, and there is no original data corresponding to the second dimension, it means that the data under the second dimension in the summary data table is empty, then the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0083] In this embodiment, when the first data corresponding to the first dimension exists in the summary data table, and the original data corresponding to the second dimension does not exist, it means that the original data under the second dimension in the summary data table is empty, then the first data corresponding to the first dimension can be directly read from the summary data table as the data to be processed.

[0084] In one embodiment, determining the first dimension and the second dimension in the summary data table includes:

[0085] Get the first dimension filter condition and the second dimension filter condition; filter the first dimension from the summary data table according to the first dimension filter condition, and filter the second dimension from the summary data table according to the second dimension filter condition.

[0086] Among them, the first dimension filter condition is a condition for filtering out the first dimension from the summary data table. The first dimension filter condition is a constraint condition on the filtered first dimension. The second dimension filter condition is a condition for filtering out the second dimension from the summary data table. The second dimension filter condition is a constraint condition on the filtered second dimension.

[0087] Specifically, the user can configure the first dimension filtering condition and the second dimension filtering condition. The computer device filters out the first dimension from the summary data table based on the first dimension filtering condition configured by the user, and filters out the second dimension from the summary data table based on the second dimension filtering condition.

[0088] In one embodiment, a user may edit and input a first dimension filtering condition and a second dimension filtering condition in a condition configuration interface.

[0089] In this embodiment, the condition configuration interface may pre-set multiple types of filter conditions, and may also configure multiple dimension categories and dimensions under each dimension category, and the user may select the type of filter condition. The computer device generates the first dimension filter condition and the second dimension filter condition based on the type of filter condition selected by the user, the dimension category selected, and the dimension selected.

[0090] The types of filter conditions include, for example, the type of "in...", the type of "not in...", etc.

[0091] In this embodiment, since there is a large amount of data in the summary data table, by obtaining the first dimension filtering condition and the second dimension filtering condition, the first dimension can be quickly filtered out from the large amount of data in the summary data table according to the first dimension filtering condition, and the second dimension can be quickly filtered out from the large amount of data in the summary data table according to the second dimension filtering condition, thereby improving data processing efficiency.

[0092] In one embodiment, based on the first dimension and the second dimension, reading the data to be processed from the summary data table includes:

[0093] Determine the type of the first dimension filter condition and the type of the second dimension filter condition; read the data to be processed from the summary data table according to the type of the first dimension filter condition, the type of the second dimension filter condition, the first dimension, and the second dimension.

[0094] The type of the filtering condition may be a retention type or an exclusion type. The retention type may be, for example, a type of "in...", and the exclusion type may be, for example, a type of "not in...".

[0095] Specifically, the computer device determines the type of the first dimension filtering condition and the type of the second dimension filtering condition, and reads the data to be processed from the summary data table according to the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension, and the second dimension.

[0096] In this embodiment, the type of the first dimension filtering condition and the type of the second dimension filtering condition are determined, and the data to be processed is read from the summary data table according to the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension and the second dimension, so that all the data required in the data allocation and aggregation process can be quickly screened out according to the type of the filtering condition.

[0097] In one embodiment, reading the to-be-processed data from the summary data table according to the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension, and the second dimension includes:

[0098] When the type of the first dimension filter condition and the type of the second dimension filter condition belong to the first preset filter type, read the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as the data to be processed; when the type of the first dimension filter condition and the type of the second dimension filter condition belong to the second preset filter type, determine the dimensional intersection between the first dimension and the second dimension; read the first data corresponding to the first dimension in the dimensional intersection and the original data corresponding to the second dimension in the dimensional intersection from the summary data table as the data to be processed.

[0099] The first preset filter type may be a retention type, and the second preset filter type may be an exclusion type. The first preset filter type may be an "in" type, and the second preset filter type may be a "not in" type.

[0100] Specifically, when the type of the first dimension filter condition and the type of the second dimension filter condition belong to the first preset filter type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as data to be processed.

[0101] like Figure 3As shown, the first dimension filter condition is "dimension A in (A1)", which means that dimension A1 under dimension category A is selected from the summary data table as the first dimension, that is, the first dimension is A=A1. When the second dimension filter condition is "dimension A in (A2, A3)", the filter condition means that dimension A2 and dimension A3 under dimension category A are selected from the summary data table as the second dimension, that is, the second dimension A=A2, and the second dimension A=A3. The first dimension filter condition "dimension A in (A1)" and the second dimension filter condition "dimension A in (A2, A3)" both belong to the first preset filter type, that is, both belong to the filter condition of the "in..." type, and the data corresponding to the first dimension A1 and the data corresponding to the second dimensions A2 and A3 are read from the summary data table as the data to be processed. The apportionment ratio of the second dimension A2 and the second dimension A3 is 50%, so the first data "100" corresponding to the first dimension A1 is apportioned to the second dimension A2 and the second dimension A3, and the second data "50" of the second dimension A2 and the second data "50" of the second dimension A3 are obtained. These data are written into the obvious data table to obtain an updated obvious data table. At the same time, the second data "50" of the second dimension A2 is aggregated with the original data "100" of the second dimension A2 in the summary data table, and the second data "50" of the second dimension A3 is aggregated with the original data "100" of the second dimension A2 in the summary data table to obtain an updated summary data table. In the updated summary data table, the data corresponding to the second dimension A2 is updated from the original "100" to "150", and the data corresponding to the second dimension A3 is updated from the original "100" to "150".

[0102] When the type of the first dimension filter condition and the type of the second dimension filter condition belong to the second preset filter type, the dimension intersection between the first dimension and the second dimension is determined. The computer device reads the first data corresponding to the first dimension in the dimension intersection and the original data corresponding to the second dimension in the dimension intersection from the summary data table as the data to be processed.

[0103] In this embodiment, when the type of the first dimension filter condition and the type of the second dimension filter condition belong to the first preset filter type, the first data corresponding to the first dimension and the original data corresponding to the second dimension are read from the summary data table as the data to be processed; when the type of the first dimension filter condition and the type of the second dimension filter condition belong to the second preset filter type, the dimensional intersection between the first dimension and the second dimension is determined, and the first data corresponding to the first dimension in the dimensional intersection and the original data corresponding to the second dimension in the dimensional intersection are read from the summary data table as the data to be processed, so that the used data that needs to be changed in data allocation and data aggregation can be read from the summary data table to perform data aggregation and improve the efficiency of data allocation and aggregation.

[0104] In one embodiment, the first data of the first dimension is apportioned according to the second dimension to obtain the second data of the second dimension, including:

[0105] The first dimension is respectively combined with each dimension in the second dimension to obtain a dimension combination; the second data is apportioned according to the dimension combination to obtain the second data of each dimension combination.

[0106] Specifically, the computer device may combine the first dimension with each dimension in the second dimension to obtain a dimension combination, and distribute the second data according to the dimension combination to obtain the second data of each dimension combination.

[0107] For example, the first dimension A1 is combined with the second dimensions A2 and A3 respectively to obtain dimension combination A1A2 and dimension combination A1A3, and the first data of the first dimension A1 is allocated to the dimension combination A1A2 and dimension combination A1A3 respectively to obtain the second data corresponding to the dimension combination A1A2 and the second data corresponding to the dimension combination A1A3.

[0108] In this embodiment, the dimension category of the first dimension and the dimension category of the second dimension are determined. When the dimension category of the first dimension is different from the dimension category of the second dimension, the first dimension is combined with each dimension in the second dimension to obtain a dimension combination. According to the dimension combination, the second data is apportioned to obtain the second data of each dimension combination.

[0109] In this embodiment, when the dimension category of the first dimension is the same as the dimension category of the second dimension, the first data of the first dimension is allocated to the second dimension according to the second dimension to obtain second data corresponding to the second dimension.

[0110] For example, if the first dimension A1 and the second dimensions A2 and A3 all belong to dimension category A, the first data of the first dimension A1 are allocated to the second dimension A2 and the second dimension A3 to obtain the second data of the second dimension A2 and the second data of the second dimension A3.

[0111] Or the first dimension A1 belongs to dimension category A, and the second dimension B1 and the second dimension B2 belong to dimension category B. The computer device can combine the first dimension A1 with the second dimensions B1 and B2 respectively to obtain dimension combination A1B1 and dimension combination A1B2. The first data of the first dimension A1 is allocated to the dimension combination A1B1 and the dimension combination A1B2 to obtain the second data of the dimension combination A1B1 and the second data of the dimension combination A1B2.

[0112] In this embodiment, the first dimension is combined with each dimension in the second dimension to obtain a dimension combination, and the second data is allocated according to the dimension combination to obtain the second data of each dimension combination, so that data allocation can be performed in a variety of ways, providing flexibility of data allocation. In addition, different data allocation methods can be set according to different dimension categories to improve the diversity of data allocation.

[0113] In one embodiment, a data processing method is provided, which is applied to a computer device, comprising:

[0114] Get the first dimension filter condition and the second dimension filter condition;

[0115] The first dimension is filtered out from the summary data table according to the first dimension filtering condition, and the second dimension is filtered out from the summary data table according to the second dimension filtering condition. The first dimension is the sender dimension of data allocation, and the second dimension is the receiver dimension of data allocation.

[0116] Determine whether there is original data corresponding to the first data of the first dimension and the second dimension data in the summary data table;

[0117] When the first data corresponding to the first dimension and the original data corresponding to the second dimension exist in the summary data table, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as the data to be processed;

[0118] When the first data and the original data exist in the summary data table, and the first data does not cover the original data, determining the type of the first dimension filtering condition and the type of the second dimension filtering condition;

[0119] When the type of the first dimension filter condition and the type of the second dimension filter condition belong to the first preset filter type, first data corresponding to the first dimension and original data corresponding to the second dimension are read from the summary data table as data to be processed;

[0120] When the type of the first dimension filter condition and the type of the second dimension filter condition belong to a second preset filter type, determining a dimensional intersection between the first dimension and the second dimension;

[0121] From the summary data table, read the first data corresponding to the first dimension in the dimension intersection and the original data corresponding to the second dimension in the dimension intersection as the data to be processed;

[0122] When the first data corresponding to the first dimension exists in the summary data table and the original data corresponding to the second dimension does not exist, the first data corresponding to the first dimension is read from the summary data table as the data to be processed.

[0123] According to the second dimension, the first data of the first dimension is apportioned to obtain second data corresponding to the second dimension;

[0124] Based on the data to be processed and the second data corresponding to the second dimension, the detailed data table is updated to obtain an updated detailed data table;

[0125] Based on the data to be processed and the second data corresponding to the second dimension, summary data is generated, and the summary data is written into a summary data table to obtain an updated summary data table.

[0126] In this embodiment, Figure 4 As shown, a schematic diagram of the architecture of a data processing method is provided. Among them, the filtering operation Filter, the apportionment operation FlatMap, and the aggregation operation CoGroup all represent related operations in the distributed stream processing framework of a general computer system. Filter is used to filter data sets, FlatMap is used to convert one piece of data into multiple pieces of data, which is suitable for processing apportionment derivation, and CoGroup is suitable for scenarios where data from multiple sources need to be processed and elements with the same key in these data need to be grouped together.

[0127] The following describes the detailed process steps of this solution:

[0128] In the first step, the big data computing framework (using Flink as an example) reads all data in the sender data set and the receiver data set in the summary data table from the database, that is, all data sets required for this allocation derivation and the data set range in which the generated data falls. The data in the sender data set is the sender data, and the data in the receiver data set is the receiver data.

[0129] Specifically, there are two ways to describe the filtering conditions of the range of the sender data set and the receiver data set. One is that dimension X is in..., and the other is that dimension X is not in..., which is used to determine the dimension member range of dimension X. After multiple dimensions are ANDed, the final data set range is obtained. The data in the final data set range is the data to be processed.

[0130] There are two scenarios: the range of the sender's dataset completely includes the range of the receiver's dataset; and the sender's dataset cannot completely include the range of the receiver's dataset.

[0131] Methods corresponding to the two scenarios: directly taking the sender data set; expanding the scope of the data set until it can also include the receiver data set.

[0132] The detailed processing flow of the filtering conditions for the database query related data sets is as follows:

[0133] 1) If the sender has a data set and the receiver does not have the dimension of the data set, the data in the sender's data set is taken as the data to be processed;

[0134] 2) If the sender has a data set and the receiver has the dimensions of the data set, it is processed as follows:

[0135] A) If the filtering conditions are all in, the data of the union of the sender dimension and the receiver dimension is taken as the data to be processed;

[0136] B) If the filtering conditions are all not in, the data of the intersection of the sender dimension and the receiver dimension is taken as the data to be processed;

[0137] C) Other types of conditions: Ignore and do nothing.

[0138] 3) The sender does not have a dataset, but the receiver has the dimension of the dataset, which is ignored and not processed.

[0139] 4) If the sender does not have a dataset and the receiver does not have the dimension of the dataset, it will be ignored and not processed.

[0140] Through the above steps, you can get a filtering condition that includes both the sender dataset range and the receiver dataset range (that is, all data within the range of the "sender dataset" and the "receiver dataset" to be read from the summary data table, and all the data includes both the data corresponding to the sender dimension and the data corresponding to the receiver dimension).

[0141] The second step is to filter out the sender data set and enter the apportionment derivation operation node (FlatMap). After the data passes the apportionment derivation operation, the receiver data will be generated and transferred to the CoGroup node for subsequent aggregation. In addition, the non-sender data set is filtered out and directly transferred to the CoGroup node.

[0142] Step 3: Complete the aggregation of all data in CoGroup, and output the recipient data to the detail data table through the corresponding Filter, and output the aggregated data to the aggregated data table, thus completing all operations.

[0143] In this embodiment, based on the big data computing platform, the "apportioned derivation" and "aggregation" parts are combined into the same computing task, which only requires one task submission, making full use of the capabilities of the big data processing platform and the distributed computing framework, reducing the read and write operations of the database, and dynamically arranging the execution sequence of multiple computing tasks by pre-analyzing the size of the data set and intelligent algorithms, making full use of computer resources and improving the system data processing capabilities.

[0144] Furthermore, the allocated derivation system based on the big data processing platform in this embodiment combines the allocated derivation and aggregation processes into one big data task, that is, one task completes the allocated derivation and aggregation, which reduces the dependence on the database and improves the utilization rate of the big data platform. Furthermore, by pre-analyzing the scope of the data set, multiple allocated derivation big data processing tasks are merged into one big data processing task for execution, further improving the processing capacity and horizontal expansion capabilities of the system. Not only does it achieve the merged execution of a single allocated derivation task, but it can also merge multiple allocated derivation tasks with the same or similar processing ranges into one task in an automated manner, and tasks with different processing ranges can be executed in parallel, which can further improve the system processing efficiency in some scenarios.

[0145] On the other hand, when summarizing, it is no longer necessary to read the detailed data table. As the detailed data table becomes larger and larger, the processing performance of the computer equipment in the traditional solution will decrease, while the data processing capacity of the computer equipment in this embodiment will basically not decrease significantly.

[0146] In one embodiment, Figure 5 As shown, a schematic diagram of the architecture of another data processing method is provided.

[0147] In order to realize the combined execution of apportionment deduction and aggregation, in addition to the technical solution described in the present invention, the processing flow of the big data processing platform can also merge the filtering operation (Filter) into the apportionment deduction operation (Flatmap). In the Flatmap operation, the filter is directly called to judge the data. When it meets the sender data, the apportionment deduction logic is used to generate the receiver data, and the sender data and the receiver data are passed to the next aggregation operation. When it does not meet the sender data, it is directly passed to the next aggregation operation. Then, the aggregation of all data is completed in the aggregation operation, and finally the corresponding detailed data and aggregated data are written into the detailed data table or the aggregated data table respectively.

[0148] In addition, the scope of the data set can be analyzed through algorithms, and through program coordination, the derivation of different scopes can be placed in different big data processing tasks and started at the same time to achieve multi-task parallel execution.

[0149] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0150] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0151] In one embodiment, Figure 6 As shown, a data processing device 600 is provided, comprising: a dimension determination module 602, a data determination module 604, an allocation module 606 and a writing module 608, wherein:

[0152] The dimension determination module 602 is used to determine the first dimension and the second dimension in the summary data table, the first dimension is the sender dimension of data allocation, and the second dimension is the receiver dimension of data allocation.

[0153] The data determination module 604 is used to determine the first data corresponding to the first dimension from the summary data table.

[0154] The allocation module 606 is used to allocate the first data of the first dimension according to the second dimension to obtain second data corresponding to the second dimension.

[0155] The writing module 608 is used to write the second data corresponding to the second dimension into the detailed data table and the summary data table respectively, so as to obtain an updated detailed data table and an updated summary data table.

[0156] In one embodiment, the data determination module 604 is further configured to read the data to be processed from the summary data table based on the first dimension and the second dimension, where the data to be processed at least includes the first data corresponding to the first dimension;

[0157] The writing module 608 is also used to update the detailed data table based on the data to be processed and the second data corresponding to the second dimension to obtain an updated detailed data table; generate summary data based on the data to be processed and the second data corresponding to the second dimension, write the summary data into the summary data table, and obtain an updated summary data table.

[0158] In one embodiment, the data determination module 604 is also used to read the first data corresponding to the first dimension from the summary data table as the data to be processed when there are first data corresponding to the first dimension and original data corresponding to the second dimension in the summary data table, and the first data covers the original data; when there are first data and original data in the summary data table, and the first data does not cover the original data, determine the data to be processed from the summary data table based on the first dimension, the second dimension and the first data.

[0159] In one embodiment, the data determination module 604 is also used to read the first data corresponding to the first dimension from the summary data table as the data to be processed when there is first data corresponding to the first dimension in the summary data table and there is no original data corresponding to the second dimension.

[0160] In one embodiment, the data determination module 604 is further used to obtain a first dimension filtering condition and a second dimension filtering condition; filter out the first dimension from the summary data table according to the first dimension filtering condition, and filter out the second dimension from the summary data table according to the second dimension filtering condition.

[0161] In one embodiment, the data determination module 604 is also used to determine the type of the first dimension filtering condition and the type of the second dimension filtering condition; and read the data to be processed from the summary data table according to the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension and the second dimension.

[0162] In one embodiment, the data determination module 604 is also used to read the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as the data to be processed when the type of the first dimension filter condition and the type of the second dimension filter condition belong to the first preset filter type; determine the dimensional intersection between the first dimension and the second dimension when the type of the first dimension filter condition and the type of the second dimension filter condition belong to the second preset filter type; and read the first data corresponding to the first dimension in the dimensional intersection and the original data corresponding to the second dimension in the dimensional intersection from the summary data table as the data to be processed.

[0163] In one embodiment, the dimension determination module 602 is further used to combine the first dimension with each dimension in the second dimension to obtain a dimension combination; and allocate the second data according to the dimension combination to obtain the second data of each dimension combination.

[0164] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0165] In one embodiment, a computer device is provided, which may be a terminal or a server. Taking a server as an example, its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store apportioned data and summarized data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0166] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] The embodiment of the present application also provides a computer-readable storage medium, which is one or more non-volatile computer-readable storage media containing computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the data processing method.

[0168] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the data processing method.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: include: Determine a first dimension and a second dimension in the summary data table, wherein the first dimension is a sender dimension of data allocation, and the second dimension is a receiver dimension of data allocation; Determining first data corresponding to the first dimension from the summary data table; According to the second dimension, the first data of the first dimension is apportioned to obtain second data corresponding to the second dimension; The second data corresponding to the second dimension are written into the detail data table and the summary data table respectively to obtain an updated detail data table and an updated summary data table.

2. The method according to claim 1, characterized in that The determining, from the summary data table, first data corresponding to the first dimension includes: Based on the first dimension and the second dimension, read the data to be processed from the summary data table, where the data to be processed at least includes the first data corresponding to the first dimension; The step of writing the second data corresponding to the second dimension into the detail data table and the summary data table respectively to obtain an updated detail data table and an updated summary data table comprises: Based on the data to be processed and the second data corresponding to the second dimension, updating the detailed data table to obtain an updated detailed data table; Based on the data to be processed and the second data corresponding to the second dimension, summary data is generated, and the summary data is written into the summary data table to obtain an updated summary data table.

3. The method according to claim 2, characterized in that The reading the to-be-processed data from the summary data table based on the first dimension and the second dimension includes: When the summary data table contains first data corresponding to the first dimension and original data corresponding to the second dimension, and the first data covers the original data, the first data corresponding to the first dimension is read from the summary data table as data to be processed; When the first data and the original data exist in the summary data table and the first data does not cover the original data, the data to be processed is determined from the summary data table based on the first dimension, the second dimension and the first data.

4. The method according to claim 3, characterized in that The method further comprises: When the first data corresponding to the first dimension exists in the summary data table, and the original data corresponding to the second dimension does not exist, the first data corresponding to the first dimension is read from the summary data table as data to be processed.

5. The method according to claim 2, characterized in that: The determining of the first dimension and the second dimension in the summary data table includes: Get the first dimension filter condition and the second dimension filter condition; The first dimension is filtered out from the summary data table according to the first dimension filtering condition, and the second dimension is filtered out from the summary data table according to the second dimension filtering condition.

6. The method according to claim 5, characterized in that The reading the to-be-processed data from the summary data table based on the first dimension and the second dimension includes: Determine the type of the first dimension filtering condition and the type of the second dimension filtering condition; The data to be processed is read from the summary data table according to the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension, and the second dimension.

7. The method according to claim 6, characterized in that The reading the to-be-processed data from the summary data table according to the type of the first dimension filtering condition, the type of the second dimension filtering condition, the first dimension, and the second dimension includes: When the type of the first dimension filtering condition and the type of the second dimension filtering condition belong to the first preset filtering type, reading the first data corresponding to the first dimension and the original data corresponding to the second dimension from the summary data table as data to be processed; When the type of the first dimension filtering condition and the type of the second dimension filtering condition belong to a second preset filtering type, determining a dimensional intersection between the first dimension and the second dimension; The first data corresponding to the first dimension in the dimensional intersection and the original data corresponding to the second dimension in the dimensional intersection are read from the summary data table as data to be processed.

8. The method according to any one of claims 1 to 7, characterized in that: The step of apportioning the first data of the first dimension according to the second dimension to obtain respective second data of the second dimension includes: Combine the first dimension with each dimension in the second dimension to obtain a dimension combination; The second data is apportioned according to the dimension combination to obtain the second data of each dimension combination.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images