Business rule-based data partitioning method, apparatus and device, and storage medium

Abstract

The application belongs to the technical field of data processing, and discloses a data splitting method and device based on business rules, equipment and a storage medium. The method comprises the following steps: configuring dimension summary data according to the business rules of mass data; determining a first dimension from the dimension summary data; pre-grouping the mass data based on the first dimension to obtain a first group; determining a first data offset of each group in the first group; updating the first group according to the first data offset to obtain an updated group; and splitting the mass data according to the updated group. The processing performance of the mass data can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data segmentation method, apparatus, device, and storage medium based on business rules. Background Technology

[0002] With the rapid development of various banking businesses, business data is growing exponentially. Batch processing programs are facing challenges in handling massive amounts of data. For example, in large batch databases like Hadoop and GaussDB, where data reaches tens of billions of records, the conventional optimization approach for batch processing programs is to split the data into batches through job splitting. However, simply splitting jobs by type is no longer suitable for rapidly changing business rules, and the performance of processing massive amounts of data will gradually decline. Summary of the Invention

[0003] The main objective of this invention is to provide a data segmentation method, apparatus, device, and storage medium based on business rules, aiming to solve the technical problem in the prior art where the processing performance of massive amounts of data degrades because existing batch processing programs cannot be adapted to rapid changes in business rules.

[0004] To achieve the above objectives, the present invention provides a data segmentation method based on business rules, the method comprising the following steps:

[0005] Data is aggregated based on the dimensions configured according to the business rules for massive amounts of data;

[0006] The first dimension is determined from the aggregated data of the aforementioned dimensions;

[0007] The massive data is pre-grouped based on the first dimension to obtain the first group;

[0008] Determine the first data offset of each group in the first group;

[0009] The first group is updated based on the first data offset to obtain the updated group;

[0010] The massive data is segmented according to the updated grouping.

[0011] Optionally, the step of configuring dimension aggregation data according to business rules for massive amounts of data includes:

[0012] Determine the business rules for the massive amount of data;

[0013] Based on the aforementioned business rules, dimensional classification is performed to obtain initial dimensional data;

[0014] The primary key data in the initial dimension data is removed to obtain the dimension summary data.

[0015] Optionally, the step of performing dimensional classification based on the business rules to obtain initial dimensional data includes:

[0016] Determine the number of groups for the massive data, and determine the multiple dimensions corresponding to the business rules;

[0017] Based on the number of groups, the multiple dimensions are classified into categories respectively to obtain the classification results of the multiple dimensions;

[0018] The initial dimension data is determined based on the partitioning results.

[0019] Optionally, determining the first data offset of each group in the first group includes:

[0020] Determine the number of groups for the massive data;

[0021] The average value of the massive data groups is determined based on the number of groups.

[0022] Determine the amount of data in each group within the first group;

[0023] Based on the group mean and the data volume, determine the first data offset of each group in the first group.

[0024] Optionally, updating the first group based on the first data offset to obtain the updated group includes:

[0025] Set upper and lower threshold values;

[0026] The threshold range of the massive data is determined based on the upper limit threshold and the lower limit threshold.

[0027] Determine whether the first data offset is within the threshold range;

[0028] If it is determined that the first data offset is within the threshold range, then the first group is used as the update group;

[0029] If it is determined that the first data offset is not within the threshold range, then the second dimension is determined from the aggregated data of the dimensions, and the update group of the massive data is determined based on the first dimension and the second dimension.

[0030] Optionally, setting the upper and lower thresholds includes:

[0031] Determine the number of groups for the massive data;

[0032] The average value of the massive data groups is determined based on the number of groups.

[0033] The upper limit threshold and the lower limit threshold are set based on the slice size and the average value of the groups.

[0034] Optionally, determining the update grouping of the massive data based on the first dimension and the second dimension includes:

[0035] S5051. Based on the total dimension, the massive data is pre-grouped to obtain a second group, wherein the total dimension includes the first dimension and the second dimension;

[0036] S5052. Determine the second data offset of each group in the second group;

[0037] S5053. Determine whether the second data offset is within the threshold range;

[0038] S5054. If it is determined that the second data offset is within the threshold range, then the second group is used as the update group;

[0039] S5055. If it is determined that the second data offset is not within the threshold range, then the target dimension is obtained from the dimension summary data, and after updating the total dimension with the target dimension, steps S5051-S5055 are repeated until it is determined that the second data offset is within the threshold range.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a data segmentation device based on business rules, the data segmentation device based on business rules comprising:

[0041] The configuration module is used to configure dimensional summary data based on business rules for massive amounts of data;

[0042] A determination module is used to determine a first dimension from the aggregated data of the dimensions;

[0043] The determining module is further configured to pre-group the massive data based on the first dimension to obtain a first group;

[0044] The determining module is used to determine the first data offset of each group in the first group;

[0045] The update module is used to update the first group according to the first data offset to obtain the updated group;

[0046] The segmentation module is used to segment the massive data according to the updated group.

[0047] Furthermore, to achieve the above objectives, the present invention also proposes a data segmentation device based on business rules, the data segmentation device based on business rules comprising: a memory, a processor, and a data segmentation program based on business rules stored in the memory and executable on the processor, the data segmentation program based on business rules being configured to implement the steps of the data segmentation method based on business rules as described above.

[0048] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a data segmentation program based on business rules, wherein when the data segmentation program based on business rules is executed by a processor, it implements the steps of the data segmentation method based on business rules as described above.

[0049] This invention proposes a data segmentation method, apparatus, device, and storage medium based on business rules. The method involves configuring dimensional summary data based on business rules for massive amounts of data; determining a first dimension from the dimensional summary data; pre-grouping the massive data based on the first dimension to obtain a first group; determining a first data offset for each group within the first group; updating the first group based on the first data offset to obtain an updated group; and segmenting the massive data based on the updated group. By configuring dimensional summary data for massive data based on business rules and then segmenting the massive data based on the dimensional summary data, the parallel processing speed of data can be effectively improved, thereby significantly enhancing the processing performance of massive data and ultimately improving the timeliness of related data links. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of a data segmentation device based on business rules in the hardware operating environment involved in the embodiments of the present invention;

[0051] Figure 2 This is a flowchart illustrating the first embodiment of the data segmentation method based on business rules of the present invention;

[0052] Figure 3 This is a schematic diagram of the data slicing operation in the first embodiment of the data slicing method based on business rules of the present invention;

[0053] Figure 4 This is a flowchart illustrating the second embodiment of the data segmentation method based on business rules of the present invention;

[0054] Figure 5 This is a diagram illustrating the threshold calculation effect in the second embodiment of the data segmentation method based on business rules of the present invention.

[0055] Figure 6 This is a schematic diagram illustrating the process of updating the second group in the second embodiment of the data segmentation method based on business rules of the present invention;

[0056] Figure 7 This is a schematic diagram of the pre-segmentation process in the second embodiment of the data segmentation method based on business rules of the present invention;

[0057] Figure 8 This is a structural block diagram of the first embodiment of the data segmentation device based on business rules of the present invention.

[0058] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0059] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0060] Reference Figure 1 , Figure 1 This is a schematic diagram of the data segmentation device structure based on business rules in the hardware operating environment involved in the embodiments of the present invention.

[0061] like Figure 1 As shown, the data segmentation device based on business rules may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the data segmentation device based on business rules, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0063] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data segmentation program based on business rules.

[0064] exist Figure 1 In the business rule-based data segmentation device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the business rule-based data segmentation device of the present invention can be set in the business rule-based data segmentation device, and the business rule-based data segmentation device calls the business rule-based data segmentation program stored in the memory 1005 through the processor 1001 and executes the business rule-based data segmentation method provided in the embodiment of the present invention.

[0065] Based on the above hardware structure, an embodiment of the data segmentation method based on business rules of the present invention is proposed.

[0066] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a data segmentation method based on business rules according to the present invention.

[0067] In this embodiment, the data segmentation method based on business rules includes the following steps:

[0068] Step S10: Configure dimensional summary data according to the business rules of massive data.

[0069] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, or personal computer, or an electronic device capable of performing the above functions or a data segmentation device based on business rules. The following description uses the data segmentation device based on business rules as an example to illustrate this embodiment and the subsequent embodiments.

[0070] It's important to note that "massive data" refers to massive amounts of business data. Dimensional summary data is configured based on the business rules governing the massive data, and the dimensional summary data will differ depending on the type of massive data. The characteristics of business rules can include dimensions such as business scenarios, channels, platforms, and attention levels. Specifically, in mobile phone application scenarios, configurable dimensions include mobile phone type, user age group, and mobile phone price range, etc., so the dimensional summary data would be the mobile phone type dimension, user age group dimension, and mobile phone price range dimension. In online shopping application scenarios, configurable dimensions include online shopping platform, user age group, and online shopping amount, etc., so the dimensional summary data would be the online shopping platform dimension, user age group dimension, and online shopping amount dimension.

[0071] Understandably, massive amounts of data can be segmented based on different single dimensions in the dimensional summary data (i.e., classification based on single-dimensional data), or multi-dimensional data can be segmented based on multiple dimensions combined in the dimensional summary data (i.e., classification based on multi-dimensional data). Single-dimensional data is mainly used when the business scenario is relatively clear, and using single-dimensional data for segmentation can accurately locate data of the same type. Single-dimensional data is very user-friendly for downstream applications. Multi-dimensional data is mainly used when the differences in business scenarios are more significant.

[0072] In one embodiment, configuring dimensional aggregation data according to business rules for massive amounts of data includes:

[0073] Determine the business rules for the massive amount of data;

[0074] Based on the aforementioned business rules, dimensional classification is performed to obtain initial dimensional data;

[0075] The primary key data in the initial dimension data is removed to obtain the dimension summary data.

[0076] Understandably, in order to avoid the primary key field being collected by adaptive grouping, it is necessary to remove key fields such as the primary key from the initial dimension data. This can effectively avoid situations such as abnormal data splitting.

[0077] In practical implementation, multiple dimensions corresponding to business rules can be determined, and then these multiple dimensions can be categorized to obtain the categorization results for multiple dimensions. The dimensional summary data includes multiple dimensions of massive data and the categorization results under these multiple dimensions. Specifically, multiple dimensions can be categorized based on the same quantity, in which case the number of categorization results for each dimension should be consistent; alternatively, multiple dimensions can be categorized based on different quantities, in which case the number of categorization results for each dimension can be inconsistent.

[0078] In one embodiment, the step of performing dimensional classification based on the business rules to obtain initial dimensional data includes:

[0079] Determine the number of groups for the massive data, and determine the multiple dimensions corresponding to the business rules;

[0080] Based on the number of groups, the multiple dimensions are classified into categories respectively to obtain the classification results of the multiple dimensions;

[0081] The initial dimension data is determined based on the partitioning results.

[0082] It should be noted that the number of groups refers to the number of segments into which massive amounts of data are divided.

[0083] Understandably, data within a single dimension can be divided into multiple categories. Specifically, in the online shopping business scenario, within the dimension of the online shopping platform, when the number of groups for the massive amount of data is 3, the data within this dimension can be divided into Platform A, Platform B, and Platform C. Data under different dimensions can be divided based on the number of groups to obtain the classification results; the initial dimension data includes multiple dimensions of the massive amount of data and the classification results under these multiple dimensions.

[0084] In this embodiment, classifying multiple dimensions by the number of groups ensures that data is segmented in each group during each pre-grouping. If the first group obtained from a pre-grouping does not meet the requirements, new dimensions can be selected for segmentation without combining two dimensions, thereby effectively improving the final segmentation rate.

[0085] Step S20: Determine the first dimension from the aggregated data of the dimensions.

[0086] It should be noted that the first dimension can be randomly selected from the dimensional summary data.

[0087] Step S30: Pre-group the massive data based on the first dimension to obtain the first group.

[0088] It should be noted that massive amounts of data can be pre-grouped based on the classification results under the first dimension. Specifically, for example, if the first dimension is the online shopping platform in the online shopping scenario, the classification results of the online shopping platform can be determined to include platform A, platform B, and platform C. Then, the massive amounts of data can be pre-grouped based on platform A, platform B, and platform C. The first grouping obtained is the platform A group, platform B group, and platform C group of the massive amounts of data.

[0089] Step S40: Determine the first data offset of each group in the first group.

[0090] It should be noted that the first data offset can be determined by comparing the amount of data in each group with the average amount of massive data.

[0091] In one embodiment, determining the first data offset of each group in the first group includes:

[0092] Determine the number of groups for the massive data;

[0093] The average value of the massive data groups is determined based on the number of groups.

[0094] Determine the amount of data in each group within the first group;

[0095] Based on the group mean and the data volume, determine the first data offset of each group in the first group.

[0096] It should be noted that the number of groups refers to the number of times the massive data is divided; the average value of the groups is the value obtained after dividing the massive data equally based on the number of groups.

[0097] Step S50: Update the first group according to the first data offset to obtain the updated group.

[0098] In practical implementation, a threshold can be preset, and the first data offset can be compared with the threshold to determine whether the first group needs to be updated. Specifically, for example, if the first data offset is less than the threshold, the first group does not need to be updated and can be directly used as the update group; if the first data offset is greater than or equal to the threshold, the first group needs to be updated to obtain the update group. New dimensions can be determined from the dimension summary information and then combined with the first dimension to update the first group.

[0099] Step S60: Divide the massive data according to the updated group.

[0100] Understandably, partitioning massive amounts of data based on update groups can ensure that the partitioned data is evenly distributed across different groups, thus avoiding the impact on the computational efficiency of parallel data computing due to large differences in the amount of data within each group.

[0101] In specific implementations, such as Figure 3 As shown, the slicing operation involves dividing the massive data into segments based on the update groups, thereby determining the activity list for each segment (i.e., determining the data within different groups of the massive data).

[0102] This embodiment configures dimension summary data according to business rules for massive data; determines a first dimension from the dimension summary data; pre-groups the massive data based on the first dimension to obtain a first group; determines a first data offset for each group in the first group; updates the first group according to the first data offset to obtain an updated group; and segments the massive data according to the updated group. By configuring dimension summary data for massive data based on business rules, and then segmenting the massive data based on the dimension summary data, the parallel processing speed of the data can be effectively improved, thereby effectively improving the processing performance of massive data and thus effectively improving the timeliness of related links.

[0103] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of a data segmentation method based on business rules according to the present invention.

[0104] Based on the first embodiment described above, the data segmentation method based on business rules in this embodiment, which updates the first group according to the first data offset to obtain the updated group, includes:

[0105] Step S501: Set the upper limit threshold and the lower limit threshold.

[0106] It should be noted that the upper threshold is the upward offset of the group mean. The upper threshold is mainly used when a single slice is continuously split into multiple small data groups to avoid the data groups being too concentrated in a single slice, so that the data can make more reasonable use of parallel resources. The lower threshold is the downward offset of the group mean. It is mainly used when a single slice is continuously split into multiple large data groups to prevent uneven splitting. The closer the lower threshold is to the absolute mean, the higher the computing power consumed.

[0107] In one embodiment, setting the upper and lower thresholds includes:

[0108] Determine the number of groups for the massive data;

[0109] The average value of the massive data groups is determined based on the number of groups.

[0110] The upper limit threshold and the lower limit threshold are set based on the slice size and the average value of the groups.

[0111] It should be noted that slice capacity refers to the maximum data capacity in a single group; when setting the upper limit threshold, it should be ensured that the upper limit threshold does not exceed the slice capacity, and that the difference between the upper limit threshold and the group mean is less than the preset difference; when setting the lower limit threshold, it should be ensured that the difference between the lower limit threshold and the group mean is less than the preset difference.

[0112] In specific implementations, such as Figure 5 As shown, the slice number is the group number, the number of slices is the amount of data in a group, and the average value is the group mean.

[0113] In this embodiment, the upper and lower thresholds are set comprehensively based on information such as the amount of massive data, the number of slices (i.e. the number of groups), and the maximum capacity of a single slice. This can prevent the occurrence of abnormal segmentation due to the amount of data in each group being too small or too large.

[0114] Step S502: Determine the threshold range of the massive data based on the upper limit threshold and the lower limit threshold.

[0115] In the specific implementation, we can first determine the first difference between the upper limit threshold and the group mean, then determine the second difference between the lower limit threshold and the group mean, determine the minimum difference between the first difference and the second difference, and then determine the threshold range of the massive data based on the minimum difference (the threshold range is the range from 0 to the minimum difference); specifically, let the group mean be A, the upper limit threshold be B, and the lower limit threshold be C, then the first difference is BA and the second difference is AC.

[0116] Step S503: Determine whether the first data offset is within the threshold range.

[0117] It is understandable that the first data offset refers to the amount of data within a group that deviates from the group mean.

[0118] Step S504: If it is determined that the first data offset is within the threshold range, then the first group is used as the update group.

[0119] It is understandable that when the first data offset of each group in the first group is within the threshold range, it indicates that the data volume of each group in the first group is relatively uniform.

[0120] Step S505: If it is determined that the first data offset is not within the threshold range, then the second dimension is determined from the aggregated data of the dimensions, and the update group of the massive data is determined based on the first dimension and the second dimension.

[0121] It should be noted that when there are groups in the first group whose first data offset is not within the threshold range, it means that the data volume of each group in the first group is uneven, and it is necessary to determine a new dimension to continue to pre-group the massive data.

[0122] In practical implementation, when the number of category division results for each dimension in the dimensional summary data is consistent, and this number matches the number of groups, the update grouping of the massive data can be directly determined based on the second dimension. When the number of category division results for each dimension in the dimensional summary data is consistent, but this number does not match the number of groups, the update grouping of the massive data needs to be determined based on both the first and second dimensions. Specifically, for example, if the online shopping platform is the first dimension, and the category division result of the first dimension is platform A and platform B, and the online shopping age group is the second dimension, and the category division result of the second dimension is age group A and age group B, then the update grouping includes groups that shop on platform A and are also in age group A, groups that shop on platform A and are also in age group B, and groups that shop on platform B and are also in age group A. And groups that both shop on platform B and belong to age group B; when the number of category division results for each dimension in the aggregated data is inconsistent, it is necessary to jointly determine the updated grouping of the massive data based on the first and second dimensions. Specifically, for example, if the online shopping platform is the first dimension, and the category division result of the first dimension is platform A and platform B, and the online shopping age group is the second dimension, and the category division result of the second dimension is age group A, age group B, and age group C, then the updated grouping includes groups that both shop on platform A and belong to age group A, groups that both shop on platform A and belong to age group B, groups that both shop on platform A and belong to age group C, groups that both shop on platform B and belong to age group A, groups that both shop on platform B and belong to age group B, and groups that both shop on platform B and belong to age group C.

[0123] In one embodiment, such as Figure 6 As shown, determining the update grouping of the massive data based on the first dimension and the second dimension includes:

[0124] S5051. Based on the total dimension, the massive data is pre-grouped to obtain a second group, wherein the total dimension includes the first dimension and the second dimension;

[0125] S5052. Determine the second data offset of each group in the second group;

[0126] S5053. Determine whether the second data offset is within the threshold range;

[0127] S5054. If it is determined that the second data offset is within the threshold range, then the second group is used as the update group;

[0128] S5055. If it is determined that the second data offset is not within the threshold range, then the target dimension is obtained from the dimension summary data, and the total dimension is updated based on the target dimension. Then, steps S5051-S5055 are repeated until it is determined that the second data offset is within the threshold range.

[0129] It is understandable that determining whether the second data offset is within the threshold range means determining whether the second data offset of all groups is within the threshold range; if the second data offset is not within the threshold range, it is necessary to continue to obtain new dimensions from the dimension summary data to update the total dimensions and then continue to pre-group until the second data offset of all groups is within the threshold range.

[0130] In specific implementations, such as Figure 7 As shown, it is necessary to first pre-segment the massive data based on the dimensional summary data, and then determine whether to re-group based on the data of each group after pre-segmentation. When re-grouping is required, new dimensions are added and pre-segmentation is performed again. When re-grouping is not required, the massive data can be segmented directly based on the currently obtained groups.

[0131] This embodiment sets an upper threshold and a lower threshold; determines the threshold range of the massive data based on the upper and lower thresholds; determines whether the first data offset is within the threshold range; if the first data offset is within the threshold range, the first group is used as the update group; if the first data offset is not within the threshold range, a second dimension is determined from the summarized dimension data, and the update group of the massive data is determined based on the first and second dimensions. Through this method, the offset of each group can be calculated after each pre-grouping is completed. When the offset of each group does not meet expectations, dimensions need to be added again and pre-grouping re-performed until the offset of each group reaches the optimal threshold range.

[0132] Furthermore, this embodiment of the invention also proposes a storage medium storing a data segmentation program based on business rules. When the data segmentation program based on business rules is executed by a processor, it implements the steps of the data segmentation method based on business rules as described above.

[0133] Reference Figure 8 ,picture 8 This is a structural block diagram of the first embodiment of the data segmentation device based on business rules of the present invention.

[0134] like Figure 8 As shown, the data segmentation device based on business rules proposed in this embodiment of the invention includes:

[0135] Configuration module 10 is used to configure dimensional summary data based on business rules for massive amounts of data.

[0136] The determination module 20 is used to determine the first dimension from the aggregated data of the dimensions.

[0137] The determining module 20 is further configured to pre-group the massive data based on the first dimension to obtain a first group.

[0138] The determining module 20 is used to determine the first data offset of each group in the first group.

[0139] The update module 30 is used to update the first group according to the first data offset to obtain the updated group.

[0140] The segmentation module 40 is used to segment the massive data according to the updated group.

[0141] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0142] This embodiment configures dimension summary data according to business rules for massive data; determines a first dimension from the dimension summary data; pre-groups the massive data based on the first dimension to obtain a first group; determines a first data offset for each group in the first group; updates the first group according to the first data offset to obtain an updated group; and segments the massive data according to the updated group. By configuring dimension summary data for massive data based on business rules, and then segmenting the massive data based on the dimension summary data, the parallel processing speed of the data can be effectively improved, thereby effectively improving the processing performance of massive data and thus effectively improving the timeliness of related links.

[0143] In one embodiment, the configuration module 10 is further configured to:

[0144] Determine the business rules for the massive amount of data;

[0145] Based on the aforementioned business rules, dimensional classification is performed to obtain initial dimensional data;

[0146] The primary key data in the initial dimension data is removed to obtain the dimension summary data.

[0147] In one embodiment, the configuration module 10 is further configured to:

[0148] Determine the number of groups for the massive data, and determine the multiple dimensions corresponding to the business rules;

[0149] Based on the number of groups, the multiple dimensions are classified into categories respectively to obtain the classification results of the multiple dimensions;

[0150] The initial dimension data is determined based on the partitioning results.

[0151] In one embodiment, the determining module 20 is further configured to:

[0152] Determining the first data offset of each group in the first group includes:

[0153] Determine the number of groups for the massive data;

[0154] The average value of the massive data groups is determined based on the number of groups.

[0155] Determine the amount of data in each group within the first group;

[0156] Based on the group mean and the data volume, determine the first data offset of each group in the first group.

[0157] In one embodiment, the update module 30 is further configured to:

[0158] Set upper and lower threshold values;

[0159] The threshold range of the massive data is determined based on the upper limit threshold and the lower limit threshold.

[0160] Determine whether the first data offset is within the threshold range;

[0161] If it is determined that the first data offset is within the threshold range, then the first group is used as the update group;

[0162] If it is determined that the first data offset is not within the threshold range, then the second dimension is determined from the aggregated data of the dimensions, and the update group of the massive data is determined based on the first dimension and the second dimension.

[0163] In one embodiment, the update module 30 is further configured to:

[0164] Determine the number of groups for the massive data;

[0165] The average value of the massive data groups is determined based on the number of groups.

[0166] The upper limit threshold and the lower limit threshold are set based on the slice size and the average value of the groups.

[0167] In one embodiment, the update module 30 is further configured to:

[0168] Based on the total dimension, the massive data is pre-grouped to obtain a second group, wherein the total dimension includes the first dimension and the second dimension;

[0169] Determine the second data offset of each group in the second group;

[0170] Determine whether the second data offset is within the threshold range;

[0171] If it is determined that the second data offset is within the threshold range, then the second group is used as the update group;

[0172] If it is determined that the second data offset is not within the threshold range, then the target dimension is obtained from the dimension summary data, and the total dimension is updated based on the target dimension. Then, a new update group is determined repeatedly until it is determined that the second data offset is within the threshold range.

[0173] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0174] In addition, for technical details not described in detail in this embodiment, please refer to the data segmentation method based on business rules provided in any embodiment of the present invention, which will not be repeated here.

[0175] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0176] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0178] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data segmentation method based on business rules, characterized in that, The data segmentation method based on business rules includes: Data is aggregated based on the dimensions configured according to the business rules for massive amounts of data; The first dimension is determined from the aggregated data of the aforementioned dimensions; The massive data is pre-grouped based on the first dimension to obtain the first group; Determine the first data offset of each group in the first group, wherein the first data offset refers to the difference between the data volume of each group and the average value of the group; The first group is updated based on the first data offset to obtain the updated group; The massive data is segmented according to the updated groups; The step of updating the first group according to the first data offset to obtain the updated group includes: Set upper and lower threshold values; The threshold range of the massive data is determined based on the upper limit threshold and the lower limit threshold. Determine whether the first data offset is within the threshold range; If it is determined that the first data offset is within the threshold range, then the first group is used as the update group; If it is determined that the first data offset is not within the threshold range, then the second dimension is determined from the aggregated data of the dimensions, and the update group of the massive data is determined based on the first dimension and the second dimension.

2. The method as described in claim 1, characterized in that, The configuration of dimension-based aggregated data according to business rules for massive amounts of data includes: Determine the business rules for the massive amount of data; Based on the aforementioned business rules, dimensional classification is performed to obtain initial dimensional data; The primary key data in the initial dimension data is removed to obtain the dimension summary data.

3. The method as described in claim 2, characterized in that, The initial dimension data is obtained by performing dimensional classification based on the business rules, including: Determine the number of groups for the massive data, and determine the multiple dimensions corresponding to the business rules; Based on the number of groups, the multiple dimensions are classified into categories respectively to obtain the classification results of the multiple dimensions; The initial dimension data is determined based on the partitioning results.

4. The method as described in claim 1, characterized in that, Determining the first data offset of each group in the first group includes: Determine the number of groups for the massive data; The average value of the massive data groups is determined based on the number of groups. Determine the amount of data in each group within the first group; Based on the group mean and the data volume, determine the first data offset of each group in the first group.

5. The method as described in claim 1, characterized in that, The setting of the upper and lower thresholds includes: Determine the number of groups for the massive data; The average value of the massive data groups is determined based on the number of groups. The upper limit threshold and the lower limit threshold are set based on the slice size and the average value of the groups.

6. The method as described in claim 1, characterized in that, Determining the update grouping of the massive data based on the first dimension and the second dimension includes: S5051. The massive data is pre-grouped based on the total dimension to obtain a second group, wherein the total dimension includes the first dimension and the second dimension. S5052. Determine the second data offset of each group in the second group; S5053. Determine whether the second data offset is within the threshold range; S5054. If it is determined that the second data offset is within the threshold range, then the second group is used as the update group; S5055. If it is determined that the second data offset is not within the threshold range, then the target dimension is obtained from the dimension summary data, and the total dimension is updated based on the target dimension. Then, steps S5051-S5055 are repeated until it is determined that the second data offset is within the threshold range.

7. A data segmentation device based on business rules, characterized in that, The data segmentation device based on business rules includes: The configuration module is used to configure dimensional summary data based on business rules for massive amounts of data; A determination module is used to determine a first dimension from the aggregated data of the dimensions; The determining module is further configured to pre-group the massive data based on the first dimension to obtain a first group; The determining module is used to determine the first data offset of each group in the first group, wherein the first data offset refers to the difference between the data volume of each group and the average value of the group. The update module is used to update the first group according to the first data offset to obtain the updated group; The segmentation module is used to segment the massive data according to the updated group; The step of updating the first group according to the first data offset to obtain the updated group includes: Set upper and lower threshold values; The threshold range of the massive data is determined based on the upper limit threshold and the lower limit threshold. Determine whether the first data offset is within the threshold range; If it is determined that the first data offset is within the threshold range, then the first group is used as the update group; If it is determined that the first data offset is not within the threshold range, then the second dimension is determined from the aggregated data of the dimensions, and the update group of the massive data is determined based on the first dimension and the second dimension.

8. A data segmentation device based on business rules, characterized in that, The device includes: a memory, a processor, and a business rule-based data segmentation program stored in the memory and executable on the processor, the business rule-based data segmentation program being configured to implement the steps of the business rule-based data segmentation method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a data segmentation program based on business rules, which, when executed by a processor, implements the steps of the data segmentation method based on business rules as described in any one of claims 1 to 6.

Images