Data segmentation methods, equipment, media and products

Abstract

The present application discloses a data segmentation method, device, medium and product, which relates to the field of data processing. The method comprises: obtaining data to be segmented; receiving a grouping attribute input by a user, wherein the grouping attribute is one of multiple attributes included in the data to be segmented; the grouping attribute comprises N grouping values, and the data to be segmented is segmented into N groups according to the N grouping values; based on a directed acyclic graph data segmentation method, the N groups of data to be segmented are re-segmented into m groups with the data corresponding to a single grouping value as the smallest unit, where m is less than N. The present application improves the flexibility and customizability of data segmentation, so that the system can quickly adapt to different data characteristics and business needs.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data segmentation method, device, medium and product. Background Art

[0002] In traditional data processing, data segmentation usually follows a hard-coded approach, that is, the data segmentation process is edited in the fixed logic of the program source code. Although this simplifies the implementation process of code editing and speeds up the implementation process of business needs. However, when new business needs emerge, developers need to modify the original code according to the business needs logic. After the modification is completed, the new code is released online. Such a process will inevitably increase the cost of development and maintenance. Moreover, with the expansion of business, the increase in task requirements, and the growth of task data volume, development and maintenance are becoming more and more complicated. At the same time, it also brings about the problem of insufficient flexibility and difficulty in adapting to changing needs. Furthermore, frequent modifications and patches may lead to the accumulation of technical debt. Therefore, the existing data segmentation logic is difficult to meet and is no longer applicable, and needs to be redesigned and adjusted. Summary of the invention

[0003] The purpose of this application is to provide a data segmentation method, device, medium and product to solve the problems described in the background technology.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a data segmentation method, comprising:

[0006] Get the data to be segmented;

[0007] receiving a grouping attribute input by a user, wherein the grouping attribute is one of a plurality of attributes included in the data to be segmented; the grouping attribute includes N grouping values, and the data to be segmented is segmented into N groups according to the N grouping values;

[0008] Based on the directed acyclic graph data segmentation method, N groups of data to be segmented are re-divided into m groups with the data corresponding to a single grouping value as the smallest unit, where m is less than N.

[0009] Optionally, after regrouping, the data corresponding to a single grouping value is recorded as subgroup data, and in the directed acyclic graph-based data segmentation method, N groups of to-be-segmented data are re-segmented into m groups with the data corresponding to a single grouping value as the smallest unit, where m is less than N, and the method further comprises:

[0010] For each of the m groups, calculate the first dependency between the group values;

[0011] Arrange the sub-group data corresponding to each grouping value according to the first dependency relationship;

[0012] For m groups, calculate the second dependency between every two groups;

[0013] Arrange the m groups according to the second dependency.

[0014] Optionally, arranging the sub-group data corresponding to each grouping value according to the first dependency relationship includes:

[0015] Connecting in parallel the subgroup data corresponding to two grouping values ​​whose correlation coefficients are smaller than a first predetermined value;

[0016] Connecting subgroup data corresponding to two grouping values ​​whose correlation coefficient is greater than or equal to a first predetermined value in series;

[0017] The step of arranging the m groups according to the second dependency relationship comprises:

[0018] connecting two groups whose correlation coefficients are smaller than a second predetermined value in parallel;

[0019] Two groups whose correlation coefficient is greater than or equal to a second predetermined value are connected in series.

[0020] Optionally, for a new task requirement, before obtaining the data to be segmented, the method further includes:

[0021] Receive task requirement information input by a user, and parse task parameters based on the task requirement information;

[0022] Pulling task data that meets the task parameters;

[0023] Determining the amount of data of the task data to be pulled based on a predetermined data amount determination method;

[0024] When the data volume is greater than or equal to a predetermined value, the task data is determined as data to be segmented.

[0025] Optionally, the predetermined data volume is determined by the following formula:

[0026]

[0027] Among them, BigData is the data volume of task data;

[0028] n is the number of data items included in the task data;

[0029] t is the preset data volume threshold that needs to be split;

[0030] c is the maximum amount of data that the system can handle;

[0031] α is the weight coefficient, which is used to adjust the ratio of the number of data items n to the system processing capacity c or the data volume threshold t;

[0032] β is a weight coefficient, which is used to adjust the impact of the amount of data exceeding the data volume threshold t on the BigData value.

[0033] Optionally, the directed acyclic graph-based data segmentation method re-divides N groups of to-be-segmented data into m groups, where m is less than N, with data corresponding to a single grouping value as the smallest unit, including:

[0034] Determine J grouping methods for dividing the N grouping values ​​into m groups, each grouping method including multiple grouping methods;

[0035] For each grouping method, calculate the grouping method with the lowest comprehensive cost value and obtain J grouping methods;

[0036] A grouping method with the lowest comprehensive cost value among the J grouping methods is determined, recorded as a first grouping method, and the first grouping method is determined as a final grouping method.

[0037] Optionally, the comprehensive cost value is calculated by the following formula:

[0038]

[0039] Among them, HSplit: the objective function of the overall horizontal segmentation, recorded as the comprehensive cost value;

[0040] i : Index of a single grouping value;

[0041] N: the number of grouped values;

[0042] w : Weight coefficient, used to balance the importance of execution time and resource requirements, the value range of w is [0, 1];

[0043] ET i : In the grouping value i The sum of the estimated execution time of the tasks assigned to it;

[0044] Res i : Grouping value i The sum of resource requirements of the tasks assigned to it;

[0045] LL i : Grouping value i Data locality score of the above distribution;

[0046] Imb: load imbalance metric;

[0047] λ: weight coefficient used to adjust the load imbalance metric. The value range of λ is [0, 1].

[0048] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0049] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0050] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0051] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0052] The data segmentation method provided in the embodiment of the present application, when receiving the data segmentation instruction, the user can input the grouping attribute according to the current task requirements, determine the grouping value included in the grouping attribute, and then based on the grouping attribute and the grouping value, use the DAG segmentation method to re-segment the data to be segmented to meet the new task requirements. For the prior art, data segmentation usually uses a hard-coded method, but the hard coding directly writes the various parameters used in the grouping into the code. Although it simplifies the grouping process, when the data needs to be regrouped, the prior art is powerless. Therefore, compared to the prior art, the present application can set the grouping attribute and the grouping value according to the user's task requirements. When the user needs to regroup according to his own task requirements, the data can be re-segmented by resetting the grouping attribute and the grouping parameter without re-modifying the code, thereby improving the flexibility and customizability of data segmentation, so that the system can quickly adapt to different data characteristics and business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 This is an application environment diagram of a data segmentation method in one embodiment of the present application;

[0055] Figure 2 A flowchart of a data segmentation method provided in one embodiment of the present application;

[0056] Figure 3for Figure 1 A detailed flow chart of step 203;

[0057] Figure 4 A flowchart of a data segmentation method provided in another embodiment of the present application;

[0058] Figure 5 A flowchart of a data segmentation method provided in another embodiment of the present application;

[0059] Figure 6 A flowchart of a data segmentation method provided in another embodiment of the present application;

[0060] Figure 7 A schematic diagram of a DAG segmentation process flow provided in another embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0064] The data segmentation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the application environment includes a terminal and a server. The terminal communicates with the server through a network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the data to be split to the server. After the server receives the data to be split, it can first store the data to be split, and then retrieve it from the storage location when it needs to be processed, or it can perform processing tasks while storing. Of course, the data to be split is more likely to be data generated by the server during data processing, or data obtained directly by the server from the storage system.

[0065] When the server obtains the data to be segmented, it uses the data segmentation method to group the data to be segmented to achieve data segmentation. The server can feed back the segmentation results to the terminal. In addition, in some embodiments, the data segmentation method can also be implemented by the server or the terminal alone, such as the terminal can directly perform data segmentation on the video to be processed, or the server can obtain the video to be processed from the data storage system and perform data segmentation on the video to be processed.

[0066] The terminals may be, but are not limited to, various desktop computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or a cloud server.

[0067] In an exemplary embodiment, Figure 2 As shown, a data segmentation method is provided, which is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 201 to 203:

[0068] Step 201, obtaining data to be segmented.

[0069] The data to be segmented may be new data to be segmented obtained from the database according to new requirements, or may be data segmented last time and needs to be segmented again.

[0070] Optionally, step 201 may be triggered by a data splitting instruction, for example, when a data splitting instruction is detected, the data to be split is obtained. The data splitting instruction may be input by a user through a user interface, or may be triggered by other conditions, for example, when the amount of data to be split is greater than a predetermined value, a data splitting instruction is generated, and the data to be split is carried in the data splitting instruction.

[0071] The data segmentation instruction may also only carry the data identifier of the data to be segmented, obtain the data to be segmented according to the data identifier, and then execute step 202. This application takes the data segmentation instruction carrying the data to be segmented as an example for explanation.

[0072] Exemplarily, the data to be segmented is the order data of a store in a store platform, and the order data is shown in Table 1 below.

[0073] Table 1 Order data

[0074]

[0075] Step 202, receiving a grouping attribute input by a user, wherein the grouping attribute is one of a plurality of attributes included in the data to be segmented; the grouping attribute includes N grouping values, and the data to be segmented is segmented into N groups according to the N grouping values.

[0076] The attributes of the data to be segmented are determined by the content of the data to be segmented. According to the task requirements, an attribute is determined from multiple attributes as a grouping attribute, and the grouping value can be regarded as a subset of the grouping attribute.

[0077] For example, still taking the above example, the attributes of the data to be segmented include user ID, purchase amount, number of purchase orders, product ID, product category, and purchase date. Furthermore, the grouping attribute input by the user is "product category" among the multiple attributes, and the product category includes shampoo, conditioner, and facial cleanser, then N is equal to 3, and the three grouping values ​​are shampoo, conditioner, and facial cleanser respectively.

[0078] Users can enter grouping attributes through the user interface or configuration files.

[0079] When the data needs to be re-divided into m groups, the size of m needs to be determined. When m is greater than N, the N groups cannot be re-divided into m groups. Therefore, there is no need to re-divide the data, and the N groups are still maintained. When m is less than N, the data is re-divided into m groups.

[0080] Step 203 , based on the directed acyclic graph data segmentation method, the N groups of data to be segmented are re-divided into m groups with the data corresponding to a single grouping value as the smallest unit, where m is less than N.

[0081] Among them, directed acyclic graph (full name: Directed Acyclic Graph; English abbreviation: DAG). In computer science and mathematics, DAG is a special graph in which the edges are directional and there are no cycles.

[0082] Exemplarily, m is 2, and the first group after re-division is the shampoo group, and the second group is the conditioner and facial cleanser.

[0083] The data segmentation method provided in the embodiment of the present application, when receiving the data segmentation instruction, the user can input the grouping attribute according to the current task requirements, determine the grouping value included in the grouping attribute, and then based on the grouping attribute and the grouping value, use the DAG segmentation method to re-segment the data to be segmented to meet the new task requirements. For the prior art, data segmentation usually uses a hard-coded method, but the hard coding directly writes the various parameters used in the grouping into the code. Although it simplifies the grouping process, when the data needs to be regrouped, the prior art is powerless. Therefore, compared to the prior art, the present application can set the grouping attribute and the grouping value according to the user's task requirements. When the user needs to regroup according to his own task requirements, the data can be re-segmented by resetting the grouping attribute and the grouping parameter without re-modifying the code, thereby improving the flexibility and customizability of data segmentation, so that the system can quickly adapt to different data characteristics and business needs.

[0084] Alternatively, see Figure 3 In another exemplary embodiment of the present application, the above step 203 may be implemented by the following steps 2031 to 2033:

[0085] Step 2031, determining J grouping methods for dividing the N grouping values ​​into m groups, each grouping method including multiple grouping methods.

[0086] For example, when the grouping values ​​are a, b, c and d, there are two groups to be divided. There are two grouping methods, the first grouping method is that two letters are the first group and the remaining two letters are the second group; the second grouping method is that three letters are the first group and the remaining one letter is the second group.

[0087] Furthermore, each of these two grouping methods includes multiple grouping methods. For example, the second grouping method includes 4 grouping methods. The first grouping method: abc is the first group, d is the second group; the second grouping method: abd is the first group, c is the second group; the third grouping method: acd is the first group, b is the second group; the fourth grouping method: bcd is the first group, a is the second group.

[0088] Step 2032: For each grouping method, calculate the grouping method with the lowest comprehensive cost value to obtain J grouping methods.

[0089] For example, for the second grouping method, first find the three letters with the smallest comprehensive cost value among abc, abd, acd, and bcd. Assuming that the smallest is abc, then among the multiple methods included in the second grouping method, the grouping method with the smallest comprehensive cost value is abc as the first group, and d as the second group, which is recorded as the first grouping method.

[0090] For the first grouping method: first find the two letters with the smallest comprehensive cost value among ab, ac, ad, bc, bd, and cd. Assuming that the smallest is ab, then among the multiple methods included in the first grouping method, the grouping method with the smallest comprehensive cost value is ab as the first group and cd as the second group, which are recorded as the first and second grouping methods.

[0091] Step 2033: determine the grouping method with the lowest comprehensive cost value among the J grouping methods, record it as the first grouping method, and determine the first grouping method as the final grouping method.

[0092] Still taking the above example, the HspIrt of the first grouping method is calculated, recorded as H1, and the HspIrt of the first second grouping method is calculated, recorded as H2. Assuming that H1 is greater than H2, the final grouping method is the grouping method corresponding to H2, that is, ab is the first group and cd is the second group.

[0093] Optionally, the comprehensive cost value is calculated by the following formula (1):

[0094] (1)

[0095] Among them, HSplit: the objective function of the overall horizontal segmentation, recorded as the comprehensive cost value;

[0096] i : Index of a single grouping value;

[0097] N: the number of grouped values;

[0098] w : Weight coefficient, used to balance the importance of execution time and resource requirements, the value range of w is [0, 1];

[0099] ET i : In the grouping value i The sum of the estimated execution time of the tasks assigned to it;

[0100] Res i : Grouping value i The sum of resource requirements of the tasks assigned to it;

[0101] LL i : Grouping value i Data locality score of the above distribution;

[0102] Imb: load imbalance metric;

[0103] λ: weight coefficient used to adjust the load imbalance metric. The value range of λ is [0, 1].

[0104] HSplit is the objective function that attempts to be minimized during splitting. It represents the overall load or cost of the entire system in task allocation. For the convenience of description in the application, it is denoted as the comprehensive cost value. The goal is to find a task allocation method that minimizes the value of HSplit, thereby achieving optimal task allocation, that is, achieving optimal data splitting. When calculating HSplit, it is necessary to perform an accumulative calculation on all N grouping values. Among them, i represents the index of a single calculated grouping value, and N is the total number of grouping values. The contribution of each grouping value i consists of the following parts: The execution time ETi is balanced with other factors by multiplying it by the weight coefficient w. The resource requirement Resi This part is weighted by (1−w), reflecting the impact of resource requirements on task allocation in the case of limited resources. The data locality LLi The data locality score of the allocated task, which usually varies between 0 and 1. A higher data locality score means that the data is more likely to be processed on the same node where it is generated, reducing the need for data transmission. Since high data locality is desired, a negative sign is used here to represent its negative impact on the total load. The load imbalance Imb is an indicator to measure the unevenness of the system load distribution. Its impact on the total load is adjusted by multiplying it by the weight coefficient λ. A higher λ value means more emphasis is placed on reducing the load imbalance. The core of the entire formula lies in balancing the execution time and resource requirements of tasks, while improving data locality and reducing load imbalance. By adjusting the values of w and λ, the task allocation can be optimized according to specific application scenarios and system characteristics to achieve the best performance and efficiency. This method enables the effective allocation of tasks in the DAG on multiple computing nodes, achieving high-performance and high-efficiency computing.

[0105] In addition, the comprehensive cost value of each grouping value can also be calculated through the above formula for HSplit. At this time, N is 1 and Imb is 0. By setting the parameter method to set the required number of groups m, when m >= n, there is no need to regroup, and each grouping value is directly allocated to each group; when m < n, regrouping is required, and according to formula (1), find the grouping method with the smallest HSplit value among all grouping methods.

[0106] Optionally, refer to Figure 4 , combined with Figure 6 In another exemplary embodiment of the present application, for a new task requirement, before step 201, the method further includes the following steps 301-step 304:

[0107] Step 301, receive the task requirement information input by the user and parse the task parameters based on the task requirement information.

[0108] The user can input task requirement information through the user interface or configuration file. The task requirement information is used to determine the data required for this task, and the task parameters are keywords obtained from the task requirements to obtain relevant task data from the database according to the task parameters.

[0109] For example, the task requirement information is "The business needs to calculate the number of purchasers, purchase amount, repurchase amount in the past month, and repurchase amount in the past two months for three products: shampoo, conditioner, and facial cleanser in January 2023." The obtained task parameters include "January 2023", "January 2023", "Conditioner", and "Facial Cleanser".

[0110] Step 302: Pull task data that meets the task parameters.

[0111] Using the task parameters as keywords, corresponding task data is pulled from the database.

[0112] Step 303: Determine the data volume of the pulled task data based on a predetermined data volume determination method.

[0113] Optionally, the method for determining the predetermined data amount is the following formula (2):

[0114] (2)

[0115] Among them, BigData is the data volume of task data;

[0116] n is the number of data items included in the task data; n is the number of data items in the task data that are automatically parsed by the written code.

[0117] t is a preset data volume threshold that requires data segmentation; t can be determined through experience.

[0118] c is the maximum amount of data that the system can process, which is estimated from the CPU and memory values. For the sake of convenience, it is also referred to as the system processing capacity in this application;

[0119] α is the weight coefficient, which is used to adjust the ratio of the number of data items n to the system processing capacity c or the data volume threshold t;

[0120] β is a weight coefficient, which is used to adjust the impact of the amount of data exceeding the data volume threshold t on the BigData value.

[0121] Furthermore, in order to better understand the present application, the above formula is further explained:

[0122] Among them, max(c, t) represents the larger value of c and t, ensuring that even if the number of data items n exceeds the system processing capacity c, a score representing the large amount of data can be calculated.

[0123] Basic ratio: The ratio of the data volume n to the system processing capacity c and the data volume threshold t is calculated, and this ratio is raised to the αth power.

[0124] Adjustments beyond threshold: When the number of data items n exceeds the data volume threshold t, this part will further increase the value of BigData. The value of β can be adjusted to control the impact of the amount of data exceeding the data volume threshold on BigData.

[0125] BigData increases as n increases, reflecting the growth of data volume. When n exceeds t, Some will further increase the value of BigData, thus more significantly reflecting the growth of data volume. The values ​​of α and β can be adjusted to control the impact of data volume on BigData, making BigData more in line with actual business needs and system processing capabilities.

[0126] Data example:

[0127] The data requirement is "Shampoo, conditioner, and facial cleanser need to be grouped in January 2023"; parsing this requirement yields the task parameters: biz_date ranges from January 1, 2023 to January 31, 2023, and sku_group includes: shampoo, conditioner, and facial cleanser.

[0128] Extract the task data related to the task parameters from the database, determine the values ​​of n, t, c, α and β corresponding to the task data, and substitute them into the above formula to calculate the data volume.

[0129] Assume t=100000, c=200000, the CPU has 32 cores and 512GB of memory, α=2, β=1, then:

[0130] If n=250000, the calculation process of BigData is:

[0131]

[0132] The result of BigData is 6.25, which is greater than the threshold of 1. Therefore, the task data is a large data volume task.

[0133] If n=120000, the calculation process of BigData is:

[0134]

[0135] The result of BigData is 0.432, which is less than the threshold of 1, so it is not a large data volume task:

[0136] When the number of tasks is not a large data volume task, grouping is not required; if the task data is a large data volume, step 304 is executed.

[0137] Step 304: when the data volume is greater than or equal to a predetermined value, the task data is determined as data to be segmented.

[0138] Optionally, after step 304, a deduplication operation may be performed on the same fields (attributes in this application) to remove duplicate records and ensure that the records in each group are unique. Through deduplication, the system generates unique group values ​​(group_value), which represent different categories in the data set or the grouping situation at the current time.

[0139] Alternatively, see Figure 5 , combined with Figure 7 In another exemplary embodiment of the present application, after step 203, the method further includes steps 204 to 207:

[0140] Step 204: For each of the m groups, calculate the first dependency relationship between the group values.

[0141] Further, the first dependency is embodied by a correlation between two grouping values. Further, the correlation can be determined by a Pearson correlation coefficient.

[0142] Step 205: Arrange the sub-group data corresponding to each grouping value according to the first dependency relationship.

[0143] After regrouping, the data corresponding to a single group value is recorded as subgroup data. Figure 7 In the graph, grouping values ​​are still used instead of subgroup data.

[0144] Further, step 205 is implemented by the following steps 2051 and 2052:

[0145] Step 2051: connect in parallel the sub-group data corresponding to two grouping values ​​whose correlation coefficients are smaller than a first predetermined value.

[0146] Among them, the subgroup data corresponding to the grouping value can be seen in parallel Figure 7 As shown, group value 6 and group value 7 are in parallel relationship, and group value 1, group value 2 and group value 3 are in series relationship.

[0147] When a data display request sent by a user is received, each sub-group data may be displayed according to the series connection and parallel connection relationship of the above sub-group data to facilitate each user's reference.

[0148] For two grouping values ​​whose correlation coefficient is less than the first predetermined value, it means that the correlation between the two grouping values ​​is small, and a small correlation means a small dependence, so the parallel connection indicates that the dependence between the two grouping values ​​is small.

[0149] The first predetermined value may be an empirical value.

[0150] Step 2052: concatenate subgroup data corresponding to two grouping values ​​whose correlation coefficients are greater than or equal to a first predetermined value.

[0151] For two grouping values ​​whose correlation coefficient is greater than or equal to the first predetermined value, it indicates that the correlation between the two grouping values ​​is relatively large. A relatively large correlation means a relatively large dependence. The series connection indicates that the dependence between the two grouping values ​​is relatively large.

[0152] For example, the two grouping values ​​are the repurchase amount of this month and the purchase amount of last month. Since the repurchase amount of this month depends on the purchase amount of last month to be determined, the two are highly dependent and the correlation coefficient will also be large. At this time, the two can be connected in series, with the purchase amount of last month in front and the repurchase amount of this month in the back.

[0153] Step 206: For the m groups, calculate the second dependency relationship between every two groups.

[0154] Further, the second dependency is embodied by a correlation between the two groups. Further, the correlation can be determined by a Pearson correlation coefficient.

[0155] Step 207, arranging m groups according to the second dependency relationship.

[0156] Further, step 207 is implemented by the following steps 2021 and 2072:

[0157] Step 2071, connecting in parallel two groups whose correlation coefficients are smaller than a second predetermined value.

[0158] The parallel connection means that there is a dependency relationship between the two groups. In step 2072, the two groups with correlation coefficients greater than or equal to the second predetermined value are connected in series.

[0159] The second predetermined value may be an empirical value.

[0160] Step 2072, connect two groups whose correlation system is greater than or equal to a second predetermined value in series.

[0161] Record the dependencies between group values ​​within each group and between groups, modify the structure of the original DAG, reassemble the tasks and dependencies, and check the assembled DAG to ensure that the load is distributed as evenly as possible on all groups.

[0162] After arranging the data groups by the above method, the user can know at a glance the relationship between the groups and obtain the overall relationship between all the data.

[0163] From the above content, we can know that after the grouping is completed, each group of data can be assigned a unique group ID (group_id) for policy attribution grouping. When all the grouping is completed, an intermediate table will be generated. The generated intermediate table is as follows Table 2:

[0164] Table 2 Intermediate table

[0165]

[0166] Data example: Assume that 20,000 data are finally extracted, where w=0.6 (more emphasis on execution time); λ=0.5 (moderate consideration of load imbalance) and divided into 2 groups. Each task that takes 5 minutes to execute requires 1 unit of resources.

[0167] There are 11,000 pieces of shampoo data, the execution time is 15 minutes, and the resource requirement is 3 units. Because the data is read only once without duplication, the data locality is 0;

[0168] 3000 pieces of conditioner data, execution time 5 minutes, resource requirement 1 unit, because the data is only read once without duplication, the locality is 0;

[0169] There are 6,000 pieces of facial cleanser data, and the execution time is 10 minutes. The resource requirement is 2 units. Because the data is read only once without duplication, the data locality is 0.

[0170] Then: HSplit shampoo = 15*0.6+(1-0.6)*3=10.2;

[0171] HSplit Conditioner = 5*0.6+(1-0.6)*1=3.4;

[0172] HSplit facial cleanser = 10*0.6+(1-0.6)*2=6.8;

[0173] Then dag split allocation:

[0174] Group 1: HSplitg1=HSplit shampoo=10.2;

[0175] Group 2: HSplitg2 = HSplit conditioner + HSplit facial cleanser = 10.2;

[0176] Calculate the load imbalance metric for each group: Imb = max(15, 15) - (15 + 15) / 2 = 0;

[0177] HSplit=HSplitg1+HSplitg2+λ*Imb=20.4.

[0178] Confirm whether HSplit is the minimum value. If so, use the segmentation result to confirm the dependencies of dag and assemble dag to obtain groups.

[0179] Two result data sets can be obtained by running the tasks of group 1 and group 2 respectively, and the results are stored in result table 3 of group 1 and result table 4 of group 2.

[0180] Table 3 Results of Group 1

[0181]

[0182] Table 4 Results of Group 2

[0183]

[0184] When the data of result table 3 is generated, the status of the basic table group_id=1 is changed to 1 and the end time is added; when the data of result table 4 is generated, the status of the basic table group_id=2 is changed to 1 and the end time is added, and the following table 5 is generated.

[0185] Table 5 Results after merging group 1 and group 2

[0186]

[0187] Furthermore, in the data display stage:

[0188] When the task status of the basic table (the table that records all group values) is displayed as completed, the system automatically triggers the DAG assembly to merge the various result sets and summarize the data processing results of each group into a unified result table. This table uses group_id and group_value as basic fields to ensure data consistency and comparability, and integrates the fields of each group for specific business scenarios, such as x1 to xn, as shown in Table 6 below.

[0189] Table 6 Results table with specific services set

[0190]

[0191] The system will clean and optimize the merged results. This step involves reviewing and deleting fields that are no longer needed for the business, streamlining the data set to remove duplicate or incomplete entries, and adjusting the format and order of the remaining fields to meet business standards. Finally, a data set containing only fields that are useful for business analysis and decision-making is output, as shown in Table 7 below:

[0192] Table 7 Optimized results

[0193]

[0194] Data example: When the status of the basic tables is 1, data is merged to obtain the following Table 8.

[0195] Table 8 Results after data merging

[0196]

[0197] Delete the group_id that is not needed for the business, and the final output is as shown in Table 9.

[0198] Table 9 Result table after deleting redundant fields

[0199]

[0200] In summary, grouping and segmentation also simplifies the maintenance and update of data processing processes, speeds up development, lowers the technical threshold, and enables non-technical users to participate in data processing. Ultimately, this strategy not only improves the efficiency and response speed of data processing, but also helps organizations stay competitive in a rapidly changing environment.

[0201] In addition, if the data to be segmented is a data system that has been obtained in the previous task requirement, there is no need to re-acquire the data, nor is there any need to pre-process the data to be segmented. The data required by the previous task can be directly used, which means that data utilization and code utilization are realized. Data reuse and code reuse can reduce resource consumption and development and maintenance costs, improve resource utilization efficiency and software development efficiency. At the same time, the simplification and automation of the process further speeds up data processing, ensures the consistency and accuracy of operations, and thus improves efficiency and responsiveness throughout the software development life cycle. These practices not only optimize the workflow, but also enhance the market competitiveness of products and services.

[0202] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data segmentation related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data segmentation method is implemented.

[0203] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0204] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0205] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0206] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0207] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0208] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0209] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0210] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A data segmentation method, characterized in that: include: Get the data to be segmented; receiving a grouping attribute input by a user, wherein the grouping attribute is one of a plurality of attributes included in the data to be segmented; the grouping attribute includes N grouping values, and the data to be segmented is segmented into N groups according to the N grouping values; Based on the directed acyclic graph data segmentation method, N groups of data to be segmented are re-divided into m groups with the data corresponding to a single grouping value as the smallest unit, where m is less than N; The directed acyclic graph-based data segmentation method re-divides N groups of to-be-segmented data into m groups, where m is less than N, with the data corresponding to a single grouping value as the smallest unit, including: Determine J grouping methods for dividing the N grouping values ​​into m groups, each grouping method including multiple grouping methods; For each grouping method, calculate the grouping method with the lowest comprehensive cost value and obtain J grouping methods; Determine a grouping method with the lowest comprehensive cost value among the J grouping methods, record it as a first grouping method, and determine the first grouping method as a final grouping method; The comprehensive cost value is calculated by the following formula: ； Among them, HSplit: the objective function of the overall horizontal segmentation, recorded as the comprehensive cost value; i : Index of a single grouping value; N: the number of grouped values; w : Weight coefficient, used to balance the importance of execution time and resource requirements, the value range of w is [0, 1]; ET i : In the grouping value i The sum of the estimated execution time of the tasks assigned to it; Res i : Grouping value i The sum of resource requirements of the tasks assigned to it; LL i : Grouping value i Data locality score of the above distribution; Imb: load imbalance metric; λ: weight coefficient used to adjust the load imbalance metric. The value range of λ is [0, 1].

2. The data segmentation method according to claim 1, characterized in that: After regrouping, the data corresponding to the single grouping value is recorded as subgroup data. In the directed acyclic graph-based data segmentation method, N groups of data to be segmented are re-segmented into m groups with the data corresponding to the single grouping value as the smallest unit, where m is less than N. The method further includes: For each of the m groups, calculate the first dependency between the group values; Arrange the sub-group data corresponding to each grouping value according to the first dependency relationship; For m groups, calculate the second dependency between every two groups; Arrange the m groups according to the second dependency.

3. The data segmentation method according to claim 2, characterized in that: Arranging the sub-group data corresponding to each grouping value according to the first dependency relationship includes: Connecting in parallel the subgroup data corresponding to two grouping values ​​whose correlation coefficients are smaller than a first predetermined value; Connecting subgroup data corresponding to two grouping values ​​whose correlation coefficient is greater than or equal to a first predetermined value in series; The step of arranging the m groups according to the second dependency relationship comprises: connecting two groups whose correlation coefficients are smaller than a second predetermined value in parallel; Two groups whose correlation coefficient is greater than or equal to a second predetermined value are connected in series.

4. The data segmentation method according to claim 1, characterized in that: For a new task requirement, before obtaining the data to be segmented, the method further includes: Receive task requirement information input by a user, and parse task parameters based on the task requirement information; Pulling task data that meets the task parameters; Determining the amount of data of the task data to be pulled based on a predetermined data amount determination method; When the data volume is greater than or equal to a predetermined value, the task data is determined as data to be segmented.

5. The data segmentation method according to claim 4, characterized in that: The method for determining the predetermined amount of data is as follows: ； Among them, BigData is the data volume of task data; n is the number of data items included in the task data; t is the preset data volume threshold that needs to be split; c is the maximum amount of data that the system can handle; α is the weight coefficient, which is used to adjust the ratio of the number of data items n to the system processing capacity c or the data volume threshold t; β is a weight coefficient, which is used to adjust the impact of the amount of data exceeding the data volume threshold t on the BigData value.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data segmentation method described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data segmentation method described in any one of claims 1 to 5 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data segmentation method described in any one of claims 1 to 5 is implemented.

Images