Data slicing method and apparatus, computer device, and storage medium

Abstract

The application relates to a data segmentation method and device, computer equipment and a storage medium. The method comprises the following steps: determining a plurality of data dimensions of to-be-processed data; obtaining a plurality of candidate data segmentation strategy information according to the plurality of data dimensions and combination results between the plurality of data dimensions; selecting a plurality of first target strategy information from the plurality of candidate data segmentation strategy information according to estimated execution time lengths of the plurality of candidate data segmentation strategy information, wherein the estimated execution time length represents data processing timeliness; scoring each first target strategy information according to corresponding base information of each first target strategy information to obtain a strategy information score of each first target strategy information, wherein the strategy information score represents data processing efficiency; selecting second target strategy information from each first target strategy information according to each strategy information score; and performing data segmentation on the to-be-processed data according to the second target strategy information. The method can take into account data processing timeliness and efficiency.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data segmentation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] For large-scale distributed systems, large support systems are often built to aggregate and statistically analyze business metrics for billing or maintenance purposes. For example, in a CDN (Content Delivery Network), cloud service providers and customers need to understand the traffic generated by online services in real time, such as by analyzing access logs generated for each request to count traffic and requests.

[0003] Different clients have different business models and require different metrics and dimensions. Cloud service providers need to provide statistical data with different dimensions for different client businesses, while also meeting the high timeliness requirements of the data. A common approach is to collect detailed client data independently and provide it to an OLAP (Online Analytical Processing) database to obtain real-time statistical indicators. To meet clients' high timeliness requirements and complete analysis and return results quickly, most OLAP solutions involve data partitioning and pre-aggregation based on partitioning rules, and using horizontal scaling to parallelize the pre-aggregation task to accelerate processing.

[0004] However, the segmentation method is closely related to the pre-aggregation result. If an inappropriate dimension is used to segment the pre-aggregation task, the pre-aggregation result may double, affecting the processing efficiency of the next level. Summary of the Invention

[0005] Therefore, it is necessary to provide a data segmentation method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can balance the timeliness and efficiency of data processing to address the aforementioned technical problems.

[0006] Firstly, this application provides a data segmentation method, including:

[0007] Determine the multiple data dimensions of the data to be processed;

[0008] Based on the combination results of multiple data dimensions, obtain information on multiple candidate data segmentation strategies;

[0009] Based on the estimated execution time of multiple candidate data segmentation strategy information, multiple first target strategy information is selected from the multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data segmentation timeliness of the candidate data segmentation strategy information;

[0010] Based on the cardinality information corresponding to each first target strategy information, each first target strategy information is scored to obtain the strategy information score corresponding to each first target strategy information. The strategy information score is used to characterize the data segmentation efficiency of the first target strategy information.

[0011] Based on the scores of each strategy information, select the second target strategy information from each first target strategy information;

[0012] Based on the second objective strategy information, the data to be processed is segmented.

[0013] Secondly, this application also provides a data segmentation apparatus, comprising:

[0014] The data dimension determination module is used to determine multiple data dimensions of the data to be processed.

[0015] The candidate segmentation strategy acquisition module is used to acquire multiple candidate data segmentation strategy information based on multiple data dimensions and the combination results between multiple data dimensions;

[0016] The first strategy selection module is used to select multiple first target strategy information from multiple candidate data segmentation strategy information based on the estimated execution time of multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data segmentation timeliness of the candidate data segmentation strategy information;

[0017] The second strategy selection module is used to score each first target strategy information according to the cardinality information corresponding to each first target strategy information, and obtain the strategy information score corresponding to each first target strategy information, wherein the strategy information score is used to characterize the data segmentation efficiency of the first target strategy information; and to select second target strategy information from each first target strategy information according to each strategy information score.

[0018] The data segmentation module is used to segment the data to be processed based on the second target strategy information.

[0019] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described data segmentation method embodiments.

[0020] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above-described data segmentation method embodiments.

[0021] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the above-described data segmentation method embodiments.

[0022] The aforementioned data segmentation method, apparatus, computer equipment, computer-readable storage medium, and computer program product first determine the data dimensions of the data to be processed, and obtain multiple candidate data segmentation strategy information based on the combination results between multiple data dimensions and the multiple data dimensions. Second, to meet the customer's timeliness requirements, the execution time of each candidate data segmentation strategy information is estimated in advance, and multiple first target strategy information is selected from each candidate data segmentation strategy information. Third, to improve data processing efficiency, each first target strategy information is scored according to the cardinality information corresponding to each first target strategy information, and a strategy information score for each first target strategy information is obtained. Finally, second target strategy information is selected based on the strategy information score, and the data to be processed is segmented. Thus, on the one hand, by automatically formulating dimensional segmentation strategies and segmenting data based on the existing data of cloud service providers, there is no need to understand the customer's business, reducing manual prior intervention and improving data processing efficiency. At the same time, for changing customer scenarios, it realizes automated switching of dimensional segmentation strategies, reducing the probability of data processing delays that may occur due to the increase in customer business and improving adaptability to different customer scenarios. On the other hand, by optimizing the data segmentation strategy through scoring, it is beneficial to reduce the amount of data to be processed, thereby improving data processing efficiency. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a diagram illustrating the application environment of a data segmentation method in one embodiment;

[0025] Figure 2 This is a flowchart illustrating a data segmentation method in one embodiment;

[0026] Figure 3 This is a flowchart illustrating the data segmentation method in another embodiment;

[0027] Figure 4 This is a flowchart illustrating the data segmentation method in yet another embodiment;

[0028] Figure 5This is a flowchart illustrating the data segmentation method in another embodiment;

[0029] Figure 6 This is a flowchart illustrating the data segmentation method in another embodiment;

[0030] Figure 7 This is a structural block diagram of a data segmentation device in one embodiment;

[0031] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] CDN (Content Delivery Network): A content delivery network is a computer network system interconnected by the Internet. It uses servers located closest to each user to deliver music, pictures, videos, applications, and other files to users faster and more reliably, providing high-performance, scalable, and low-cost network content delivery to users.

[0034] Access logs: Each user accessing the CDN network generates one or more access logs, recording basic information about the request, as well as some customer-customized information, such as the values ​​of certain request headers. Internally, cloud service providers typically collect access logs for billing and scheduling analysis, and display the corresponding analysis results in the console; however, some larger customers prefer to request cloud service providers to directly provide access logs for their own analysis.

[0035] OLAP: Online Analytical Processing, is a technique for performing high-speed, complex queries or multidimensional analyses on large amounts of data stored in data warehouses, data lakes, or other data repositories. OLAP is used in business intelligence, decision support, and various business forecasting and reporting applications. Most business data has multiple dimensions, meaning the data is broken down into multiple categories to allow for the presentation, tracking, or analysis of specific metrics.

[0036] Dimension: A multidimensional dataset contains one or more organized hierarchical levels that users understand and use as the basis for data analysis. For example, a geographic dimension might include levels for country / region, province / autonomous region / municipality, and city / autonomous region / municipality. Or, a time dimension might include hierarchical levels for year, quarter, month, and day. In a pivot table or pivot chart report, each hierarchy becomes a set of fields that can be expanded and collapsed to display lower or higher levels.

[0037] Cardinality: The number of distinct values ​​for a given dimension in a dataset. For example, if "country" is a dimension and has 200 distinct values, then the cardinality of this dimension is 200.

[0038] Metrics: A set of values ​​in a cube, based on columns in the cube's fact table, typically numerical. A metric is a central value in a cube that has been preprocessed, aggregated, and analyzed. Common examples include traffic, bandwidth, and the number of requests.

[0039] Cloud service providers offer standardized services to different clients. However, due to varying business models, these clients require different metrics and dimensions. Some clients choose to pull detailed data for self-analysis, while others prefer cloud providers to offer OLAP services tailored to their specific business needs, providing customized analysis of designated metrics and dimensions. Furthermore, some clients have strict timeliness requirements, such as needing complete analysis results within a specified timeframe. Therefore, businesses dealing with big data processing need to provide different statistical data for different client needs while ensuring data timeliness.

[0040] Due to the widespread demand for data analysis, especially for real-time analysis of large datasets, pre-aggregation and horizontal scaling are commonly used technologies to ensure customer queries return results within seconds. The specific process involves: dividing pre-aggregation tasks using certain rules to ensure that individual tasks do not become too large to exceed single-machine processing capabilities; and then processing these tasks in parallel across multiple nodes to meet the timeliness requirements of customer queries. Pre-aggregation, in particular, involves aggregating data with the same dimension columns from multiple rows into a single row during data import, while keeping the dimension column values ​​unchanged. The indicator columns are then aggregated according to the associated aggregation function. Pre-aggregation can reduce the original hundreds of millions of rows of data to only a few million rows, significantly improving query speed.

[0041] However, as the business grows, it is necessary to dynamically adjust the partitioning and bucketing rules and pre-aggregation task splitting to meet the timeliness of queries. However, cloud services are often unaware of the customer's business model, which may lead to pre-aggregation delays due to changes in the customer's business and failure to adjust in a timely manner, thus affecting the timeliness of the data.

[0042] To meet clients' high timeliness requirements and complete analysis and return results quickly, various OLAP solutions employ pre-aggregation and use horizontal scaling to parallelize pre-aggregation tasks to accelerate processing. This process requires specifying one or more dimensions and partitioning rules to divide the complete data into a series of local data, which are then distributed to different processing nodes to calculate pre-aggregation. Finally, all pre-aggregated results are aggregated and written to the database's query node for client access. For example, if a client wants to calculate bandwidth data by province and carrier, detailed logs from China Telecom can be sent to one node, while logs from China Mobile can be sent to another. Each node calculates the province-specific bandwidth data for that carrier. Finally, the data is aggregated into a single table, providing the province-specific and carrier-specific bandwidth data.

[0043] The choice of segmentation dimension is closely related to the processing result. If an inappropriate business dimension is used for data segmentation, it may lead to the doubling of the pre-aggregation result and affect the processing efficiency of the next level. However, choosing the right dimension for segmentation requires certain business insights or manual heuristic attempts. At the same time, it is also necessary to consider that different customers have different business forms and calculation dimensions. Therefore, there is an urgent need for a data segmentation solution that can adapt to different customer needs, automatically select segmentation dimensions, and adjust in a timely manner when customer business changes.

[0044] The data segmentation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0045] Specifically, server 104 may determine multiple data dimensions of the data to be processed. Next, based on the multiple data dimensions and their combinations, it may obtain multiple candidate data segmentation strategy information. Then, based on the estimated execution time of the multiple candidate data segmentation strategy information, it may select multiple first target strategy information from the multiple candidate data segmentation strategy information. The estimated execution time is used to characterize the data processing timeliness of the candidate data segmentation strategy information. Next, based on the cardinality information corresponding to each first target strategy information, it may score each first target strategy information to obtain a strategy information score corresponding to each first target strategy information. The strategy information score is used to characterize the data processing efficiency of the first target strategy information. Finally, based on the strategy information scores, it may select second target strategy information from the first target strategy information, and based on the second target strategy information, perform data segmentation on the data to be processed. The segmented data is then processed and fed back to terminal 102.

[0046] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] In one exemplary embodiment, such as Figure 2 As shown, a data partitioning method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps (hereinafter referred to as S): S100 to S400. Wherein:

[0048] S100, determine the multiple data dimensions of the data to be processed.

[0049] The data to be processed varies depending on the big data statistical analysis scenario, and this embodiment does not limit the data segmentation scenario. For example, in a cloud service provider's big data analysis scenario, online business generates 10 million HTTP access logs. Each access log contains fields including: timestamp, client IP, request path, country, network operator, and number of bytes sent. The timestamp, client IP, request path, country, network operator, and number of bytes sent represent multiple data dimensions of the data to be processed.

[0050] In practice, multiple data dimensions of the data to be processed can be determined by the original fields of the data to be processed, or multiple data dimensions of the data to be processed in the current batch can be determined by the results of data segmentation of the previous batch of data to be processed.

[0051] S200 obtains multiple candidate data segmentation strategy information based on multiple data dimensions and the combination results between multiple data dimensions.

[0052] Among them, the candidate data splitting strategy information represents possible data splitting (bucketing) schemes.

[0053] In practice, the data to be processed can be segmented based on a single data dimension, thereby generating candidate data segmentation strategy information containing a single data dimension, consistent with the number of data dimensions. For example, candidate data segmentation strategy information can be generated using timestamp, client IP, request path, country, network operator, and number of bytes sent, respectively.

[0054] Alternatively, the data to be processed can be segmented using multiple data dimensions. In this case, the candidate data segmentation strategy information can be obtained by first combining multiple data dimensions to obtain multiple combination results. Each combination result contains multiple data dimensions, thus yielding multiple candidate data segmentation strategy information. For example, the multiple candidate data segmentation strategy information could be, in sequence, "timestamp-request path," "country-network operator," and "country-network operator-client IP," etc. It is understood that this is merely an example and not a definitive limitation.

[0055] S300: Based on the estimated execution time of multiple candidate data segmentation strategy information, select multiple first target strategy information from multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data processing timeliness of the candidate data segmentation strategy information.

[0056] An ideal data dimension does not guarantee that it will meet the customer's timeliness requirements. The processing time for a single task may still be too long, failing to effectively utilize parallelism. Therefore, when selecting a data partitioning strategy, the time required for multiple nodes to process the partitioned data in parallel must be considered.

[0057] The estimated execution time of multiple candidate data segmentation strategy information can be the predicted time required for multiple nodes to process the segmented data in parallel (such as statistical indicators) after the data to be processed is segmented according to the candidate data segmentation strategy information. The first target strategy information is represented as the candidate data segmentation strategy information selected from multiple candidate data segmentation strategy information.

[0058] In practical applications, in order to meet timeliness requirements, execution time constraints can be preset according to the customer's timeliness needs. Specifically, the execution time constraints can be that the estimated execution time is less than a preset estimated execution time threshold, or the execution completion time does not exceed a preset time. The execution completion time can be determined based on the estimated execution time and the current time.

[0059] In specific implementation, for each candidate data segmentation strategy, the estimated execution time of the candidate data segmentation strategy can be estimated based on the amount of data to be processed corresponding to the different dimensional values ​​in the data segmentation strategy. After obtaining the estimated execution times of multiple candidate data segmentation strategies, it is determined whether each candidate data segmentation strategy meets a preset execution time constraint. Then, multiple first target strategy information that meets the preset execution time constraint is selected from the candidate data segmentation strategies. Specifically, the estimated execution time of each candidate data segmentation strategy can be compared with a preset estimated execution time threshold, and first target strategy information with an estimated execution time less than the preset estimated execution time threshold can be selected; alternatively, the execution completion time of each candidate data segmentation strategy can be determined based on the estimated execution time of each candidate data segmentation strategy and the current time, and first target strategy information with an execution completion time not exceeding a preset time can be selected.

[0060] S400, based on the base information corresponding to each first target strategy information, score each first target strategy information to obtain the strategy information score corresponding to each first target strategy information, wherein the strategy information score is used to characterize the data processing efficiency of the first target strategy information.

[0061] The cardinality information may include the values ​​of different dimensions under each dimension in the first target strategy information, and the amount of data to be processed corresponding to the values ​​of different dimensions.

[0062] In practice, multiple evaluation indicators and their corresponding scoring criteria can be pre-set. Then, for each target strategy information, the score of each evaluation indicator is determined based on the target strategy information and its base information. Weights are assigned to each evaluation indicator. Finally, the strategy information score is determined by weighted summation or other methods.

[0063] For example, evaluation metrics may include load balancing and parallelism. Load balancing measures the distribution of data across different processing nodes after data partitioning, and this metric can be measured using the variance and coefficient of variation of the data volume corresponding to different dimension values. Parallelism can be measured based on the cardinality of the data dimensions.

[0064] In other implementations, a machine learning model can be used to score each first target strategy information to obtain a strategy information score corresponding to each first target strategy information. Specifically, the cardinality information of the first target strategy information within a historical time period can be obtained, scoring features can be extracted from the first target strategy information and its cardinality information, training samples can be constructed, and then the constructed scoring model can be iteratively trained using the training samples until a preset training termination condition is met to obtain a scoring model used to score the first target strategy information. In specific implementation, scoring features are extracted from the first target strategy information and its cardinality information, and the scoring features are input into the scoring model to obtain the strategy information score.

[0065] S500: Based on the scoring of each strategy information, select the second target strategy information from each first target strategy information.

[0066] In practice, the first target strategy information with the highest score can be selected from the first target strategy information, and the selected first target strategy information can be determined as the second target strategy information.

[0067] S600 performs data segmentation on the data to be processed based on the second target strategy information.

[0068] In practice, the data to be processed can be segmented according to the data dimensions in the second target strategy information to obtain multiple data shards, which are then distributed to different processing nodes for pre-aggregation processing. Subsequently, each processing node writes the pre-aggregation results into the database query node for customers to query.

[0069] In the above data segmentation method, firstly, by determining the data dimensions of the data to be processed, multiple candidate data segmentation strategy information is obtained based on the combination results between multiple data dimensions and the multiple data dimensions. Secondly, to meet the customer's timeliness requirements, the execution time of each candidate data segmentation strategy information is estimated in advance, and multiple first target strategy information is selected from each candidate data segmentation strategy information. Thirdly, to improve data processing efficiency, each first target strategy information is scored according to the cardinality information corresponding to each first target strategy information, and a strategy information score for each first target strategy information is obtained. Finally, second target strategy information is selected based on the strategy information score, and the data to be processed is segmented. Thus, on the one hand, by automatically formulating dimensional segmentation strategies and segmenting data based on the existing data of cloud service providers, there is no need to understand the customer's business, reducing manual prior intervention and improving data processing efficiency. At the same time, for changing customer scenarios, it realizes automated switching of dimensional segmentation strategies, reducing the probability of data processing delays that may occur due to the increase in customer business and improving adaptability to different customer scenarios. On the other hand, by optimizing the data segmentation strategy through scoring, it is beneficial to reduce the amount of data to be processed, thereby improving data processing efficiency.

[0070] In an exemplary embodiment, as shown Figure 3 in the figure, the manner for obtaining the estimated execution duration of candidate data splitting strategy information includes S101 to S103, where:

[0071] S101: Obtain the historical data splitting result and historical execution duration of the to-be-processed data of the previous batch.

[0072] The historical data splitting result may include the number of data entries of each data shard obtained after splitting the to-be-processed data of the previous batch, and the number of data entries corresponding to each dimension value. The historical execution duration represents the processing duration for the processing node to process the split to-be-processed data of the previous batch.

[0073] In specific implementation, the number of data entries of each data shard obtained after splitting the to-be-processed data of the previous batch, and the number of data entries corresponding to each dimension value in the data shards may be obtained from the pre-aggregation table of each data processing node.

[0074] In this embodiment, after data splitting is performed on the to-be-processed data, a plurality of pre-aggregation tables are created, and an additional metric is added to the pre-aggregation tables, which is used to represent the number of data entries of the to-be-processed data. For example, if a customer needs to count the traffic of operators by province, when creating the table, the dimension columns are province and operator, and in addition to traffic, the metric column also needs to add the number of to-be-processed data entries. For example, the table structure is as follows:

[0075] CREATE TABLE cdn_access_stats (

[0076] event_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT 'service time',

[0077] province VARCHAR(50) NOT NULL COMMENT 'province',

[0078] isp VARCHAR(50) NOT NULL COMMENT 'operator',

[0079] traffic BIGINT NOT NULL COMMENT 'traffic (bytes)',

[0080] _count BIGINT NOT NULL COMMENT 'number of original to-be-processed data entries');

[0081] The `_count` column is used to count the number of raw, pending data items and is not accessible to clients. Some OLAP engines may use different representations; for example, some analytics engines allow adding fields at runtime, which will not be elaborated upon here.

[0082] For example, suppose the data dimension "Province" in the data to be processed includes "Guangdong", "Guangxi", and "Hunan". Then the cardinality of this data dimension is 3, and "Guangdong", "Guangxi", and "Hunan" are the dimension values. The number of data records corresponding to the dimension values ​​represents the distribution of data in the dimension space. Specifically, suppose the distribution of 10 million data records to be processed in the "Province" dimension is as follows: 5 million data records for "Guangdong", 3 million data records for "Guangxi", and 1 million data records for "Hunan".

[0083] S102, based on the historical data segmentation results, estimate the cardinality information of each candidate data segmentation strategy.

[0084] In this embodiment, considering that customer business generally does not undergo sudden changes, the cardinality information of the current batch's data segmentation result can be estimated based on the cardinality information of the data segmentation result of the previous micro-batch.

[0085] In practice, this can be achieved by determining the ratio between the total number of data entries in the historical data segmentation results and the total number of data entries in the current batch of data to be processed. Based on this ratio, the number of data entries corresponding to each dimension value in the historical data segmentation results is transformed to estimate the number of data entries corresponding to each dimension value in each candidate data segmentation strategy information. The total number of data entries in the current batch of data to be processed can be obtained as follows: For systems using message queues, the difference between the data read offset of the previous batch provided by the message queue and the current data read offset can be calculated to obtain the total number of data entries in the current batch. For file-exchange based systems, such as S3 or HDFS, additional metadata can be used to obtain the sum of the data entries in all files, thus determining the total number of data entries in the current batch of data to be processed.

[0086] S103. Based on the historical execution time and the base information of each candidate data segmentation strategy, estimate the execution time of each candidate data segmentation strategy to obtain the estimated execution time of each candidate data segmentation strategy.

[0087] In practice, the estimated execution time is determined by the data dimension with the largest number of data entries. Therefore, it is possible to fit the maximum number of data entries corresponding to each dimension value in the historical data segmentation results with the historical execution time to obtain an execution time prediction model for estimating the execution time based on the maximum number of data entries corresponding to each dimension value. Then, based on the maximum number of data entries corresponding to each dimension value of each candidate data segmentation strategy information, the estimated execution time corresponding to each candidate data segmentation strategy information is predicted.

[0088] The impact of the number of data entries on execution time may be linear or non-linear. For example, more data entries lead to longer transmission times, which is linear. In computationally intensive stages, if the aggregated metrics involve joint queries, such as calculating the maximum bandwidth for each ISP for each URL, the time consumption of this stage is quadratic. Therefore, in this embodiment, an execution time prediction model can be constructed by performing linear regression, multinomial fitting, and curve fitting on the maximum number of data entries and historical execution times. For example, consider a certain data dimension. If its base is 20, then there are 20 dimension values ​​in total. Let its dimensional value be denoted as the 1st. The number of data entries corresponding to each dimension value is ,but This represents the maximum number of data entries corresponding to each of its dimensions. Let the execution time for parallel data processing by nodes after data partitioning using this data dimension be... (Unit: seconds), the execution time prediction model can be:

[0089]

[0090] in, The selected fitting function includes logarithmic, linear, and quadratic terms. This is used to predict execution time. This can be obtained from the historical data segmentation results of the previous batch. (The data segmentation results are...) One task can earn you By combining samples of "number of data entries - execution time", a model for predicting execution time can be fitted by combining samples obtained from multiple executions.

[0091] Subsequently, for each candidate data segmentation strategy, the maximum number of data entries corresponding to each dimension value in the cardinality information of the candidate data segmentation strategy is used as input, and the constructed execution time prediction model is called to obtain the estimated execution time corresponding to the candidate data segmentation strategy.

[0092] In this embodiment, the data to be processed is segmented using different dimensions based on the data segmentation results of the previous batch and the execution time prediction. The processing time of the segmented data is then used to improve the prediction accuracy and efficiency. Furthermore, it is beneficial to select data segmentation strategy information based on the predicted estimated execution time to meet timeliness requirements.

[0093] In one exemplary embodiment, such as Figure 4 As shown, the cardinality information includes the number of data entries to be processed corresponding to each dimension value. Based on the execution time and the cardinality information of each candidate data segmentation strategy, the execution time of each candidate data segmentation strategy is estimated, resulting in the estimated execution time of each candidate data segmentation strategy, including S301 to S302, where:

[0094] S301, for each candidate data segmentation strategy information, determine the maximum number of data items to be processed.

[0095] S302, based on the maximum value, historical data segmentation results and historical execution time, estimate the execution time of each candidate data segmentation strategy information, and obtain the estimated execution time of each candidate data segmentation strategy information.

[0096] For S301, in specific implementation, it can be that for each candidate data segmentation strategy information, the maximum value is selected from the number of data items to be processed corresponding to each dimension value of the candidate data segmentation strategy information.

[0097] For S302, in specific implementation, the historical data segmentation results and historical execution times are fitted to construct an execution time prediction model for predicting execution times. Subsequently, the maximum value corresponding to each candidate data segmentation strategy information is input into the execution time prediction model to obtain the estimated execution time of each candidate data segmentation strategy information. Specifically, the execution time prediction model can be fitted using a regression model (such as logistic regression), by estimating the coefficients of a linear model using the least squares method, or by using a small fully connected neural network. In this embodiment, considering that each batch needs to fit the execution time prediction model based on the historical data segmentation results and historical execution times of the previous batch, a fitting method that can converge quickly is preferred, such as using a support vector machine with kernel functions to construct the execution time prediction model.

[0098] Furthermore, to ensure the standardization of execution time prediction, in this embodiment, after the data to be processed is segmented, multiple pre-aggregation tasks are generated from these segmented data fragments. These tasks are then distributed to various nodes via a cloud container platform, where each node processes the pre-aggregation tasks (e.g., statistical metrics). Each task operates in a consistent execution environment to avoid scheduling to heterogeneous nodes, such as a mix of x86 and ARM processors. Differences in single-core CPU performance can lead to varying execution times for processing the same number of data entries. This reduces interference from other factors on execution conditions and improves the accuracy of execution time prediction.

[0099] If, during actual implementation, the hardware performance parameters of each node differ (e.g., using a heterogeneous cloud platform or a hybrid cloud (containing different CPUs from different cloud providers), then the execution time prediction model needs to be modified. Add hardware performance parameter correction items. Hardware performance parameters can include, but are not limited to, CPU type, CPU frequency, and number of processor cores.

[0100] After constructing the execution time prediction model, for each candidate data segmentation strategy information, the maximum value of the number of data entries corresponding to each dimension value is input into the execution time prediction model to obtain the estimated execution time of the candidate data segmentation strategy information.

[0101] In this embodiment, the execution time is estimated by using the correlation between historical data segmentation results and historical execution time of historical batches, as well as the cardinality information of candidate data segmentation strategy information, thereby improving the accuracy of execution time prediction.

[0102] In one exemplary embodiment, cardinality information includes the cardinality number and the number of data entries corresponding to each dimension value. For example... Figure 5 As shown, based on the cardinality information corresponding to each first objective strategy information, each first objective strategy information is scored to obtain the strategy information score corresponding to each first objective strategy information, including S321 to S323, wherein:

[0103] S321, For each piece of first target strategy information, evaluate the uniformity of the dimension distribution of the first target strategy information based on the number of data entries corresponding to each dimension value, and obtain the dimension distribution score.

[0104] S322, determine the score for the number of base numbers based on the base number and the minimum value among the base numbers.

[0105] S323, Determine the score of the first objective strategy information based on the dimensional distribution score and the cardinality score.

[0106] In this embodiment, for the data segmentation strategy, selecting a data segmentation strategy with uniform data distribution is beneficial to make the execution time of processing nodes similar and reduce the difficulty of task generation. The cardinality of data dimensions determines the resource utilization rate of dimension parallelization. Under the premise of meeting the preset execution time constraints, the smaller the cardinality of data dimensions, the better. Furthermore, using fewer processing nodes to process the segmented data to be processed will also reduce the probability of failure.

[0107] Therefore, in this embodiment, the first target strategy information is evaluated from two aspects: the uniformity of dimensional distribution and the number of cardinalities. Regarding S321, the method for obtaining the number of data entries corresponding to each dimensional value is the same as that in the above embodiment, and will not be repeated here.

[0108] For each piece of information related to the first objective strategy, define its dimensions. ,have Let the nth distinct dimension value be denoted as its nth dimension value. The number of data entries corresponding to each dimension value is: Then, the standard deviation and mean of the number of data points to be processed corresponding to each dimension value can be used to determine the uniformity of the dimension distribution of the first target strategy information, and the dimension distribution score S1 can be obtained.

[0109]

[0110]

[0111]

[0112] in, That is, the standard deviation of the number of data points corresponding to each dimension value, and This represents the average number of data rows corresponding to each dimension value. Ideally, if the distribution were perfectly uniform, meaning all dimension values ​​corresponded to the same number of data rows, then... S1=1. If the dimension values ​​in this segmentation strategy have a clearly long-tailed distribution, leading to... If the score is less than 0, then the score for that item is less than 0.

[0113] For S322, it can be the ratio between the minimum cardinality of each data dimension and the cardinality of the data dimensions in the first target strategy information, to obtain the cardinality score S2:

[0114]

[0115] in, It is the minimum cardinality across all dimensions.

[0116] In other methods, an exponential approach can be used to obtain the base number score.

[0117] For S323, the strategy information score of the first target strategy information can be determined based on the dimensionality distribution score and the cardinality count score. :

[0118]

[0119] Understandably, in an ideal situation, if this dimension has the smallest cardinality among all dimensions and is perfectly uniformly distributed, then... , have .Right now Therefore, the higher the score of the solution, the closer it is to the ideal dimension.

[0120] In this embodiment, the quality of the first target strategy information is evaluated from two aspects: uniformity of dimensional distribution and number of cardinalities. This is beneficial for selecting the better first target strategy information for data segmentation based on the evaluation results, thereby improving the processing efficiency of the data to be processed.

[0121] In one exemplary embodiment, such as Figure 6 As shown, based on multiple data dimensions and the combination results between multiple data dimensions, multiple candidate data segmentation strategy information is obtained, including S401 to S403, where:

[0122] S401 combines multiple data dimensions into pairs to obtain multiple data segmentation strategy information.

[0123] In this embodiment, considering that in some scenarios, using a single dimension for data partitioning may result in an insufficient cardinality of the partitioned data bucket, low parallelism, and difficulty in meeting processing time requirements, and considering that multi-dimensional partitioning may lead to the problem of dimensional cardinality explosion, this embodiment adopts a two-dimensional partitioning method.

[0124] In practice, this can be achieved by combining two data dimensions into one data dimension combination, iterating through each data dimension to obtain multiple data dimension combinations, and each data dimension combination represents a data segmentation strategy. Understandably, for a query scenario with n data dimensions, there can be at most a total of... This refers to information on data partitioning strategies, specifically the bucketing scheme.

[0125] S402, perform correlation analysis on the data dimensions in each data segmentation strategy information to obtain the correlation degree of each data segmentation strategy information.

[0126] In this embodiment, considering that not all two-dimensional bucketing schemes can improve parallelism, if the two data dimensions are strongly correlated, the pre-aggregation task obtained after data partitioning using the bucketing scheme generated by these two data dimensions will not become more uniform. For example, for different scenarios, most Mondays are workdays. If the data dimensions "weekday" and "whether it is a workday" are used for joint partitioning, the amount of data in "Monday - workday" will be much larger than the amount of data in "Monday - non-workday". Using a combination of weakly correlated or uncorrelated dimensions for partitioning will have a better effect. For example, the values ​​of "weekday - weather" will have less overlap. Therefore, in this embodiment, when formulating a bucketing scheme using two data dimensions, in order to reduce the subsequent computational load, a combination of two weakly correlated dimensions is preferred as a candidate bucketing scheme.

[0127] In practical implementation, for each data segmentation strategy, a correlation analysis algorithm can be used to analyze the correlation between the data dimension combinations in the data segmentation strategy information to obtain the correlation degree of the data dimension combinations. The correlation analysis algorithm includes, but is not limited to, Pearson correlation coefficient, Spearman rank correlation coefficient, and mutual information. For example, in this embodiment, mutual information is used as the correlation degree of the data dimension combinations. Specifically, for two data dimensions... ,remember for One of the possible values, for One possible value, the mutual information between these two data dimensions is:

[0128]

[0129] In this embodiment, regarding the marginal probabilities and joint probability The mutual information between the two data dimensions is approximated by using the proportion of the number of data records to the total number of data records:

[0130]

[0131] in, For data dimensions Values The number of data entries to be processed. For data dimensions Values The number of data entries to be processed. For data dimensions Values And data dimensions Values The number of data entries to be processed. This represents the total number of data entries to be processed in the current batch. If the two data dimensions are completely independent, then for each... The possible values ​​for are:

[0132]

[0133] That is, the logarithmic terms are close to Therefore, the sum of all combinations is close to That is, the range of values ​​for mutual information is .

[0134] The method for obtaining the total number of data rows n in the current batch of data to be processed can include: For systems using message queues, the total number of data rows in the current batch can be obtained by calculating the difference between the data read offset of the previous batch provided by the message queue and the current data read offset. For file exchange-based systems, such as S3 or HDFS, additional metadata can be used to obtain the sum of the data rows of all files, thus obtaining the total number of data rows in the current batch of data to be processed.

[0135] Among them, for The joint frequency, or frequency of association, can be obtained by querying the pre-aggregation table. For example, following the pre-aggregation table example above, querying the joint frequency can be done as follows:

[0136] SELECT province, isp SUM(_count) as count FROM cdn_access_stats GROUPBY province, isp.

[0137] Correspondingly, to count the cardinality of a dimension, you can use:

[0138] SELECT COUNT(DISTINCT province) as n_province FROM cdn_access_stats.

[0139] S403: Select N candidate data segmentation strategies with the lowest relevance from multiple data segmentation strategy information, where N is a positive integer.

[0140] N can be set according to the data splitting requirements.

[0141] In practice, after determining the correlation between the data dimension combinations in each data segmentation strategy information, the data segmentation strategy information is sorted according to the correlation. Then, N data segmentation strategy information with the lowest correlation are selected and determined as candidate data segmentation strategy information.

[0142] In this embodiment, multiple bucketing schemes are first generated by combining data dimensions. Then, candidate bucketing schemes are selected based on the relevance of the data dimension combinations, which helps to improve the efficiency of subsequent data processing.

[0143] In an exemplary embodiment, obtaining multiple candidate data segmentation strategy information based on multiple data dimensions and the combination results between multiple data dimensions further includes: generating second candidate data segmentation strategy information based on each data dimension, wherein the candidate data segmentation strategy information further includes the second candidate data segmentation strategy information.

[0144] The second candidate data segmentation strategy information includes data dimensions.

[0145] In practice, a single data dimension can be used to segment the data to be processed, generating a pre-aggregation task. Specifically, for each of the multiple data dimensions, a second candidate data segmentation strategy information including that data dimension is generated. For example, if there are n data dimensions, a segmentation strategy is generated for each data dimension, resulting in n segmentation strategies.

[0146] To provide a clearer explanation of the data segmentation method provided in this application, a specific embodiment is described below, which includes the following steps:

[0147] S1, determine the multiple data dimensions of the data to be processed.

[0148] S2 generates candidate data segmentation strategy information based on each data dimension.

[0149] S3: Group multiple data dimensions into pairs to obtain multiple data segmentation strategy information. Perform correlation analysis on the data dimensions in each data segmentation strategy information to obtain the correlation of each data segmentation strategy information. Select N candidate data segmentation strategy information with the lowest correlation from the multiple data segmentation strategy information information, where N is a positive integer.

[0150] S4 retrieves the historical data segmentation results and historical execution time of the previous batch of data to be processed.

[0151] S5. Based on the historical data segmentation results, estimate the number of data items to be processed corresponding to each dimension value of the candidate data segmentation strategy information.

[0152] S6. Based on the historical execution time and the base information of each candidate data segmentation strategy, estimate the execution time of each candidate data segmentation strategy to obtain the estimated execution time of each candidate data segmentation strategy.

[0153] S7. For each candidate data segmentation strategy, determine the maximum number of data items to be processed.

[0154] S8. Based on the maximum value, historical data segmentation results and historical execution time, estimate the execution time of each candidate data segmentation strategy information to obtain the estimated execution time of each candidate data segmentation strategy information.

[0155] S9. Based on the estimated execution time of multiple candidate data segmentation strategy information, select multiple first target strategy information from multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data processing timeliness of the candidate data segmentation strategy information.

[0156] S10: For each piece of first target strategy information, evaluate the uniformity of the dimension distribution of the first target strategy information based on the number of data entries corresponding to each dimension value, and obtain the dimension distribution score. Determine the cardinality score based on the cardinality number and the minimum value among the cardinality numbers. Determine the score of the first target strategy information based on the dimension distribution score and the cardinality score.

[0157] S11. Based on the scoring of each strategy information, select the second target strategy information from each first target strategy information, and perform data segmentation on the data to be processed based on the second target strategy information.

[0158] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] In one exemplary embodiment, such as Figure 7 As shown, a data segmentation device 600 is provided, including: a data dimension determination module 610, a candidate segmentation strategy acquisition module 620, a first strategy selection module 630, a second strategy selection module 640, and a data segmentation module 650, wherein:

[0160] The data dimension determination module 610 is used to determine multiple data dimensions of the data to be processed.

[0161] The candidate segmentation strategy acquisition module 620 is used to acquire multiple candidate data segmentation strategy information based on multiple data dimensions and the combination results between multiple data dimensions;

[0162] The first strategy selection module 630 is used to select multiple first target strategy information from multiple candidate data segmentation strategy information based on the estimated execution time of multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data segmentation timeliness of the candidate data segmentation strategy information;

[0163] The second strategy selection module 640 is used to score each first target strategy information according to the cardinality information corresponding to each first target strategy information, and obtain the strategy information score corresponding to each first target strategy information, wherein the strategy information score is used to characterize the data segmentation efficiency of the first target strategy information; and select second target strategy information from each first target strategy information according to each strategy information score.

[0164] The data segmentation module 650 is used to segment the data to be processed based on the second target strategy information.

[0165] In an exemplary embodiment, the data segmentation device 600 further includes a historical data acquisition module 660, which is used to acquire the historical data segmentation results and historical execution time of the previous batch of data to be processed; estimate the cardinality information of each candidate data segmentation strategy information based on the historical data segmentation results; and estimate the execution time of each candidate data segmentation strategy information based on the historical execution time and the cardinality information of each candidate data segmentation strategy information, thereby obtaining the estimated execution time of each candidate data segmentation strategy information.

[0166] In an exemplary embodiment, the first strategy selection module 630 is further configured to determine the maximum number of data items to be processed for each candidate data segmentation strategy information; and to estimate the execution time of each candidate data segmentation strategy information based on the maximum value, historical data segmentation results and historical execution time, thereby obtaining the estimated execution time of each candidate data segmentation strategy information.

[0167] In an exemplary embodiment, the second strategy selection module 640 is further configured to evaluate the uniformity of the dimension distribution of the first target strategy information based on the number of data entries corresponding to each dimension value for each first target strategy information, and obtain a dimension distribution score.

[0168] The score for the number of cardinals is determined based on the number of cardinals and the minimum value among all cardinals.

[0169] The score for the first objective strategy information is determined based on the dimensionality distribution score and the cardinality score.

[0170] In an exemplary embodiment, the data segmentation module 650 is further configured to combine multiple data dimensions in pairs to obtain multiple data segmentation strategy information; perform correlation analysis on the data dimensions in each data segmentation strategy information to obtain the correlation of each data segmentation strategy information; and select N candidate data segmentation strategy information with the lowest correlation from the multiple data segmentation strategy information, where N is a positive integer.

[0171] In an exemplary embodiment, the data segmentation module 650 is further configured to generate candidate data segmentation strategy information based on each data dimension.

[0172] Each module in the aforementioned data segmentation device 600 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0173] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data partitioning method.

[0174] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described data segmentation method embodiments.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps in any of the above data segmentation method embodiments.

[0177] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in any of the above data segmentation method embodiments.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0181] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data segmentation method, characterized in that, The method includes: Determine the multiple data dimensions of the data to be processed; Based on the multiple data dimensions and the combination results between the multiple data dimensions, obtain multiple candidate data segmentation strategy information; Based on the estimated execution time of the multiple candidate data segmentation strategy information, multiple first target strategy information is selected from the multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data processing timeliness of the candidate data segmentation strategy information; Based on the base information corresponding to each of the first target strategy information, each of the first target strategy information is scored to obtain the strategy information score corresponding to each of the first target strategy information, wherein the strategy information score is used to characterize the data processing efficiency of the first target strategy information. Based on the scoring of each of the strategy information, select the second target strategy information from each of the first target strategy information; Based on the data dimensions in the second target strategy information, the data to be processed is segmented to obtain multiple data fragments, and the multiple data fragments are distributed to different processing nodes for pre-aggregation processing.

2. The method according to claim 1, characterized in that, The methods for obtaining the estimated execution time of the candidate data segmentation strategy information include: Retrieve the historical data segmentation results and historical execution time of the previous batch of data to be processed; Based on the historical data segmentation results, estimate the cardinality information of each candidate data segmentation strategy; Based on the historical execution time and the cardinality information of each candidate data segmentation strategy, the execution time of each candidate data segmentation strategy is estimated to obtain the estimated execution time of each candidate data segmentation strategy.

3. The method according to claim 2, characterized in that, The cardinality information includes the number of data entries to be processed corresponding to each dimension value; The step of estimating the execution time of each candidate data segmentation strategy based on the execution time and the cardinality information of each candidate data segmentation strategy includes: For each candidate data segmentation strategy, determine the maximum number of data items to be processed; Based on the maximum value, the historical data segmentation results, and the historical execution time, the execution time of each candidate data segmentation strategy information is estimated to obtain the estimated execution time of each candidate data segmentation strategy information.

4. The method according to claim 1, characterized in that, The cardinality information includes the cardinality number and the number of data entries corresponding to each dimension value; The step of scoring each first target strategy information according to the cardinality information corresponding to each first target strategy information to obtain the strategy information score corresponding to each first target strategy information includes: For each piece of the first target strategy information, the uniformity of the dimension distribution of the first target strategy information is evaluated based on the number of data entries corresponding to each dimension value, and a dimension distribution score is obtained. The score for the number of base elements is determined based on the number of base elements and the minimum value among the base elements. The score of the first target strategy information is determined based on the dimensional distribution score and the cardinality score.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining multiple candidate data segmentation strategy information based on the multiple data dimensions and the combination results between the multiple data dimensions includes: By grouping the data dimensions into pairs, multiple data dimensions are combined to obtain multiple data segmentation strategy information; A correlation analysis is performed on the data dimensions in each of the data segmentation strategy information to obtain the correlation degree of each of the data segmentation strategy information; From the multiple data segmentation strategy information, select N candidate data segmentation strategy information with the lowest relevance, where N is a positive integer.

6. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining multiple candidate data segmentation strategy information based on the multiple data dimensions and the combination results between the multiple data dimensions further includes: Based on each of the aforementioned data dimensions, candidate data segmentation strategy information is generated.

7. A data segmentation device, characterized in that, The device includes: The data dimension determination module is used to determine multiple data dimensions of the data to be processed. The candidate segmentation strategy acquisition module is used to acquire multiple candidate data segmentation strategy information based on the multiple data dimensions and the combination results between the multiple data dimensions; The first strategy selection module is used to select multiple first target strategy information from the multiple candidate data segmentation strategy information based on the estimated execution time of the multiple candidate data segmentation strategy information, wherein the estimated execution time is used to characterize the data segmentation timeliness of the candidate data segmentation strategy information; The second strategy selection module is used to score each first target strategy information according to the cardinality information corresponding to each first target strategy information, to obtain a strategy information score corresponding to each first target strategy information, wherein the strategy information score is used to characterize the data segmentation efficiency of the first target strategy information; and to select second target strategy information from each first target strategy information according to each strategy information score. The data segmentation module is used to segment the data to be processed according to the data dimensions in the second target strategy information to obtain multiple data fragments, and distribute the multiple data fragments to different processing nodes for pre-aggregation processing.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images