A spark multi-objective data partition optimization method and system based on a genetic algorithm, a terminal, and a storage medium

Abstract

The application relates to the field of data processing and discloses a Spark multi-target data partition optimization method, system, terminal and storage medium based on a genetic algorithm, which comprises the following steps: modeling data partition as a multi-target optimization problem of load balancing, Shuffle cost and data locality; constructing a comprehensive fitness function based on the multi-target optimization problem, randomly generating a partition scheme by using a genetic algorithm to serve as an initial population, iteratively searching for an optimal partition scheme through selection, crossover, mutation and elite reservation operations; and further establishing an optimized partition mapping for a high-frequency key and configuring a hash fallback function for a low-frequency key based on the scheme; and finally querying a mapping or calculating a fallback function according to a data key type in Spark task execution to complete data partition. The application significantly improves the load balancing among nodes, simultaneously reduces the network transmission overhead in the Shuffle stage, and realizes comprehensive performance improvement on a highly skewed data set.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a Spark multi-objective data partitioning optimization method, system, terminal, and computer-readable storage medium based on a genetic algorithm. Background Technology

[0002] In Spark (a distributed computing framework), data partitioning is the core mechanism for dividing a large dataset into multiple smaller data blocks (partitions) and distributing them to different computing nodes for parallel processing. The quality of partitioning directly determines the load balancing, network transmission overhead, and overall execution efficiency of the job. Therefore, an ideal partitioning scheme needs to optimize three key objectives simultaneously: load balancing, minimizing the cost of shuffle (the combined resource and performance overhead of performing shuffle operations in a distributed computing framework), and maintaining data locality. To achieve these objectives, the industry has developed two basic technical approaches: one is hash partitioning based on hash functions, which achieves simple and efficient data distribution through deterministic mapping; the other is range partitioning based on data sampling, which attempts to achieve ordered and balanced partitioning by sensing the data distribution.

[0003] However, existing technologies suffer from systemic flaws when processing highly skewed data: hash partitioning and its improvements, unable to perceive data distribution, inevitably lead to severe load unevenness in hot key scenarios; range partitioning and sampling-based improvements attempt to perceive data distribution, but the sampling mechanism is prone to distortion under extreme skew and cannot handle unsortable keys, and their strategy of pursuing record balance ignores the differences in actual data volume. More importantly, all existing methods only perform local optimization for a single objective, and the greedy strategies they employ are prone to getting trapped in local optima, lacking global search capabilities. This inevitably leads to the sacrifice of other key metrics when optimizing a particular objective. Therefore, when facing real-world data, existing technologies lack a unified optimization framework that can systematically balance the three objectives of load balancing, shuffle cost, and data locality, and adapt to large-scale datasets, resulting in common bottlenecks such as task tailing, high network overhead, and low resource utilization.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a Spark multi-objective data partitioning optimization method, system, terminal, and computer-readable storage medium based on a genetic algorithm. This invention aims to solve the problem that existing Spark data partitioning methods cannot effectively balance multiple objectives such as load balancing, shuffle cost, and data locality when processing skewed data, resulting in performance bottlenecks.

[0006] To achieve the above objectives, this invention provides a Spark multi-objective data partitioning optimization method based on a genetic algorithm. This method includes the following steps:

[0007] Distributed sampling is performed on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective.

[0008] A comprehensive fitness function is constructed based on the load balancing objective, the shuffle cost objective, and the data locality objective.

[0009] Multiple partitioning schemes are randomly generated, and each partitioning scheme is used as an individual in the genetic algorithm. An initial population is formed based on all individuals, and the fitness value of each individual in the initial population is calculated based on the comprehensive fitness function.

[0010] Starting from the initial population and the fitness value of each individual in the initial population, multiple iterations are performed. In the new population obtained in each iteration, the fitness of the individuals in the new population is evaluated, and the partitioning scheme represented by the individual with the highest fitness during the iteration process is taken as the optimal partitioning scheme.

[0011] Based on the optimal partitioning scheme, construct an optimized partitioning map for high-frequency keys and configure a hash rollback mechanism for low-frequency keys;

[0012] Obtain the data key input during Spark task execution. If the data key is a high-frequency key, determine the target partition for the high-frequency key by querying the optimized partition mapping. If the data key is a low-frequency key, execute the hash backoff mechanism to determine the target partition for the low-frequency key.

[0013] Optionally, the Spark multi-objective data partitioning optimization method based on genetic algorithms, wherein the distributed sampling of the Spark dataset to be processed to obtain the key frequency distribution of the data, and the data partitioning problem is formalized into a multi-objective optimization problem based on the key frequency distribution, specifically includes:

[0014] The Spark dataset to be processed is randomly sampled at a pre-configured sampling rate, and the frequency of each key is counted through distributed aggregation operations to obtain the key frequency distribution.

[0015] If the number of unique keys in the key frequency distribution exceeds a preset threshold, then the top preset number of keys with the highest frequency are selected as the high-frequency key subset to be optimized.

[0016] Based on the key frequency distribution and the high-frequency key subset, the load balancing objective, the shuffle cost objective, and the data locality objective are quantified into mathematical expressions to form the multi-objective optimization problem.

[0017] Optionally, the Spark multi-objective data partitioning optimization method based on genetic algorithm, wherein the load balancing objective is to minimize the normalized standard deviation of the partition load and convert the normalized standard deviation into a load balancing fitness that adapts to the genetic algorithm to complete the quantification, so as to balance the computational load of each partition.

[0018] Wherein, the partition load is the sum of the frequencies of all keys selected from the high-frequency key subset within the partition, and the normalized standard deviation is the normalized value of the deviation between the actual load and the ideal uniform load of each partition based on the high-frequency key subset.

[0019] The Shuffle cost target is quantified by maximizing the data centrality coefficient, which is used to reduce the overhead of cross-partition data transfer in the Shuffle operation;

[0020] The data concentration coefficient is calculated based on the frequency distribution of the high-frequency key subset and is used to measure whether the data of the high-frequency key subset is concentrated in a few partitions. The larger the data concentration coefficient, the smaller the expected amount of data to be transferred across partitions.

[0021] The data locality objective is quantified by maximizing the key adjacency metric, which is used to place related keys close together in physical storage to improve access efficiency.

[0022] The key adjacency metric is the ratio of the number of adjacent key pairs assigned to the same or adjacent partitions to the total number of adjacent key pairs in the key sequence of the high-frequency key subset arranged in a predetermined order.

[0023] Optionally, the Spark multi-objective data partitioning optimization method based on genetic algorithms, wherein constructing a comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective specifically includes:

[0024] Based on the load balancing objective, the shuffle cost objective, and the data locality objective, a comprehensive fitness function is constructed using a weighted sum method:

[0025] ;

[0026] in, Represents the comprehensive fitness function, This indicates the load balancing fitness level. This represents the centrality coefficient of the dataset. This indicates the maximization of the key adjacency metric. , , These represent the configurable first weight coefficient, second weight coefficient, and third weight coefficient, respectively. Indicates the partitioning scheme.

[0027] Optionally, the Spark multi-objective data partitioning optimization method based on genetic algorithm, wherein each partitioning scheme is treated as an individual in the genetic algorithm, an initial population is formed based on all individuals, and the fitness value of each individual in the initial population is calculated based on the comprehensive fitness function, specifically includes:

[0028] Each partitioning scheme is defined as an individual in the genetic algorithm, and each individual is represented by a chromosome encoding.

[0029] The chromosome encoding generation process is as follows: sampling to obtain a subset of high-frequency bonds selected from the bond frequency distribution, and randomly assigning a partition index with a value range of [0, N-1] to each bond in the high-frequency bond subset, where N represents the total number of partitions;

[0030] Based on a preset comprehensive fitness function, the fitness value corresponding to each individual in the initial population is calculated. The fitness value is used to comprehensively reflect the quantitative performance of the partitioning scheme represented by the individual in the multi-objective optimization problem.

[0031] Optionally, the Spark multi-objective data partitioning optimization method based on genetic algorithms, wherein the step of iterating multiple times starting from the initial population and the fitness value of each individual in the initial population, and evaluating the fitness of individuals in the new population obtained in each iteration, and taking the partitioning scheme represented by the individual with the highest fitness during the iteration as the optimal partitioning scheme, specifically includes:

[0032] Set the number of iterations and execute the iteration loop. When the number of iterations is reached, output the partitioning scheme represented by the recorded global best individual as the optimal partitioning scheme.

[0033] The process of each iteration is as follows: based on the fitness value of individuals in the current population, the selection operation of the genetic algorithm is performed to select parent individuals for reproduction;

[0034] Perform crossover on the selected parent individuals and mutation on the generated offspring individuals to produce a new set of candidate individuals;

[0035] Based on the comprehensive fitness function, the fitness value of each individual in the newly generated candidate individual set is calculated, and combined with the preset elite retention strategy, a new generation of population is formed.

[0036] Compare and record the individual with the highest fitness in the current iteration, and update the record of the globally optimal individual.

[0037] Optionally, the Spark multi-objective data partitioning optimization method based on genetic algorithms, wherein obtaining the data key input during Spark task execution, and if the data key is a high-frequency key, determining the target partition for the high-frequency key by querying the optimized partitioning mapping; and if the data key is a low-frequency key, executing the hash backoff mechanism to determine the target partition for the low-frequency key, specifically includes:

[0038] On each executor of a Spark task, a data record to be processed is received, and a data key is extracted from the data record.

[0039] Determine whether the data key exists in the key set of the optimized partition mapping. If it exists, the data key is a high-frequency key. Find the partition index corresponding to the high-frequency key from the optimized partition mapping and use the partition index as the target partition of the high-frequency key.

[0040] If it does not exist, the data key is a low-frequency key. The low-frequency key is input into a preset hash function to calculate the hash value, and the hash value is moduloed with the total number of partitions. The result of the operation is used as the target partition of the low-frequency key.

[0041] Furthermore, to achieve the above objectives, the present invention also provides a Spark multi-objective data partitioning optimization system based on a genetic algorithm, wherein the Spark multi-objective data partitioning optimization system based on a genetic algorithm includes:

[0042] The optimization problem modeling module is used to perform distributed sampling on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective.

[0043] The fitness function construction module is used to construct a comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective.

[0044] The individual fitness value calculation module is used to randomly generate multiple partition schemes, treat each partition scheme as an individual in the genetic algorithm, form an initial population based on all individuals, and calculate the fitness value of each individual in the initial population based on the comprehensive fitness function.

[0045] The optimal partitioning scheme acquisition module is used to perform multiple iterations starting from the initial population and the fitness value of each individual in the initial population, and to evaluate the fitness of individuals in the new population obtained in each iteration, and to take the partitioning scheme represented by the individual with the highest fitness during the iteration as the optimal partitioning scheme.

[0046] The partition scheme conversion module is used to construct an optimized partition mapping for high-frequency keys based on the optimal partition scheme, and to configure a hash rollback mechanism for low-frequency keys;

[0047] The data partitioning execution module is used to obtain the data key input during the execution of the Spark task. If the data key is a high-frequency key, the target partition of the high-frequency key is determined by querying the optimized partition mapping. If the data key is a low-frequency key, the hash backoff mechanism is executed to determine the target partition of the low-frequency key.

[0048] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a Spark multi-objective data partitioning optimization program based on a genetic algorithm stored in the memory and executable on the processor. When the Spark multi-objective data partitioning optimization program based on a genetic algorithm is executed by the processor, it implements the steps of the Spark multi-objective data partitioning optimization method based on a genetic algorithm as described above.

[0049] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a Spark multi-objective data partitioning optimization program based on a genetic algorithm, and when the Spark multi-objective data partitioning optimization program based on a genetic algorithm is executed by a processor, it implements the steps of the Spark multi-objective data partitioning optimization method based on a genetic algorithm as described above.

[0050] In this invention, data partitioning is modeled as a multi-objective optimization problem involving load balancing, shuffle cost, and data locality. A comprehensive fitness function is constructed based on this multi-objective optimization problem. A genetic algorithm is used to randomly generate partitioning schemes as the initial population. Through iterative search using selection, crossover, mutation, and elite retention operations, the optimal partitioning scheme is obtained. Then, based on this scheme, an optimized partitioning mapping is established for high-frequency keys, and a hash backoff function is configured for low-frequency keys. Finally, during Spark task execution, the mapping is queried or the backoff function is calculated based on the data key type to complete the data partitioning. This invention effectively alleviates task tailing caused by data skew, significantly improves load balancing among nodes, reduces network transmission overhead during the shuffle stage, and enhances the storage and access locality of related data, achieving comprehensive performance improvement on highly skewed datasets. Attached Figure Description

[0051] Figure 1 This is a flowchart of a preferred embodiment of the Spark multi-objective data partitioning optimization method based on genetic algorithm of the present invention;

[0052] Figure 2 This is a schematic diagram comparing the load balancing performance of various partitioning algorithms on different datasets in the Spark multi-objective data partitioning optimization method based on genetic algorithms of this invention;

[0053] Figure 3 This is a schematic diagram illustrating the improvement rate of load variance of the GA-Optimized method relative to the baseline method in the Spark multi-objective data partitioning optimization method based on genetic algorithms of this invention;

[0054] Figure 4 This is a schematic diagram comparing the partition load distribution of each partitioning algorithm in the Spark multi-objective data partitioning optimization method based on genetic algorithm of this invention;

[0055] Figure 5 This is a schematic diagram illustrating the performance of different partitioning methods in the three-dimensional target space of the Spark multi-objective data partitioning optimization method based on genetic algorithm in this invention;

[0056] Figure 6 This is a schematic diagram of the evolutionary process of the genetic algorithm and the changes in population diversity in the Spark multi-objective data partitioning optimization method based on genetic algorithm of this invention;

[0057] Figure 7 This is a structural diagram of a preferred embodiment of the Spark multi-objective data partitioning optimization system based on genetic algorithms of the present invention;

[0058] Figure 8 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0059] This application provides a method, system, and terminal for Spark multi-objective data partitioning optimization based on genetic algorithms. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0061] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0062] The preferred embodiment of the present invention describes a Spark multi-objective data partitioning optimization method based on genetic algorithms, such as... Figure 1 As shown, the Spark multi-objective data partitioning optimization method based on genetic algorithms includes the following steps:

[0063] Step S10: Distributed sampling is performed on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective.

[0064] The Spark dataset to be processed undergoes distributed sampling to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, specifically including:

[0065] The Spark dataset to be processed is randomly sampled at a pre-configured sampling rate, and the frequency of each key is counted through distributed aggregation operations to obtain the key frequency distribution.

[0066] If the number of unique keys in the key frequency distribution exceeds a preset threshold, then the top preset number of keys with the highest frequency are selected as the high-frequency key subset to be optimized.

[0067] Based on the key frequency distribution and the high-frequency key subset, the load balancing objective, the shuffle cost objective, and the data locality objective are quantified into mathematical expressions to form the multi-objective optimization problem.

[0068] In this embodiment, random sampling is performed using rdd.sample(withReplacement=false, fraction=0.01) (the sampling rate is configurable), and the key frequency distribution key_freq is obtained through distributed aggregation operations of map and reduceByKey. This process avoids a full scan and prevents memory overflow on the driver side.

[0069] The high-frequency key subset selection is based on a Top-K mechanism. When the number of unique keys exceeds a preset threshold (e.g., max_keys=10000), the system selects only the top K most frequent keys to form an optimized subset. This design is based on the Pareto principle, focusing on optimizing the hot keys that have the greatest impact on system load, while leaving the remaining low-frequency keys to be handled by an efficient hash rollback mechanism, thus achieving a balance between optimization effect and computational overhead.

[0070] Furthermore, based on the key frequency distribution and the high-frequency key subset, the load balancing objective, the shuffle cost objective, and the data locality objective are quantitatively modeled, transforming the three abstract objectives into a mathematical form that can be processed by the genetic algorithm.

[0071] Specifically, the load balancing objective is to minimize the normalized standard deviation of the partition load and convert the normalized standard deviation into a load balancing fitness suitable for the genetic algorithm to achieve quantification, thereby balancing the computational load of each partition; wherein, the partition load is the sum of the frequencies of all keys selected from the high-frequency key subset within the partition, and the normalized standard deviation is the normalized value of the deviation between the actual load of each partition and the ideal uniform load based on the high-frequency key subset.

[0072] Understandably, uneven load can cause some partitions to become performance bottlenecks. For each unique key... Its frequency is ,in, Represents the key set, Indicates the index of the key. Indicator key frequency, Indicates the first One key, This indicates that the condition is met. The number of indices i.

[0073] In partitioning scheme Below, partitions The load is:

[0074] ;

[0075] in, In the partition scheme Below, partitions load, This represents all elements belonging to the key set. And in the partitioning scheme Downmap to any partition The key The corresponding frequencies are summed. The system consists of m partitions, denoted as . , Represents a partition set. This indicates the first partition. This indicates the second partition. This indicates the third partition. This represents the j-th partition (any partition). This represents the m-th partition (the last partition).

[0076] Ideally, each partition should bear a uniform load of , This represents the total number of records in the dataset. This invention uses the normalized standard deviation to quantify load imbalance.

[0077] ;

[0078] in, This represents the standard deviation after normalization.

[0079] Finally, to facilitate its use as a fitness function in the genetic algorithm (the larger the better), the normalized standard deviation is transformed into a load-balanced fitness suitable for the genetic algorithm to complete the quantification:

[0080] ;

[0081] in, This indicates the load balancing adaptability.

[0082] When all partitions have completely equal load The fitness level reaches its maximum value of 1; the more unbalanced the load, the better. The larger the value, the lower the fitness. Accordingly, the load balancing objective is defined as:

[0083] ;

[0084] in, This indicates the load balancing target.

[0085] The Shuffle cost target is quantified by maximizing the data concentration coefficient to reduce the overhead of cross-partition data transfer in the Shuffle operation; wherein, the data concentration coefficient is calculated based on the frequency distribution of the high-frequency key subset and is used to measure whether the data of the high-frequency key subset is concentrated in a few partitions. The larger the data concentration coefficient, the smaller the expected value of the amount of data transferred across partitions.

[0086] Understandably, cross-partition data transfer is a major performance bottleneck in Spark's Shuffle operation. This invention introduces a data centrality coefficient to model the Shuffle cost:

[0087] ;

[0088] in, This represents the data centrality coefficient, which measures the degree to which the data is concentrated. The larger the data centrality coefficient, the more concentrated the data is in a few partitions, and the less data is transferred across partitions during shuffle.

[0089] Under the uniform shuffle assumption, the expected shuffle data volume is:

[0090] ;

[0091] in, This indicates the expected amount of Shuffle data. It indicates a direct proportional relationship.

[0092] Therefore, maximize This is equivalent to minimizing the Shuffle cost, and the Shuffle cost objective is defined as:

[0093] ;

[0094] in, This indicates the Shuffle cost target.

[0095] The data locality objective is quantified by maximizing the key adjacency metric, which is used to place associated keys close together in physical storage to improve access efficiency; wherein the key adjacency metric is the ratio of the number of adjacent key pairs assigned to the same or adjacent partitions to the total number of adjacent key pairs in the key sequence of the high-frequency key subset arranged in a predetermined order.

[0096] Understandably, data locality emphasizes that relevant keys should be placed in adjacent partitions to improve cache hit rate. Assuming the keys are arranged in some order (e.g., lexicographical or frequency order), this invention uses a key adjacency metric:

[0097] ;

[0098] in, This represents the key adjacency metric, used to measure the degree of adjacency of keys. It reflects the proximity of keys in data locality; the higher the value, the better the adjacency of the keys, which is more conducive to improving cache hit rate. Key set The number of middle keys, and Represent the key set respectively The first number arranged in a certain order (such as lexicographical or frequency order) The key and the first One key, Indicates key Arrange the mapped results in a certain order. Indicates key The result after mapping.

[0099] Understandable The larger the value, the higher the proportion of adjacent keys being assigned to the same or adjacent partitions. Therefore, the data locality objective is:

[0100] ;

[0101] in, This represents the goal of data locality.

[0102] As can be seen, this invention clearly defines how to transform actual data into a well-defined multi-objective optimization problem containing three quantifiable sub-objectives, providing specific inputs and evaluation criteria for subsequent genetic algorithm solutions.

[0103] Step S20: Construct a comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective.

[0104] Based on the load balancing objective, the shuffle cost objective, and the data locality objective, a comprehensive fitness function is constructed using a weighted sum method:

[0105] ;

[0106] in, Represents the comprehensive fitness function, This indicates the load balancing fitness level. This represents the centrality coefficient of the dataset. This indicates the maximization of the key adjacency metric. , , These represent the configurable first weight coefficient, second weight coefficient, and third weight coefficient, respectively. Indicates the partitioning scheme.

[0107] In this embodiment, based on empirical analysis, the default weight configuration can be used: (Load balancing is the primary goal). (Shuffle cost is a secondary objective). (Locality is a secondary objective). This configuration reflects the fact that load balancing typically has the most significant impact on performance.

[0108] Understandably, this invention transforms the three abstract objectives proposed in multi-objective optimization problem modeling into a single evaluation criterion that can be directly computed by a genetic algorithm. Weighted summation is used as the basic form of multi-objective fusion. This is a classic and effective method in multi-objective optimization, transforming multiple conflicting objectives into a maximizable scalar value through linear combination, enabling the genetic algorithm to directly compare and rank different partitioning schemes accordingly.

[0109] Furthermore, a key feature of this invention is that the weighting coefficients are explicitly configurable. This endows the invention with great practical flexibility. Users can dynamically adjust the relative importance of different objectives based on specific application scenarios (such as network bandwidth sensitivity or computational cache sensitivity) and workload characteristics, thereby guiding the genetic algorithm to search for the partitioning scheme that best meets the current needs. The default configuration given above is an experience-based recommended configuration with load balancing as the primary objective. The weighted sum structure and the design of configurable weights are key technical manifestations of this method's ability to achieve adaptive optimization and avoid the limitations of a single objective.

[0110] Step S30: Randomly generate multiple partition schemes, treat each partition scheme as an individual in the genetic algorithm, form an initial population based on all individuals, and calculate the fitness value of each individual in the initial population based on the comprehensive fitness function.

[0111] In this embodiment, the process of randomly generating multiple partition schemes is as follows: For the high-frequency key subset determined by the sampling and selection mechanism, each key is independently and uniformly assigned an integer index within the range of the preset total number of partitions, thereby constructing an integer array (i.e., a partition scheme or genetic algorithm individual) representing the complete key-partition mapping relationship; this process is repeated until a preset number of such arrays are generated, and the set of these arrays constitutes the initial population of the genetic algorithm. This random initialization strategy aims to broadly cover the solution space, providing a rich and diverse starting point for subsequent evolution.

[0112] Furthermore, the step of treating each partitioning scheme as an individual in the genetic algorithm, constructing an initial population based on all individuals, and calculating the fitness value of each individual in the initial population based on the comprehensive fitness function specifically includes:

[0113] Each partitioning scheme is defined as an individual in the genetic algorithm, and each individual is represented by a chromosome encoding.

[0114] The chromosome encoding generation process is as follows: sampling to obtain a subset of high-frequency bonds selected from the bond frequency distribution, and randomly assigning a partition index with a value range of [0, N-1] to each bond in the high-frequency bond subset, where N represents the total number of partitions;

[0115] Based on a preset comprehensive fitness function, the fitness value corresponding to each individual in the initial population is calculated. The fitness value is used to comprehensively reflect the quantitative performance of the partitioning scheme represented by the individual in the multi-objective optimization problem.

[0116] Understandably, this embodiment first establishes a bridge between the problem domain and the algorithm domain: a complete partitioning scheme (i.e., the mapping rule from high-frequency keys to partitions) is explicitly defined as an individual in the genetic algorithm. This is the first step in applying genetic algorithms to solve any combinatorial optimization problem, ensuring that each solution searched by the algorithm directly corresponds to a partitioning strategy to be evaluated.

[0117] Furthermore, in this embodiment, direct integer encoding is employed. An individual is represented by an integer array (chromosome), the array length of which is equal to the number of high-frequency keys. The value of each array element (gene) is the partition index assigned to the corresponding key. The generation of the encoding is randomized. For each key in the high-frequency key subset, a partition index (ranging from [0, N-1], where N is the preset total number of partitions in the Spark job) is independently and uniformly randomly assigned. The purpose of this process is to explore the entire solution space as uniformly as possible at the beginning of the algorithm, avoiding getting trapped in local optima due to initial solution bias. Repeating this process to generate multiple individuals constitutes a highly diverse initial population.

[0118] Furthermore, the fitness value is calculated based on the comprehensive fitness function constructed in the independent claim. This function is a quantitative fusion of three objectives: load balancing, shuffle cost, and data locality. The calculated fitness value is a scalar that comprehensively and quantitatively reflects the overall performance of the partitioning scheme represented by the individual in solving the multi-objective optimization problem. The higher the fitness value, the better the partitioning scheme is under the user-configured weight system.

[0119] As can be seen, this invention clearly defines the initialization paradigm of genetic algorithms for the specific problem of Spark data partitioning optimization. It transforms the complex combinatorial optimization problem into a standard process that can be iteratively searched and evolved using genetic algorithms, laying the necessary foundation for subsequent evolutionary operations such as selection, crossover, and mutation.

[0120] Step S40: Starting from the initial population and the fitness value of each individual in the initial population, perform multiple iterations, and in the new population obtained in each iteration, evaluate the fitness of the individuals in the new population, and take the partitioning scheme represented by the individual with the highest fitness in the iteration process as the optimal partitioning scheme.

[0121] Specifically, the number of iterations is set and the iteration loop is executed. When the number of iterations is reached, the partitioning scheme represented by the recorded global optimal individual is output as the optimal partitioning scheme.

[0122] The process of each iteration is as follows: based on the fitness values ​​of individuals in the current population, a selection operation of the genetic algorithm is performed to select parent individuals for reproduction; a crossover operation is performed on the selected parent individuals, and a mutation operation is performed on the generated offspring individuals to generate a new set of candidate individuals; based on the comprehensive fitness function, the fitness value of each individual in the newly generated set of candidate individuals is calculated, and combined with a preset elite retention strategy, a new generation of population is formed; the individual with the highest fitness in the current iteration is compared and recorded, and the record of the globally optimal individual is updated.

[0123] Understandably, this embodiment defines a standard procedure for the algorithm to search for the optimal solution through multiple generations of iterative iterations, starting from the initial population. Specifically, it first establishes the termination condition for iterations (reaching a preset number of iterations) and the final output (the partitioning scheme represented by the globally optimal individual). This sets clear boundaries for the algorithm's operation, ensuring that it converges within predictable computational overhead and ultimately outputs the best solution found across all historical generations.

[0124] Furthermore, based on the fitness values ​​of individuals in the current population (i.e., the quantitative merits of the partitioning schemes they represent), selection operations are performed. A tournament selection strategy is typically used, aiming to select individuals with higher fitness as "parents," simulating natural selection and guiding the search direction. During the breeding and exploration phases, crossover and mutation operations are performed on the selected parent individuals.

[0125] The crossover operation is specifically implemented as a single-point crossover at the key level. As explained by the user, this operation randomly selects a crossover point and swaps the gene fragments (i.e., partial key partition indices) after the crossover point between two parent individuals, thereby generating a "offspring" that incorporates parental characteristics. Its core design objective is to preserve potentially excellent "building blocks" (continuous and logical key-partition mapping fragments) that may have already formed in the parent generation, while simultaneously exploring new combinations of solutions through gene recombination. If the key space is ordered, this operation also helps maintain the adjacent key structure upon which data locality depends.

[0126] Mutation is a gene-level mutation. It randomly alters one or more genes (partition indices of the genes) in offspring individuals with a certain probability. This is a key mechanism for introducing randomness and maintaining population diversity, preventing the algorithm from prematurely converging to a local optimum due to excessive selection pressure, and ensuring global search capability.

[0127] Furthermore, the newly generated set of candidate individuals (offspring) is immediately evaluated using a comprehensive fitness function. Subsequently, combined with an elite retention strategy (directly retaining the individuals with the highest fitness from the previous generation), they are merged to form a new generation of population. Elite retention ensures the monotonically convergent nature of the algorithm and prevents the discovery of optimal solutions from being lost during evolution.

[0128] Furthermore, at the end of each generation, the best individual in the current iteration is compared and recorded, and the record of the globally best individual is updated accordingly. This ensures that the algorithm always remembers the best solution it has discovered throughout the entire run.

[0129] As can be seen, this invention precisely adapts and applies the classic genetic algorithm framework to the specific optimization problem of Spark data partitioning. It defines a complete and operable algorithm process that starts from the initial solution and gradually approaches and outputs the globally optimal (or near-optimal) partitioning scheme through a cyclical process of "selection-crossover-mutation-evaluation-elite retention".

[0130] Step S50: Construct an optimized partition mapping for high-frequency keys based on the optimal partitioning scheme, and configure a hash rollback mechanism for low-frequency keys.

[0131] Specifically, this embodiment engineers the optimal partitioning scheme output by the genetic algorithm into an executable runtime strategy: for high-frequency keys, a hash mapping (optimized partitioning mapping) is constructed from the key to the partition index according to the predetermined allocation in the scheme, thereby achieving accurate and fast lookup with O(1) complexity (the most ideal type of algorithm time complexity, also called constant time complexity). This mapping is directly constructed on the Driver side based on the optimal partitioning scheme, and its internal implementation usually uses an efficient hash table (such as Java HashMap) to ensure constant time query performance. After construction, the mapping table is distributed to all executors in the cluster through Spark's serialization and broadcasting mechanism, so that each computing node can hold the complete optimized partitioning strategy locally, avoiding the network overhead caused by remote queries at runtime. This design cleverly transforms the high-quality static scheme obtained by global optimization in the offline stage into a dynamic routing table that can be accessed immediately during the execution of distributed tasks. For high-frequency keys (i.e. hot spots in data skew), the executor does not need to perform any complex calculations. It only needs a local memory lookup to find the best partition after global optimization. This fundamentally solves the problem of hot data accumulation caused by blind mapping in hash partitioners, which is a key operational guarantee for achieving the load balancing goal.

[0132] For low-frequency keys, a standard hash modulo function is configured as a fallback mechanism. This mechanism is a lightweight deterministic computation function. For any key not included in the optimized partitioning map (i.e., low-frequency long-tail keys), the executor will automatically trigger this fallback path. By calculating its hash value and taking the modulo of the total number of partitions, a deterministic partition index can be quickly obtained. This mechanism has a dual advantage: firstly, it ensures the completeness of the method, meaning that all input data can be assigned to a deterministic partition without omission, avoiding boundary cases where data has nowhere to go due to limited optimization scope; secondly, it maintains computational efficiency, with extremely low time complexity and computational overhead. By combining the optimized mapping with the fallback function, this method ensures intelligent and optimized allocation of hot data while ensuring that all data (including long-tail keys not involved in optimization) can be deterministically processed, thus achieving multi-objective optimization benefits while maintaining comparable running efficiency and robustness to the native Spark partitioner.

[0133] Step S60: Obtain the data key input during the execution of the Spark task. If the data key is a high-frequency key, determine the target partition of the high-frequency key by querying the optimized partition mapping. If the data key is a low-frequency key, execute the hash rollback mechanism to determine the target partition of the low-frequency key.

[0134] Specifically, on each executor of a Spark task, a data record to be processed is received, and a data key is extracted from the data record;

[0135] Determine whether the data key exists in the key set of the optimized partition mapping. If it exists, the data key is a high-frequency key. Find the partition index corresponding to the high-frequency key from the optimized partition mapping and use the partition index as the target partition of the high-frequency key.

[0136] If it does not exist, the data key is a low-frequency key. The low-frequency key is input into a preset hash function to calculate the hash value, and the hash value is moduloed with the total number of partitions. The result of the operation is used as the target partition of the low-frequency key.

[0137] It is understood that this embodiment implements an intelligent and differentiated data routing mechanism to solve the performance bottleneck caused by the inability of a single processing strategy (such as pure hashing or pure range partitioning) to simultaneously handle hot data and long-tail data in the prior art.

[0138] Specifically, each Spark executor first receives the raw data records and extracts their data keys. Then, the system performs an efficient key existence query to instantly classify the data stream into two categories: high-frequency keys that exist in a pre-built optimized partition map key set, and low-frequency keys that do not exist. For high-frequency keys, the process directly retrieves their target partition index from the map using an O(1) complexity hash table lookup, calculated globally by a genetic algorithm. This path ensures that the allocation of a few key hotspots in the dataset (typically following the Pareto principle, i.e., 20% of the keys carry 80% of the data) is optimal, directly achieving multiple goals such as load balancing, reducing shuffles, and maintaining locality.

[0139] For low-frequency keys, a hash rollback mechanism is triggered: the key is input into a pre-defined, efficient hash function (such as the Spark-optimized version of java.lang.Object.hashCode()) for calculation, and the resulting hash value is moduloed by the total number of partitions to determine its target partition. This rollback path, though simple, is significant: it ensures the algorithm's completeness in processing all data, leaving no data untouched; moreover, its computational overhead is comparable to Spark's native HashPartitioner (the most core and commonly used partitioner in Spark), ensuring the overall processing efficiency and preventing performance degradation.

[0140] As can be seen, this invention provides a hybrid partitioning strategy deployed at the forefront of distributed computing. It successfully combines the global optimization intelligence for high-frequency keys obtained through genetic algorithms in the offline phase with the efficient and robust processing capabilities for all data at runtime. This design allows this invention to enjoy the significant performance improvements brought by complex optimization algorithms (especially in mitigating data skew) while maintaining seamless compatibility with the existing Spark ecosystem and stable and efficient execution characteristics.

[0141] The present invention has the following beneficial effects:

[0142] (1) First systematic application of genetic algorithms to Spark data partitioning optimization: Unlike existing research on using genetic algorithms for task scheduling, this invention focuses on data partitioning, a more fundamental performance influencing factor. Data partitioning optimization can reduce data skew and shuffle overhead from the source, providing a better foundation for upper-level task scheduling. This is the first time that a customized design and systematic study of genetic algorithms has been conducted for the Spark data partitioning problem.

[0143] (2) Constructing a multi-objective optimization framework for data partitioning: This invention innovatively models the data partitioning problem as a multi-objective optimization problem, simultaneously considering three key but conflicting optimization objectives: load balancing, shuffle cost, and data locality. A flexible trade-off is achieved through a weighted fitness function, allowing users to adjust the relative importance of each objective according to actual application scenarios (e.g., increasing the shuffle weight in network-constrained environments, and increasing the locality weight in cache-sensitive scenarios), thus avoiding the limitations of single-objective optimization.

[0144] (3) Customized design of genetic algorithm for data partitioning problem: Direct integer encoding: Makes chromosomes correspond one-to-one with partitioning schemes, ensuring the legality of all candidate solutions and avoiding complex decoding and constraint processing; Key-level single-point crossover: Preserves the building blocks (continuous gene segments) of the parent generation, maintains the adjacency relationship of adjacent keys, and improves the efficiency of the algorithm; Gene-level adaptive mutation: Preserves population diversity while avoiding excessive destruction, and the mutation rate of 0.1 has been verified by a large number of experiments; Elite retention strategy: Ensures the monotonic convergence of the algorithm and accelerates the convergence to a high-quality solution; These designs enable the genetic algorithm to efficiently explore the solution space of data partitioning problem and converge to a high-quality solution within 50 generations of iteration.

[0145] (4) Scalable optimization strategy based on sampling: The key space size is effectively controlled by distributed sampling (1% sampling rate) and Top-K selection (max_keys=10000), avoiding the memory bottleneck on the Driver side.

[0146] (5) O(1) Partition lookup and hash rollback mechanism: A hash table is used to store the mapping relationship between keys and partitions, providing constant time complexity partition lookup. For low-frequency keys that are not in the Top-K, automatic rollback to the hash partition is performed, ensuring robustness while maintaining performance. This design makes the optimization overhead completely offline, and the runtime performance is comparable to the standard HashPartitioner.

[0147] (6) Seamless integration with the Spark framework: Implements the standard Partitioner interface without modifying the Spark kernel or application code. Provides a concise factory method API, allowing users to use the optimized partitioner with just one line of code. Supports serialization and broadcasting for efficient transmission in distributed environments. These engineering designs ensure ease of use and production suitability of the methods.

[0148] (7) Global search capability avoids local optima: Unlike greedy methods such as load-aware partitioning, genetic algorithms can escape the trap of local optima and search for high-quality partitioning schemes globally through mechanisms such as population evolution and crossover mutation. Experiments show that this method still has a 10%-15% performance improvement compared to the greedy load-aware method, fully demonstrating the advantages of global optimization.

[0149] Furthermore, this invention verifies the validity of the Spark multi-objective data partitioning optimization method based on genetic algorithms through a systematic evaluation of 750 independent experiments (5 real datasets × 5 workloads × 6 partitioning algorithms × 5 repetitions). This is mainly reflected in the following six aspects:

[0150] Firstly, overall performance has been significantly improved:

[0151] Table 1 below shows the average execution time (in milliseconds) of six partitioning algorithms across five datasets and five workloads. Each value is the average of five iterations. As can be seen from the table, GA-Optimized achieves optimal or near-optimal performance in the vast majority of scenarios.

[0152] Specifically, GA-Optimized (an optimized genetic algorithm) shows a particularly significant advantage on highly skewed data. On highly skewed datasets such as Wikimedia and Amazon, GA-Optimized achieves a 28%-35% performance improvement over Hash Partitioner and approximately a 10%-15% improvement over Load-Aware. Different workloads exhibit different performance characteristics, with grouping and aggregation workloads benefiting the most because these operations are most sensitive to load balancing. Join operations show the second largest improvement, which is related to their requirement for bidirectional shuffle. Notably, Range Partitioner performs relatively well in sorting operations because its inherent ability to generate ordered partitions matches the requirements of sorting operations. On other low-skewed datasets, the performance differences between algorithms are relatively small (5%-10%), but GA-Optimized still maintains a slight advantage or performs on par.

[0153] Table 1: Comparison of average execution time for algorithms in each partition

[0154]

[0155] Secondly, load balancing has been significantly improved:

[0156] Table 2 below shows a comparison of partition size variance for different partitioning algorithms. Lower variance indicates a more balanced load. Experimental results show that GA-Optimized Partitioner achieved the lowest or near-lowest partition variance in all test scenarios. Table 2 quantifies the load balancing performance of each algorithm, using the coefficient of variation (CV) as a normalized load balancing metric.

[0157] Table 2 shows that GA-Optimized significantly improves load balancing performance. Compared to HashPartitioner, GA-Optimized reduces load imbalance by 44%-71%, achieving a 71% improvement on the highly skewed Amazon dataset. This improvement exhibits a clear trend: the higher the data skew, the more pronounced the advantage of GA-Optimized. For example, on the Amazon dataset, the CV of hash partitioning is as high as 4.28, while the optimized genetic algorithm is only 1.24, representing a qualitative leap. More importantly, even compared to load-aware methods specifically optimized for load balancing, the optimized genetic algorithm still has 30%-40% room for improvement. This fully demonstrates that the global search capability of the optimized genetic algorithm is superior to greedy heuristic strategies, enabling it to find better solutions in complex optimization spaces.

[0158] Table 2: Comparison of Zone Load Balancing

[0159]

[0160] like Figure 2 As shown, the load balancing performance comparison of the six algorithms on five datasets is further visualized. From Figure 2 A significant trend can be observed: GA-Optimized achieves the lowest CV values ​​on all five datasets, demonstrating consistent superiority. On the low-skew Netflix dataset, the CV values ​​of each algorithm are generally low (between 0.48 and 0.87), with relatively small differences, because the data itself is relatively evenly distributed, limiting the optimization space for partitioning strategies. However, on the high-skew dataset, the differences widen significantly: on the Wikimedia dataset, the CV value of hash partitioning reaches 2.84, while GA-Optimized is only 0.92, a reduction of 67.6%; on the Amazon dataset, this gap is even larger, with the CV value of hash partitioning reaching as high as 4.28, meaning that the standard deviation of the partition size is 4.28 times the mean, indicating severe load imbalance, which GA-Optimized reduces to 1.24, achieving a qualitative leap. It is worth noting that Load-Aware, as a heuristic method specifically designed for load balancing, significantly outperforms traditional hash partitioning, range partitioning, and consistent hashing methods on all datasets, but still cannot reach the level of GA-Optimized. This verifies the core assumption of this invention: the data partitioning problem is a complex multi-objective optimization problem. Greedy algorithms are prone to getting trapped in local optima, while the optimized genetic algorithm can explore a wider solution space and find a better partitioning scheme through population evolution and global search.

[0161] like Figure 3 As shown, the improvement of GA-Optimized over each baseline method in terms of load variance reduction is demonstrated. Figure 3The horizontal bar chart visually illustrates the optimization effect of GA-Optimized in load balancing. Compared to hash partitioning, GA-Optimized achieves a 71.0% reduction in load variance. This means that on the extremely skewed Amazon dataset, the coefficient of variation, which was originally as high as 4.28, is reduced to 1.24, resulting in a fundamental improvement in load distribution. Compared to range partitioning and consistent hashing, the improvement rates are 60.8% and 67.7%, respectively, indicating that GA-Optimized still has a significant advantage even compared to traditional methods with some load balancing capabilities. More importantly, even compared to load-aware and sampling range methods specifically optimized for load balancing, GA-Optimized still achieves improvement rates of 42.1% and 51.9%, respectively. This result is particularly important because it demonstrates that the optimized genetic algorithm not only performs better than naive methods but also surpasses existing heuristic optimization methods. While load-aware methods consider key frequency through a greedy strategy, their sequential decision-making nature limits the optimization space. In contrast, GA-Optimized explores multiple candidate solutions in parallel, introduces random perturbations through crossover and mutation operations, and comprehensively evaluates multiple objectives through a fitness function. This allows it to escape the trap of local optima and find a better key-to-partition mapping relationship at the global level.

[0162] like Figure 4 The figure shows the actual load distribution across 16 partitions (in relative units), with the dashed line representing the ideal uniform load level (approximately 5.5 units). It is clear from the figure that the load distribution of the hash partitions is extremely uneven. Some partitions (such as partitions 0, 2, 6, 10, and 14) have loads as high as 10-13 units, more than twice the ideal value, while some partitions (such as partitions 1, 3, 5, 7, 9, 11, 13, and 15) have loads of only 1-3 units, causing serious resource waste and task tailing problems. Range partitioning is improved through sampling and range division, but high loads still exist on some partitions (such as partitions 0, 2, 6, and 12). Consistent hashing performs similarly to hash partitioning; although the virtual node mechanism introduces some randomness, its ability to process hot data is limited. Load awareness significantly improves the load distribution, with most partitions having loads between 5-7 units, but a few partitions (such as partitions 0, 6, and 12) still slightly exceed the ideal value. Sampling range is further optimized through weighted sampling, but the effect is still not as good as GA-Optimized. GA-Optimized offers the most even load distribution, with almost all partitions experiencing load fluctuations within a narrow range of 4-6 units, closely resembling an ideal uniform distribution. This balanced load distribution directly translates to shorter task execution times and higher resource utilization—when the loads of all tasks are similar, the job completion time is determined by the average task time, rather than being dragged down by the slowest tailing task.

[0163] Understandably, improved load balancing directly translates to shorter job completion times. When all partitions are at near-average load, job completion times are determined by average task times, rather than being dragged down by the slowest tail tasks, thus improving overall execution efficiency and resource utilization.

[0164] Thirdly, the problem of task tailing has been significantly alleviated:

[0165] Task tailing refers to a situation where a few tasks take significantly longer to execute than others, thus slowing down the overall job completion time. This is one of the most direct manifestations of data skew. This code uses the ratio of the P95 task time (95th percentile) to the median task time to measure the degree of tailing; the closer this ratio is to 1, the more balanced the task execution times.

[0166] Table 3 below shows a comparison of task tailing levels. GA-Optimized reduced task tailing by 60%-65%, significantly alleviating performance bottlenecks caused by data skew. This means that after using GA-Optimized, the execution time of the slowest task is closer to that of the median task, and the overall load distribution is more even. Taking the Amazon dataset as an example, the hash partition ratio is as high as 6.28, meaning that the execution time of the slowest 5% of tasks is 6.28 times that of the median task. GA-Optimized reduces this ratio to 2.18, significantly mitigating the impact of task tailing on overall performance. This improvement is of great value in real-world production environments because job completion time is often determined by the slowest task, and reducing task tailing directly improves the user experience.

[0167] Understandably, mitigating task tailing means that the execution time of the slowest task is closer to that of the median task, preventing a few tailing tasks from becoming the performance bottleneck of the entire job. This is of great value in production environments because job completion time is often determined by the slowest task, and reducing task tailing can directly improve user experience and system throughput.

[0168] Table 3: Comparison of Task Tail-up Degree

[0169]

[0170] Fourthly, Shuffle costs are significantly reduced:

[0171] Shuffle is one of the most expensive operations in Spark, directly impacting network bandwidth consumption and disk I / O overhead. Table 4 below shows the average Shuffle cost (in GB) for different partitioning algorithms: GA-OptimizedPartitioner effectively reduces cross-node data transfer by optimizing the key-to-partition mapping. On average, Shuffle cost is reduced by 24%-28%, which directly translates into savings in network bandwidth and reduced disk I / O. For applications running in network-constrained environments or processing large-scale data, this reduction in Shuffle cost has significant practical implications, not only improving the performance of individual jobs but also enhancing the resource utilization efficiency of the entire cluster.

[0172] Understandably, the reduction in shuffle costs directly translates to savings in network bandwidth and reduced disk I / O. For applications running in network-constrained environments or processing large-scale data, this reduction in shuffle costs has significant economic and performance implications, not only improving the performance of individual jobs but also enhancing the resource utilization efficiency of the entire cluster.

[0173] Table 4: Shuffle Cost Comparison

[0174]

[0175] Fifthly, the algorithm overhead is controllable:

[0176] Preprocessing overhead includes: sampling phase, GA optimization phase, and total preprocessing overhead.

[0177] The preprocessing process consists of several stages: Sampling phase: For datasets with millions to hundreds of millions of records, 1% sampling typically takes 5-10 seconds. GA optimization phase: With a population size of 100 and 50 generations, optimization typically takes 10-30 seconds. Total preprocessing overhead: Approximately 30-40 seconds.

[0178] Runtime overhead includes partition lookup and memory overhead.

[0179] Partition lookup: O(1) constant time, comparable to Hash Partitioner. Memory overhead: The mapping table for the Top-10000 keys occupies approximately 40KB, which is negligible.

[0180] For processing large-scale datasets (GB to TB level), the actual execution time of a job typically ranges from several minutes to several hours. A one-time preprocessing overhead of around 30 seconds represents a 28%-35% improvement in job execution time (saving several minutes to tens of minutes), resulting in an extremely high return on investment.

[0181] Sixthly, stability under different scenarios:

[0182] Stability improvements were achieved at different levels of data skew: 5%-12% at low skew, 18%-25% at medium skew, and 28%-35% at high skew. A clear positive correlation was also observed: the higher the skew, the more pronounced the advantage.

[0183] Stability improvements were achieved to varying degrees across different workload types: Aggregation: 30%-35% improvement (most beneficial), Multi-metric aggregation: 28%-32% improvement, Connections: 22%-28% improvement, Word frequency statistics: 25%-30% improvement, Ranking: 15%-20% improvement.

[0184] The method achieves varying degrees of stability improvement depending on the number of partitions: 16 partitions: 26%-30% improvement; 32 partitions: 28%-35% improvement (optimal); 64 partitions: 24%-28% improvement (increased task scheduling overhead). The method is low in sensitivity to the number of partitions parameter and highly adaptable.

[0185] Furthermore, Figure 5 This illustrates the performance of different partitioning methods in a three-dimensional target space. The X-axis in the figure represents the data locality score. The Y-axis represents the Shuffle cost score. The Z-axis represents the load balancing score. The ideal solution is located at coordinate point (1,1,1), indicating that all three objectives are simultaneously optimized. Figure 5 The paper presents the performance distribution of six partitioning methods: hash partitioner, range partitioner, consistent hash partitioner, load-aware partitioner, sampled range partitioner, and the proposed GA-Optimized method (weight configuration). =0.5, =0.3, =0.2). The figure clearly shows that traditional methods mostly fall into the lower performance range, with unsatisfactory scores across all three dimensions. The GA-Optimized method in this paper is located closest to the ideal point, achieving the best trade-off among the three objectives. The Pareto front (a core concept in multi-objective optimization problems) is formed by several specifically configured GA methods (Genetic Algorithms). These solutions represent the optimal trade-offs under different weight configurations; improvement in any objective inevitably leads to deterioration in other objectives. This figure effectively illustrates the multi-objective nature of the data partitioning optimization problem and the advantages of genetic algorithms in finding Pareto optimal solutions.

[0186] like Figure 6 As shown, Figure 6The convergence characteristics and population diversity evolution of the genetic algorithm during 50 generations of iteration are demonstrated. Figure 6 Figure (a) shows the trends of the three fitness curves with the number of iterations. The top solid line represents the fitness of the best individual in the population, exhibiting a typical monotonically increasing characteristic, rising rapidly from an initial value of about 0.3, reaching above 0.8 around generation 20, and finally converging to around 0.95 at generation 50, close to the theoretical convergence threshold. The middle dashed line represents the average fitness of the population, steadily increasing from an initial value of about 0.25 to 0.70, reflecting the continuous improvement in the overall quality of the population. The bottom dotted line represents the fitness of the worst individual in the population, gradually increasing from an initial value of about 0.15 to 0.40, indicating that elite retention and selection pressure effectively eliminated low-quality solutions. The gradually decreasing spacing between the three curves indicates that the population is gradually concentrating in the high-fitness region. Figure 6 Figure (b) illustrates the evolution of population diversity (measured by genetic difference) over iterations. The shaded area below represents the range of population diversity variation. Initially, the population diversity is extremely high (approximately 110 different genotypes), due to the wide distribution of individuals resulting from random initialization. As iterations progress, selection pressure and convergence effects cause diversity to decline rapidly, with the most significant decrease occurring in the first 10 generations. However, thanks to the continuous effect of mutation operations, diversity does not completely disappear, stabilizing at around 20 in later stages. This moderate maintenance of diversity is crucial to preventing premature convergence to local optima. Both subgraphs together demonstrate that the genetic algorithm achieves a good balance between rapid convergence and maintaining exploratory capabilities, validating the rationality of the algorithm design.

[0187] Furthermore, such as Figure 7 As shown, based on the above-mentioned Spark multi-objective data partitioning optimization method based on genetic algorithms, this invention also provides a Spark multi-objective data partitioning optimization system based on genetic algorithms, wherein the Spark multi-objective data partitioning optimization system based on genetic algorithms includes:

[0188] The optimization problem modeling module 51 is used to perform distributed sampling on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective.

[0189] The fitness function construction module 52 is used to construct a comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective.

[0190] The individual fitness value calculation module 53 is used to randomly generate multiple partition schemes, treat each partition scheme as an individual in the genetic algorithm, form an initial population based on all individuals, and calculate the fitness value of each individual in the initial population based on the comprehensive fitness function.

[0191] The optimal partitioning scheme acquisition module 54 is used to perform multiple iterations starting from the initial population and the fitness value of each individual in the initial population, and to evaluate the fitness of individuals in the new population obtained in each iteration, and to take the partitioning scheme represented by the individual with the highest fitness during the iteration as the optimal partitioning scheme.

[0192] The partition scheme conversion module 55 is used to construct an optimized partition mapping for high-frequency keys based on the optimal partition scheme, and to configure a hash rollback mechanism for low-frequency keys.

[0193] The data partitioning execution module 56 is used to obtain the data key input during the execution of the Spark task. If the data key is a high-frequency key, the target partition of the high-frequency key is determined by querying the optimized partition mapping. If the data key is a low-frequency key, the hash backoff mechanism is executed to determine the target partition of the low-frequency key.

[0194] Furthermore, such as Figure 8 As shown, based on the above-mentioned Spark multi-objective data partitioning optimization method and system based on genetic algorithm, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 8 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0195] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a Spark multi-objective data partitioning optimization program 40 based on a genetic algorithm, which can be executed by the processor 10 to implement the Spark multi-objective data partitioning optimization method based on a genetic algorithm in this application.

[0196] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the Spark multi-objective data partitioning optimization method based on genetic algorithms.

[0197] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0198] In one embodiment, when processor 10 executes Spark multi-objective data partitioning optimization program 40 based on genetic algorithm in memory 20, the following steps are performed:

[0199] Distributed sampling is performed on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective.

[0200] A comprehensive fitness function is constructed based on the load balancing objective, the shuffle cost objective, and the data locality objective.

[0201] Multiple partitioning schemes are randomly generated, and each partitioning scheme is used as an individual in the genetic algorithm. An initial population is formed based on all individuals, and the fitness value of each individual in the initial population is calculated based on the comprehensive fitness function.

[0202] Starting from the initial population and the fitness value of each individual in the initial population, multiple iterations are performed. In the new population obtained in each iteration, the fitness of the individuals in the new population is evaluated, and the partitioning scheme represented by the individual with the highest fitness during the iteration process is taken as the optimal partitioning scheme.

[0203] Based on the optimal partitioning scheme, construct an optimized partitioning map for high-frequency keys and configure a hash rollback mechanism for low-frequency keys;

[0204] Obtain the data key input during Spark task execution. If the data key is a high-frequency key, determine the target partition for the high-frequency key by querying the optimized partition mapping. If the data key is a low-frequency key, execute the hash backoff mechanism to determine the target partition for the low-frequency key.

[0205] Specifically, the Spark dataset to be processed undergoes distributed sampling to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which specifically includes:

[0206] The Spark dataset to be processed is randomly sampled at a pre-configured sampling rate, and the frequency of each key is counted through distributed aggregation operations to obtain the key frequency distribution.

[0207] If the number of unique keys in the key frequency distribution exceeds a preset threshold, then the top preset number of keys with the highest frequency are selected as the high-frequency key subset to be optimized.

[0208] Based on the key frequency distribution and the high-frequency key subset, the load balancing objective, the shuffle cost objective, and the data locality objective are quantified into mathematical expressions to form the multi-objective optimization problem.

[0209] The load balancing objective is achieved by minimizing the normalized standard deviation of the partition load and converting the normalized standard deviation into a load balancing fitness suitable for the genetic algorithm, thereby quantifying the load balancing of each partition.

[0210] Wherein, the partition load is the sum of the frequencies of all keys selected from the high-frequency key subset within the partition, and the normalized standard deviation is the normalized value of the deviation between the actual load and the ideal uniform load of each partition based on the high-frequency key subset.

[0211] The Shuffle cost target is quantified by maximizing the data centrality coefficient, which is used to reduce the overhead of cross-partition data transfer in the Shuffle operation;

[0212] The data concentration coefficient is calculated based on the frequency distribution of the high-frequency key subset and is used to measure whether the data of the high-frequency key subset is concentrated in a few partitions. The larger the data concentration coefficient, the smaller the expected amount of data to be transferred across partitions.

[0213] The data locality objective is quantified by maximizing the key adjacency metric, which is used to place related keys close together in physical storage to improve access efficiency.

[0214] The key adjacency metric is the ratio of the number of adjacent key pairs assigned to the same or adjacent partitions to the total number of adjacent key pairs in the key sequence of the high-frequency key subset arranged in a predetermined order.

[0215] The construction of the comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective specifically includes:

[0216] Based on the load balancing objective, the shuffle cost objective, and the data locality objective, a comprehensive fitness function is constructed using a weighted sum method:

[0217] ;

[0218] in, Represents the comprehensive fitness function, This indicates the load balancing fitness level. This represents the centrality coefficient of the dataset. This indicates the maximization of the key adjacency metric. , , These represent the configurable first weight coefficient, second weight coefficient, and third weight coefficient, respectively. Indicates the partitioning scheme.

[0219] Specifically, the step of treating each partitioning scheme as an individual in the genetic algorithm, constructing an initial population based on all individuals, and calculating the fitness value of each individual in the initial population based on the comprehensive fitness function includes:

[0220] Each partitioning scheme is defined as an individual in the genetic algorithm, and each individual is represented by a chromosome encoding.

[0221] The chromosome encoding generation process is as follows: sampling to obtain a subset of high-frequency bonds selected from the bond frequency distribution, and randomly assigning a partition index with a value range of [0, N-1] to each bond in the high-frequency bond subset, where N represents the total number of partitions;

[0222] Based on a preset comprehensive fitness function, the fitness value corresponding to each individual in the initial population is calculated. The fitness value is used to comprehensively reflect the quantitative performance of the partitioning scheme represented by the individual in the multi-objective optimization problem.

[0223] The process of iterating multiple times starting from the initial population and the fitness value of each individual in the initial population, and evaluating the fitness of individuals in the new population obtained in each iteration, and taking the partitioning scheme represented by the individual with the highest fitness during the iteration as the optimal partitioning scheme, specifically includes:

[0224] Set the number of iterations and execute the iteration loop. When the number of iterations is reached, output the partitioning scheme represented by the recorded global best individual as the optimal partitioning scheme.

[0225] The process of each iteration is as follows: based on the fitness value of individuals in the current population, the selection operation of the genetic algorithm is performed to select parent individuals for reproduction;

[0226] Perform crossover on the selected parent individuals and mutation on the generated offspring individuals to produce a new set of candidate individuals;

[0227] Based on the comprehensive fitness function, the fitness value of each individual in the newly generated candidate individual set is calculated, and combined with the preset elite retention strategy, a new generation of population is formed.

[0228] Compare and record the individual with the highest fitness in the current iteration, and update the record of the globally optimal individual.

[0229] Specifically, the process of obtaining the data key input during Spark task execution includes: if the data key is a high-frequency key, the target partition for the high-frequency key is determined by querying the optimized partition mapping; if the data key is a low-frequency key, the hash backoff mechanism is executed to determine the target partition for the low-frequency key.

[0230] On each executor of a Spark task, a data record to be processed is received, and a data key is extracted from the data record.

[0231] Determine whether the data key exists in the key set of the optimized partition mapping. If it exists, the data key is a high-frequency key. Find the partition index corresponding to the high-frequency key from the optimized partition mapping and use the partition index as the target partition of the high-frequency key.

[0232] If it does not exist, the data key is a low-frequency key. The low-frequency key is input into a preset hash function to calculate the hash value, and the hash value is moduloed with the total number of partitions. The result of the operation is used as the target partition of the low-frequency key.

[0233] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a Spark multi-objective data partitioning optimization program based on a genetic algorithm, and the Spark multi-objective data partitioning optimization program based on a genetic algorithm, when executed by a processor, implements the steps of the Spark multi-objective data partitioning optimization method based on a genetic algorithm as described above.

[0234] In summary, this invention provides a Spark multi-objective data partitioning optimization method, system, terminal, and storage medium based on a genetic algorithm. The method includes: modeling data partitioning as a multi-objective optimization problem involving load balancing, shuffle cost, and data locality; constructing a comprehensive fitness function based on the multi-objective optimization problem; using a genetic algorithm to randomly generate a partitioning scheme as the initial population; iteratively searching through selection, crossover, mutation, and elite retention operations to obtain the optimal partitioning scheme; then establishing an optimized partitioning mapping for high-frequency keys and configuring a hash backoff function for low-frequency keys based on this scheme; finally, querying the mapping or calculating the backoff function according to the data key type during Spark task execution to complete the data partitioning. This invention effectively alleviates task tailing caused by data skew, significantly improves load balancing among nodes, reduces network transmission overhead during the shuffle stage, and enhances the storage and access locality of related data, achieving comprehensive performance improvement on highly skewed datasets.

[0235] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0236] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0237] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A Spark multi-objective data partitioning optimization method based on genetic algorithm, characterized in that, The Spark multi-objective data partitioning optimization method based on genetic algorithms includes: Distributed sampling is performed on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective. A comprehensive fitness function is constructed based on the load balancing objective, the shuffle cost objective, and the data locality objective. Multiple partitioning schemes are randomly generated, and each partitioning scheme is used as an individual in the genetic algorithm. An initial population is formed based on all individuals, and the fitness value of each individual in the initial population is calculated based on the comprehensive fitness function. Starting from the initial population and the fitness value of each individual in the initial population, multiple iterations are performed. In the new population obtained in each iteration, the fitness of the individuals in the new population is evaluated, and the partitioning scheme represented by the individual with the highest fitness during the iteration process is taken as the optimal partitioning scheme. Build an optimized partition mapping for high-frequency keys based on the optimal partitioning scheme, and configure a hash rollback mechanism for low-frequency keys; Obtain the data key input during Spark task execution. If the data key is a high-frequency key, determine the target partition of the high-frequency key by querying the optimized partition mapping. If the data key is a low-frequency key, execute the hash backoff mechanism to determine the target partition of the low-frequency key. The Spark dataset to be processed undergoes distributed sampling to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, specifically including: The Spark dataset to be processed is randomly sampled at a pre-configured sampling rate, and the frequency of each key is counted through distributed aggregation operations to obtain the key frequency distribution. If the number of unique keys in the key frequency distribution exceeds a preset threshold, then the top preset number of keys with the highest frequency are selected as the high-frequency key subset to be optimized. Based on the key frequency distribution and the high-frequency key subset, the load balancing objective, the shuffle cost objective, and the data locality objective are quantified into mathematical expressions to form the multi-objective optimization problem. The process of obtaining the data key input during Spark task execution involves several steps. If the data key is a high-frequency key, the target partition for the high-frequency key is determined by querying the optimized partition mapping. If the data key is a low-frequency key, the hash backoff mechanism is executed to determine the target partition for the low-frequency key. Specifically, this includes: On each executor of a Spark task, a data record to be processed is received, and a data key is extracted from the data record. Determine whether the data key exists in the key set of the optimized partition mapping. If it exists, the data key is a high-frequency key. Find the partition index corresponding to the high-frequency key from the optimized partition mapping and use the partition index as the target partition of the high-frequency key. If it does not exist, the data key is a low-frequency key. The low-frequency key is input into a preset hash function to calculate the hash value, and the hash value is moduloed with the total number of partitions. The result of the operation is used as the target partition of the low-frequency key.

2. The Spark multi-objective data partitioning optimization method based on genetic algorithm according to claim 1, characterized in that, The load balancing objective is to minimize the normalized standard deviation of the partition load and convert the normalized standard deviation into a load balancing fitness that fits the genetic algorithm to achieve quantification, thereby balancing the computational load of each partition. Wherein, the partition load is the sum of the frequencies of all keys selected from the high-frequency key subset within the partition, and the normalized standard deviation is the normalized value of the deviation between the actual load and the ideal uniform load of each partition based on the high-frequency key subset. The Shuffle cost target is quantified by maximizing the data centrality coefficient, which is used to reduce the overhead of cross-partition data transfer in the Shuffle operation; The data concentration coefficient is calculated based on the frequency distribution of the high-frequency key subset and is used to measure whether the data of the high-frequency key subset is concentrated in a few partitions. The larger the data concentration coefficient, the smaller the expected amount of data to be transferred across partitions. The data locality objective is quantified by maximizing the key adjacency metric, which is used to place related keys close together in physical storage to improve access efficiency. The key adjacency metric is the ratio of the number of adjacent key pairs assigned to the same or adjacent partitions to the total number of adjacent key pairs in the key sequence of the high-frequency key subset arranged in a predetermined order.

3. The Spark multi-objective data partitioning optimization method based on genetic algorithm according to claim 2, characterized in that, The construction of a comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective specifically includes: Based on the load balancing objective, the shuffle cost objective, and the data locality objective, a comprehensive fitness function is constructed using a weighted sum method: ； in, Represents the comprehensive fitness function, This indicates the load balancing fitness level. This represents the centrality coefficient of the dataset. This indicates the maximization of the key adjacency metric. , , These represent the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient, respectively. Indicates the partitioning scheme.

4. The Spark multi-objective data partitioning optimization method based on genetic algorithm according to claim 1, characterized in that, The step of treating each partitioning scheme as an individual in the genetic algorithm, constructing an initial population based on all individuals, and calculating the fitness value of each individual in the initial population based on the comprehensive fitness function specifically includes: Each partitioning scheme is defined as an individual in the genetic algorithm, and each individual is represented by a chromosome encoding. The chromosome encoding generation process is as follows: sampling to obtain a subset of high-frequency bonds selected from the bond frequency distribution, and randomly assigning a partition index with a value range of [0, N-1] to each bond in the high-frequency bond subset, where N represents the total number of partitions; Based on a preset comprehensive fitness function, the fitness value corresponding to each individual in the initial population is calculated. The fitness value is used to comprehensively reflect the quantitative performance of the partitioning scheme represented by the individual in the multi-objective optimization problem.

5. The Spark multi-objective data partitioning optimization method based on genetic algorithm according to claim 1, characterized in that, The process involves multiple iterations starting from the initial population and the fitness value of each individual in the initial population. In each iteration, the fitness of individuals in the new population is evaluated, and the partitioning scheme represented by the individual with the highest fitness during the iteration is taken as the optimal partitioning scheme. Specifically, this includes: Set the number of iterations and execute the iteration loop. When the number of iterations is reached, output the partitioning scheme represented by the recorded global best individual as the optimal partitioning scheme. The process of each iteration is as follows: based on the fitness value of individuals in the current population, the selection operation of the genetic algorithm is performed to select parent individuals for reproduction; Perform crossover on the selected parent individuals and mutation on the generated offspring individuals to produce a new set of candidate individuals; Based on the comprehensive fitness function, the fitness value of each individual in the newly generated candidate individual set is calculated, and combined with the preset elite retention strategy, a new generation of population is formed. Compare and record the individual with the highest fitness in the current iteration, and update the record of the globally best individual.

6. A Spark multi-objective data partitioning optimization system based on genetic algorithm, characterized in that, The Spark multi-objective data partitioning optimization system based on genetic algorithm is used to implement the Spark multi-objective data partitioning optimization method based on genetic algorithm as described in any one of claims 1-5. The Spark multi-objective data partitioning optimization system based on genetic algorithm includes: The optimization problem modeling module is used to perform distributed sampling on the Spark dataset to be processed to obtain the key frequency distribution of the data. Based on the key frequency distribution, the data partitioning problem is formalized into a multi-objective optimization problem, which includes load balancing objective, shuffle cost objective and data locality objective. The fitness function construction module is used to construct a comprehensive fitness function based on the load balancing objective, the shuffle cost objective, and the data locality objective. The individual fitness value calculation module is used to randomly generate multiple partition schemes, treat each partition scheme as an individual in the genetic algorithm, form an initial population based on all individuals, and calculate the fitness value of each individual in the initial population based on the comprehensive fitness function. The optimal partitioning scheme acquisition module is used to perform multiple iterations starting from the initial population and the fitness value of each individual in the initial population, and to evaluate the fitness of individuals in the new population obtained in each iteration, and to take the partitioning scheme represented by the individual with the highest fitness during the iteration as the optimal partitioning scheme. The partition scheme conversion module is used to construct an optimized partition mapping for high-frequency keys based on the optimal partition scheme, and to configure a hash rollback mechanism for low-frequency keys; The data partitioning execution module is used to obtain the data key input during the execution of the Spark task. If the data key is a high-frequency key, the target partition of the high-frequency key is determined by querying the optimized partition mapping. If the data key is a low-frequency key, the hash backoff mechanism is executed to determine the target partition of the low-frequency key.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a Spark multi-objective data partitioning optimization program based on a genetic algorithm stored in the memory and executable on the processor. When the Spark multi-objective data partitioning optimization program based on a genetic algorithm is executed by the processor, it implements the steps of the Spark multi-objective data partitioning optimization method based on a genetic algorithm as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a Spark multi-objective data partitioning optimization program based on a genetic algorithm, which, when executed by a processor, implements the steps of the Spark multi-objective data partitioning optimization method based on a genetic algorithm as described in any one of claims 1-5.

Images