Intelligent design method for hybrid parallel distributed training

Abstract

The invention discloses an intelligent design method for hybrid parallel distributed training, relates to the technical field of distributed training optimization algorithms, and aims to improve the efficiency of hybrid parallel training. Carrying out density clustering on the preprocessed training data set by adopting a DBSCAN algorithm; extracting a network architecture of the distributed training model, and exporting a task graph in an ONNX format; graph division is realized through a genetic algorithm to obtain an optimized model slicing decision; and selecting an individual with the highest fitness as an optimal solution, and taking a corresponding model segmentation decision as a final output result. According to the method, the model training speed can be increased, the optimization strategy is adaptively adjusted according to different data, model structures and training environments, the requirements of manual design and hybrid parallel strategy adjustment are reduced, and the method is suitable for machine learning models of various scales and types.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed training optimization algorithms, and in particular to an intelligent design method for hybrid parallel distributed training. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (large models) have been widely used in the field of natural language processing (NLP), such as Baidu's Wenxin Yiyan and Tencent's Hunyuan. However, the tens of thousands of huge parameters in the NLP large model, coupled with the model's learning and understanding of complex semantics in a large amount of text data, lead to slow convergence speed during NLP large model training and huge tuning overhead. Therefore, if the NLP large model training process can be optimized, the learning speed of the NLP model can be greatly improved, its convergence can be accelerated, and to a certain extent, the training cost can be compressed and the tuning overhead can be reduced, thereby promoting the further development of the NLP large prediction model.

[0003] At present, the technologies for improving the performance of distributed training mainly include: data parallelism, model parallelism and hybrid parallelism. Among them, hybrid parallelism combines the advantages of the first two, taking their strengths and making up for their weaknesses, and is the most used in distributed training performance optimization technologies. The key research point of hybrid parallel technology is: when adopting hybrid parallel distributed training, it is necessary to consider how to better split data and models and deploy them in the case of distributed nodes. Therefore, this leads to the multiplicity of the final optimization goals: how to divide data, how to split and deploy models.

[0004] According to the components and architecture types, distributed training can be modeled as a knowledge graph with a slightly simpler structure: in the distributed training architecture, each computing node is a vertex. When the model is sliced ​​and deployed on different nodes, the node devices can be divided according to the computing content, communication volume, and computational complexity corresponding to different nodes, so as to better analyze the factors affecting distributed training performance and improve the overall performance of distributed training. From the above, it can be seen that improving the performance of distributed training based on hybrid parallelism is a graph partitioning problem. At present, graph partitioning is an NP-hard problem, and there is no approximate algorithm with constant factors.

[0005] With the continuous development of evolutionary learning, intelligent learning strategies have been widely used in many fields. Different from general machine learning, the core idea of ​​evolutionary learning is to continuously screen the target population to obtain the optimal decision. This idea can be used for the graph partitioning problem of distributed training optimization to obtain the best strategy for distributed training model partitioning.

[0006] Based on this, this paper proposes an intelligent design method for hybrid parallel distributed training. By introducing adaptive data segmentation technology based on the DBSCAN algorithm, intelligent optimization design of data segmentation is realized; through evolutionary learning, the optimal model segmentation strategy is learned. Summary of the invention

[0007] In response to the needs and shortcomings of current technological development, the present invention provides an intelligent design method for hybrid parallel distributed training, aiming to improve the efficiency of hybrid parallel training of large NLP models, optimize resource allocation, and provide an end-to-end intelligent design solution for distributed training of large-scale machine learning models.

[0008] The present invention provides an intelligent design method for hybrid parallel distributed training, and the technical solution adopted to solve the above technical problems is as follows:

[0009] An intelligent design method for hybrid parallel distributed training, comprising the following steps:

[0010] S1, obtain and preprocess the training data set;

[0011] S2. Use DBSCAN algorithm to perform density clustering on the preprocessed training data set;

[0012] S3, extract the network architecture of the distributed training model and export it as a task graph in ONNX format;

[0013] S4, implement graph partitioning through genetic algorithm to obtain the optimal model slicing decision;

[0014] S5. Select the individual with the highest fitness as the optimal solution, and use its corresponding model segmentation decision as the final output result.

[0015] Optionally, the step S1 specifically includes the following operations:

[0016] S1.1, let the training data set be X = {X1, X2, ..., X i , ..., X n}, where X i represents a data point, is a d-dimensional vector, that is, each X i Contains d features;

[0017] S1.2. Standardize the training data set. The standardization formula is as follows:

[0018]

[0019]

[0020] In the formula, represents the jth feature of the i-th data point after standardization; X ij represents the jth feature of the i-th data point in the original data set; μ jrepresents the mean of the jth feature; σ j represents the standard deviation of the jth feature; n represents the total number of data points;

[0021] S1.3, based on the standardized training data set, the principal component analysis algorithm is used for dimensionality reduction. First, the mean vector of the sample is calculated. The calculation formula is as follows:

[0022]

[0023] Where n is the total number of samples in the training data set after standardization; X i is the value of the i-th sample in the training data set after standardization;

[0024] Then the data is decentralized, and the formula involved is as follows:

[0025]

[0026] Where, X i-cen is the decentralized value of the i-th sample in the training data set after standardization; X i is the value of the i-th sample in the training data set after standardization; is the sample mean vector of the standardized training data set;

[0027] Next, calculate the covariance matrix of the data set, involving the following formula:

[0028]

[0029] C v =λv,

[0030] V=[v1,v2,...,v k ],

[0031] X'=X cen V,

[0032] Where, X cen represents a decentralized data matrix, which contains the decentralized values ​​of all samples in the standardized training data set; (X cen ) T Represents the data matrix X cen is the transpose of the covariance matrix; C represents the covariance matrix; λ represents the eigenvalue; v represents the eigenvector; V represents the first k eigenvectors selected [v1,v2,...,v k ]; X' represents the data matrix or data set after dimensionality reduction using the principal component analysis algorithm; k represents the number of principal components selected, satisfying t is the set threshold.

[0033] Further optionally, the expression of the DBSCAN algorithm involved is:

[0034] X”=f dbscn (X',ε,MinPts),

[0035] MinPts = β * minestSample,

[0036] In the formula, X' is the data set after clustering using the DBSCAN algorithm; X' is the data set after preprocessing; f dbscn Represents the DBSCAN algorithm; ε represents the neighborhood radius, which defines the neighborhood range of the data point and takes values ​​between 0.1 and 0.5, which is an adjustable value; MinPts represents the minimum number of points, which is the minimum number of neighborhood points required to form a core point and is defined according to the minimum effective sample size of financial data; β is the scaling factor and takes values ​​between 0.05 and 0.2; minestSample represents the minimum effective sample size of financial data.

[0037] Further optionally, the DBSCAN algorithm involved performs density clustering on the preprocessed training data set through the following steps:

[0038] S2.1, Neighborhood exploration: For each point p in the preprocessed training data set, determine its ε-neighborhood N_ε(p), the specific formula is as follows:

[0039] N_ε(p)={q∈X'|dist(p,q)≤ε},

[0040] Where N_ε(p) represents the set of all points q whose distance from point p does not exceed ε; X' is the preprocessed data set; dist(p,q) is the Euclidean distance function between points p and q; ε represents the neighborhood radius, which defines the neighborhood range of a data point and takes values ​​between 0.1 and 0.5, which is an adjustable value;

[0041] S2.2. If the ε-neighborhood of point p contains at least MinPts objects, that is, |N_ε(p)|≥MinPts, then p is identified as a core point. The specific formula is as follows:

[0042] CorePoint(p)=True if|N_ε(p)|≥MinPts,

[0043] Where |N_ε(p)| represents the number of objects contained in the ε-neighborhood of point p; MinPts defines the parameter that determines the minimum number of neighbors required for a point to be considered a core point; CorePoint(p) is a function or variable used to determine whether point p is a core point. If the condition |N_ε(p)|≥MinPts is met, CorePoint(p) returns True, indicating that p is a core point, otherwise it returns False;

[0044] S2.3, Cluster initialization: For each identified core point p, if p has not been assigned to any cluster, create a new cluster C centered on p. The specific formula is as follows:

[0045] C={p}∪N_ε(p),

[0046] Where C represents a cluster, which is a set containing multiple data points; p represents a core point, which is a data point in the preprocessed training data set, and its neighborhood contains at least MinPts objects; N_ε(p) represents the set consisting of all points q whose distance from point p does not exceed ε; {p} represents a set containing only point p; ∪ is the union operator of sets, which is used to merge two sets into a new set, which contains all data points that appear in either set;

[0047] S2.4, Cluster expansion: For each point q in the newly created cluster C,

[0048] a) If q is a core point, that is, it satisfies the condition |N_ε(q)|≥MinPts, then all points in its ε-neighborhood are added to cluster C. The formula is as follows:

[0049] C=C∪N_ε(q),

[0050] In the formula, C represents a cluster, which is a set of multiple data points; q is a data point in cluster C, which has a chance to be a new core point; N_ε(q) is the set of all points contained in the neighborhood with radius ε centered on point q; C = C∪N_ε(q) is used to add all points in the neighborhood N_ε(q) of point q to cluster C through the union operator ∪;

[0051] b) Repeat step a) until no new core points are added to cluster C;

[0052] S2.5, Density reachability propagation: During cluster expansion,

[0053] a) If the ε-neighborhoods of two core points p and q intersect, that is, Then the clusters to which the core points p and q belong are merged;

[0054] b) If there is a data point p belonging to cluster C1 and a data point q belonging to cluster C2, so that the distance dist(p,q) between data point p and data point q is less than or equal to ε, then merge C1 and C2 into a new cluster C0;

[0055] S2.6, Termination condition: When all points in the data set are assigned to a cluster or marked as noise points, the DBSCAN algorithm terminates. The formula involved in this process is as follows:

[0056]

[0057] Where D represents the set of all data points; p∈D represents that data point p belongs to data set D; SomeCluster represents a cluster, which is a set of multiple data points; Noise(p) is a Boolean function used to determine whether data point p is marked as a noise point. If Noise(p)=True, it means that p is a noise point, otherwise, p is not a noise point.

[0058] S2.7, Noise point processing: Points that do not belong to any cluster are marked as noise points.

[0059]

[0060] Where D represents the set containing all data points; p∈D represents that data point p belongs to data set D; Noise is a set used to store data points marked as noise points; Ci represents the i-th cluster, which is a set containing multiple data points; ∪iCi represents the union of all clusters, that is, the set of all data points in all clusters; Indicates that the noise point set Noise contains data points that do not belong to any cluster.

[0061] Further optionally, the execution step S3 involves exporting the network architecture of the distributed training model into a task graph in ONNX format by importing the ONNX library in the PyTorch or TensorFlow deep learning framework.

[0062] Further optionally, the genetic algorithm involved sequentially performs six processes of population initialization, excellent individual selection based on fitness, crossover, mutation, population update, and termination condition check, and then returns to continue to perform excellent individual selection based on fitness, or outputs the individual with the highest fitness.

[0063] Preferably, the genetic algorithm involved sequentially performs six processes of population initialization, excellent individual selection based on fitness, crossover, mutation, population update, and termination condition check, specifically including:

[0064] S4.1. Population initialization: Initialize the chromosome population by setting the chromosome decision coding scheme;

[0065] Before initialization: Set the composition of chromosome to chromosome0 = {k(v i )=j}, where chromosome0 represents the initialization chromosome, v i represents the i-th node, k(v i )=j means that the ith node is assigned to the position numbered j in the subgraph k. The values ​​of k and j are both 0 to n, and n represents the length of the array. It is set that the difference between the serial numbers of adjacent nodes cannot be greater than 1, that is, k(v i )-k(v i-1 )>=0, otherwise it is considered invalid;

[0066] During the initialization process: each chromosome represents a graph partitioning scheme, base = [0, 1, 2, 3, ..., n] is used as the benchmark, assuming that each node is assigned to a different subgraph, and a chromosome population that meets the requirements is generated based on a normal distribution random number, and a chromosome population of 1000 is obtained, chromosome0 = random (base), chromosome0 = random (base) means that 1000 different nodes are randomly selected from base to form the chromosome population chromosome0;

[0067] S4.2, Selection of excellent individuals based on fitness:

[0068] First, the quality of each chromosome individual is evaluated according to the fitness function fit(c). The fitness function takes into account the communication cost and computational load balance. The specific formula is:

[0069] fit(c)=w v (i)*compute(c)+w e (i) *commu(c),

[0070] In the formula, w v (i) represents the weight of the computational variance, which is used to measure the degree of computational load balance of individual chromosomes, w v The larger the (i) is, the greater the impact of the computational variance on the fitness. compute(c) represents the computational variance of chromosome individual c, which is used to measure the balance of the chromosome individual in the allocation of computing resources. The smaller the computational variance is, the more balanced the allocation of computing resources of the chromosome individual is, and the higher the fitness is. e (i) represents the sum of the weights of the cross-device edges, which is used to measure the communication cost of the chromosome individual, w e(i) The larger the value, the greater the impact of the sum of the weights of the cross-device edges on the fitness. commu(c) represents the sum of the weights of the cross-device edges of chromosome individual c, which is used to measure the balance of the chromosome individual in the allocation of communication resources. The smaller the sum of the weights of the cross-device edges, the more balanced the allocation of communication resources of the chromosome individual and the higher the fitness.

[0071] Secondly, the compromise method is used to calculate the fitness of all individuals in the population to obtain the fitness of different individuals, and then arrange them according to the fitness size, select the middle value as the fitness threshold, and select individuals with a fitness greater than the fitness threshold to enter the next round;

[0072] Finally, repeat the above screening process until the termination condition is met;

[0073] S4.3, Crossover: Select the parent generation and perform random crossover operation to generate new offspring;

[0074] When performing random crossover operations, a single-point crossover method is used, that is, a crossover point o is randomly selected, o∈[1,n], and the genes of the two parents before the crossover point are copied to the offspring, while the genes of the parents after the crossover point are exchanged;

[0075] S4.4, mutation: Adopt adaptive mutation mode and perform mutation operation according to the actual situation of the node;

[0076] S4.5. Update the population: add the newly generated offspring to the population, keeping the population size unchanged;

[0077] S4.6, termination condition check: Check whether the termination condition of "reaching the maximum number of iterations or there is no significant improvement in the optimal solution for multiple consecutive generations" is met. If the termination condition is not met, return to step S4.2 to continue iterating; otherwise, output the individual with the highest fitness, that is, the model segmentation decision Y.

[0078] Preferably, during the execution of S4.2, the calculation formula of the adaptive threshold is as follows:

[0079]

[0080] In the formula, fitness_threshold represents the adaptive threshold, which is used to calculate a threshold based on the total number of individuals C in the population in the current iteration and the fitness fit of each individual, and then screen the next generation chromosome population based on this threshold; assuming that the current population is updated for the i-th generation, the next generation chromosome population chromosome is obtained after screening. i+1 ,i+1 represents the i+1th iteration.

[0081] Preferably, when executing step S4.4, the calculation formula of the mutation rate mu involved is as follows:

[0082] mu=base_rate*(1+α*(max_fitness-avg_fitness) / max_fitness),

[0083] In the formula, mu represents the probability of chromosome mutation; base_rate represents the basic mutation rate, which is a fixed constant with a value of 0.3 and is used to control the frequency of mutation operations; α represents the adjustment factor, which is a fixed constant with a value of 0.5 and is used to adjust the relationship between mutation rate and fitness. The larger α is, the greater the influence of fitness on mutation rate; max_fitness represents the maximum fitness value in the population, that is, the maximum value of all individual fitnesses; avg_fitness represents the average fitness value of the population, that is, the sum of all individual fitnesses divided by the total number of individuals.

[0084] The intelligent design method for hybrid parallel distributed training of the present invention has the following beneficial effects compared with the prior art:

[0085] The present invention adopts the DBSCAN algorithm to density cluster the training data and adopts the genetic algorithm to obtain the optimal model slicing decision, which can speed up the training of large NLP models and adaptively adjust the optimization strategy according to different data, model structure and training environment. It is applicable to machine learning models of various sizes and types and has good versatility. Through the automated intelligent optimization process, it reduces the need for manual design and adjustment of hybrid parallel strategies, reduces the dependence on professional knowledge, and improves development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Attached Figure 1 It is a flow chart of the method of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0087] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0088] Embodiment 1:

[0089] Combined with Figure 1 This embodiment proposes an intelligent design method for hybrid parallel distributed training, aiming to speed up the training of large NLP models, which includes the following steps:

[0090] S1. Obtain and preprocess the training dataset.

[0091] This embodiment takes obtaining a financial training data set as an example.

[0092] S1.1, let the training data set be X = {X1, X2, ..., X i, ..., X n}, where X i represents a financial data point, is a d-dimensional vector, that is, each X i Contains d features;

[0093] S1.2. Standardize the training data set. The standardization formula is as follows:

[0094]

[0095] In the formula, represents the jth feature of the i-th data point after standardization; X ij represents the jth feature of the i-th data point in the original data set; μ j represents the mean of the jth feature; σ j represents the standard deviation of the jth feature; n represents the total number of data points;

[0096] S1.3, based on the standardized training data set, the principal component analysis algorithm is used for dimensionality reduction. First, the mean vector of the sample is calculated. The calculation formula is as follows:

[0097]

[0098] Where n is the total number of samples in the training data set after standardization; X i is the value of the i-th sample in the training data set after standardization;

[0099] Then the data is decentralized, and the formula involved is as follows:

[0100]

[0101] Where, X i-cen is the decentralized value of the i-th sample in the training data set after standardization; X i is the value of the i-th sample in the training data set after standardization; is the sample mean vector of the standardized training data set;

[0102] Next, calculate the covariance matrix of the data set, involving the following formula:

[0103]

[0104] C v =λv,

[0105] V=[v1,v2,...,v k ],

[0106] X'=X cen V,

[0107] In the formula, X cen represents a decentralized data matrix, which contains the decentralized values ​​of all samples in the standardized training data set; (X cen ) T Represents the data matrix X cen is the transpose of the covariance matrix; C represents the covariance matrix; λ represents the eigenvalue; v represents the eigenvector; V represents the first k eigenvectors selected [v1,v2,...,v k ]; X' represents the data matrix or data set after dimensionality reduction using the principal component analysis algorithm; k represents the number of principal components selected, satisfying t is the set threshold, and the threshold set in this step is 0.95.

[0108] S2. Use DBSCAN algorithm to perform density clustering on the preprocessed training data set.

[0109] The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is a density-based clustering algorithm. Its name is the abbreviation of "Density-Based Spatial Clustering of Applications with Noise". The algorithm clusters the density areas in space and divides the interconnected areas of high-density areas into the same category, thereby achieving the purpose of dividing the data set into several clusters.

[0110] The expression of DBSCAN algorithm is:

[0111] X”=f dbscn (X',ε,MinPts),

[0112] MinPts = β * minestSample,

[0113] In the formula, X' is the data set after clustering using the DBSCAN algorithm; X' is the data set after preprocessing; f dbscn Represents the DBSCAN algorithm; ε represents the neighborhood radius, which defines the neighborhood range of the financial data point and takes values ​​between 0.1 and 0.5, which is an adjustable value; MinPts represents the minimum number of points, which is the minimum number of neighborhood points required to form a core point and is defined according to the minimum effective sample size of the financial data; β is the scaling factor, which takes values ​​between 0.05 and 0.2; minestSample represents the minimum effective sample size of the financial data.

[0114] The DBSCAN algorithm involved performs density clustering on the preprocessed training dataset through the following steps:

[0115] S2.1, Neighborhood exploration: For each point p in the preprocessed training data set, determine its ε-neighborhood N_ε(p), the specific formula is as follows:

[0116] N_ε(p)={q∈X'|dist(p,q)≤ε},

[0117] Where N_ε(p) represents the set of all points q whose distance to point p does not exceed ε; X' is the preprocessed data set; dist(p,q) is the Euclidean distance function between points p and q; ε represents the neighborhood radius, which defines the neighborhood range of a financial data point and takes values ​​between 0.1 and 0.5, which is an adjustable value.

[0118] S2.2. If the ε-neighborhood of point p contains at least MinPts objects, that is, |N_ε(p)|≥MinPts, then p is identified as a core point. The specific formula is as follows:

[0119] CorePoint(p)=True if|N_ε(p)|≥MinPts,

[0120] Where |N_ε(p)| represents the number of objects contained in the ε-neighborhood of point p; MinPts defines the parameter that determines the minimum number of neighbors required for a point to be considered a core point; CorePoint(p) is a function or variable used to determine whether point p is a core point. If the condition |N_ε(p)|≥MinPts is met, CorePoint(p) returns True, indicating that p is a core point, otherwise it returns False.

[0121] S2.3, Cluster initialization: For each identified core point p, if p has not been assigned to any cluster, create a new cluster C centered on p. The specific formula is as follows:

[0122] C={p}∪N_ε(p),

[0123] Where C represents a cluster, which is a set containing multiple data points; p represents a core point, which is a data point in the preprocessed training data set, and its neighborhood contains at least MinPts objects; N_ε(p) represents the set of all points q whose distance from point p does not exceed ε; {p} represents a set containing only point p; ∪ is the union operator of sets, which is used to merge two sets into a new set, and the new set contains all the data points that appear in either set.

[0124] S2.4, Cluster expansion: For each point q in the newly created cluster C,

[0125] a) If q is a core point, that is, it satisfies the condition |N_ε(q)|≥MinPts, then all points in its ε-neighborhood are added to cluster C. The formula is as follows:

[0126] C=C∪N_ε(q),

[0127] In the formula, C represents a cluster, which is a set of multiple data points; q is a data point in cluster C, which has a chance to be a new core point; N_ε(q) is the set of all points contained in the neighborhood with radius ε centered on point q; C = C∪N_ε(q) is used to add all points in the neighborhood N_ε(q) of point q to cluster C through the union operator ∪;

[0128] b) Repeat step a) until no new core points are added to cluster C.

[0129] S2.5, Density reachability propagation: During cluster expansion,

[0130] a) If the ε-neighborhoods of two core points p and q intersect, that is, Then the clusters to which the core points p and q belong are merged;

[0131] b) If there is a data point p belonging to cluster C1 and a data point q belonging to cluster C2, so that the distance dist(p,q) between data point p and data point q is less than or equal to ε, then C1 and C2 are merged into a new cluster C0; that is, So that dist(p,q)≤ε, then C0=C1∪C2.

[0132] S2.6, Termination condition: When all points in the data set are assigned to a cluster or marked as noise points, the DBSCAN algorithm terminates. The formula involved in this process is as follows:

[0133]

[0134] Where D represents the set of all data points; p∈D represents that data point p belongs to data set D; SomeCluster represents a cluster, which is a set of multiple data points; Noise(p) is a Boolean function used to determine whether data point p is marked as a noise point. If Noise(p)=True, it means that p is a noise point, otherwise, p is not a noise point.

[0135] S2.7, Noise point processing: Points that do not belong to any cluster are marked as noise points.

[0136]

[0137] Where D represents the set containing all data points; p∈D represents that data point p belongs to data set D; Noise is a set used to store data points marked as noise points; Ci represents the i-th cluster, which is a set containing multiple data points; ∪iCi represents the union of all clusters, that is, the set of all data points in all clusters; Indicates that the noise point set Noise contains data points that do not belong to any cluster.

[0138] S3. By importing the ONNX library in the PyTorch or TensorFlow deep learning framework, the network architecture of the distributed training model is exported as a task graph in ONNX format.

[0139] S4, implement graph partitioning through genetic algorithm to obtain the optimal model slicing decision;

[0140] The genetic algorithm involved sequentially executes six processes: population initialization, selection of excellent individuals based on fitness, crossover, mutation, population update, and termination condition check, and then returns to continue to select excellent individuals based on fitness, or outputs the individual with the highest fitness.

[0141] The six processes involved specifically include the following operations:

[0142] S4.1. Population initialization: Initialize the chromosome population by setting the chromosome decision coding scheme;

[0143] Before initialization: Set the composition of chromosome to chromosome0 = {k(v i )=j}, where chromosome0 represents the initialization chromosome, v i represents the i-th node, k(v i )=j means that the ith node is assigned to the position numbered j in the subgraph k. The values ​​of k and j are both 0 to n, and n represents the length of the array. It is set that the difference between the serial numbers of adjacent nodes cannot be greater than 1, that is, k(v i )-k(v i-1 )>=0, otherwise it is considered invalid;

[0144] During the initialization process: each chromosome represents a graph partitioning scheme, base = [0, 1, 2, 3, ..., n] is used as the benchmark, assuming that each node is assigned to a different subgraph, and a chromosome population that meets the requirements is generated based on a normally distributed random number, and a chromosome population of 1000 is obtained, chromosome0 = random (base), chromosome0 = random (base) means that 1000 different nodes are randomly selected from base to form the chromosome population chromosome0.

[0145] S4.2, Selection of excellent individuals based on fitness:

[0146] First, the quality of each chromosome individual is evaluated according to the fitness function fit(c). The fitness function takes into account the communication cost and computational load balance. The specific formula is:

[0147] fit(c)=w v (i)*compute(c)+w e (i) *commu(c),

[0148] In the formula, w v (i) represents the weight of the computational variance, which is used to measure the degree of computational load balance of individual chromosomes, w v The larger the (i) is, the greater the impact of the computational variance on the fitness. compute(c) represents the computational variance of chromosome individual c, which is used to measure the balance of the chromosome individual in the allocation of computing resources. The smaller the computational variance is, the more balanced the allocation of computing resources of the chromosome individual is, and the higher the fitness is. e (i) represents the sum of the weights of the cross-device edges, which is used to measure the communication cost of the chromosome individual, w e (i) The larger the value, the greater the impact of the sum of the weights of the cross-device edges on the fitness. commu(c) represents the sum of the weights of the cross-device edges of chromosome individual c, which is used to measure the balance of the chromosome individual in the allocation of communication resources. The smaller the sum of the weights of the cross-device edges, the more balanced the allocation of communication resources of the chromosome individual and the higher the fitness.

[0149] Secondly, the compromise method is used to calculate the fitness of all individuals in the population to obtain the fitness of different individuals, and then arrange them according to the fitness size, select the middle value as the fitness threshold, and select individuals with a fitness greater than the fitness threshold to enter the next round. The calculation formula for the adaptive threshold involved is as follows:

[0150]

[0151] In the formula, fitness_threshold represents the adaptive threshold, which is used to calculate a threshold based on the total number of individuals C in the population in the current iteration and the fitness fit of each individual, and then screen the next generation chromosome population based on this threshold; assuming that the current population is updated for the i-th generation, the next generation chromosome population chromosome is obtained after screening. i+1 ,i+1 represents the i+1th iteration;

[0152] Finally, the above screening process is repeated until the termination condition is met: the maximum number of iterations is reached or there is no significant improvement in the optimal solution for multiple consecutive generations.

[0153] S4.3, Crossover: Select the parent generation and perform random crossover operation to generate new offspring;

[0154] When performing random crossover operations, a single-point crossover method is used, that is, a crossover point o is randomly selected, o∈[1,n], and the genes of the two parents before the crossover point are copied to the offspring, while the genes of the parents after the crossover point are exchanged;

[0155] For example, for parent 1: [0, 0, 1, 1, 2, 2] and parent 2: [1, 2, 0, 2, 1, 0], if the intersection is 3, then child 1 is [0, 0, 1, 2, 1, 0] and child 2 is [1, 2, 0, 1, 2, 2].

[0156] S4.4, mutation: Adopt adaptive mutation mode and perform mutation operation according to the actual situation of the node;

[0157] The calculation formula for the mutation rate mu is as follows:

[0158] mu=base_rate*(1+α*(max_fitness-avg_fitness) / max_fitness),

[0159] In the formula, mu represents the probability of chromosome mutation; base_rate represents the basic mutation rate, which is a fixed constant with a value of 0.3 and is used to control the frequency of mutation operations; α represents the adjustment factor, which is a fixed constant with a value of 0.5 and is used to adjust the relationship between mutation rate and fitness. The larger α is, the greater the influence of fitness on mutation rate; max_fitness represents the maximum fitness value in the population, that is, the maximum value of all individual fitnesses; avg_fitness represents the average fitness value of the population, that is, the sum of all individual fitnesses divided by the total number of individuals.

[0160] S4.5. Update the population: add the newly generated offspring to the population, keeping the population size unchanged.

[0161] S4.6, termination condition check: Check whether the termination condition of "reaching the maximum number of iterations or there is no significant improvement in the optimal solution for multiple consecutive generations" is met. If the termination condition is not met, return to step S4.2 to continue iterating; otherwise, output the individual with the highest fitness, that is, the model segmentation decision Y.

[0162] S5. Select the individual with the highest fitness as the optimal solution, and use its corresponding model segmentation decision Y as the final output result.

[0163] In summary, the intelligent design method for hybrid parallel distributed training of the present invention can speed up the training of large NLP models, reduce the need for manual design and adjustment of hybrid parallel strategies, improve development efficiency, adaptively adjust optimization strategies according to different data, model structures and training environments, and is suitable for machine learning models of various sizes and types.

[0164] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. An intelligent design method for hybrid parallel distributed training, characterized in that: The steps include: S1, obtain and preprocess the training data set; S2. Use DBSCAN algorithm to perform density clustering on the preprocessed training data set; S3, extract the network architecture of the distributed training model and export it as a task graph in ONNX format; S4, implement graph partitioning through genetic algorithm to obtain the optimal model slicing decision; S5. Select the individual with the highest fitness as the optimal solution, and use its corresponding model segmentation decision as the final output result.

2. The intelligent design method for hybrid parallel distributed training according to claim 1, characterized in that: The step S1 specifically includes the following operations: S1.1, let the training data set be X = {X1, X2, ..., X i , ..., X n }, where X i represents a data point, is a d-dimensional vector, that is, each X i Contains d features; S1.

2. Standardize the training data set. The standardization formula is as follows: In the formula, represents the jth feature of the i-th data point after standardization; X ij represents the jth feature of the i-th data point in the original data set; μ j represents the mean of the jth feature; σ j represents the standard deviation of the jth feature; n represents the total number of data points; S1.3, based on the standardized training data set, the principal component analysis algorithm is used for dimensionality reduction. First, the mean vector of the sample is calculated. The calculation formula is as follows: Where n is the total number of samples in the training data set after standardization; X i is the value of the i-th sample in the training data set after standardization; Then the data is decentralized, and the formula involved is as follows: Where, X i-cen is the decentralized value of the i-th sample in the training data set after standardization; X i is the value of the i-th sample in the training data set after standardization; is the sample mean vector of the standardized training data set; Next, calculate the covariance matrix of the data set, involving the following formula: C v =λv, V=[v1,v2,...,v k ], X'=X cen V, Where, X cen represents a decentralized data matrix, which contains the decentralized values ​​of all samples in the standardized training data set; (X cen ) T Represents the data matrix X cen is the transpose of the covariance matrix; C represents the covariance matrix; λ represents the eigenvalue; v represents the eigenvector; V represents the first k eigenvectors selected [v1,v2,...,v k ]; X' represents the data matrix or data set after dimensionality reduction using the principal component analysis algorithm; k represents the number of principal components selected, satisfying t is the set threshold.

3. The intelligent design method for hybrid parallel distributed training according to claim 2, characterized in that: The expression of the DBSCAN algorithm is: X”=f dbscn (X',ε,MinPts), MinPts = β * minestSample, In the formula, X' is the data set after clustering using the DBSCAN algorithm; X' is the data set after preprocessing; f dbscn Represents the DBSCAN algorithm; ε represents the neighborhood radius, which defines the neighborhood range of the data point and takes values ​​between 0.1 and 0.5, which is an adjustable value; MinPts represents the minimum number of points, which is the minimum number of neighborhood points required to form a core point and is defined according to the minimum effective sample size of financial data; β is the scaling factor, which ranges from 0.05 to 0.2; minestSample represents the minimum valid sample size of financial data.

4. The intelligent design method for hybrid parallel distributed training according to claim 3, characterized in that: The DBSCAN algorithm performs density clustering on the preprocessed training data set through the following steps: S2.1, Neighborhood exploration: For each point p in the preprocessed training data set, determine its ε-neighborhood N_ε(p), the specific formula is as follows: N_ε(p)={q∈X'|dist(p,q)≤ε}, Where N_ε(p) represents the set of all points q whose distance from point p does not exceed ε; X' is the preprocessed data set; dist(p,q) is the Euclidean distance function between points p and q; ε represents the neighborhood radius, which defines the neighborhood range of a data point and takes values ​​between 0.1 and 0.5, which is an adjustable value; S2.

2. If the ε-neighborhood of point p contains at least MinPts objects, that is, |N_ε(p)|≥MinPts, then p is identified as a core point. The specific formula is as follows: CorePoint(p)=True if|N_ε(p)|≥MinPts, Where |N_ε(p)| represents the number of objects contained in the ε-neighborhood of point p; MinPts defines the parameter that determines the minimum number of neighbors required for a point to be considered a core point; CorePoint(p) is a function or variable used to determine whether point p is a core point. If the condition |N_ε(p)|≥MinPts is met, CorePoint(p) returns True, indicating that p is a core point, otherwise it returns False; S2.3, Cluster initialization: For each identified core point p, if p has not been assigned to any cluster, create a new cluster C centered on p. The specific formula is as follows: C={p}∪N_ε(p), Where C represents a cluster, which is a set containing multiple data points; p represents a core point, which is a data point in the preprocessed training data set, and its neighborhood contains at least MinPts objects; N_ε(p) represents the set consisting of all points q whose distance from point p does not exceed ε; {p} represents a set containing only point p; ∪ is the union operator of sets, which is used to merge two sets into a new set, which contains all data points that appear in either set; S2.4, Cluster expansion: For each point q in the newly created cluster C, a) If q is a core point, that is, it satisfies the condition |N_ε(q)|≥MinPts, then all points in its ε-neighborhood are added to cluster C. The formula is as follows: C=C∪N_ε(q), In the formula, C represents a cluster, which is a set of multiple data points; q is a data point in cluster C, which has a chance to be a new core point; N_ε(q) is the set of all points contained in the neighborhood with radius ε centered on point q; C = C∪N_ε(q) is used to add all points in the neighborhood N_ε(q) of point q to cluster C through the union operator ∪; b) Repeat step a) until no new core points are added to cluster C; S2.5, Density reachability propagation: During cluster expansion, a) If the ε-neighborhoods of two core points p and q intersect, that is, Then the clusters to which the core points p and q belong are merged; b) If there is a data point p belonging to cluster C1 and a data point q belonging to cluster C2, so that the distance dist(p,q) between data point p and data point q is less than or equal to ε, then merge C1 and C2 into a new cluster C0; S2.6, Termination condition: When all points in the data set are assigned to a cluster or marked as noise points, the DBSCAN algorithm terminates. The formula involved in this process is as follows: Where D represents the set of all data points; p∈D represents that data point p belongs to data set D; SomeCluster represents a cluster, which is a set of multiple data points; Noise(p) is a Boolean function used to determine whether data point p is marked as a noise point. If Noise(p)=True, it means that p is a noise point, otherwise, p is not a noise point. S2.7, Noise point processing: Points that do not belong to any cluster are marked as noise points. Where D represents the set containing all data points; p∈D represents that data point p belongs to data set D; Noise is a set used to store data points marked as noise points; Ci represents the i-th cluster, which is a set containing multiple data points; ∪iCi represents the union of all clusters, that is, the set of all data points in all clusters; Indicates that the noise point set Noise contains data points that do not belong to any cluster.

5. The intelligent design method for hybrid parallel distributed training according to claim 4, characterized in that: Execute step S3 to export the network architecture of the distributed training model into a task graph in ONNX format by importing the ONNX library in the PyTorch or TensorFlow deep learning framework.

6. The intelligent design method for hybrid parallel distributed training according to claim 4, characterized in that: The genetic algorithm sequentially performs six processes: population initialization, excellent individual selection based on fitness, crossover, mutation, population update, and termination condition check, and then returns to continue to perform excellent individual selection based on fitness, or outputs the individual with the highest fitness.

7. The intelligent design method for hybrid parallel distributed training according to claim 6, characterized in that: The genetic algorithm sequentially performs six processes: population initialization, selection of excellent individuals based on fitness, crossover, mutation, population update, and termination condition check, specifically including: S4.

1. Population initialization: Initialize the chromosome population by setting the chromosome decision coding scheme; Before initialization: Set the composition of chromosome to chromosome0 = {k(v i )=j}, where chromosome0 represents the initialization chromosome, v i represents the i-th node, k(v i )=j means that the ith node is assigned to the position numbered j in the subgraph k. The values ​​of k and j are both 0 to n, and n represents the length of the array. It is set that the difference between the serial numbers of adjacent nodes cannot be greater than 1, that is, k(v i )-k(v i-1 )>=0, otherwise it is considered invalid; During the initialization process: each chromosome represents a graph partitioning scheme, base = [0, 1, 2, 3, ..., n] is used as the benchmark, assuming that each node is assigned to a different subgraph, and a chromosome population that meets the requirements is generated based on a normal distribution random number, and a chromosome population of 1000 is obtained, chromosome0 = random (base), chromosome0 = random (base) means that 1000 different nodes are randomly selected from base to form the chromosome population chromosome0; S4.2, Selection of excellent individuals based on fitness: First, the quality of each chromosome individual is evaluated according to the fitness function fit(c). The fitness function takes into account the communication cost and computational load balance. The specific formula is: fit(c)=w v (i)*compute(c)+w e (i)*commu(c), In the formula, w v (i) represents the weight of the computational variance, which is used to measure the degree of computational load balance of individual chromosomes, w v The larger the (i) is, the greater the impact of the computational variance on the fitness. compute(c) represents the computational variance of chromosome individual c, which is used to measure the balance of the chromosome individual in the allocation of computing resources. The smaller the computational variance is, the more balanced the allocation of computing resources of the chromosome individual is, and the higher the fitness is. e (i) represents the sum of the weights of the cross-device edges, which is used to measure the communication cost of the chromosome individual, w e (i) The larger the value, the greater the impact of the sum of the weights of the cross-device edges on the fitness. commu(c) represents the sum of the weights of the cross-device edges of chromosome individual c, which is used to measure the balance of the chromosome individual in the allocation of communication resources. The smaller the sum of the weights of the cross-device edges, the more balanced the allocation of communication resources of the chromosome individual and the higher the fitness. Secondly, the compromise method is used to calculate the fitness of all individuals in the population to obtain the fitness of different individuals, and then arrange them according to the fitness size, select the middle value as the fitness threshold, and select individuals with a fitness greater than the fitness threshold to enter the next round; Finally, repeat the above screening process until the termination condition is met; S4.3, Crossover: Select the parent generation and perform random crossover operation to generate new offspring; When performing random crossover operations, a single-point crossover method is used, that is, a crossover point o is randomly selected, o∈[1,n], and the genes of the two parents before the crossover point are copied to the offspring, while the genes of the parents after the crossover point are exchanged; S4.4, mutation: Adopt adaptive mutation mode and perform mutation operation according to the actual situation of the node; S4.

5. Update the population: add the newly generated offspring to the population, keeping the population size unchanged; S4.6, termination condition check: Check whether the termination condition of "reaching the maximum number of iterations or no significant improvement in the best solution for multiple consecutive generations" is met. If the termination condition is not met, return to step S4.2 to continue iterating; otherwise, output the individual with the highest fitness, that is, the model segmentation decision Y.

8. The intelligent design method for hybrid parallel distributed training according to claim 7, characterized in that: During the execution of S4.2, the calculation formula of the adaptive threshold is as follows: In the formula, fitness_threshold represents the adaptive threshold, which is used to calculate a threshold based on the total number of individuals C in the population in the current iteration and the fitness fit of each individual, and then screen the next generation chromosome population based on this threshold; assuming that the current population is updated for the i-th generation, the next generation chromosome population chromosome is obtained after screening. i+1 ,i+1 represents the i+1th iteration.

9. The intelligent design method for hybrid parallel distributed training according to claim 7, characterized in that: When executing step S4.4, the calculation formula of the mutation rate mu involved is as follows: mu=base_rate*(1+α*(max_fitness-avg_fitness) / max_fitness), In the formula, mu represents the probability of chromosome mutation; base_rate represents the basic mutation rate, which is a fixed constant with a value of 0.3 and is used to control the frequency of mutation operations; α represents the adjustment factor, which is a fixed constant with a value of 0.5 and is used to adjust the relationship between mutation rate and fitness. The larger α is, the greater the influence of fitness on mutation rate; max_fitness represents the maximum fitness value in the population, that is, the maximum value of all individual fitnesses; avg_fitness represents the average fitness value of the population, that is, the sum of all individual fitnesses divided by the total number of individuals.

Images