A hybrid multi-learning mode selection method of counting cooperation

Abstract

The application discloses a mixed multi-learning mode selection method of communication and calculation cooperation, and relates to the field of machine learning. On the basis of a mixed segmentation and federated learning architecture, the selection of a segmentation layer in segmentation learning is optimized. The segmentation layer can cause changes in the data volume of transmitted activation values and gradients, thereby causing differentiated time delay. On the basis of the segmentation layer optimization, different terminals participating in different learning modes in the mixed segmentation and federated learning can bring different influences on time delay and convergence effect, and the selection of a terminal learning mode is further optimized. The application can make training faster and improve learning performance.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to a hybrid multi-learning mode selection method that combines computational and analytical approaches. Background Technology

[0002] The widespread adoption of mobile devices has led to an unprecedented surge in data generation, propelling the development of wireless distributed intelligence. Federated Learning (FL), a well-known distributed learning paradigm, is gaining increasing attention for its application in wireless networks. In FL, mobile devices simultaneously download a global deep neural network (DNN) model from a central server, train their local model using their own data, and upload the updated model back to the server. The server aggregates these updates and creates a new global model. This process is repeated multiple times until convergence. Federated Learning enables parallel and efficient DNN model training across numerous mobile devices. However, a major limitation of Federated Learning is that it requires each device to have sufficient computing resources to train the entire model locally, which can be a constraint for devices with limited processing power.

[0003] To alleviate these issues, split learning (SL Split Learning) has been introduced and studied in wireless networks. SL Split Learning innovatively splits the complete DNN model into two independent parts: a marginal model and a local model, which are trained separately on a server and a terminal, respectively. In SL Split Learning, the terminals train the models sequentially. Specifically, the first terminal downloads the local model from the server, jointly performs forward and backward propagation with the server to complete the entire model training, and uploads the updated local model back to the server. This process is then executed sequentially by subsequent terminals. While SL Split Learning effectively reduces the computational load on terminals by offloading part of the model training and improves learning performance by overcoming the negative impact of non-independent and identically distributed datasets between terminals, its sequential nature does significantly increase training time.

[0004] To fully leverage the advantages of these two learning models, numerous researchers have dedicated themselves to integrating federated learning and split learning. Hybrid Split and Federated Learning (HSFL) involves a subset of terminals participating in federated learning training while others participate in split learning training.

[0005] Therefore, those skilled in the art are dedicated to developing a hybrid multi-learning mode selection method that integrates computation and learning, providing more flexible learning mode selection and further improving learning performance. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to improve the learning performance of hybrid segmentation and federated learning.

[0007] To achieve the above objectives, this invention provides a hybrid multi-learning mode selection method with integrated computing and collaboration, comprising the following steps:

[0008] Step 1: The terminal's learning modes are divided into federated learning and segmented learning;

[0009] Step 2: Select the optimal segmentation layer;

[0010] Step 3: Change the terminal learning mode. Continue with step 2 and select the optimal terminal learning mode.

[0011] Furthermore, the terminal learning mode is selected in each round of training.

[0012] Furthermore, the learning mode of the terminal is selected based on the training effect and training latency.

[0013] Furthermore, the objective function for selecting the terminal learning mode is:

[0014]

[0015] Where ρ is a hyperparameter. It's training latency. It represents the number of segmented learning terminals in the t-th communication round.

[0016] Furthermore, ρ = 0.002.

[0017] Furthermore, the objective function is minimized.

[0018] Furthermore, in step 1, the learning mode of the terminal is randomly initialized, and divided into user clusters participating in federated learning and user clusters participating in segmented learning.

[0019] Further, in step 2, the latency of the terminal selecting different segmentation layers is calculated, and the optimal segmentation layer is selected from them.

[0020] Furthermore, in step 3, the terminal learning mode is changed using the Gibbs sampling algorithm.

[0021] Furthermore, the goal of the terminal learning is to minimize the global loss function.

[0022] Some existing studies only address architectural changes but fail to consider the inherent relationships within the new architecture. For example, compared to federated learning, segmentation learning has a higher communication load, requiring more communication capabilities from the terminal, while federated learning has a high local training load, necessitating higher computational power from the terminal. Furthermore, the choice of segmentation layer in segmentation learning affects latency, making layer selection a critical issue. This invention, based on a hybrid segmentation and federated learning architecture, considers the selection of segmentation layers in segmentation learning. Since segmentation layers alter the amount of transmitted activation values ​​and gradient data, leading to latency variations, a good segmentation layer selection can accelerate training. Building upon the segmentation layer optimization, this invention addresses the impact of different terminals participating in different learning modes on latency and convergence performance in hybrid segmentation and federated learning. Therefore, it introduces optimization for terminal learning mode selection. In the proposed architecture, terminals with lower computational power can participate in segmentation learning to reduce local computational load, while terminals with higher computational power participate in federated learning to reduce communication load. For the selection of segmentation layers in segmentation learning, given the limited number of model layers, an exhaustive search approach is used to find the optimal segmentation layer. Then switch between different terminal selection combinations and continue the above optimization process until the change in latency is considered the optimal solution within a very small range.

[0023] Compared with the prior art, the present invention has the following obvious substantive features and significant advantages:

[0024] 1. This invention achieves optimal training results, with significant advantages even with the gains from the optimization algorithm.

[0025] 2. This invention can accelerate training and help machine learning models be deployed more quickly.

[0026] 3. This invention is applied to the medical record diagnostic model in hospitals, enabling rapid preliminary diagnosis of patients' symptoms and improving medical efficiency.

[0027] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0028] Figure 1 This represents the test accuracy of different training architectures on the CIFAR-10 dataset.

[0029] Figure 2 These are the loss function values ​​for different training architectures in the CIFAR-10 dataset;

[0030] Figure 3 This is a flowchart of the hybrid segmentation and federated learning training process;

[0031] Figure 4 This is a comparison chart of the training effects with different ρ values. Detailed Implementation

[0032] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0033] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0034] Machine learning aims to enable computers to automatically learn patterns and rules, thereby making predictions or decisions based on unknown data. For example, a learning task like predicting house prices can be accomplished by constructing a polynomial (i.e., mathematically similar to Ax). n +Bx n The model uses a polynomial form (e.g., -1…+Cx+D) (this is a machine learning model; different tasks may have different model settings, increasing complexity, but overall it can be viewed as a polynomial). However, since directly conceiving the coefficients is difficult, computers predict the current house price using given feature values ​​such as room area, room location, and room floor (forward propagation: a forward calculation process). The difference is then calculated by comparing this prediction with the actual house price data, using a formula such as E = (yt). 2 The loss function is calculated using the formula, and the loss function value is obtained. The derivative of the loss function with respect to y is 2(yt). Therefore, the gradient value is calculated using the predicted and actual values, and is used to update the model's coefficients and other parameters (backpropagation: the difference between the predicted and actual values ​​obtained through forward calculation is compared, and the model is updated in reverse by taking the derivative). One forward propagation and one backpropagation can be considered as one round.

[0035] Deep learning: Machine learning assumes only one polynomial, but of course there can be many polynomials. Each polynomial is programmed with different parameters to produce multiple results in parallel, and these results are then fed into the next individual polynomial. This can form a two-layer machine learning model. Of course, through this reuse, multi-layer models can be formed. The meaning of deep learning is to build more layers on the basis of machine learning, which can be used to solve more complex problems. It is a subset of machine learning.

[0036] Loss function: A formula used to measure the difference between the predicted value and the actual value of a machine learning model.

[0037] Federated Learning (FL) is a model where the server receiving the training data has all the data on its local machine. However, some data may involve privacy concerns, or data on certain terminals may be updated rapidly. Transmitting all data to the server would place a heavy communication load on the network. Therefore, distributed machine learning was proposed, where the model is distributed to the terminal for training. After training, the terminal uploads the model back to the server. The server then only needs to perform model aggregation (such as parameter averaging). This preserves privacy and reduces communication load to some extent.

[0038] The learning steps are as follows:

[0039] 1. At the beginning of each global round (one round is defined as the server aggregating the model once), the terminal downloads the latest machine learning model from the server;

[0040] 2. The terminal uses its own data for training (forward propagation, backward propagation). The terminal may perform forward and backward propagation locally multiple times, which is called multiple local rounds (one forward and one backward propagation).

[0041] 3. The terminal uploads the model updated in the current global round to the server;

[0042] 4. The server aggregates the models for the next global training round;

[0043] This process is parallel across terminals, with multiple terminals simultaneously downloading, training, and uploading the model.

[0044] Split Learning (SL) Because federated learning requires each terminal to train the model completely once (a complete forward and backward propagation), it places certain demands on the terminal's computing power (otherwise, some terminals will train too slowly, slowing down the entire training process). Some terminals have a lot of high-quality data, and training them independently would be difficult. Therefore, split learning was proposed, which involves splitting a complete model into two parts. For example, suppose the current deep learning model consists of four fully connected layers (a set of multiple polynomials, each layer has multiple polynomials, and the result of each polynomial calculation is used as the input data for the next layer's polynomial; the first layer uses basic information such as house prices as input data). The complete model has four layers. In split learning mode, if the third layer is used as the splitting layer of the current model, then layers 1 to 3 will be trained on the client side (training means forward and backward propagation), and the fourth layer will be calculated on the server. This splits the complete computational load into two (or even more) parts, which can reduce the terminal's computation time and increase speed (because servers generally have strong computing power). However, because the terminal needs to send a copy of the splitting layer's calculation result to the server for each piece of data, the communication load will increase.

[0045] The learning steps are as follows:

[0046] 1. At the beginning of each global round, the terminal downloads the latest terminal-side machine learning model from the server;

[0047] 2. The terminal uses its own data for forward propagation;

[0048] 3. The terminal uploads the calculation results of the segmentation layer (such as the calculation results of the polynomial in the third layer above) to the server;

[0049] 4. The server uses the obtained calculation results as input to the server-side model, performs forward propagation to calculate the results, and calculates the loss function value;

[0050] 5. Then, backpropagation of the server-side model is performed to update the server-side model and obtain the gradient value (this gradient extends all the way to the next layer of the segmentation layer. In the case above, the server only needs to calculate the gradient of the fourth layer, and the rest are calculated by the terminal). This gradient will be transmitted to the terminal side.

[0051] 6. After the terminal receives the gradient, it will continue the backpropagation of the terminal model to update the terminal model.

[0052] 7. The terminal uploads the updated model to the server;

[0053] 8. The server saves the terminal-side model until the next terminal downloads it;

[0054] This process is sequential across different terminals; once one terminal completes its training, the next terminal begins its training.

[0055] HSFL (Hybrid Split and Federated Learning): The terminal involved in training is divided into two parts, one part participates in federated learning and the other part participates in split learning, so it is hybrid.

[0056] like Figure 3 As shown, the left side of the flowchart represents the process of terminals participating in federated learning. Each terminal first downloads the global model, then performs local forward and backward propagation to obtain updated model training data. At the end of each round, the local model is uploaded. This operation is processed in parallel among different terminals participating in federated learning.

[0057] The right side of the flowchart shows the process of terminal participation in segmentation learning. The first terminal downloads its terminal-side model and then trains it. The training process includes: 1. Forwarding the terminal-side model, 2. Uploading activation values, 3. Forwarding the server-side model, 4. Backwarding the server-side model, 5. Downloading gradient values, and 6. Backwarding the terminal-side model. After training, the terminal-side model is uploaded to the server, and the next terminal continues to execute the above steps until all terminals have participated in the training sequentially.

[0058] Segmentation layer: From Figure 3 You can see the dotted line. Figure 3 The complete four-layer model on the right is split into upper and lower parts. The third layer serves as the splitting layer of the current model. Terminals 3 and 4 will train up to the third layer locally, uploading the activation values ​​of the third layer as output to the server. The server, upon receiving the activation values, uses them as input to the fourth layer and continues training. Backpropagation is a similar operation; the server only performs backpropagation on the fourth layer, transmitting the gradient values ​​obtained from the backpropagation to terminals 3 and 4. Terminals c and d then use these gradients to perform backpropagation calculations on the gradients of the first three layers. Segmentation learning means dividing the model into multiple parts, trained by different terminals. Note that the forward and backward propagation training of terminals 3 and 4 above is sequential, not simultaneous. Figure 3 The FL section on the left is parallel.

[0059] Wireless distributed learning includes K terminals (terminals can be mobile phones, computers, or various mobile devices) and one edge server (edge ​​server: a high-performance computer). k = 1, ..., K represents the k-th terminal. The dataset of the k-th terminal is generated by D. k It consists of data samples, where the i-th data sample is represented as (x represents sample feature values, such as room area and floor information needed for house price prediction; y represents the model label, which is the value to be predicted, such as the actual house price of a specific room.) The goal of distributed learning is for the terminal to find the optimal global model ω through multiple rounds of data or model interaction with the server. * Minimize the global loss function L(ω), that is:

[0060]

[0061] In the formula, the global loss function L(ω) can be expressed as the single-sample loss function. Average form:

[0062]

[0063] Typically, the optimal global model ω can be obtained through gradient descent. * Therefore, in the (t+1)th communication round, the federated learning terminal updates as follows:

[0064] ω k，t+1 =ω t -ηg k，t (ω t (3)

[0065] In the formula, η represents the learning rate (learning rate: the magnitude of each update in gradient descent), and the segmentation learning terminal updates as follows:

[0066]

[0067] In the formula, It is the number of users segmented for learning in the t-th communication round, where It is the set of all users, x k,t It is the training model for the kth user, x k,t =1 indicates participation in federated learning, x k,t =0 indicates participation in segmentation learning. Because segmentation learning is serialized, subsequent user model parameter updates are based on the updates from the previous user.

[0068] For FL terminals, the latency per round (latency: delay caused by time expenditure, meaning the time spent) is...

[0069]

[0070] in It represents the number of users segmented for learning in the t-th communication round. This refers to the latency of downloading the global model (the complete machine learning model) from the federated learning terminal (FL model download). It refers to the model training latency (FL model training) of the federated learning terminal. This is the model upload latency (FL model upload) of the federated learning terminal. Because federated learning involves parallel downloading, computation, and uploading, the total latency is the maximum value.

[0071] For SL terminals, the latency per round is:

[0072]

[0073] in This is the global model download latency (SL model download) of the segmentation learning terminal, which includes the download of the terminal-side model for segmentation learning and the download of gradients back. This refers to the model training latency (SLmodel training) of the segmentation learning terminal, which includes the training latency of the terminal side (server-side model: if the model has a total of L layers, with l as the segmentation layer, then [l+1,L] belongs to the server training, so this part is called the server-side model; terminal-side model: the remaining [1,l] is the terminal-side model) and the server-side model training latency. It is the model upload latency (SL modelupload) of the segmentation learning terminal, which includes the upload of the terminal-side model and the upload of activation values ​​(activation values: the output of the current layer of the machine learning model). Since segmentation learning is executed serially, the total latency is the sum of the values.

[0074] Therefore, the latency for a single training round of hybrid segmentation and federated learning is:

[0075]

[0076] Assuming l-smooth, μ-strongly convex and Under the assumption that three assumptions hold true (these three assumptions are common in machine learning analysis: l-smooth means that above any given function, there can always be a quadratic function whose value is greater than this function in any domain, defining the upper bound of the quadratic function and the maximum rate of change of the gradient, where l is the coefficient of this quadratic function; μ-strongly convex, on the other hand, means that below any given function, there can always be a quadratic function whose value is less than this function in any domain, defining the lower bound of the quadratic function and the minimum rate of change of the gradient, where μ is the coefficient of this quadratic function; the third assumption means that the gradient calculated in each update will not be too large, that is, the update will not cause the model parameters to change too much, and the model is relatively stable), we can obtain the upper bound of convergence for hybrid segmentation and federated learning (representing the difference between the final convergence effect of the model and the theoretical optimal model parameters, which usually means that the machine learning model will make 100% accurate predictions every time, which is the ideal and cannot occur in real life).

[0077]

[0078] in Φ is a constant representing the heterogeneity of data between terminals (usually representing the distribution of data tags; the greater the difference in distribution, the higher the heterogeneity). If the terminal data is independent and identically distributed (meaning that the data tag distribution of each terminal can be considered to be exactly the same), then this value will tend to 0.

[0079] As can be seen from the convergence upper bound above, the convergence upper bound decreases with the increase in the number of users participating in segmentation learning. This reflects that more users participating in segmentation learning can lead to better convergence results (the difference between the trained model parameters and the optimal model parameters is smaller, and closer to the ideal situation). Because segmentation learning is serialized, more users participating in segmentation learning can bring more convergence times to the overall model, resulting in better training performance. However, because segmentation learning is serialized, more terminals participating in segmentation learning usually lead to greater latency. To achieve a better convergence result faster, it is necessary to choose the number of terminals for segmentation learning to balance the training effect and latency of a single round.

[0080] To balance the relationship between the two, an optimization objective function is proposed:

[0081]

[0082] Here, ρ is a hyperparameter, and different values ​​of ρ will lead to different training effects. The objective function aims to continuously reduce its value. There are two approaches: 1. Reduce the training latency per round (faster training per round); 2. Increase the number of single-round training users (better training effect per round). Both will improve the training effect per round, so the goal is to find the minimum value of the objective function, ensuring optimal training performance in each round. Subsequent optimizations of the Gibbs algorithm follow this principle, minimizing the objective function.

[0083] like Figure 4 The image shows a comparison of the impact of changing only the weight hyperparameters of the objective function on training performance when the joint algorithm is in effect. This time delay was statistically obtained under the premise that the same training accuracy is achieved in each training iteration. Figure 4 It can be seen that the hyperparameter ρ has a significant impact on training latency and user selection. The training latency per round increases monotonically with the increase of the ρ value (as shown in the bar chart in Figure 4). Figure 4 The horizontal axis in the diagram is the reciprocal of ρ (i.e., ρ increases from right to left). This aligns with the objective function, which is characterized by focusing more on the number of SL users in a single round and downplaying the training latency in a single round as the value of ρ increases. However, more SL users, due to their serial nature, will lead to more training latency. In other words, the training latency in a single round and the number of SL users in a single round generally show the same trend.

[0084] Given a large ρ, increasing the number of SL users will result in a greater gain in reducing the objective function value. Conversely, given a small ρ, reducing the single-round latency will result in a greater gain in reducing the objective function value. Therefore, during the continuous optimization process, we tend to optimize this parameter more.

[0085] Conclusion: The overall training latency exhibits a trend of first decreasing and then increasing (not monotonic). Therefore, achieving the same training speed faster is not necessarily better with more segmentation learning users. Once the number of segmentation learning users reaches a certain threshold, further increasing the number may increase training latency and worsen the results. Therefore, optimizing the user training mode is necessary. Furthermore, the number of segmentation learning users will vary under different data environments, requiring a choice of user training mode for each training round.

[0086] from Figure 4 The optimal point can be found at ρ = 0.002.

[0087] When choosing the terminal learning mode, the selection of the segmentation layer is necessary. This doesn't affect the model's convergence performance, but only the latency (because different segmentation layers result in different training layers on the terminal and server sides, and their computational capabilities differ, affecting the latency of a single training round). Additionally, the number of activation values ​​in each layer differs, leading to varying communication data volumes and thus different single-round communication latencies since all activation values ​​and their corresponding gradients need to be transmitted. In other words, the choice of segmentation layer significantly impacts training latency (due to changes in the communication and computational capabilities of each terminal in different rounds, a choice must be made in each round). Regardless of the segmentation layer chosen, the overall training is complete and doesn't affect the training effect. We need to compare the cost of each layer to select the optimal solution. Since directly optimizing these two problems is complex, a multi-stage algorithm is proposed for the joint optimization of these two problems:

[0088] 1. The first step randomly initializes the learning mode of each terminal and divides them into two clusters: one for users participating in federated learning and the other for users participating in segmentation learning. If the total number of users is 10, then each cluster is an array of length 10 (an array is a structure that can store multiple values, such as an array of length 3 represented as [0,0,0], where each value is 0). All users are numbered from 1 to 10. If the current user k participates in federated learning, then the value of this user at the k-th position in the federated learning user cluster is 1, and the value of the k-th position in the segmentation learning user cluster is 0, and vice versa. Therefore, the number of 1s in each cluster represents the number of users participating in this learning mode.

[0089] 2. For each user participating in segmentation learning:

[0090] 2.1 Calculate the computational cost of forward and backward propagation for each layer (calculation method: can be calculated using the torchstat package in Python), and calculate the computational latency of each layer on different terminals by dividing by the current terminal's computational capability and the server's computational capability.

[0091] 2.2 Calculate the output data volume generated by each layer. This can be obtained using tools such as the `torchstat` package in Python. This output data volume is generated for each data point (e.g., individual house price information). Therefore, the data volume generated by the current layer at the current terminal needs to be multiplied by the total data volume of the current terminal, and then multiplied by 2, because each layer, in addition to uploading the output data volume, also needs to download the corresponding backpropagation gradient value. Then, divide by the communication rate of the current terminal to calculate the required communication latency for each layer as a segmentation layer.

[0092] 2.3 Iteratively select each layer l as the segmentation layer, calculate the terminal-side computation latency of [0, l], add the server-side computation latency of [l+1, L], and add the communication latency generated by layer l (because l is the segmentation layer, the preceding layer l is calculated on the terminal side, and the subsequent layers l+1 to the total number of layers L of the model are calculated on the server side). Select the layer with the minimum latency among all layers as the segmentation layer as the segmentation layer for the current user.

[0093] 3. From the above calculations, we can obtain the computation and communication latency for each user participating in segmentation learning in a single round (here, the communication latency includes not only the data generated by the segmentation layer but also the communication latency caused by the upload and download of data from the user layer model). Summing these two amounts gives the training latency for each user participating in segmentation learning. Since segmentation learning is serialized (serialization: after one user finishes training, the next user begins training), the training latency of segmentation learning is the sum of the latency of all users participating in segmentation learning.

[0094] 3.1 Using a similar method as above, calculate the computation (computation latency of training the complete model locally on the terminal) and communication latency (latency of downloading and uploading the complete model) for each user participating in federated learning. Summing these two amounts yields the training latency for each user. Because federated learning is parallelized (parallelization: all users participating in federated learning train simultaneously), the federated learning latency is the maximum value of the latencies of all users participating in federated learning.

[0095] 3.2, through The total latency of a single training epoch can be calculated. Because the learning between the two clusters, segmentation learning and federated learning, is parallel, the latency of a single training epoch is the maximum of the two.

[0096] 3.3, At this point, it appeared The number of users participating in segmentation learning has also increased. We can calculate the objective function u(t), where we set ρ = 0.002. Figure 4 The optimal value is selected by comparison.

[0097] 4. In the initial learning mode, the learning mode is changed using the Gibbs sampling algorithm. The process is as follows:

[0098] 4.1. Randomly select one user from all users and swap the values ​​of this user at the same position in different clusters. For example, if a second user is selected, swap the value at the second position in the federated learning cluster with the value at the second position in the segmentation learning cluster. If the user's value at the corresponding position in the federated learning cluster is 1, and the corresponding values ​​in the segmentation learning cluster are all 0, then the number of users participating in segmentation learning will increase by 1 after the swap (this conclusion can be derived from the definition of corresponding positions in the two clusters above). Of course, it is also possible that the opposite occurs, with the number of users participating in segmentation learning increasing by 1. If the user's value is the same in both clusters, the swap is invalid, and a new user is selected.

[0099] 4.2 Because the user distribution has changed, we can obtain a new objective function value through step two.

[0100] 4.3 Substituting the two objective function values ​​into equation (10) yields a value ∈, which represents the probability that the update will be adopted. Random numbers will be generated. If the random number is greater than this value, the update will be adopted; if the random number is less than this value, the update will not be adopted.

[0101] 4.4 If the new objective function value increases, it means that the current choice will lead to a relatively poor result: either the time per round increases (training becomes slower), or the number of users learning per round decreases (training effect deteriorates). Either of these will worsen the training effect, so the current update should be avoided as much as possible. In equation (10), if the new objective value is greater than the old objective value, the exponent of e will be a positive value. The larger the value, the larger the exponent, and the larger the denominator. The larger the value, the smaller the probability value, and therefore the smaller the probability of updating.

[0102] 4.5 Conversely, if a better result is obtained, the update probability will increase.

[0103] 4.7 Since a relatively poor result may lead to a better result the next time, the strategy of not updating will not be used directly to avoid getting trapped in a local optimum.

[0104] As this update continues, it will continuously move towards a better solution (achieving better training results in a shorter time), eventually arriving at a theoretically optimal solution for the current global round.

[0105]

[0106] In the above formula, e is the natural logarithm. and u t Let represent the objective function values ​​calculated using the new strategy and the old strategy, respectively. δ is a hyperparameter used to control the magnitude of the update probability; here we set δ to a relatively small value of 0.00075. This is because when δ is small, regardless of the numerator... Is it a positive or negative number? Both will produce a relatively large absolute value, so if A positive number indicates a larger new objective function value, which is a poor strategy (this conclusion can be drawn from optimization algorithm 4.4 above). The calculation result leads to a large overall denominator in equation (10), meaning the overall update probability value is very small, tending towards 0, in order to maintain the current learning pattern. Conversely, if... A negative number indicates that the new objective function is small, which is a better strategy. Calculations show that this leads to a large overall denominator in equation (10), meaning the update probability value ∈ [0, 1] is large overall, tending towards 1, in order to adopt the new strategy. Conversely, if δ is relatively large, then... The absolute value of δ will be relatively small, and regardless of whether it is positive or negative, it will tend to 0. This will cause the denominator of equation (10) to tend to 2 (assuming that in the extreme case δ tends to infinity, then it will lead to...). The denominator is 1 + e, which equals 0. 0 As shown in 2), the probability of each update tends to be 0.5, making it difficult to update to a good result.

[0107] Figure 1 This represents the test accuracy of different training architectures on the CIFAR-10 dataset. Figure 2 These are the loss function values ​​for different training architectures on the CIFAR-10 dataset; and a comparison chart showing the training effects of hybrid segmentation and federated learning under different optimization algorithms compared to ordinary hybrid segmentation and federated learning without optimization, segmentation learning, and federated learning. From... Figure 1 and Figure 2 As can be seen, ordinary HSFL can bring certain gains to training, and if the selection of training mode is added, it will bring more significant gains to the training effect. If the selection of terminal training mode is added, the training effect can be seen to be optimal. Therefore, the architecture proposed in this invention has a very significant advantage under the gain of the optimization algorithm.

[0108] This invention can be used to accelerate machine learning scenarios with privacy concerns, such as hospital medical record diagnostic models. By training a machine learning model, rapid pre-diagnosis of patient symptoms can be achieved, improving medical efficiency. Since patient information in hospitals is private and cannot be centrally hosted on a single server, a distributed learning approach can be considered. This invention can accelerate training, helping machine learning models to be deployed more quickly. It can also be applied to other scenarios where uploading local data is inconvenient.

[0109] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A hybrid multi-learning mode selection method for collaborative computing, characterized in that, Includes the following steps: Step 1: The terminal's learning modes are divided into federated learning and segmented learning; Step 2: Select the optimal segmentation layer; Step 3: Change the terminal learning mode. Continue with step 2 and select the optimal terminal learning mode. Each round of training involves selecting the terminal learning mode; The learning mode of the terminal is selected based on the training effect and training latency; The objective function for selecting the terminal learning mode is: in, It's a hyperparameter. It's training latency. It is the first The number of learning terminals segmented in each communication round; The terminal learning mode selection optimization is introduced: terminals with weak computing power choose to participate in segmentation learning to reduce local computing load, while terminals with strong computing power participate in federated learning to reduce communication load. The latency for a single training round of hybrid segmentation and federated learning is: The training latency of segmentation learning is the sum of the latencies of all users participating in segmentation learning. ; Federated learning latency is the maximum latency of all users participating in federated learning. .

2. The hybrid multi-learning mode selection method for collaborative computing as described in claim 1, characterized in that, = 0.002。 3. The hybrid multi-learning mode selection method for collaborative computing as described in claim 1, characterized in that, Minimize the objective function.

4. The hybrid multi-learning mode selection method for collaborative computing as described in claim 1, characterized in that, In step 1, the learning mode of the terminal is randomly initialized, and divided into user clusters participating in federated learning and user clusters participating in segmented learning.

5. The hybrid multi-learning mode selection method for collaborative computing as described in claim 1, characterized in that, In step 2, the latency of the terminal selecting different segmentation layers is calculated, and the optimal segmentation layer is selected from them.

6. The hybrid multi-learning mode selection method for collaborative computing as described in claim 1, characterized in that, Step 3 involves changing the terminal learning mode using the Gibbs sampling algorithm.

7. The hybrid multi-learning mode selection method for collaborative computing as described in claim 1, characterized in that, The goal of the terminal learning is to minimize the global loss function.

Images