Distributed training method, device and equipment based on mobile chip cluster server

By dividing logical groups on the mobile chip cluster server and optimizing logical group mapping and communication, the problems of low training energy efficiency and performance in existing training methods are solved, and efficient model training and performance improvement are achieved.

CN118193221BActive Publication Date: 2024-12-03北京泛睿云图科技有限公司
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410409781.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-07
Publication Date
2024-12-03
Estimated Expiration
2044-04-07

AI Technical Summary

Technical Problem

The existing training methods based on mobile chip cluster servers have low training efficiency and performance, and cannot fully utilize the computing density and energy efficiency of mobile chip cluster servers.

Method used

By dividing the mobile chips in the mobile chip cluster server into multiple logical groups and assigning the training data set and the model to be trained to these logical groups, multiple mobile chips in each logical group are simultaneously trained, using minimization The number of logical groups for communication between physical groups is the goal. The logical groups are mapped to the physical group and divided into logical group sets according to the communication competition between logical groups, and the logical group sets are executed sequentially to achieve synchronous training results.

Benefits of technology

Parallel distributed training on mobile chip cluster servers is realized, which improves the energy efficiency and performance of model training, ensures training accuracy while reducing training time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118193221B_ABST
    Figure CN118193221B_ABST
Patent Text Reader

Abstract

The present application provides a distributed training method, device and equipment based on a mobile chip cluster server, the method comprising: dividing the mobile chips in the mobile chip cluster server into multiple logical groups, and assigning the training data set and the model to be trained to the multiple logical groups, and dividing the mobile chips in the mobile chip cluster server into multiple physical groups according to the connection mode; with the goal of minimizing the number of logical groups for inter-physical group communication, mapping multiple logical groups to multiple physical groups to obtain multiple mapped logical groups; dividing the mapped multiple logical groups into multiple logical group sets according to whether there is communication competition between the logical groups; sequentially executing multiple logical group sets, and synchronizing the execution results of the multiple logical group sets to obtain the training results of this round. In this way, the mobile chip cluster server is efficiently utilized to give full play to the capabilities of the mobile chip cluster server and improve the training energy efficiency and performance of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a distributed training method, device and equipment based on a mobile chip cluster server. Background Art

[0002] Mobile chip cluster servers are a new type of server composed of large-scale mobile chips. Compared with traditional servers deployed in cloud / edge data centers, mobile chip cluster servers can provide higher computing density and higher energy efficiency. In addition, mobile chip cluster servers can run local mobile applications without any modification, so a large number of mobile chip cluster servers are deployed in the edge cloud to provide mobile applications (for example, cloud games and real-time streaming) and conduct deep learning training based on mobile chip cluster servers.

[0003] However, the existing training methods based on mobile chip cluster servers have low training energy efficiency and performance. Therefore, there is an urgent need for a training method that can efficiently utilize mobile chip cluster servers to give full play to the capabilities of mobile chip cluster servers to improve training energy efficiency and performance. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a distributed training method, apparatus and device based on a mobile chip cluster server to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a distributed training method based on a mobile chip cluster server is disclosed, the method comprising:

[0006] The mobile chips in the mobile chip cluster server are divided into multiple logical groups, and the training data sets and the models to be trained are allocated to the multiple logical groups, and the multiple mobile chips in each logical group perform model training at the same time. The mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode;

[0007] With the goal of minimizing the number of logical groups that perform inter-physical group communication, the plurality of logical groups are mapped to the plurality of physical groups to obtain a plurality of mapped logical groups;

[0008] According to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, and there is no communication competition between the multiple logical groups in each logical group set;

[0009] The multiple logic group sets are executed sequentially, and the execution results of the multiple logic group sets are synchronized to obtain the training results of this round.

[0010] Optionally, the mobile chips in the mobile chip cluster server are divided into a plurality of logical groups, including:

[0011] Initially grouping the mobile chips in the mobile chip cluster server in order of group size from small to large, and determining the training accuracy of the initial grouping corresponding to each group size;

[0012] Determine the group size of the initial grouping whose training accuracy meets the accuracy threshold as the target group size;

[0013] According to the target group size, the mobile chips in the mobile chip cluster server are divided into a plurality of logical groups.

[0014] Optionally, with the goal of minimizing the number of logical groups that perform inter-physical group communication, the plurality of logical groups are mapped to the plurality of physical groups to obtain the mapped plurality of logical groups, including:

[0015] In a mapping manner that fully satisfies mapping of multiple mobile chips in a logical group to the same physical group, a first part of the logical groups in the multiple logical groups are mapped to the multiple physical groups to obtain a first mapping result;

[0016] In a manner of sequentially mapping the plurality of mobile chips in the logical group, mapping a second portion of the logical groups except the first portion of the logical groups in the plurality of logical groups to the plurality of physical groups to obtain a second mapping result;

[0017] A plurality of mapped logical groups are obtained according to the first mapping result and the second mapping result, and each of the plurality of mapped logical groups has communication competition with at most two logical groups.

[0018] Optionally, the plurality of logical group sets include a first logical group set and a second logical group set;

[0019] According to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, including:

[0020] Draw a bipartite graph according to the mapped multiple logical groups, where the nodes in the bipartite graph represent the logical groups, and the edges in the bipartite graph represent whether there is communication competition between the logical groups;

[0021] The bipartite graph is solved according to a depth-first search method to obtain the first logic group set and the second logic set.

[0022] Optionally, sequentially executing the multiple sets of logic groups includes:

[0023] Calling the first logical group set to perform calculation;

[0024] After the first logical group set is calculated, the first logical group set communicates, and calls the second logical group set to calculate, and the calculation time of the second logical group set is greater than the communication time of the first logical group set.

[0025] Optionally, the plurality of mobile chips in the logic group include: a heterogeneous processor and a central processing unit, the heterogeneous processor performs calculations based on an integer operation model, and the central processing unit performs calculations based on a single-precision floating-point number model;

[0026] Calling the first logic group set to perform calculation and calling the second logic group set to perform calculation are performed according to the following steps:

[0027] Scheduling part of the training data in the training data set to the heterogeneous processor for calculation according to a calculation power ratio of the calculation capability ratio of the heterogeneous processor;

[0028] According to the confidence of the error gap between the integer operation model and the single-precision floating-point model and the computing power ratio, the training data control value of the central processing unit is determined, and part of the training data in the training data set is scheduled to the central processing unit for calculation according to the training data control value.

[0029] Optionally, allocating the training data set and the to-be-trained models to the multiple logical groups includes:

[0030] Dividing the training data set into a plurality of sub-training data sets, the number of the sub-training data sets being the same as the number of the plurality of logical groups;

[0031] The multiple sub-training data sets and the models to be trained are allocated to the multiple logical groups, each logical group corresponds to a sub-training data set and the model to be trained.

[0032] Optionally, the time of each round of training is negatively correlated with the number of the multiple logical groups, and the training accuracy of the model to be trained is negatively correlated with the number of the multiple logical groups.

[0033] A second aspect of the embodiment of the present application discloses a distributed training device based on a mobile chip cluster server, the device comprising:

[0034] A first division module is used to divide the mobile chips in the mobile chip cluster server into multiple logical groups, and allocate the training data set and the model to be trained to the multiple logical groups, and the multiple mobile chips in each logical group perform model training at the same time, and the mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode;

[0035] A first mapping module, configured to map the plurality of logical groups to the plurality of physical groups with the goal of minimizing the number of logical groups performing inter-physical group communication, to obtain a plurality of mapped logical groups;

[0036] A second division module is used to divide the mapped multiple logical groups into multiple logical group sets according to whether there is communication competition between the logical groups, and there is no communication competition between the multiple logical groups in each logical group set;

[0037] The first execution module is used to sequentially execute the multiple logic group sets and synchronize the execution results of the multiple logic group sets to obtain the training results of this round.

[0038] A third aspect of an embodiment of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the distributed training method based on the mobile chip cluster server described in the first aspect of the embodiment of the present application are implemented.

[0039] The embodiments of the present application include the following advantages:

[0040] In an embodiment of the present application, the mobile chips in the mobile chip cluster server are divided into multiple logical groups, and the training data set and the model to be trained are assigned to the multiple logical groups, the multiple mobile chips in each logical group perform model training at the same time, and the mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection method; and with the goal of minimizing the number of logical groups that communicate between physical groups, the multiple logical groups are mapped to the multiple physical groups to obtain multiple mapped logical groups; then, according to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, and there is no communication competition between the multiple logical groups in each logical group set; finally, the multiple logical group sets are executed sequentially, and the execution results of the multiple logical group sets are synchronized to obtain the training results of this round.

[0041] Since there is no communication competition between multiple logical groups in a logical group set, and multiple mobile chips in a logical group can perform model training at the same time, executing multiple logical group sets in sequence can achieve parallel distributed training on the mobile chip cluster server, that is, fast model training can be achieved under the cooperation of multiple mobile chips without affecting accuracy. In this way, the mobile chip cluster server is efficiently utilized to give full play to the capabilities of the mobile chip cluster server and improve the training energy efficiency and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 It is a flow chart of the steps of a distributed training method based on a mobile chip cluster server provided in an embodiment of the present application;

[0044] Figure 2 This is a schematic diagram of mapping a logical group to a physical group provided in an embodiment of the present application;

[0045] Figure 3 It is a structural diagram of a distributed training system based on a mobile chip cluster server provided in an embodiment of the present application;

[0046] Figure 4 It is a structural diagram of a distributed training device based on a mobile chip cluster server provided in an embodiment of the present application;

[0047] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The reasons for conducting deep learning training on mobile chip cluster servers are: 1) Mobile chip cluster servers are usually deployed on the edge cloud, where mobile user data is aggregated. Compared with training models in centralized data centers, it reduces the network routing between data and training infrastructure, thereby reducing the network pressure on the backbone network. In addition, training on the edge cloud also alleviates privacy issues by using data early. It also allows model training to be personalized geographically, for example, item recommendation tasks. 2) Deep learning training is a relatively predictable and latency-tolerant workload. For example, an input method prediction model can be updated regularly (e.g., once a month) in the edge cloud in each region and sent to the mobile client the next morning. 3) Mobile chips usually contain accelerators (e.g., NPUs), which can achieve significant performance improvements in performing deep learning tasks. At the same time, the algorithm foundation for low-precision training adapted to the data format of mobile chip accelerators has also been established, especially for relatively small models and simple computer vision tasks in the edge cloud.

[0050] However, the existing training methods based on mobile chip cluster servers have low training efficiency and performance. To this end, an embodiment of the present application provides a distributed training method based on a mobile chip cluster server to give full play to the capabilities of the mobile chip cluster server to improve training efficiency and performance.

[0051] The present application embodiment provides a distributed training method based on a mobile chip cluster server, referring to Figure 1 As shown, Figure 1 1 is a flow chart of the steps of a distributed training method based on a mobile chip cluster server provided in an embodiment of the present application. Figure 1 As shown, the method may include steps S110 to S140:

[0052] Step S110: Divide the mobile chips in the mobile chip cluster server into multiple logical groups, and assign training data sets and models to be trained to the multiple logical groups. Multiple mobile chips in each logical group perform model training simultaneously. The mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode.

[0053] In an embodiment of the present application, a mobile chip cluster server refers to a computing resource composed of multiple mobile chips. The mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection method. That is, in the mobile chip cluster server, mobile chips with a physical connection relationship are regarded as a physical group. For example, there are 10 mobile chips connected together in the mobile chip cluster server, and these 10 mobile chips connected together are regarded as a physical group.

[0054] A logical group refers to multiple mobile chips that are logically grouped together for model training at the same time. The mobile chips in the mobile chip cluster server are divided into multiple logical groups, specifically including: dividing several chips in the mobile chip cluster together for model training at the same time according to the model training rate, model training accuracy, and training tasks, and treating multiple mobile chips that are trained at the same time as a logical group. For example, if mobile chip 1, mobile chip 2, mobile chip 3, and mobile chip 4 in the mobile chip cluster server are grouped together for training at the same time, then mobile chip 1, mobile chip 2, mobile chip 3, and mobile chip 4 form a logical group.

[0055] After dividing the mobile chips in the mobile chip cluster server into multiple logical groups, the training data set and the model to be trained are allocated to the multiple logical groups, each logical group is allocated part of the training data and the model to be trained in the training data set, and then each logical group performs model training on the model to be trained according to the allocated part of the training data to obtain the model training result of the logical group.

[0056] In an optional embodiment, the time of each round of training is negatively correlated with the number of the multiple logical groups, and the training accuracy of the model to be trained is negatively correlated with the number of the multiple logical groups.

[0057] Specifically, assume that the M mobile chips in the mobile chip cluster server are divided into N logical groups, and the global batch size of each group is BS g , the time for each round of training is expressed as:

[0058]

[0059] Among them, NUM sample Represents the amount of data in the training dataset; represents the calculation time; T sync Data synchronization time, T sync Including communication time within the logical group and communication time between logical groups; M represents the number of mobile chips in the mobile chip cluster server, and the value of M is greater than or equal to 1; N represents the number of logical groups, and the value of N is less than or equal to M.

[0060] Since the computation time and synchronization time are constant, according to the above formula (1), it can be seen that the time for each round of training is T epoch It is negatively correlated with the number of logical groups N, and the training accuracy of the model to be trained is negatively correlated with the number of logical groups N. Because increasing the batch size often has an adverse effect on the convergence accuracy, in order to determine the maximum number of logical groups that guarantees the minimum training time while maintaining the convergence accuracy, considering that the behavior of the training accuracy in the preparation stage of model training is closely similar to that of the convergence accuracy, it is proposed to divide the logical groups based on the training accuracy in the initial stage.

[0061] Specifically, the mobile chips in the mobile chip cluster server are divided into multiple logical groups, including steps A1 to A3:

[0062] Step A1: Initially grouping the mobile chips in the mobile chip cluster server in order of group size from small to large, and determining the training accuracy of the initial grouping corresponding to each group size.

[0063] Step A2: The group size of the initial grouping whose training accuracy meets the accuracy threshold is determined as the target group size.

[0064] Step A3: Divide the mobile chips in the mobile chip cluster server into multiple logical groups according to the target group size.

[0065] In the embodiment of the present application, in the preparation stage of model training, the mobile chip cluster servers are initially grouped from a smaller group size to a larger group size, and the training accuracy is analyzed, and the training is stopped at the group size where the training accuracy is first found to be significantly reduced, that is, the group size of the initial grouping where the training accuracy meets the accuracy threshold is used as the target group size. For example, the accuracy threshold is 15%, and when it is found that the training accuracy is higher than 15%, the corresponding initial group size is determined as the target group size.

[0066] For example, a mobile chip cluster server composed of 60 mobile chips is used as an example to explain the division method of logical groups in detail. First, the initial grouping is performed with a group size of 1, and the mobile chip cluster server is divided into 60 logical groups, and the training accuracy corresponding to the initial grouping method is calculated; when the training accuracy does not meet the accuracy threshold, the initial grouping is performed with a group size of 2, and the mobile chip cluster server is divided into 30 logical groups, and the training accuracy corresponding to the grouping method is calculated. When the training accuracy meets the accuracy threshold, the target group size is determined to be 2, otherwise the initial grouping is performed with a group size of 3 until the training accuracy meets the accuracy threshold. Subsequently, the mobile chips in the mobile chip cluster server are divided according to the target group size to obtain multiple logical groups.

[0067] In an optional embodiment, allocating the training data set and the to-be-trained model to the plurality of logical groups includes steps B1 to B2:

[0068] Step B1: Divide the training data set into a plurality of sub-training data sets, where the number of the sub-training data sets is the same as the number of the plurality of logical groups.

[0069] Step B2: assigning the multiple sub-training data sets and the models to be trained to the multiple logical groups, each logical group corresponds to a sub-training data set and the model to be trained.

[0070] In an embodiment of the present application, each logical group uses part of the training data to train the model, so the training data set is divided into multiple sub-training data sets, and a sub-training data set is assigned to each logical group, so that the logical group subsequently trains the model to be trained based on the assigned sub-training data set.

[0071] Step S120: with the goal of minimizing the number of logical groups that perform inter-physical group communication, the plurality of logical groups are mapped to the plurality of physical groups to obtain a plurality of mapped logical groups.

[0072] In an embodiment of the present application, the mobile chips in the mobile chip cluster server are organized into different physical groups. The interaction between different physical groups requires the use of an external network card, and the logical group needs to be mapped to the physical group to implement subsequent model training. In the specific implementation, in order to facilitate intensive data exchange within the logical group without going through the external network card to minimize contention, the logical group is preferably mapped to the mobile chip in the same physical group. However, the sizes of logical groups and physical groups are usually unequal, and mixed partitioning is inevitably required, that is, the same logical group will be mapped to at least two physical groups. Therefore, the embodiment of the present application is based on a complete greedy mapping, with the goal of minimizing the number of logical groups that communicate between physical groups, and mapping multiple logical groups to multiple physical groups.

[0073] In an optional embodiment, with the goal of minimizing the number of logical groups that perform inter-physical group communication, the plurality of logical groups are mapped to the plurality of physical groups to obtain the mapped plurality of logical groups, including steps C1 to C3:

[0074] Step C1: Map the first part of the multiple logical groups to the multiple physical groups in a mapping manner that fully satisfies mapping multiple mobile chips in the logical group to the same physical group to obtain a first mapping result.

[0075] Step C2: Mapping the plurality of mobile chips in the logical group in sequence, mapping the second part of the logical groups except the first part of the logical groups in the plurality of logical groups to the plurality of physical groups to obtain a second mapping result.

[0076] Step C3: obtaining a plurality of mapped logical groups according to the first mapping result and the second mapping result, wherein each of the plurality of mapped logical groups has communication competition with at most two logical groups.

[0077] Specifically, the mapping problem is defined as follows: Assume there are K physical groups, each logical group contains M / N mobile chips, and the i-th physical group contains Si logical groups, denoted as L i ={L i,0 ,L i,1 ,…L i,Si}; If the number of chips in the j-th logical group of the i-th physical group is less than M / N, it is represented as |L i,j |<M / N. At this time, there must be at least one mobile chip in other physical groups, so inter-physical-group communication is required. Use to represent the logical group for inter-physical-group communication within the i-th physical group, specifically represented as:

[0078]

[0079] The logical group assistant for inter-physical-group communication among K physical groups is represented as:

[0080]

[0081] In specific implementation, as many logical groups as possible are mapped to the same physical group without splitting. First, execute step C1 to map each logical group in the first part of the logical group mapping to a physical group; then execute step C2 to sequentially map the remaining second part of the logical groups to physical groups, and at most two logical groups have communication competition after each mapping; finally, execute step C3 to obtain multiple mapped logical groups according to the first mapping result of step C1 and the second mapping result of step C2.

[0082] Exemplarily, Figure 2 is a mapping schematic diagram of mapping logical groups to physical groups provided by an embodiment of the present application. Taking the logical group size of 3 and the physical group size of 5 as an example, the mapping method of mapping logical groups to physical groups is described. First, every 3 mobile chips in logical groups 1 to 3 (i.e., the first part of logical groups) are placed in the first 3 mobile chips in the corresponding physical group. Secondly, the nodes of the remaining logical groups (i.e., the second part of logical groups) are mapped in sequence, that is, by continuously placing the mobile chips in a logical group in a physical group, 3 mobile chips in logical group 4 span the 1st and 2nd physical groups, and 3 mobile chips in logical group 5 span the 2nd physical group and the 3rd physical group.

[0083] Step S130: Divide the multiple mapped logical groups into multiple logical group sets according to whether there is communication competition between logical groups, and there is no communication competition between multiple logical groups in each logical group set.

[0084] In the embodiment of the present application, communication contention refers to the existence of network card contention between different logical groups. When the size of the logical group does not match the size of the physical group, communication contention between logical groups is inevitable. In order to alleviate communication contention, the implementation of the present application combines logical groups into different logical group sets. In the same logical group set, the intra-group synchronization of different logical groups is staggered, and there is no communication contention between different logical groups. After dividing the mapped multiple logical groups into multiple logical group sets, the logical group synchronizations in different logical group sets are communicated separately in sequence to avoid network contention.

[0085] For example, Figure 2 Logical groups 1 to 4 in the diagram form a logical group set, and logical group 5 is placed in another logical group set. This is because there is no physical inter-group communication between logical groups 1 to 3, so they can be placed anywhere. However, there is a physical inter-group communication conflict between logical groups 4 and 5, so logical groups 4 and 5 cannot be placed in the same logical group set.

[0086] In an optional embodiment, the multiple logical group sets include a first logical group set and a second logical group set; and according to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, including steps D1 to D2:

[0087] Step D1: Draw a bipartite graph according to the mapped multiple logical groups, where the nodes in the bipartite graph represent logical groups and the edges in the bipartite graph represent whether there is communication contention between the logical groups.

[0088] Step D2: Solve the bipartite graph according to a depth-first search method to obtain the first logic group set and the second logic set.

[0089] In the embodiment of the present application, each logical group in the mapped multiple logical groups has communication competition with at most two logical groups, thereby converting the logical group set partitioning problem into a minimum bipartite graph coloring problem, and drawing a bipartite graph based on the mapped multiple logical groups. In the bipartite graph, each node has at most two edges, so two feasible solutions can be obtained by solving the bipartite graph, namely, a first logical group set and a second logical group set, and there is no communication competition between the logical groups in the first logical group set and the second logical group set.

[0090] Step S140: sequentially execute the multiple logic group sets, and synchronize the execution results of the multiple logic group sets to obtain the training results of this round.

[0091] In the embodiment of the present application, there is communication competition among the logic groups in different logic group sets, so only the logic groups in one logic group set can be synchronized at a certain time, and the other logic group sets should wait until there is no communication competition. Therefore, in order to better plan the communication order of multiple logic group sets to minimize the idle time of the processor, multiple logic group sets are executed in a sequential execution manner.

[0092] Then, the execution results of the multiple logic group sets are synchronized to obtain the training results of this round, wherein the synchronization of the execution results of the multiple logic group sets includes data synchronization within the logic group and data synchronization between the logic groups. Since the logic group set contains multiple logic groups, sequentially executing multiple logic group sets can realize parallel calculation of multiple logic groups, thereby improving the training accuracy of the model.

[0093] In an optional embodiment, the plurality of logic group sets are sequentially executed, including step E1 to step E2:

[0094] Step E1: calling the first logic group set to perform calculation.

[0095] Step E2: After the calculation of the first logical group set is completed, the first logical group set communicates, and calls the second logical group set to calculate, and the calculation time of the second logical group set is greater than the communication time of the first logical group set.

[0096] In an embodiment of the present application, considering that the calculation time of a logic group set is greater than the communication time of the logic group set, in order to hide the communication time within the logic group sets executed sequentially, each logic group set first performs calculation and then communicates. Therefore, when the first logic group set communicates, the second logic group set starts to calculate, thereby hiding the communication time of the first logic group set, thereby improving the overall model training efficiency.

[0097] In an optional embodiment, the plurality of mobile chips in the logic group include: heterogeneous processors and a central processing unit, the heterogeneous processors perform calculations based on an integer operation model, and the central processing unit performs calculations based on a single-precision floating-point number model.

[0098] In an embodiment of the present application, in order to improve training energy efficiency and training performance, the mobile chip will rely on heterogeneous processors to use integer operation models (i.e., INT8 data format) to accelerate the training of deep neural networks. However, there are two main problems in algorithm design when using heterogeneous processors to accelerate training: 1) Quantizing a single-precision floating-point model (i.e., FP32) to INT8 on a heterogeneous processor will cause numerical errors, which may grow exponentially as training progresses. Therefore, when training the model, simply averaging the gradients from the central processing unit (i.e., CPU) and the heterogeneous processor may result in significant accuracy loss, so it is necessary to minimize the quantization error to ensure convergence accuracy. 2) Unlike distributed training scenarios in cloud data centers, mobile chips face a huge gap in training speed. Therefore, based on the above two problems, it is necessary to schedule the calculations of mobile chips within a logical group.

[0099] Specifically, calling the first logic group set for calculation and calling the second logic group set for calculation are called according to the following steps F1 to F2:

[0100] Step F1: according to the computing power ratio of the computing capability ratio of the heterogeneous processor, dispatch part of the training data in the training data set to the heterogeneous processor for computing.

[0101] Step F2: Determine the training data control value of the central processing unit according to the confidence of the error gap between the integer operation model and the single-precision floating-point model and the computing power ratio, and dispatch part of the training data in the training data set to the central processing unit for calculation according to the training data control value.

[0102] In an embodiment of the present application, by controlling the relative proportion of the amount of data received by the model running on the heterogeneous processor and the central processing unit, without redesigning the network structure or the training process, two indicators are introduced, and the two indicators include: the confidence α of the error gap between the integer operation model and the single-precision floating-point model, and the computing power ratio β of the computing power ratio of the heterogeneous processor. Among them, α is calculated by simply analyzing the validation set on the heterogeneous processor or the central processing unit before each training period. When α is close to zero, the accuracy of the integer operation model is low, so more data is allocated to the central processing unit training to reduce the loss of training accuracy, otherwise more data should be provided to the heterogeneous processor to increase the training speed. β can simply be used as an analysis of the performance gap between the central processing unit and the heterogeneous processor before the training task begins. In order to avoid processor idleness, the portion of the input data fed to the heterogeneous processor should be exactly the same as β.

[0103] Therefore, when considering accuracy requirements and performance issues, the embodiment of the present application determines the training data control value of the central processing unit, and the training data control value of the central processing unit is {e -α ,1-β}, that is, always provide the CPU with the data with the maximum training data control value, because when e -α When it is greater than 1-β, the processing power of the heterogeneous processors is insufficient and the central processor is idle. At this time, providing more data to the heterogeneous processors will not gain any benefit. Otherwise, due to the quantization error of the integer operation model, more data should be provided to the central processor, even if the heterogeneous processor is idle. This will achieve efficient model training while ensuring the training performance of the model.

[0104] In order to better illustrate the distributed training method based on the mobile chip cluster server provided by the present application, the method is fully implemented on the MNN on the mobile chip (MNN is the most advanced training library on mobile SoCs). The method provided in the embodiment of the present application supports models exported from TensorFlow and PyTorch, and the system is fully evaluated on a commercial mobile chip cluster server with five popular DNNs (Deep Neural Networks) and five data sets representing edge cloud scenarios. By comparing the present application with multiple competitive benchmarks, the competitive benchmarks include: traditional parameter servers, ring global reduction topologies, methods with hierarchical architectures, and gradient compression methods. Experimental results show that the method provided by the implementation of the present application can significantly and consistently outperform all benchmarks in terms of training speed while maintaining convergence accuracy (<1% loss), for example, using 32 mobile chips to accelerate 1.6×–740×. In most cases, the method provided by the implementation of the present application is the only method that can complete training within a typical idle time frame and allows model updates every day. In addition, compared with a general-purpose GPU (Graphics Processing Unit) widely used for deep learning training, that is, compared with NVIDIA V100, the method provided by the present application is implemented at a similar training speed, but the energy consumption is reduced by 2.31×–10.23×.

[0105] In summary, in the embodiment of the present application, the mobile chips in the mobile chip cluster server are divided into multiple logical groups, and the training data set and the model to be trained are assigned to the multiple logical groups. The multiple mobile chips in each logical group perform model training at the same time, and the mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode; and with the goal of minimizing the number of logical groups that communicate between physical groups, the multiple logical groups are mapped to the multiple physical groups to obtain multiple mapped logical groups; then, according to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, and there is no communication competition between the multiple logical groups in each logical group set; finally, the multiple logical group sets are executed sequentially, and the execution results of the multiple logical group sets are synchronized to obtain the training results of this round. Since there is no communication competition between the multiple logical groups in the logical group set, and the multiple mobile chips in the logical group can perform model training at the same time, the method of executing multiple logical group sets in sequence can realize parallel distributed training on the mobile chip cluster server, that is, fast model training can be realized under the cooperation of multiple mobile chips without affecting the accuracy. In this way, the mobile chip cluster server is efficiently utilized to give full play to the capabilities of the mobile chip cluster server, thereby improving the training energy efficiency and performance of the model.

[0106] The present application also provides a distributed training system based on a mobile chip cluster server. Figure 3 As shown, Figure 3 It is a structural diagram of a distributed training system based on a mobile chip cluster server provided in an embodiment of the present application, the system including a global scheduler and a distributed training engine.

[0107] Specifically, the global scheduler divides the mobile chips in the mobile chip cluster server into multiple logical groups based on the logical group division module, and allocates the training data set and the model to be trained to the multiple logical groups; based on the mapping module, with the goal of minimizing the number of logical groups that communicate between physical groups, the multiple logical groups are mapped to the multiple physical groups to obtain multiple mapped logical groups, and the mapped multiple logical groups are divided into multiple logical group sets according to whether there is communication contention between the logical groups.

[0108] The distributed training engine sequentially executes multiple logical group sets so that the global scheduler synchronizes the results calculated by the distributed training engine based on the synchronization module to obtain the training results of this round. Specifically, the logical group set includes a first logical group set and a second logical group set, and the multiple mobile chips in the logical group include heterogeneous processors and central processing units. The distributed training engine schedules part of the training data in the training data set to the heterogeneous processor for calculation according to the computing power ratio of the computing power ratio of the heterogeneous processor; determines the training data control value of the central processing unit according to the confidence and computing power ratio of the error gap between the integer operation model and the single-precision floating point model, and schedules part of the training data in the training data set to the central processing unit for calculation according to the training data control value.

[0109] The present application also provides a distributed training device based on a mobile chip cluster server, referring to Figure 4 As shown, Figure 4 : is a structural diagram of a distributed training device based on a mobile chip cluster server provided in an embodiment of the present application, the device comprising:

[0110] A first division module 410 is used to divide the mobile chips in the mobile chip cluster server into multiple logical groups, and assign the training data set and the model to be trained to the multiple logical groups, and the multiple mobile chips in each logical group perform model training at the same time. The mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode;

[0111] A first mapping module 420, configured to map the plurality of logical groups to the plurality of physical groups with the goal of minimizing the number of logical groups performing inter-physical group communication, to obtain a plurality of mapped logical groups;

[0112] A second division module 430 is used to divide the mapped multiple logical groups into multiple logical group sets according to whether there is communication competition between the logical groups, and there is no communication competition between the multiple logical groups in each logical group set;

[0113] The first execution module 440 is used to execute the multiple logic group sets in sequence and synchronize the execution results of the multiple logic group sets to obtain the training result of this round.

[0114] In an optional embodiment, the first division module includes:

[0115] A first division submodule is used to initially group the mobile chips in the mobile chip cluster server in order of group size from small to large, and determine the training accuracy of the initial grouping corresponding to each group size;

[0116] A first determination module is used to determine the group size of the initial grouping whose training accuracy meets the accuracy threshold as the target group size;

[0117] The second division submodule is used to divide the mobile chips in the mobile chip cluster server into multiple logical groups according to the target group size.

[0118] In an optional embodiment, the first mapping module includes:

[0119] A first mapping submodule, configured to map a first part of the logical groups in the plurality of logical groups to the plurality of physical groups in a mapping manner that fully satisfies mapping the plurality of mobile chips in the logical group to the same physical group, to obtain a first mapping result;

[0120] A second mapping submodule is used to map a second part of the logical groups except the first part of the logical groups in the multiple logical groups to the multiple physical groups in a manner of sequentially mapping the multiple mobile chips in the logical groups to obtain a second mapping result;

[0121] The third mapping submodule is used to obtain a plurality of mapped logical groups according to the first mapping result and the second mapping result, wherein each of the plurality of mapped logical groups has communication competition with at most two logical groups.

[0122] In an optional embodiment, the multiple logical group sets include a first logical group set and a second logical group set; and the second division module includes:

[0123] A drawing module, used for drawing a bipartite graph according to the mapped multiple logical groups, wherein the nodes in the bipartite graph represent the logical groups, and the edges in the bipartite graph represent whether there is communication competition between the logical groups;

[0124] A solution module is used to solve the bipartite graph according to a depth-first search method to obtain the first logic group set and the second logic set.

[0125] In an optional embodiment, the first execution module includes:

[0126] A first calculation module, used for calling the first logical group set to perform calculation;

[0127] The second calculation module is used for the first logic group set to communicate after the first logic group set is calculated, and to call the second logic group set to calculate, the calculation time of the second logic group set being greater than the communication time of the first logic group set.

[0128] In an optional embodiment, the plurality of mobile chips in the logic group include: a heterogeneous processor and a central processing unit, the heterogeneous processor performs calculations based on an integer operation model, and the central processing unit performs calculations based on a single-precision floating-point number model; the first execution module further includes:

[0129] A first calling module is used to schedule part of the training data in the training data set to the heterogeneous processor for calculation according to the computing power ratio of the computing capability ratio of the heterogeneous processor;

[0130] The second calling module is used to determine the training data control value of the central processing unit according to the confidence of the error gap between the integer operation model and the single-precision floating-point model and the computing power ratio, and schedule part of the training data in the training data set to the central processing unit for calculation according to the training data control value.

[0131] In an optional embodiment, the first division module further includes:

[0132] A data partitioning module, used for partitioning the training data set into a plurality of sub-training data sets, wherein the number of the sub-training data sets is the same as the number of the plurality of logical groups;

[0133] The data allocation module is used to allocate the multiple sub-training data sets and the models to be trained to the multiple logical groups, each logical group corresponds to a sub-training data set and the model to be trained.

[0134] In an optional embodiment, the time of each round of training is negatively correlated with the number of the multiple logical groups, and the training accuracy of the model to be trained is negatively correlated with the number of the multiple logical groups.

[0135] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 includes: a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus communication. A computer program is stored in the memory 510. The computer program can be run on the processor 520, thereby implementing the steps of the distributed training method based on the mobile chip cluster server described in the embodiment of the present application.

[0136] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0137] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0141] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0142] The above is a detailed introduction to the distributed training method, device and equipment based on the mobile chip cluster server provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A distributed training method based on a mobile chip cluster server, characterized in that: The method comprises: The mobile chips in the mobile chip cluster server are divided into multiple logical groups, and the training data sets and the models to be trained are allocated to the multiple logical groups, and the multiple mobile chips in each logical group perform model training at the same time. The mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode; With the goal of minimizing the number of logical groups that perform inter-physical group communication, the plurality of logical groups are mapped to the plurality of physical groups to obtain a plurality of mapped logical groups; According to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, and there is no communication competition between the multiple logical groups in each logical group set, wherein the communication competition refers to the existence of network card contention between different logical groups, and when the size of the logical group does not match the size of the physical group, there is communication competition between the logical groups; Execute the multiple logic group sets in sequence, and synchronize the execution results of the multiple logic group sets to obtain the training results of this round; Determining the size of the logical group includes: In the preparation stage of model training, the mobile chip cluster servers are initially grouped from smaller group sizes to larger group sizes, and the training accuracy is analyzed; When it is determined that the corresponding training accuracy is higher than the accuracy threshold, the initial grouping size corresponding to the training accuracy is determined as the size of the logical group.

2. The method according to claim 1, characterized in that The mobile chips in the mobile chip cluster server are divided into multiple logical groups, including: Initially grouping the mobile chips in the mobile chip cluster server in order of group size from small to large, and determining the training accuracy of the initial grouping corresponding to each group size; Determine the group size of the initial grouping whose training accuracy meets the accuracy threshold as the target group size; According to the target group size, the mobile chips in the mobile chip cluster server are divided into a plurality of logical groups.

3. The method according to claim 1, characterized in that With the goal of minimizing the number of logical groups for inter-physical group communication, the plurality of logical groups are mapped to the plurality of physical groups to obtain the mapped plurality of logical groups, including: In a mapping manner that fully satisfies mapping of multiple mobile chips in a logical group to the same physical group, a first part of the logical groups in the multiple logical groups are mapped to the multiple physical groups to obtain a first mapping result; In a manner of sequentially mapping the plurality of mobile chips in the logical group, mapping a second part of the logical groups except the first part of the logical groups in the plurality of logical groups to the plurality of physical groups to obtain a second mapping result; A plurality of mapped logical groups are obtained according to the first mapping result and the second mapping result, and each of the plurality of mapped logical groups has communication competition with at most two logical groups.

4. The method according to claim 1, characterized in that: The plurality of logical group sets include a first logical group set and a second logical group set; According to whether there is communication competition between the logical groups, the mapped multiple logical groups are divided into multiple logical group sets, including: Draw a bipartite graph according to the mapped multiple logical groups, where the nodes in the bipartite graph represent the logical groups, and the edges in the bipartite graph represent whether there is communication competition between the logical groups; The bipartite graph is solved according to a depth-first search method to obtain the first logic group set and the second logic set.

5. The method according to claim 4, characterized in that Sequentially executing the plurality of sets of logic groups comprises: Calling the first logical group set to perform calculation; After the first logical group set is calculated, the first logical group set communicates, and calls the second logical group set to calculate, and the calculation time of the second logical group set is greater than the communication time of the first logical group set.

6. The method according to claim 5, characterized in that: The plurality of mobile chips in the logic group include: a heterogeneous processor and a central processing unit, wherein the heterogeneous processor performs calculations based on an integer operation model, and the central processing unit performs calculations based on a single-precision floating-point number model; Calling the first logic group set to perform calculation and calling the second logic group set to perform calculation are performed according to the following steps: Scheduling part of the training data in the training data set to the heterogeneous processor for calculation according to a calculation power ratio of the calculation capability ratio of the heterogeneous processor; According to the confidence of the error gap between the integer operation model and the single-precision floating-point model and the computing power ratio, the training data control value of the central processing unit is determined, and part of the training data in the training data set is scheduled to the central processing unit for calculation according to the training data control value.

7. The method according to claim 1, characterized in that Allocating the training data set and the models to be trained to the multiple logical groups includes: Dividing the training data set into a plurality of sub-training data sets, the number of the sub-training data sets being the same as the number of the plurality of logical groups; The multiple sub-training data sets and the models to be trained are allocated to the multiple logical groups, each logical group corresponds to a sub-training data set and the model to be trained.

8. The method according to any one of claims 1 to 7, characterized in that: The time of each round of training is negatively correlated with the number of the multiple logical groups, and the training accuracy of the model to be trained is negatively correlated with the number of the multiple logical groups.

9. A distributed training device based on a mobile chip cluster server, characterized in that: The device comprises: A first division module is used to divide the mobile chips in the mobile chip cluster server into multiple logical groups, and allocate the training data set and the model to be trained to the multiple logical groups, and the multiple mobile chips in each logical group perform model training at the same time, and the mobile chips in the mobile chip cluster server are divided into multiple physical groups according to the connection mode; A first mapping module, configured to map the plurality of logical groups to the plurality of physical groups with the goal of minimizing the number of logical groups performing inter-physical group communication, to obtain a plurality of mapped logical groups; A second division module is used to divide the mapped multiple logical groups into multiple logical group sets according to whether there is communication competition between the logical groups, and there is no communication competition between the multiple logical groups in each logical group set, wherein the communication competition refers to the existence of network card contention between different logical groups, and when the size of the logical group does not match the size of the physical group, there is communication competition between the logical groups; A first execution module, used for sequentially executing the multiple logic group sets and synchronizing the execution results of the multiple logic group sets to obtain a training result of this round; Among them, determining the size of the logical group in the first partitioning module includes: in the preparation stage of model training, initially grouping the mobile chip cluster servers from smaller group sizes to larger group sizes, and analyzing the training accuracy; when it is determined that the corresponding training accuracy is higher than the accuracy threshold, determining the initial grouping size corresponding to the training accuracy as the size of the logical group.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the distributed training method based on the mobile chip cluster server described in any one of claims 1-8 are implemented.

Citation Information

Patent Citations

  • Multi-agent transmission method for cognitive radio network signals and terminal equipment

    CN112383926A

  • Networking layout method and device, equipment and storage medium

    CN114781316A