Distributed training method, device, intelligent terminal and computer-readable storage medium

Abstract

The present invention discloses a distributed training method, device, intelligent terminal and computer-readable storage medium. The method includes: obtaining the server intelligence level of each server respectively, and each of the above server intelligence levels respectively reflects the data processing capability of the corresponding server; obtaining the target group number information based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each server, and grouping all servers based on the target group number information and the server intelligence level of each server to obtain multiple server groups; distributing data resources in the data set to be trained and model resources in the model to be trained to each server group based on the group intelligence level of each server group, and each group intelligence level respectively reflects the data processing capability of the corresponding server group; and controlling each server group to perform training based on the obtained data resources and model resources. Compared with the prior art, the present invention can improve the training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a distributed training method, device, intelligent terminal and computer-readable storage medium. Background Art

[0002] Deep learning is an intelligent algorithm for modern information processing. It solves the local optimal problem of traditional neural network algorithms when training multi-layer neural networks by fitting training sample data through multi-layer neural networks. At present, with the development of science and technology, especially the rapid development of Internet technology, the growth rate of various data is getting faster and faster, which brings great challenges to the training of deep learning models. At the same time, as artificial intelligence tasks become more and more complex, the requirements for models are also getting higher and higher. In the field of deep learning, it has now tended to develop in the direction of large models and big data. Big data and large models have put forward higher requirements for computing power and storage capacity, and therefore have brought huge challenges to traditional deep learning training methods.

[0003] In the prior art, distributed model training is usually achieved by randomly sampling or scrambling data, horizontally or vertically segmenting the model, and distributing the segmented data and / or model to each server. The problem with the prior art is that the actual processing capacity of each server is not considered when dividing and distributing data and models, resulting in low training efficiency of the server, which is not conducive to improving training efficiency.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a distributed training method, device, intelligent terminal and computer-readable storage medium, aiming to solve the problem in the prior art that the actual processing capacity of each server is not considered when dividing and distributing data and models, resulting in low training efficiency of the server, which is not conducive to improving training efficiency.

[0006] In order to achieve the above object, the first aspect of the present invention provides a distributed training method, wherein the method comprises:

[0007] Obtaining server intelligence levels of each server respectively, wherein each of the server intelligence levels respectively reflects the data processing capability of the corresponding server;

[0008] Based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers, information on the number of target groups is obtained, and based on the information on the number of target groups and the server intelligence level of each of the above servers, all of the above servers are grouped to obtain multiple server groups;

[0009] Distributing the data resources in the dataset to be trained and the model resources in the model to be trained to each server group based on the group intelligence level of each server group, wherein each of the group intelligence levels reflects the data processing capability of the corresponding server group;

[0010] The server groups are respectively controlled to perform training based on the obtained data resources and model resources.

[0011] Optionally, the above separately obtaining the server intelligence level of each server includes:

[0012] Obtain the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each of the above servers respectively;

[0013] The server intelligence level of each of the above servers is calculated based on the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each of the above servers.

[0014] Optionally, the intelligence level of the server is the ratio of the capability product of the corresponding server to the energy consumption cost, and the capability product is the product of the video processing capability, general computing capability, storage capability, communication capability and device access capability of the server.

[0015] Optionally, the target group number information is obtained based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the servers, and all of the servers are grouped based on the target group number information and the server intelligence level of each of the servers to obtain multiple server groups, including:

[0016] Obtaining data set information of the data set to be trained, and calculating the complexity of obtaining the data set to be trained based on the data set information, wherein the data set information includes the feature dimension, required memory size, and required computing power size of the data set to be trained;

[0017] Obtain model information of the model to be trained, and calculate the complexity of the model to be trained based on the model information, wherein the model information includes the number of network layers and the size of parameters of the model to be trained;

[0018] The target grouping number information is obtained based on the intelligence level of each of the above servers, the complexity of the above data set to be trained and the complexity of the above model to be trained, and all of the above servers are divided into groups based on the above target grouping number information and the server intelligence level of each of the above servers to obtain multiple server groups.

[0019] Optionally, the target group number information includes the target group number and the target group number corresponding to each server group, the target group number information is obtained by calculating based on the intelligence level of each of the servers, the complexity of the obtained data set to be trained and the complexity of the model to be trained, and all of the servers are grouped based on the target group number information and the server intelligence level of each of the servers to obtain multiple server groups, including:

[0020] The number of target groups is calculated based on the intelligence level of each of the above servers, the complexity of the above data set to be trained and the complexity of the above model to be trained, wherein the number of target groups is the number of server groups that need to be obtained by dividing all the above servers;

[0021] Divide all the above servers into groups based on the above target group number to obtain server groups with the target group number;

[0022] The data sets to be trained and the models to be trained are divided based on the group intelligence levels corresponding to the server groups to obtain group data sets and group models corresponding to the server groups;

[0023] Calculating and obtaining the number of target groups corresponding to each of the server groups based on the group intelligence level of each of the server groups, the complexity of the group data set, and the complexity of the group model;

[0024] The servers in each of the server groups are divided based on the target group numbers corresponding to each of the server groups, so as to obtain a plurality of server groups in each of the server groups.

[0025] Optionally, the calculation of the number of target groups based on the intelligence level of each of the servers, the complexity of the obtained data set to be trained and the complexity of the obtained model to be trained includes:

[0026] Calculate the value obtained by multiplying the overall complexity by the overall intelligence level and take the integer value, and use the integer value obtained as the number of the above target groups;

[0027] The overall complexity product is the product of the complexity of the dataset to be trained and the complexity of the model to be trained, and the overall intelligence level is the sum of the server intelligence levels of all the servers.

[0028] Optionally, the above-mentioned obtaining the number of target groups corresponding to each of the server groups by respectively calculating based on the group intelligence level of each of the server groups, the complexity of the group data set and the complexity of the group model includes:

[0029] Calculate and obtain the value of the group complexity product corresponding to each of the above server groups divided by the group intelligence level and take the integer, and use the obtained integer value as the target group number corresponding to each of the above server groups;

[0030] The product of each of the above group complexity is the product of the complexity of the group data set corresponding to each of the above server groups and the complexity of the group model, and the intelligence level of each of the above group is the sum of the server intelligence levels of all servers in each of the above server groups.

[0031] Optionally, the servers in each of the server groups are divided based on the number of target groups corresponding to each of the server groups, and a plurality of server groups are obtained in each of the server groups, including:

[0032] Based on the target group number corresponding to each of the above server groups and the server intelligence level of each server in each of the above server groups, the servers in each of the above server groups are divided respectively by a clustering algorithm, and multiple server groups are obtained in each of the above server groups.

[0033] Optionally, the group intelligence level based on each server group distributes data resources in the data set to be trained and model resources in the model to be trained to each server group, wherein each of the group intelligence levels reflects the data processing capability of the corresponding server group, including:

[0034] In each of the above-mentioned server groups, the data resources in the above-mentioned group data set and the model resources in the above-mentioned group model are distributed to each server group based on the group intelligence level corresponding to each of the above-mentioned server groups. Each of the above-mentioned group intelligence levels is the sum of the server intelligence levels of all servers in each of the above-mentioned server groups.

[0035] Optionally, the above-mentioned separately controlling each of the above-mentioned server groups to perform training based on the obtained data resources and model resources includes:

[0036] In each of the above server groups, when the complexity of the obtained data resources is greater than the complexity of the obtained model resources, training is performed using a data parallel method;

[0037] In each of the above server groups, when the complexity of the obtained data resources is less than the complexity of the obtained model resources, training is performed using a model parallel method.

[0038] A second aspect of the present invention provides a distributed training device, wherein the device comprises:

[0039] A server intelligence level acquisition module, used to respectively acquire the server intelligence level of each server, wherein each of the above server intelligence levels respectively reflects the data processing capability of the corresponding server;

[0040] A server group division module, used to obtain target group number information based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers, and to divide all of the above servers into groups based on the above target group number information and the server intelligence level of each of the above servers to obtain multiple server groups;

[0041] A resource distribution module, used to distribute the data resources in the above-mentioned to-be-trained data set and the model resources in the above-mentioned to-be-trained model to each server group based on the group intelligence level of each server group, wherein each of the above-mentioned group intelligence levels respectively reflects the data processing capability of the corresponding server group;

[0042] The training control module is used to control each of the above server groups to perform training based on the obtained data resources and model resources.

[0043] A third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and a distributed training program stored in the memory and executable on the processor, wherein the distributed training program implements any step of the distributed training method when executed by the processor.

[0044] A fourth aspect of the present invention provides a computer-readable storage medium, on which a distributed training program is stored. When the distributed training program is executed by a processor, the steps of any one of the above-mentioned distributed training methods are implemented.

[0045] As can be seen from the above, in the scheme of the present invention, the server intelligence level of each server is obtained respectively, wherein each of the above server intelligence levels reflects the data processing capability of the corresponding server; based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each server, the target group number information is obtained, and all the above servers are grouped based on the target group number information and the server intelligence level of each server to obtain multiple server groups; based on the group intelligence level of each server group, the data resources in the above data set to be trained and the model resources in the above model to be trained are distributed to each server group respectively, wherein each of the above group intelligence levels reflects the data processing capability of the corresponding server group; and each of the above server groups is controlled to perform training based on the obtained data resources and model resources. Compared with the method in the prior art that only randomly samples or scrambles the data and divides the model horizontally or vertically to distribute the divided data and / or model to each server, the present invention considers the actual processing capability of each server and distributes the corresponding data resources and model resources. At the same time, the servers are grouped according to the server intelligence level, which is conducive to the collaboration between the servers in the server group, thereby facilitating the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a flow chart of a distributed training method provided by an embodiment of the present invention;

[0048] Figure 2 Embodiment of the present invention Figure 1 Specific flow diagram of step S100;

[0049] Figure 3 Embodiment of the present invention Figure 1 Specific flow diagram of step S200;

[0050] Figure 4 Embodiment of the present invention Figure 3 Specific flow diagram of step S203;

[0051] Figure 5 Embodiment of the present invention Figure 1 Specific flow diagram of step S400;

[0052] Figure 6is a structural schematic diagram of a distributed training device provided by an embodiment of the present invention;

[0053] Figure 7 is an architecture diagram of a dynamic adaptive training system provided by an embodiment of the present invention;

[0054] Figure 8 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0056] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0057] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0058] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0059] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0060] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0062] Deep learning is an intelligent algorithm for modern information processing. It solves the local optimal problem of traditional neural network algorithms when training multi-layer neural networks by fitting training sample data with multi-layer neural networks. At present, with the development of science and technology, especially the rapid development of Internet technology, the growth rate of various data (especially video data) is getting faster and faster, which brings great challenges to the training of deep learning models. At the same time, as artificial intelligence tasks become more and more complex, the requirements for models (such as visual models) are also getting higher and higher. In addition, the emergence of a large amount of training data provides the basic conditions for model training, so currently visual models are developing in the direction of large models, and the number of parameters of large visual models is growing very rapidly. In the field of deep learning, it has tended to develop in the direction of large models and big data. Big data and large models have put forward higher requirements for computing power and storage capacity, which has brought huge challenges to traditional deep learning training methods. Traditional single-machine training methods cannot withstand huge computing power and storage requirements, so some distributed model training methods have emerged. That is, by dividing training data, dividing model parameters, allocating training tasks, deploying computing resources and integrating the final training results, the problem of large model training in the context of big data is solved. In the prior art, distributed model training is usually achieved by randomly sampling or scrambling data, horizontally or vertically segmenting the model, and distributing the segmented data and / or model to each server. However, the amount of video data and model parameters is large, and the actual processing capacity of each server is not considered when dividing and distributing data and models in the prior art, resulting in low server training efficiency, which is not conducive to improving the overall model training efficiency.

[0063] At present, there are some studies on distributed training methods, such as training methods based on data parallelism and training methods based on model parallelism. The data parallel method mainly solves the problem that the training set is too large and sometimes it is impossible to load all the training data into a single machine at one time. Therefore, the data set can be divided into multiple parts and then placed in multiple nodes (such as servers) for model training. The model parallel method mainly solves the problem that the model is too large and it is difficult to load the entire model into one machine because the model has too many layers and too many parameters. Therefore, the model can be divided into multiple parts and loaded into different nodes respectively. However, the above method still cannot solve the problem that the single-layer parameter amount exceeds the single-card video memory, causing the model to be unable to be trained. In terms of data partitioning, the cost of global sampling in the existing method is relatively high, and the sampling method makes it difficult to select low-frequency training samples, resulting in some labeled training data not being fully utilized. The existing model partitioning method also cannot meet the requirements of large model distributed training. The existing data and model partitioning methods do not fully consider the processing capacity of the server, resulting in low training efficiency. In this embodiment, a hybrid parallel distributed training method is adopted, and the actual processing capacity of the server is considered, which is conducive to improving training efficiency. Among them, the hybrid parallel distributed training method combines data parallelism and model parallelism in different parts of the same distributed training.

[0064] In this embodiment, in order to solve the problems of the prior art, the server intelligence level of each server is obtained respectively, wherein each of the above server intelligence levels reflects the data processing capability of the corresponding server; based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each server, the target group number information is obtained, and all the above servers are grouped based on the target group number information and the server intelligence level of each server to obtain multiple server groups; based on the group intelligence level of each server group, the data resources in the above data set to be trained and the model resources in the above model to be trained are distributed to each server group respectively, wherein each of the above group intelligence levels reflects the data processing capability of the corresponding server group; and each of the above server groups is controlled to perform training based on the obtained data resources and model resources. Compared with the method in the prior art that only randomly samples or scrambles the data and divides the model horizontally or vertically to distribute the divided data and / or model to each server, the present invention considers the actual processing capability of each server and distributes the corresponding data resources and model resources. At the same time, the servers are grouped according to the server intelligence level, which is conducive to the collaboration between the servers in the server group, thereby improving the training efficiency.

[0065] Exemplary Methods

[0066] like Figure 1As shown, an embodiment of the present invention provides a distributed training method. Specifically, the method includes the following steps:

[0067] Step S100, obtaining the server intelligence level of each server respectively, wherein each of the above server intelligence levels respectively reflects the data processing capability of the corresponding server.

[0068] Step S200, based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers, obtain the target group number information, and divide all the above servers into groups based on the above target group number information and the server intelligence level of each of the above servers to obtain multiple server groups.

[0069] Step S300, based on the group intelligence level of each server group, distribute the data resources in the above-mentioned data set to be trained and the model resources in the above-mentioned model to be trained to each server group, wherein each of the above-mentioned group intelligence levels respectively reflects the data processing capability of the corresponding server group.

[0070] Step S400, controlling each of the above server groups to perform training based on the obtained data resources and model resources.

[0071] In this embodiment, the server is a device for training the model to be trained according to the data set to be trained. In this embodiment, the server is used as an example for specific description, but in actual use, the model can be trained through a specific training device or node. The specific training process can refer to the description of the server training process in this embodiment, and is not specifically limited here.

[0072] In this embodiment, the model to be trained is a model that needs to be trained, and the data set to be trained is a collection of data used to train the model to be trained. Specifically, the model to be trained is a visual model, and the data in the data set to be trained is video data. In actual use, the model to be trained can also be other types of models (such as auditory models, etc.), and the data in the data set to be trained can also be other types of data (such as image data, audio data, etc.), which are not specifically limited here. The complexity of the data set to be trained reflects the complexity of each data in the data set to be trained, which can be represented by parameters such as the feature dimension of the data set, the required memory size and / or the computing power size; the complexity of the model to be trained reflects the complexity of the model to be trained in the model structure, which can be represented by parameters such as the number of network layers of the model to be trained and the size of the parameters. The target group number information is calculated to indicate the information related to the number of groups when grouping all servers.

[0073] In one application scenario, all servers are directly divided into multiple server groups, and the above target group number information may include the number of server groups to be divided. In another application scenario, all servers are first divided into multiple server groups, and then multiple server groups are obtained from each server group. The above target group number information may include the number of server groups to be divided and the number of server groups to be divided in each server group. In actual use, if the servers are divided and grouped at multiple levels, the above target group number information may include the number of groups corresponding to each level, which is not specifically limited here.

[0074] In this embodiment, when the above-mentioned servers are divided into groups, servers with similar server intelligence levels are divided into the same server group, which helps the collaboration between servers in the server group and further improves the training efficiency.

[0075] As can be seen from the above, in the distributed training method provided by the embodiment of the present invention, the server intelligence level of each server is obtained respectively, wherein each of the above server intelligence levels respectively reflects the data processing capability of the corresponding server; based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers, the target group number information is obtained, and based on the target group number information and the server intelligence level of each of the above servers, all of the above servers are grouped to obtain multiple server groups; based on the group intelligence level of each server group, the data resources in the above data set to be trained and the model resources in the above model to be trained are distributed to each server group respectively, wherein each of the above group intelligence levels respectively reflects the data processing capability of the corresponding server group; and each of the above server groups is controlled to perform training based on the obtained data resources and model resources. Compared with the prior art method of only distributing the divided data and / or model to each server by randomly sampling or scrambling the data and horizontally or vertically dividing the model, the present invention takes into account the actual processing capacity of each server and distributes the corresponding data resources and model resources. At the same time, the servers are grouped according to their intelligence levels, so that servers with similar intelligence levels are divided into a server group, which facilitates the collaboration between the servers in the server group, thereby helping to improve the training efficiency.

[0076] In one implementation, the server intelligence level corresponding to each server is obtained respectively, wherein the server intelligence level corresponding to each server can reflect the data processing capability of the server, and further, the server intelligence level can also comprehensively reflect the energy consumption of the server. In an application scenario, the server intelligence level corresponding to the server can be calculated by one or more data of the server's video processing capability, general computing capability, storage capability, communication capability, device access capability, and energy consumption cost.

[0077] In this embodiment, if Figure 2 As shown, the above step S100 specifically includes the following steps:

[0078] Step S101, respectively obtaining the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each of the above servers.

[0079] Step S102, based on the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each of the above servers, the server intelligence level of each of the above servers is calculated and obtained. In a specific implementation, all servers are first encoded (i.e., each server is numbered separately, and the specific numbering method is not limited) to describe the specific operations on each server. Specifically, the code of the current server is set to a, a = {1, 2, ..., N}, where N is the total number of servers. Then, the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each server are calculated in sequence according to the encoding order. Specifically, the video processing capability is the ability of the server to encode and decode the original video stream, and the video processing capability = video compression ratio × video quality / video compression time. The general computing capability is the sum of the computing levels of various types of data by the server's central processing unit and graphics processing unit, and the general computing capability = central processing unit computing power + graphics processing unit computing power. The server storage capacity is the sum of the server's storage levels for various types of data, which is mainly calculated by the storage size and storage time of the storage module. Storage capacity = storage capacity / storage time; where the above storage time is the time spent on transferring data. Storage capacity divided by time can represent both the size of storage capacity and the size of storage speed. The larger the storage capacity and the faster the speed, the stronger the storage capacity. The communication capacity of the server is the efficiency of the communication equipment supporting the server for various types of data transmission levels. Communication capacity = amount of data transmitted / transmission time. Device access capacity mainly refers to the number, type and stability of device access that the server can support. Device access capacity = types of devices that can be supported for access × number of devices that can be supported for access × device failure rate; device access capacity can affect the stability of the training process, and device access capacity also represents the scalability of the server. Energy consumption cost refers to the cost of resources such as electricity generated during the operation and maintenance of the server. Energy consumption cost is mainly calculated by the cost of electricity resources.

[0080] In a specific implementation, the server intelligence level is the ratio of the capability product of the corresponding server to the energy consumption cost, and the capability product is the product of the video processing capability, general computing capability, storage capability, communication capability and device access capability of the server. Specifically, as shown in the following formula (1):

[0081]

[0082] Among them, I a represents the server intelligence level of the a-th server, v a 、x a ,y a 、z a 、r aand e a They represent the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of the a-th server respectively. In this way, the data processing capability corresponding to each server can be reflected by the server intelligence level, so that servers with similar server intelligence levels can be divided into a server group, and corresponding data resources and model resources can be allocated to each group according to the group intelligence level corresponding to each group. For example, for server groups with high group intelligence levels, more data resources and more complex model resources (or more complex data resources and model resources) can be allocated to make full use of their data processing capabilities. For server groups with low group intelligence levels, fewer data resources and simpler model resources (or less complex data resources and model resources) can be allocated to ensure that they can complete tasks quickly and improve the training efficiency of the entire model.

[0083] In this embodiment, if Figure 3 As shown, the above step S200 specifically includes the following steps:

[0084] Step S201, obtaining data set information of the data set to be trained, and calculating the complexity of the data set to be trained based on the data set information, wherein the data set information includes the feature dimension of the data set to be trained, the required memory size, and the required computing power size.

[0085] Step S202, obtaining model information of the model to be trained, and calculating the complexity of the model to be trained based on the model information, wherein the model information includes the number of network layers and parameter size of the model to be trained.

[0086] Step S203, based on the intelligence level of each of the above servers, the complexity of the above data set to be trained and the complexity of the above model to be trained, calculate the target group number information, and divide all the above servers into groups based on the above target group number information and the server intelligence level of each of the above servers to obtain multiple server groups.

[0087] In a specific implementation, the above-mentioned data set information and the above-mentioned model information are pre-set information that needs to be obtained based on the actual data set to be trained and the model to be trained, and are related to the data itself and the model structure itself. In this embodiment, the pre-set data set information includes the characteristic dimensions, required memory size and required computing power size of the above-mentioned data set to be trained, and the model information includes the number of network layers and parameter size of the above-mentioned model to be trained. In another application scenario, the above-mentioned data set information and the above-mentioned model information may also include other information, which is not specifically limited here. Among them, the above-mentioned characteristic dimension is the number of variables in the data set, and the more variables, the higher the complexity. The above-mentioned parameter size is the number of parameters contained in the model to be trained, and the parameters are the variables that need to be learned when training the model to be trained.

[0088] It should be noted that the complexity of the above-mentioned training data set is equal to the product of the feature dimension of the above-mentioned training data set, the required memory size and the required computing power size. The complexity of the above-mentioned training model is equal to the product of the number of network layers and the size of the parameters of the above-mentioned training model. Specifically, S d =f·α·β, where S d represents the complexity of the training data set, f, α, and β represent the feature dimension, required memory size, and required computing power of the training data set, respectively; S m = l·δ,S m represents the complexity of the model to be trained, l and δ represent the number of network layers and the size of parameters of the model to be trained respectively.

[0089] In this embodiment, if Figure 4 As shown, the above step S203 specifically includes the following steps:

[0090] Step S2031, calculating the number of target groups based on the intelligence level of each of the above servers, the complexity of the above data set to be trained and the complexity of the above model to be trained, wherein the above number of target groups is the number of server groups that need to be obtained by dividing all the above servers.

[0091] Step S2032: Divide all the above servers into groups based on the above target group number to obtain target group number server groups.

[0092] Step S2033: dividing the data set to be trained and the model to be trained based on the group intelligence level corresponding to each of the server groups, to obtain the group data set and group model corresponding to each of the server groups.

[0093] Step S2034, calculating and obtaining the number of target groups corresponding to each of the server groups based on the group intelligence level of each of the server groups, the complexity of the group data set and the complexity of the group model.

[0094] Step S2035 , dividing the servers in each of the server groups based on the target group numbers corresponding to each of the server groups, and obtaining a plurality of server groups in each of the server groups.

[0095] In a specific implementation, each server may be pre-divided, for example, an initial value of the number of target groups (such as h 0 ) and the initial value of the number of target groups (such as k 0 ), where h 0 and k 0 The specific value can be set and adjusted according to actual needs, and is not limited here. Sort all servers by their intelligence level, and then evaluate and split them into h 0 groups, and in each group, k 0 server groups to improve the efficiency of subsequent specific grouping.

[0096] In this embodiment, all servers are first divided into multiple server groups, and then multiple server groups are obtained in each server group to achieve double-layer nesting. Therefore, the number of server groups (i.e., the number of target groups) and the number of server groups in each server group (i.e., the number of target groups) can be dynamically and adaptively adjusted according to actual task needs, with a wider adjustment range, thus improving the adaptive ability to various types of training tasks.

[0097] In a specific implementation, the value obtained by dividing the overall complexity product by the overall intelligence level is calculated and an integer is taken, and the obtained integer value is used as the number of the target group; wherein the overall complexity product is the product of the complexity of the dataset to be trained and the complexity of the model to be trained, and the overall intelligence level is the sum of the server intelligence levels of all the servers. Specifically, the number of the target group can be calculated by the following formula (2):

[0098]

[0099] Where a is the server number, h is the number of target groups, S d Represents the complexity of the training data set, S m Represents the complexity of the model to be trained, I aRepresents the server intelligence level of the ath server, and N is the total number of servers. After obtaining the above-mentioned target group number h, all servers are sorted according to the server intelligence level, and then evaluated and split into h server groups. Among them, the division of server groups is directly divided according to the size of the server intelligence level, which belongs to a one-dimensional method. There are many ways to implement it, such as selecting h target servers according to the ranking of server intelligence levels, and other servers and the h target servers form a server group nearby, etc., which is not specifically limited here. It should be noted that the formula (2) adopts the method of rounding down to an integer, and in actual use, it can also be rounded up to an integer.

[0100] In this embodiment, after all servers are divided into groups to obtain h server groups, a corresponding group data set and group model are allocated to each server group. The group data set is a collection of data in the training data set that needs to be trained with the corresponding server group; the group model is a model in the training model that needs to be trained with the corresponding server group (it can be a certain layer, a certain part or a certain structure of the training model, etc.).

[0101] In a specific implementation, a corresponding group data set and group model can be allocated to each server group through an intelligent parallel allocation algorithm. Specifically, the group intelligence level of each server group is calculated respectively, and model resources and data resources are allocated according to the group intelligence level of each server group. The group intelligence level is the sum of the server intelligence levels of all servers in the corresponding server group.

[0102] In a specific implementation, the value of the group complexity product corresponding to each of the above-mentioned server groups divided by the group intelligence level is calculated and the integer is taken, and the obtained integer value is used as the number of target groups corresponding to each of the above-mentioned server groups; wherein each of the above-mentioned group complexity products is the product of the complexity of the group data set corresponding to each of the above-mentioned server groups and the complexity of the group model, and each of the above-mentioned group intelligence levels is the sum of the server intelligence levels of all servers in each of the above-mentioned server groups. The complexity of the group data set can be obtained by calculating the characteristic dimension of the group data set, the required memory size and the required computing power size, and the complexity of the group model can be obtained by calculating the number of network layers and the size of parameters of the group model. The specific calculation method can refer to the calculation method of the complexity of the data set to be trained and the complexity of the model to be trained, which will not be repeated here. Specifically, the above-mentioned number of target groups can be calculated by the following formula (3):

[0103]

[0104] Where t is the number of the server group, t = [1····h], h is the number of server groups (i.e. the number of target groups), k t is the number of target groups corresponding to the t-th server group, N is the total number of servers, is the complexity of the group data set corresponding to the t-th server group, is the complexity of the group model corresponding to the t-th server group, is the server intelligence level of the ath server in the tth server group, a is the server number, and in formula (3), for the tth server group, the value range of a is arrive Thus through The group intelligence level corresponding to the t-th server group is calculated. It should be noted that the formula (3) uses the method of rounding down to an integer, and in actual use, it can also be rounded up to an integer.

[0105] In this embodiment, after calculating the target number k corresponding to each server group, t Afterwards, based on the number of target groups corresponding to each of the server groups and the server intelligence level of each server in each of the server groups, the servers in each of the server groups are divided by a clustering algorithm, and multiple server groups are obtained in each of the server groups. Specifically, the number of server groups obtained in each server group is equal to the number of target groups corresponding to the server group.

[0106] In a specific implementation, the grouping of a server group is taken as an example for explanation. Specifically, the number of target groups corresponding to the server group is set to k, and k servers are randomly selected from all servers in the server group as the initial cluster centers, and the number of iterations is initialized. Then the following steps are performed: the first step is to calculate the distance from each server to the k cluster centers and divide it into the class corresponding to the cluster center with the smallest distance, wherein, when calculating the distance between each server, the video processing capacity, general computing capacity, storage capacity, communication capacity, device access capacity and energy consumption of each server are used for calculation. The specific distance calculation formula is shown in the following formula (4):

[0107]

[0108] That is, d i,j =|I i -I j |, where d i,j is the distance between the i-th server and the j-th server, v i 、x i ,y i、z i 、r i and e i represents the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of the i-th server, respectively. j 、x j ,y j 、z j 、r j and e j Represent the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of the jth server respectively. i is the server intelligence level of the ith server, I j is the server intelligence level of the jth server.

[0109] In the second step, after all servers are divided into groups, there are a total of k server groups in the server group. For each server group, the cluster center of the server group is recalculated using the mean method, with the goal of minimizing the square error Among them, C i represents the i-th server group, μ i is the average intelligence level of the i-th server group, |C i | represents the number of servers included in the i-th server group, I a Indicates the server intelligence level of the a-th server.

[0110] The third step is to update the number of iterations (the number of iterations plus 1), and determine whether the updated number of iterations reaches the preset iteration threshold. If it reaches the preset iteration threshold, save the current cluster center and stop the iteration. If it does not reach the preset iteration threshold, determine whether the distance difference between the recalculated cluster center and the original cluster center (the cluster center saved last time, and the initial cluster center is selected when there is no cluster center saved last time) is less than the preset distance threshold. If it is less than the preset distance threshold, save the current cluster center and stop the iteration. If it is not less than the preset distance threshold, save the current cluster center and return to the second step. Among them, the iteration threshold and the distance threshold can be set and adjusted according to actual needs, and are not specifically limited here. In this way, the corresponding square error E is made smaller and smaller through iteration, thereby ensuring that the intelligence level of the servers in each server group is similar, so that it is also easy to distinguish the overall intelligence level of different server groups, and can better cope with training tasks of different types and sizes.

[0111] In a specific implementation, the above step S300 specifically includes: in each of the above server groups, based on the group intelligence level corresponding to each of the above server groups, the data resources in the above group data set and the model resources in the above group model are distributed to each server group respectively, and the group intelligence level of each of the above server groups is the sum of the server intelligence levels of all servers in each of the above server groups. The specific operations in a server group are described below. In actual use, the following specific operations are performed for each server group, so that the corresponding data resources and model resources are allocated to different server groups through an intelligent parallel allocation algorithm. Specifically, according to the result of group division, the intelligence level of each group is obtained respectively, and then the weight of each server group in the server group is determined according to the intelligence level of each group. Specifically, the weight of each server group is first standardized, the weight of the server group with the smallest group intelligence level is set to 1, and the weights of other server groups are set in sequence according to the proportion of the group intelligence level. The weight sum of all server groups is calculated and recorded as W, and the group data set corresponding to the server group and the data resources and model resources corresponding to the group model are divided into W parts in an equal manner. In each round of data resource and model resource allocation, the data resources and model resources are divided according to the proportion of the weights between the server groups. Check whether the data resources and model resources allocated to each server group exceed its own group intelligence level. If there is a server group that exceeds its own group intelligence level, all excess data resources and model resources will be reallocated to other server groups that have not been satisfied (i.e., the data resources and model resources obtained do not reach their own group intelligence level) according to the weight ratio and repeat the above steps until the data resources and model resources obtained by all server groups do not exceed their own group intelligence level, and the final data resource and model resource allocation results are obtained. It should be noted that the relationship between the intelligence level and the complexity of the maximum data resources or model resources that can be processed can be preset, for example, the relationship between the group intelligence level and the corresponding maximum complexity of the data resources and the maximum complexity of the model resources can be preset, so as to determine whether the resources obtained by each server group exceed its own processing capacity.

[0112] In this embodiment, if Figure 5 As shown, the above step S400 specifically includes the following steps:

[0113] Step S401, in each of the above server groups, when the complexity of the obtained data resources is greater than the complexity of the obtained model resources, training is performed using a data parallel method.

[0114] Step S402: In each of the above server groups, when the complexity of the obtained data resources is less than the complexity of the obtained model resources, training is performed using a model parallel method.

[0115] In a specific implementation, data parallelism and model parallelism are dynamically combined within each server group to further improve the training efficiency of each server group. When , it means that the comprehensive complexity of the data in the i-th server group is greater than the comprehensive complexity of the model, that is, the data parallel method is used to train the layers with fewer parameters and larger computational complexity in the front of the neural network. When , it means that the comprehensive complexity of the data in the i-th server group is less than the comprehensive complexity of the model, and the model parallel method is used for training. Among them, i is the number of the server group, is the complexity of the data resources obtained in the i-th server group (the calculation method is similar to the calculation method of the complexity of the training data set), It is the complexity corresponding to the model resources obtained in the i-th server group (the calculation method is similar to the calculation method of the complexity of the model to be trained).

[0116] In an application scenario, in order to solve the problem of low training efficiency of large visual models on large data sets due to improper data and model division, data resources and model resources are reasonably distributed and trained based on the above-mentioned distributed training method. Specifically, a server intelligence level measurement method is proposed, which calculates the server intelligence level of each server through indicators such as video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption. Then, based on the server intelligence level of each server, all servers are divided into groups through a clustering algorithm to ensure that servers with similar capabilities are in one group, providing a basis for the parallel allocation of large models and large data sets. In terms of data division and model division, unlike traditional random sampling and scrambling segmentation methods, the present invention allocates data resources and model resources to different server groups through an intelligent parallel allocation algorithm, and reasonably allocates data and models to server groups based on the size and feature dimension of the training data volume, the number of network layers and the number of parameters of the large model. Within each server group, the data parallel and model parallel methods are dynamically combined. For example, for the layers with fewer parameters and large computational complexity in the front of the neural network, data parallelism is used; for the layers with many parameters and small computational complexity, model parallelism is used. Within each server group, dynamic control of the server group can be achieved through model parameterization adjustment. Specifically, parameters such as the number of target groups, the number of target groups, and the distance threshold can be adjusted according to the complexity corresponding to the data resources, the complexity corresponding to the model resources, and / or actual needs, thereby achieving dynamic adaptive adjustment of the server group. There is no need to manually divide the corresponding server groups for different data sets and models to be trained each time, thereby improving practicality and realizing dynamic resource allocation and efficient training in the context of big data and large models.

[0117] Exemplary Devices

[0118] like Figure 6 As shown in , corresponding to the above-mentioned distributed training method, an embodiment of the present invention further provides a distributed training device, and the above-mentioned distributed training device includes:

[0119] The server intelligence level acquisition module 510 is used to respectively acquire the server intelligence level of each server, wherein each of the above server intelligence levels respectively reflects the data processing capability of the corresponding server.

[0120] The server group division module 520 is used to obtain target group number information based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers, and to group all of the above servers based on the above target group number information and the server intelligence level of each of the above servers to obtain multiple server groups.

[0121] The resource distribution module 530 is used to distribute the data resources in the above-mentioned data set to be trained and the model resources in the above-mentioned model to be trained to each server group based on the group intelligence level of each server group, wherein each of the above-mentioned group intelligence levels reflects the data processing capability of the corresponding server group.

[0122] The training control module 540 is used to control each of the above server groups to perform training based on the obtained data resources and model resources.

[0123] In this embodiment, the server is a device for training the model to be trained according to the data set to be trained. In this embodiment, the server is used as an example for specific description, but in actual use, the model can be trained through a specific training device or node. The specific training process can refer to the description of the server training process in this embodiment, and is not specifically limited here.

[0124] In this embodiment, the model to be trained is a model that needs to be trained, and the data set to be trained is a collection of data used to train the model to be trained. Specifically, the model to be trained is a visual model, and the data in the data set to be trained is video data. In actual use, the model to be trained can also be other types of models (such as auditory models, etc.), and the data in the data set to be trained can also be other types of data (such as image data, audio data, etc.), which are not specifically limited here. The complexity of the data set to be trained reflects the complexity of each data in the data set to be trained, which can be represented by parameters such as the feature dimension of the data set, the required memory size and / or the computing power size; the complexity of the model to be trained reflects the complexity of the model to be trained in the model structure, which can be represented by parameters such as the number of network layers of the model to be trained and the size of the parameters. The target group number information is calculated to indicate the information related to the number of groups when grouping all servers.

[0125] In one application scenario, all servers are directly divided into multiple server groups, and the above target group number information may include the number of server groups to be divided. In another application scenario, all servers are first divided into multiple server groups, and then multiple server groups are obtained from each server group. The above target group number information may include the number of server groups to be divided and the number of server groups to be divided in each server group. In actual use, if the servers are divided and grouped at multiple levels, the above target group number information may include the number of groups corresponding to each level, which is not specifically limited here.

[0126] In this embodiment, when the above-mentioned servers are divided into groups, servers with similar server intelligence levels are divided into the same server group, which helps the collaboration between servers in the server group and further improves the training efficiency.

[0127] As can be seen from the above, the distributed training device provided by the embodiment of the present invention obtains the server intelligence level of each server through the server intelligence level acquisition module 510, wherein each of the above server intelligence levels reflects the data processing capability of the corresponding server; obtains the target group number information based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers through the server group division module 520, and divides all the above servers into groups based on the target group number information and the server intelligence level of each of the above servers to obtain multiple server groups; distributes the data resources in the above data set to be trained and the model resources in the above model to be trained to each server group based on the group intelligence level of each server group through the resource distribution module 530, wherein each of the above group intelligence levels reflects the data processing capability of the corresponding server group; and controls each of the above server groups to perform training based on the obtained data resources and model resources through the training control module 540. Compared with the prior art method of only distributing the divided data and / or model to each server by randomly sampling or scrambling the data and horizontally or vertically dividing the model, the present invention takes into account the actual processing capacity of each server and distributes the corresponding data resources and model resources. At the same time, the servers are grouped according to their intelligence levels, so that servers with similar intelligence levels are divided into a server group, which facilitates the collaboration between the servers in the server group, thereby helping to improve the training efficiency.

[0128] In a specific implementation, the specific functions of the above-mentioned distributed training device and its modules can refer to the corresponding description in the above-mentioned distributed training method, which will not be repeated here.

[0129] In another specific embodiment, the above distributed training method is performed by a dynamic adaptive training system. Figure 7 is an architecture diagram of a dynamic adaptive training system provided by an embodiment of the present invention, such as Figure 7As shown, the above-mentioned dynamic adaptive training system includes a server intelligent measurement module, a server group clustering module, a dynamic parameterized group adaptive control module and an intelligent parallel allocation module. The server intelligent measurement module calculates the intelligence level of each server through video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption. The server group clustering module divides all servers into groups based on the intelligence level of each server through a clustering algorithm. The dynamic parameterized group adaptive control module performs parameterized adjustment according to the feature dimension and data volume of the data set to be trained, as well as the number of network layers and parameter volume of the model to be trained, to achieve dynamic adaptive control of the server group. The intelligent parallel allocation module allocates data and models to different server groups through an intelligent parallel allocation algorithm. Thereby, dynamic resource allocation and efficient training in the context of big data and large models are achieved.

[0130] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 8 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a distributed training program. The internal memory provides an environment for the operation of the operating system and the distributed training program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the distributed training program is executed by the processor, the steps of any of the above-mentioned distributed training methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0131] Those skilled in the art will understand that Figure 8 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In one embodiment, a smart terminal is provided, the smart terminal comprising a memory, a processor, and a distributed training program stored in the memory and executable on the processor, wherein the distributed training program performs the following operation instructions when executed by the processor:

[0133] Obtaining server intelligence levels of each server respectively, wherein each of the server intelligence levels respectively reflects the data processing capability of the corresponding server;

[0134] Based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each of the above servers, information on the number of target groups is obtained, and based on the information on the number of target groups and the server intelligence level of each of the above servers, all of the above servers are grouped to obtain multiple server groups;

[0135] Distributing the data resources in the dataset to be trained and the model resources in the model to be trained to each server group based on the group intelligence level of each server group, wherein each of the group intelligence levels reflects the data processing capability of the corresponding server group;

[0136] The server groups are respectively controlled to perform training based on the obtained data resources and model resources.

[0137] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0138] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0139] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0140] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0141] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0142] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0143] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A distributed training method, characterized in that: The method comprises: Obtaining server intelligence levels of each server respectively, wherein each server intelligence level reflects the data processing capability of the corresponding server; Based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each server, information about the number of target groups is obtained, and based on the information about the number of target groups and the server intelligence level of each server, all the servers are grouped to obtain a plurality of server groups; Distributing the data resources in the to-be-trained data set and the model resources in the to-be-trained model to each server group based on the group intelligence level of each server group, wherein each group intelligence level reflects the data processing capability of the corresponding server group; Respectively controlling each of the server groups to perform training based on the obtained data resources and model resources; The obtaining of the server intelligence level of each server respectively includes: Respectively obtain the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each server; The server intelligence level of each server is calculated based on the video processing capability, general computing capability, storage capability, communication capability, device access capability and energy consumption cost of each server; The step of obtaining target group number information based on the complexity of the data set to be trained, the complexity of the model to be trained, and the intelligence level of each server, and grouping all the servers based on the target group number information and the server intelligence level of each server to obtain multiple server groups includes: Obtaining data set information of a data set to be trained, and calculating the complexity of obtaining the data set to be trained based on the data set information, wherein the data set information includes a feature dimension, a required memory size, and a required computing power size of the data set to be trained; Obtain model information of a model to be trained, and calculate the complexity of the model to be trained based on the model information, wherein the model information includes the number of network layers and the size of parameters of the model to be trained; Calculating target group number information based on the intelligence level of each server, the complexity of the data set to be trained, and the complexity of the model to be trained, and grouping all the servers based on the target group number information and the server intelligence level of each server to obtain multiple server groups; The target group number information includes the target group number and the target group number corresponding to each server group, the target group number information is obtained by calculating based on the intelligence level of each server, the complexity of the obtained data set to be trained and the complexity of the model to be trained, and all the servers are grouped based on the target group number information and the server intelligence level of each server to obtain multiple server groups, including: The number of target groups is calculated based on the intelligence level of each server, the complexity of the obtained data set to be trained and the complexity of the model to be trained, wherein the number of target groups is the number of server groups that need to be obtained by dividing all the servers; Divide all the servers into groups based on the target group number to obtain server groups with the target group number; The to-be-trained data set and the to-be-trained model are divided based on the group intelligence level corresponding to each of the server groups to obtain a group data set and a group model corresponding to each of the server groups; Calculating and obtaining the number of target groups corresponding to each of the server groups based on the group intelligence level of each of the server groups, the complexity of the group data set, and the complexity of the group model; The servers in each of the server groups are divided based on the target group numbers corresponding to each of the server groups, and a plurality of server groups are obtained in each of the server groups.

2. The distributed training method according to claim 1, characterized in that: The server intelligence level is the ratio of the capability product of the corresponding server to the energy consumption cost, and the capability product is the product of the video processing capability, general computing capability, storage capability, communication capability and device access capability of the server.

3. The distributed training method according to claim 1, characterized in that: The calculating and obtaining the number of target groups based on the intelligence level of each server, the complexity of the obtained data set to be trained and the complexity of the model to be trained includes: Calculate the value obtained by multiplying the overall complexity by the overall intelligence level and take the integer, and use the obtained integer value as the number of the target group; The overall complexity product is the product of the complexity of the data set to be trained and the complexity of the model to be trained, and the overall intelligence level is the sum of the server intelligence levels of all the servers.

4. The distributed training method according to claim 1, characterized in that: The obtaining the number of target groups corresponding to each of the server groups by respectively calculating based on the group intelligence level of each of the server groups, the complexity of the group data set and the complexity of the group model comprises: Calculate and obtain the value of the group complexity product corresponding to each of the server groups divided by the group intelligence level and take the integer, and use the obtained integer value as the target group number corresponding to each of the server groups; The product of each group complexity is the product of the complexity of the group data set corresponding to each server group and the complexity of the group model, and the intelligence level of each group is the sum of the server intelligence levels of all servers in each server group.

5. The distributed training method according to claim 1, characterized in that: The dividing the servers in each of the server groups based on the number of target groups corresponding to each of the server groups to obtain a plurality of server groups in each of the server groups includes: Based on the target group number corresponding to each server group and the server intelligence level of each server in each server group, the servers in each server group are divided respectively by a clustering algorithm, and multiple server groups are obtained in each server group.

6. The distributed training method according to claim 1, characterized in that: The group intelligence level based on each server group distributes the data resources in the to-be-trained data set and the model resources in the to-be-trained model to each server group, wherein each group intelligence level reflects the data processing capability of the corresponding server group, including: In each of the server groups, data resources in the group data set and model resources in the group model are distributed to each server group based on the group intelligence level corresponding to each server group. Each group intelligence level is the sum of the server intelligence levels of all servers in each server group.

7. The distributed training method according to claim 1, characterized in that: The respectively controlling each of the server groups to perform training based on the obtained data resources and model resources includes: In each of the server groups, when the complexity of the obtained data resources is greater than the complexity of the obtained model resources, training is performed using a data parallel method; In each of the server groups, when the complexity of the obtained data resources is less than the complexity of the obtained model resources, training is performed using a model parallel method.

8. A distributed training device, used to implement the distributed training method according to any one of claims 1 to 7, characterized in that: The device comprises: A server intelligence level acquisition module, used to respectively acquire the server intelligence level of each server, wherein each server intelligence level respectively reflects the data processing capability of the corresponding server; A server group division module, used to obtain target group number information based on the complexity of the data set to be trained, the complexity of the model to be trained and the intelligence level of each server, and to divide all the servers into groups based on the target group number information and the server intelligence level of each server to obtain multiple server groups; A resource distribution module, used to distribute the data resources in the to-be-trained data set and the model resources in the to-be-trained model to each server group based on the group intelligence level of each server group, wherein each group intelligence level reflects the data processing capability of the corresponding server group; The training control module is used to control each of the server groups to perform training based on the obtained data resources and model resources.

9. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a distributed training program stored in the memory and executable on the processor. When the distributed training program is executed by the processor, the steps of the distributed training method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a distributed training program, which, when executed by a processor, implements the steps of the distributed training method according to any one of claims 1 to 7.

Images