A distributed modeling method and system

Abstract

The present application relates to the field of neural network technology, and in particular to a distributed modeling method and system, the method comprising: splitting a neural network model into multiple sub-models; inputting a training sample into the neural network model, forward propagating between each sub-model, obtaining an output result, calculating a preset loss between training sample label data, backpropagating each sub-model, obtaining a first parameter and a second parameter of each sub-model, wherein the freezing coefficient corresponding to the first parameter is 1, and the freezing coefficient corresponding to the second parameter is 0; obtaining a parameter combination of each freezing strategy; obtaining a performance score of each parameter combination in a test sample, and using the parameter combination with the maximum performance score as the neural network model of this iteration; performing multiple iterations until the number of iterations is greater than the preset number. The technical solution of the present application can improve the training accuracy and training efficiency of the neural network.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a distributed modeling method and system. Background Art

[0002] With the rapid development of computer technology, neural network models are widely used in various fields due to their powerful generalization ability. In order for the neural network model to output accurate results, it is necessary to train the neural network model to complete the modeling process of the neural network model. However, as the amount of data in the neural network model gradually increases, it becomes increasingly difficult to train the neural network model through a single device; in response to this situation, the distributed modeling method of the neural network model came into being.

[0003] At present, the patent application document with authorization announcement number CN115660034B discloses a method, device and system for distributed model training, wherein the method includes: inputting sample data into a first network to perform forward calculation to obtain an intermediate result; sending the intermediate result to a server; in response to receiving the output result sent by the server, calculating the loss value based on the label corresponding to the sample data and the output result, wherein the output result is obtained by forward calculation of the second network in the server based on the intermediate result; calculating the gradient of the output result according to the loss value; sending the gradient of the output result to the server; in response to receiving the parameter gradient of the second network sent by the server, performing reverse calculation of the first network to obtain the parameter gradient of the first network, wherein the parameter gradient of the second network is obtained by reverse calculation based on the gradient of the output result; updating the parameters of the first network based on the parameter gradient of the first network.

[0004] The above method deploys the first network and the second network on different nodes and realizes distributed model training through communication between nodes. However, updating the parameters of the first network and the second network at the same time in each iterative training will lead to overfitting and cannot guarantee the training accuracy and training efficiency of the neural network. Summary of the invention

[0005] In order to solve the technical problem of low training accuracy and training efficiency of neural networks, the present application provides a distributed modeling method and system, which can improve the training accuracy and training efficiency of neural networks.

[0006] In a first aspect, the present application provides a distributed modeling method, which includes: splitting a neural network model into multiple sub-models, and deploying each sub-model in a different computing node; inputting a training sample into the neural network model, performing forward propagation between each sub-model to obtain an output result, calculating a preset loss between the output result and the training sample label data, and performing backpropagation on each sub-model to obtain a first parameter and a second parameter of each sub-model, wherein the freezing coefficient corresponding to the first parameter is 1, the freezing coefficient corresponding to the second parameter is 0, and the second parameter is an update result of the first parameter; obtaining a parameter combination of each freezing strategy, wherein the freezing strategy is a combination of freezing coefficients of each sub-model, and the parameter combination includes the first parameter or the second parameter of each sub-model; obtaining a performance score of each parameter combination in a test sample, and taking the parameter combination with the maximum performance score as the neural network model of this iteration, wherein the performance score is negatively correlated with the preset loss of the test sample; performing multiple iterations until the number of iterations is greater than the preset number, and the distributed modeling is completed.

[0007] In one embodiment, the neural network model includes a fully connected neural network, a convolutional neural network, a recurrent neural network, or a hybrid neural network composed of multiple neural networks.

[0008] In one embodiment, the process of obtaining the output result includes: transmitting the training data to the computing node where the starting sub-model is located for forward propagation to obtain the intermediate output of the starting sub-model; transmitting the intermediate output to the computing node where the next adjacent sub-model is located until the intermediate output of the terminal sub-model is obtained, and the intermediate output of the terminal sub-model corresponds to the output result.

[0009] In one embodiment, the process of obtaining the first parameter and the second parameter of each sub-model includes: performing back propagation on the terminal sub-model to obtain the parameter gradient of each model parameter in the terminal sub-model, wherein each model parameter in the terminal sub-model corresponds to the first parameter of the terminal sub-model; updating the model parameters along the direction of the parameter gradient using the gradient descent method, and using the updated model parameters as the second parameter of the terminal sub-model; transmitting the parameter gradient of each model parameter in the terminal sub-model to the computing node where the previous adjacent sub-model is located, until the first parameter and the second parameter of the starting sub-model are obtained.

[0010] Freezing the model parameters of the sub-model can reduce the complexity of the neural network model and reduce the risk of overfitting. Here, the first parameter and the second parameter are generated for each sub-model to provide a data basis for avoiding overfitting and improving training accuracy in the future.

[0011] In one embodiment, the performance score is the inverse of a preset loss of the test sample.

[0012] In one embodiment, the parameter combination Performance Rating Satisfies the relationship:

[0013] , For this iteration Medium parameter combination The preset loss of the test sample, For the previous iteration The preset loss of the test sample in , is the preset loss of the training samples in this iteration.

[0014] when The larger the value, the greater the parameter combination. The greater the reduction in the preset loss of the test sample compared to the previous iteration, the greater the parameter combination The better the output accuracy improvement effect; Ability to characterize parameter combinations The better the output accuracy improvement and the smaller the overfitting degree, the better the performance score. The bigger.

[0015] In one embodiment, the modeling method further includes: in response to the current iteration belonging to the first K iterations, setting the freezing coefficient of each sub-model to 0, where K is a preset ratio of the preset number of times.

[0016] In the first K iterations, the model parameters of all sub-models will be updated, that is, the freezing coefficient of each sub-model in the freezing strategy is 0. The model parameters of each sub-model are fully updated first. Starting from the K+1th iteration, each sub-model will correspond to a first parameter and a second parameter, which ensures the training efficiency and improves the training accuracy, avoiding overfitting of the neural network model.

[0017] In one embodiment, splitting the neural network model into multiple sub-models includes: setting multiple initial splitting points and obtaining sub-models corresponding to the initial splitting points; constructing an objective function:

[0018] ,in, , and Respectively represent The number of model parameters, model calculation amount, and output size of intermediate outputs of each sub-model. is the number of all sub-models, For calculation The variance of the sum of the model parameters and the model calculation in each sub-model, is the adjustment coefficient, is the value of the objective function; the multiple initial splitting points are continuously adjusted, the initial splitting point when the objective function takes the minimum value is used as the preset splitting point, and the sub-model corresponding to the preset splitting point is obtained.

[0019] when When the value of approaches 0, it means that the sum of the model parameters and the model calculation amount in each sub-model is basically the same, ensuring the load balance among multiple computing nodes; It can reflect the transmission volume of the intermediate results of each sub-model between multiple computing nodes. The smaller the objective function is, the more it ensures the load balancing of all computing nodes while reducing the overhead of data transmission between computing nodes.

[0020] In a second aspect of the present application, a distributed modeling system is provided, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a distributed modeling method according to the first aspect of the present application is implemented.

[0021] The technical solution of this application has the following beneficial technical effects:

[0022] First, the neural network model is split into multiple sub-models, and each sub-model is deployed in a different computing unit, and a neural network model is trained with the help of multiple computing units; in one iteration, by setting the freezing coefficient, the parameters of a sub-model in two cases where the model parameters are frozen (the freezing coefficient is 1) and the model parameters are not frozen (the freezing coefficient is 0), namely the first parameter and the second parameter; the combination of the freezing coefficients of each sub-model is used as the freezing strategy, and the performance scores of the parameter combinations corresponding to different freezing strategies are calculated in the test samples, and the parameter combination with the maximum performance score is selected as the neural network model of this iteration to complete one iteration. In this way, the best parameter combination corresponding to the freezing strategy can be selected in each iteration to avoid the decrease in training accuracy caused by overfitting, until the number of iterations is greater than the preset number, the distributed modeling is completed, and the training accuracy and training efficiency of the neural network are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0024] Figure 1 It is a schematic diagram of pipeline parallelism;

[0025] Figure 2 is a flow chart of a distributed modeling method according to an embodiment of the present application;

[0026] Figure 3 It is a structural block diagram of a distributed modeling system according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0028] It should be understood that when the terms "first", "second", etc. are used in the claims, specification and drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0029] According to the first aspect of the present application, the present application provides a distributed modeling method for distributed training of a neural network model. In the era of large models, the number of parameters in a neural network model is huge, and it is difficult to obtain a neural network model with high training accuracy on a single device. In order to solve this problem, pipeline parallelism is an effective solution. Specifically, please refer to Figure 1 , is a schematic diagram of pipeline parallelism, which divides a neural network model into several model blocks by layer, and assigns each model block to a GPU device; in the process of forward propagation, each GPU device passes the intermediate output to the GPU device where the next model block is located until the output result of the neural network model is obtained; in the process of backward propagation, each GPU device passes the parameter gradient of each parameter back to the GPU device where the previous model block is located, and uses gradient descent to update the model parameters of all model blocks in turn; in this way, the training of the neural network model is completed in multiple GPU devices, and in the process of training, data flows on multiple GPU devices like water, realizing distributed modeling of the neural network model.

[0030] Figure 2 is a flow chart of a distributed modeling method according to an embodiment of the present application. Figure 2 As shown, the distributed modeling method 100 includes steps S101 to S105, which are described in detail below.

[0031] S101, split the neural network into multiple sub-models, and deploy each sub-model in a different computing node.

[0032] In one embodiment, the neural network model is a layered structure, and each layer includes multiple model parameters. The neural network model includes a fully connected neural network, a convolutional neural network, a recurrent neural network, or a hybrid neural network composed of multiple neural networks. The present application does not limit the type of neural network model; for example, a hybrid neural network composed of a convolutional neural network and a recurrent neural network. The neural network model is split into multiple sub-models with a sequence according to a preset split point, and each sub-model is deployed in a different computing node, and the computing node is a GPU device or a CPU device.

[0033] For example, if the neural network model is a convolutional neural network ResNet50, ResNet50 includes 175 layer structures, each of which corresponds to a name, output size, and model parameter quantity. Taking the layer structure named conv2_block1_2_conv as an example, its output size is 56×56×64, and the number of model parameters it contains is 36928; taking the layer structure named conv4_block3_2_conv as an example, its output size is 14×14×256, and the number of model parameters it contains is 590080; Block1_2_conv and conv4_block3_2_conv are used as preset splitting points to split the neural network model. The layer structure between the output layer and conv2_block1_2_conv constitutes sub-model 1, the layer structure between conv2_block1_2_conv and conv4_block3_2_conv constitutes sub-model 2, and the layer structure after conv4_block3_2_conv constitutes sub-model 3; that is to say, all sub-models are spliced ​​together in order to obtain the neural network model.

[0034] S102, input the training sample into the neural network model, perform forward propagation between each sub-model to obtain the output result, calculate the preset loss between the output result and the training sample label data, perform back propagation on each sub-model, and obtain the first parameter and the second parameter of each sub-model, the freezing coefficient corresponding to the first parameter is 0, the freezing coefficient corresponding to the second parameter is 1, and the second parameter is the update result of the first parameter.

[0035] In one embodiment, one training sample corresponds to one label data. For example, when using the convolutional neural network ResNet50 for image classification, the training sample is the image data, and the label data is the category label corresponding to the label data.

[0036] Specifically, the process of obtaining the output result includes: transmitting the training data to the computing node where the starting sub-model is located for forward propagation to obtain the intermediate output of the starting sub-model; transmitting the intermediate output to the computing node where the next adjacent sub-model is located until the intermediate output of the terminal sub-model is obtained, and the intermediate output of the terminal sub-model corresponds to the output result.

[0037] Among them, there is a sequence between multiple sub-models, and the intermediate output of the previous sub-model is the input of the subsequent sub-model. Therefore, in order to obtain the output result of the training data, it is necessary to perform forward propagation in multiple computing nodes in sequence according to the sequence of the sub-models. The preset loss is calculated based on the output result and the label data of the training sample. The preset loss is used to characterize the difference between the output result and the label data. Specifically, the preset loss is related to the task type of the neural network model. When the neural network model is a classification task, the preset loss is the cross entropy loss function. When the neural network model is a regression task, the preset loss is the mean square error loss function.

[0038] In one embodiment, the process of obtaining the first parameter and the second parameter of each sub-model includes: performing back propagation on the terminal sub-model to obtain the parameter gradient of each model parameter in the terminal sub-model, wherein each model parameter in the terminal sub-model corresponds to the first parameter of the terminal sub-model; updating the model parameters along the direction of the parameter gradient using the gradient descent method, and using the updated model parameters as the second parameter of the terminal sub-model; transmitting the parameter gradient of each model parameter in the terminal sub-model to the computing node where the previous adjacent sub-model is located, until the first parameter and the second parameter of the starting sub-model are obtained.

[0039] Among them, during the back propagation process, the freezing coefficient can be set to control whether the model parameters in the sub-model are updated; when the freezing coefficient of a sub-model is 1, it means that the model parameters in the sub-model are frozen and the model parameters in the sub-model are not updated; when the freezing coefficient of a sub-model is 0, it means that the model parameters in the sub-model are not frozen, and the model parameters in the sub-model are updated using the gradient descent method during the back propagation process. By freezing the model parameters of the sub-model, the complexity of the neural network model can be reduced, the risk of overfitting can be reduced, and over-training of the model parameters of the sub-model can be avoided.

[0040] Specifically, the model parameters As an example, the process of updating the model parameters along the direction of the parameter gradient using the gradient descent method is: ,in is the model parameter The parameter gradient of is the preset learning rate in this iteration, is the updated model parameter. The preset learning rate in this iteration is 0.1. It can be understood that each sub-model corresponds to a first parameter and a second parameter, the freezing coefficient corresponding to the first parameter is 1, and the freezing coefficient corresponding to the second parameter is 0.

[0041] It should be noted that the preset learning rate is negatively correlated with the number of iterations, and as the number of iterations increases, the preset learning rate gradually decreases.

[0042] In this way, during the back propagation process, the freezing coefficients 0 and 1 are set for each sub-model respectively to obtain two sets of model parameters. The model parameters when the freezing coefficient is 1 are used as the first parameters, and the model parameters when the freezing coefficient is 0 are used as the second parameters.

[0043] S103, obtaining a parameter combination of each freezing strategy, wherein the freezing strategy is a combination of freezing coefficients of each sub-model, and the parameter combination includes a first parameter or a second parameter of each sub-model.

[0044] In one embodiment, the freezing strategy is a combination of freezing coefficients of each sub-model. For example, if there are 3 sub-models and the freezing coefficient corresponding to each sub-model is 0 or 1, then the total number of sub-models that can be obtained is A freezing strategy.

[0045] For any freezing strategy, the first parameter or the second parameter of the corresponding sub-model is selected according to the freezing coefficient of each sub-model in the freezing strategy to obtain the parameter combination of the freezing strategy. For example, there are three sub-models, namely, sub-model 1, sub-model 2 and sub-model 3, and the freezing strategy is 101; then the first parameter of sub-model 1, the second parameter of sub-model 2 and the first parameter of sub-model 3 are selected to obtain the parameter combination of freezing strategy 101.

[0046] S104, obtaining the performance score of each parameter combination in the test sample, and taking the parameter combination with the maximum performance score as the neural network model of this iteration, and the performance score is negatively correlated with the preset loss of the test sample.

[0047] In one embodiment, the test sample is sample data different from the training sample, and each test sample also corresponds to a label data. For a parameter combination, the test sample is input into each sub-model in the parameter combination, and the preset loss between the output result and the test sample label data is calculated to obtain the preset loss of the test sample; the smaller the preset loss of the test sample, the higher the output accuracy of the corresponding parameter combination on the test sample, and the greater the performance score. Specifically, the performance score is the opposite of the preset loss of the test sample.

[0048] In another embodiment, the modeling process of the neural network model requires multiple iterations. In order to ensure that the output accuracy of the neural network model is improved after each iteration, the performance score also needs to reflect the accuracy improvement effect of each parameter combination relative to the neural network model of the previous iteration, as well as the degree of overfitting of each parameter combination.

[0049] Specifically, the parameter combination Performance Rating Satisfies the relationship: , For this iteration Medium parameter combination The preset loss of the test sample, For the previous iteration The preset loss of the test sample in , is the preset loss of the training samples in this iteration.

[0050] in, The smaller the value, the greater the value of this iteration. Medium parameter combination The higher the output accuracy on the test sample, the The larger the value, the greater the parameter combination. The greater the reduction in the preset loss of the test sample compared to the previous iteration, the greater the parameter combination The better the output accuracy improvement, the better the performance score. The bigger; Represents the difference in preset loss between training samples and test samples. When the parameter combination The default loss of the test sample Approaching the preset loss of training samples When , it means that the output accuracy of the neural network model on the test sample and the training sample is close, and the parameter combination There is no overfitting, and the parameter combination The default loss of the test sample The preset loss of the training sample When the difference between them is large, it means that the output accuracy of the neural network model on the test sample and the training sample is very different, and the parameter combination There is overfitting, so Ability to characterize parameter combinations The smaller the overfitting degree, the better the performance score. The bigger.

[0051] In this way, the performance score of each parameter combination is obtained. The larger the performance score, the smaller the overfitting degree of the corresponding parameter combination is, and the better the accuracy improvement effect is. Since one parameter combination includes the model parameters of all sub-models, the parameter combination with the maximum performance score can be directly used as the neural network model of this iteration to complete this iteration.

[0052] S105, executing multiple iterations until the number of iterations is greater than a preset number, and then stopping to complete the distributed modeling.

[0053] In one embodiment, new training samples are continuously selected and input into the neural network model, and the iterative operation is repeated until the number of iterations is greater than a preset number of times, and the distributed modeling is completed to obtain a trained neural network model, and the preset number of times is 300.

[0054] In this way, the modeling process of the neural network model is completed. The overfitting degree of the trained neural network model is smaller and the accuracy is higher. By splitting the model, the modeling process of the neural network model is completed on multiple computing nodes, which improves the training accuracy and training efficiency of the neural network.

[0055] In another embodiment, in the above distributed modeling method, during one iteration, each sub-model has two sets of parameters, namely, the first parameter and the second parameter, and it is necessary to calculate the performance score of the test sample in each parameter combination. Although the training accuracy of the neural network is improved and overfitting is avoided, one iteration will increase the amount of calculation. Therefore, the modeling method also includes: in response to this iteration belonging to the first K iterations, the freezing coefficients of each sub-model are set to 0.

[0056] The value of K is a preset ratio of a preset number of times, and the preset ratio is 50%.

[0057] Exemplarily, the preset number of times is 300. In the first 150 iterations, the model parameters of all sub-models will be updated, that is, the freezing coefficient of each sub-model in the freezing strategy is 0. In other words, in the first 150 iterations, each sub-model only corresponds to the second parameter; starting from the 151st iteration, each sub-model will correspond to a first parameter and a second parameter, which ensures the training efficiency, improves the training accuracy, and avoids overfitting of the neural network model.

[0058] In another embodiment, due to the different model parameter quantities and output sizes of different layer structures, after each sub-model is deployed on different computing nodes, there will be differences in the storage load and computing load in different computing nodes, resulting in unbalanced loads among multiple computing nodes. In addition, intermediate output results need to be transmitted between different sub-models. In order to ensure load balancing of all computing nodes in the distributed modeling process, the neural network model can be adaptively split.

[0059] Specifically, splitting the neural network model into multiple sub-models includes: setting multiple initial splitting points and obtaining sub-models corresponding to the initial splitting points; constructing an objective function:

[0060] , , and Respectively represent The number of model parameters, model calculation amount, and output size of intermediate outputs of each sub-model. is the number of all sub-models, For calculation The variance of the sum of the model parameters and the model calculation in each sub-model, is the adjustment coefficient, is the value of the objective function; the multiple initial split points are continuously adjusted, the initial split point when the objective function takes the minimum value is used as the preset split point, and the sub-model corresponding to the preset split point is obtained. Among them, the adjustment coefficient The value of is 0.001.

[0061] The calculation method of the model parameter amount and the model calculation amount is related to the type of neural network model, which is common knowledge in the field and will not be repeated here. When the value of approaches 0, it means that the sum of the model parameters and the model calculation amount in each sub-model is basically the same, ensuring the load balance among multiple computing nodes; at the same time, It can reflect the transmission volume of the intermediate results of each sub-model between multiple computing nodes. The smaller the objective function is, the more it ensures the load balancing of all computing nodes while reducing the overhead of data transmission between computing nodes.

[0062] The technical principles and implementation details of the distributed modeling method of the present application are introduced above through specific embodiments. First, the neural network model is split into multiple sub-models, and each sub-model is deployed in different computing units, and a neural network model is trained with the help of multiple computing units; in one iteration, by setting the freezing coefficient, the parameters of a sub-model in two cases where the model parameters are frozen (the freezing coefficient is 1) and the model parameters are not frozen (the freezing coefficient is 0), that is, the first parameter and the second parameter; the combination of the freezing coefficients of each sub-model is used as the freezing strategy, and the performance scores of the parameter combinations corresponding to different freezing strategies are calculated in the test samples, and the parameter combination with the maximum performance score is selected as the neural network model of this iteration, and one iteration is completed. In this way, the best parameter combination corresponding to the freezing strategy can be selected in each iteration to avoid the decrease in training accuracy caused by overfitting, until the number of iterations is greater than the preset number of times, the distributed modeling is completed, and the training accuracy and training efficiency of the neural network are improved.

[0063] According to the second aspect of the present application, the present application also provides a distributed modeling system. Figure 3 is a structural block diagram of a distributed modeling system according to an embodiment of the present application. Figure 3 As shown, the system 50 includes a processor and a memory, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a distributed modeling method according to the first aspect of the present application is implemented. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, whose settings and functions are known in the art, so they are not described here.

[0064] In the present application, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in this application may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0065] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A distributed modeling method, characterized in that: The modeling method comprises: Splitting the neural network model into multiple sub-models, and deploying each sub-model in a different computing node, wherein the computing node is a GPU device or a CPU device; Input the training sample into the neural network model, perform forward propagation between each sub-model to obtain an output result, calculate the preset loss between the output result and the training sample label data, perform back propagation on each sub-model, obtain the first parameter and the second parameter of each sub-model, the freezing coefficient corresponding to the first parameter is 1, the freezing coefficient corresponding to the second parameter is 0, and the second parameter is the update result of the first parameter; Obtaining a parameter combination of each freezing strategy, wherein the freezing strategy is a combination of freezing coefficients of each sub-model, and the parameter combination includes a first parameter or a second parameter of each sub-model; The performance score of each parameter combination is obtained in the test sample, and the parameter combination with the maximum performance score is used as the neural network model of this iteration. The performance score is negatively correlated with the preset loss of the test sample. Multiple iterations are performed until the number of iterations exceeds the preset number, and the distributed modeling is completed.

2. A distributed modeling method according to claim 1, characterized in that: The neural network model includes a fully connected neural network, a convolutional neural network, a recurrent neural network or a hybrid neural network composed of multiple neural networks.

3. A distributed modeling method according to claim 1, characterized in that: The process of obtaining the output results includes: The training data is transmitted to the computing node where the starting sub-model is located for forward propagation to obtain the intermediate output of the starting sub-model; The intermediate output is transmitted to the computing node where the next adjacent sub-model is located until the intermediate output of the terminal sub-model is obtained, and the intermediate output of the terminal sub-model corresponds to the output result.

4. A distributed modeling method according to claim 1, characterized in that: The process of obtaining the first parameter and the second parameter of each sub-model includes: Performing back propagation on the endpoint sub-model to obtain parameter gradients of each model parameter in the endpoint sub-model, wherein each model parameter in the endpoint sub-model corresponds to a first parameter of the endpoint sub-model; The model parameters are updated along the direction of the parameter gradient using the gradient descent method, and the updated model parameters are used as the second parameters of the endpoint sub-model; The parameter gradient of each model parameter in the end sub-model is transmitted to the calculation node where the previous adjacent sub-model is located, until the first parameter and the second parameter of the starting sub-model are obtained.

5. A distributed modeling method according to claim 1, characterized in that: The performance score is the inverse of the preset loss of the test sample.

6. A distributed modeling method according to claim 1, characterized in that: Parameter combination Performance Rating Satisfies the relationship: , For this iteration Medium parameter combination The preset loss of the test sample, For the previous iteration The preset loss of the test sample in , is the preset loss of the training samples in this iteration.

7. A distributed modeling method according to claim 1, characterized in that: The modeling method also includes: In response to the current iteration belonging to the first K iterations, the freezing coefficient of each sub-model is set to 0, and K is a preset ratio of the preset number of times.

8. A distributed modeling method according to claim 1, characterized in that: The splitting of the neural network model into multiple sub-models includes: Set multiple initial split points and obtain the sub-models corresponding to the initial split points; Construct the objective function: ,in, , and Respectively represent The number of model parameters, model calculation amount, and output size of intermediate outputs of each sub-model. is the number of all sub-models, For calculation The variance of the sum of the model parameters and the model calculation in each sub-model, is the adjustment coefficient, is the value of the objective function; The multiple initial splitting points are continuously adjusted, the initial splitting point when the objective function takes the minimum value is used as the preset splitting point, and the sub-model corresponding to the preset splitting point is obtained.

9. A distributed modeling system, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a distributed modeling method according to any one of claims 1 to 8 is implemented.

Images