Parameter processing method, device and system
By using the parameter backup and recovery method of the master node to the computing node in distributed training, the resource waste caused by training interrupts is solved, breakpoint training is achieved, and training efficiency is improved.
Patent Information
- Application Number
- CN202310866370.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-13
- Publication Date
- 2025-01-14
AI Technical Summary
Training interruptions caused by abnormalities such as power outages and downtime in distributed training scenarios need to be retrained from scratch, wasting time and resources.
Provides a parameter processing method, which sends parameter export commands to the calculation node through the master node, backups and restores the parameter values of the training task, and realizes breakpoint training and avoids recalculating the previous model and parameters.
It saves training time and resources for distributed training, realizes efficient backup and recovery of parameters, and avoids resource waste caused by interrupts.
Smart Images
Figure CN119311470A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence (AI) model training technology, and in particular to parameter processing methods, devices and systems. Background Art
[0002] The parameters in the AI model are the values that can be obtained by training the AI model, such as weight, global step, counter for random numbers, etc. These parameters need to be continuously updated during the training process, and the parameter values affect the results of each iterative training.
[0003] For distributed training scenarios, the parameters used for these training need to be deployed to different devices respectively, and then continuously updated iteratively. During the distributed training process, it is inevitable that the distributed training will be interrupted due to power outages, downtime, etc. If the distributed training is restarted from the beginning, training time and training resources will be wasted. Summary of the invention
[0004] The embodiments of the present application provide a parameter processing method, device and system, which can save training time and training resources for distributed training.
[0005] In the first aspect, the present application provides a parameter processing method, including: when determining to perform parameter backup of a first training job, sending a first parameter export command to a first computing node; receiving a parameter value of a first training task sent by the first computing node in response to the parameter export command; the first training task is a training task of the first training job; storing the parameter value of the first training task as a backup parameter value of the first training task; when determining to perform breakpoint resume training of the first training job, sending the backup parameter value of the first training task to the first computing node, for use as the parameter value of the breakpoint resume training of the first training task. This method can realize the backup and recovery of the parameter value in the first computing node of the first training job by the master node, saving the training time and training resources of distributed training.
[0006] In a possible implementation, it also includes: when it is determined to perform parameter backup of the first training job, sending a second parameter export command to the second computing node; receiving the parameter value of the second training task sent by the second computing node in response to the parameter export command; the second training task is the training task of the first training job; storing the parameter value of the second training task as the backup parameter value of the second training task; when it is determined to perform breakpoint resume training of the first training job, sending the backup parameter value of the second training task to the second computing node, for use as the parameter value for breakpoint resume training of the second training task.
[0007] In a possible implementation manner, the method further includes: when the number of iterations of the first training job reaches a preset first threshold, determining to perform parameter backup of the first training job.
[0008] In a possible implementation, the method further includes: when it is detected that at least one computing node corresponding to the first training job fails, instructing a computing node corresponding to the first training job that has not failed to stop a training task of the first training job.
[0009] In a possible implementation, determining to perform breakpoint resume of the first training job includes: receiving a breakpoint resume request for the first training job from a user, and determining to perform breakpoint resume of the first training job.
[0010] In the second aspect, the embodiment of the present application provides a parameter processing method, including: receiving a first parameter export command for a first training job sent by a master node; the first parameter export command is sent by the master node when determining to perform parameter backup of the first training job; in response to the first parameter export command, sending a parameter value of the first training task to the master node, so that the master node stores the parameter value of the first training task as a backup parameter value of the first training task; the first training task is a training task of the first training job; receiving the backup parameter value of the first training task sent by the master node, which is used as the parameter value for breakpoint resumption of training of the first training task. In this method, the first computing node can cooperate with the master node to realize the backup and recovery of the parameter value in the first computing node of the first training job by the master node, saving training time and training resources for distributed training.
[0011] In a possible implementation, before receiving the backup parameter value of the first training task sent by the master node, the method further includes: receiving a first stop instruction for the first training job; and stopping the first training task or stopping parameter update of the first training task in response to the first stop instruction.
[0012] In the third aspect, an embodiment of the present application provides a master node, comprising: one or more processors; a memory; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, enable the master node to execute any one of the methods of the first aspect.
[0013] In a fourth aspect, an embodiment of the present application provides a computing node, comprising: one or more processors; a memory; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, enable the computing node to execute any one of the methods of the second aspect.
[0014] In a fifth aspect, an embodiment of the present application provides a parameter processing system, comprising the master node of the third aspect and at least one computing node of the fourth aspect.
[0015] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes any one of the methods of the first aspect or the second aspect.
[0016] In a seventh aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method of the first aspect.
[0017] In one possible design, the program in the seventh aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A schematic diagram of a scenario structure to which the parameter processing method provided in the embodiment of the present application is applicable;
[0019] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0020] Figure 3 A flow chart of a parameter processing method provided in an embodiment of the present application;
[0021] Figure 4 Another schematic diagram of a flow chart of a parameter processing method provided in an embodiment of the present application;
[0022] Figure 5 A schematic diagram of the structure of the CC training architecture provided in an embodiment of the present application;
[0023] Figure 6 A flow chart of a parameter processing method based on a CC training architecture provided in an embodiment of the present application;
[0024] Figure 7 A schematic diagram of the structure of the PS training architecture provided in an embodiment of the present application;
[0025] Figure 8 Another flowchart of a parameter processing method based on a PS training architecture provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.
[0027] The parameters in the AI model are the values that can be obtained by training the AI model, such as weight, global step, counter for random numbers, etc. These parameters need to be continuously updated during the training process, and the parameter values affect the results of each iteration of training. In some scenarios, these parameters can also be called variables.
[0028] For distributed training scenarios, the parameters used for these calculations need to be deployed to different node devices, and then continuously updated iteratively. During the distributed training process, it is inevitable that some node devices will be interrupted due to power outages or downtime. If the AI model training is repeated from scratch, it will waste training time and resources.
[0029] To this end, the embodiments of the present application provide a parameter processing method, device and system that can resume distributed training at breakpoints, saving training time and training resources for distributed training.
[0030] To achieve breakpoint resume training, you must first save the previously trained AI model and parameter information during the training process. In addition, you need to export and back up all parameters at regular intervals, so that when training is interrupted and training is resumed, the model and parameters backed up at a certain time can be imported into the corresponding node device. In this way, you can use the model and parameters backed up at a certain time as the starting point for retraining, and the model and parameters before that time do not need to be recalculated, thereby achieving breakpoint resume training and saving training time and training resources.
[0031] Tensorflow, pytorch and other frameworks for stand-alone AI training support variable backup (tf.train.saver / torch.save) and recovery (tf.train.restore / torch.load) interfaces, but lack the processing of distributed scenarios. In the embodiment of the present application, for distributed training scenarios, the master node can uniformly perform parameter backup and parameter recovery.
[0032] First, the nouns involved in the embodiments of the present application are exemplified.
[0033] AI model is a kind of mathematical algorithm model that uses machine learning ideas to solve practical problems. The AI model includes a large number of parameters and calculation formulas (or calculation rules). The AI model involved in the embodiments of the present application is a general term and may include deep learning models, machine learning models, etc. AI models generally need to be trained before they are used to complete a task.
[0034] Parameters in an AI model: are numerical values that can be obtained by training an AI model with a training data set. For example, parameters of an AI model are calculation formulas or weights of calculation factors in the AI model.
[0035] Hyper parameters in AI models: Parameters that cannot be obtained by training AI models through training data sets. Hyper parameters can be used to guide the construction of AI models or the training of AI models. Hyper parameters may include: the number of iterations of AI model training, the learning rate, the batch size, the number of layers of the AI model, the number of neurons in each layer, etc.
[0036] The main difference between hyperparameters and parameters in AI models is that the values of AI model hyperparameters cannot be obtained by analyzing the training data set, while the values of AI model parameters can be modified and determined based on the analysis of the training data set during the training process.
[0037] The parameters backed up and restored in the embodiments of the present application may be parameters whose values change during AI model training. Therefore, the parameters backed up and restored in the embodiments of the present application may also be referred to as variables in the AI model, and may include, for example: parameters in the above-mentioned AI model, and some hyperparameters, such as the number of iterations.
[0038] The parameters for backup and recovery in the embodiments of the present application can support generalization, without limiting specific parameters and business scenarios, and are therefore not limited to the variable parameters used in the optimizer or initializer, and can be freely expanded and named independently, and can be parsed and processed in the computing node.
[0039] Training an AI model means using existing data to fit the AI model to the rules of existing data through certain methods and determine the parameters in the AI model. Training an AI model requires preparing a training data set. Depending on whether the training data in the training data set is labeled (i.e., whether the data corresponds to specific label information, such as type, name, and label box contained in the data), the training of the AI model can be divided into supervised training and unsupervised training. When the AI model is supervised, the training data in the training data set used for training is labeled. When training the AI model, the training data in the training data set is used as the input of the AI model, and the AI model calculates the input training data to obtain the output value of the AI model. The label corresponding to the training data is used as a reference for the output value of the AI model, and the loss function is used to calculate the loss value of the output value of the AI model and the label corresponding to the training data, and the parameters in the AI model are adjusted according to the loss value. The AI model is iteratively trained with each training data in the training data set, and the parameters of the AI model are continuously adjusted until the AI model can output the same or similar output value as the label corresponding to the training data with high accuracy based on the input training data. When the AI model is trained unsupervised, the training data in the training data set is not labeled, and the training data in the training data set is input into the AI model in sequence, and the AI model gradually identifies the associations and potential rules between the training data in the training data set until the AI model can be used to judge or identify the type or characteristics of the input data. For example, in clustering, after receiving a large amount of training data, the AI model used for clustering can learn the characteristics of each training data and the associations and differences between the training data, and automatically divide the training data into multiple types. Different AI models can be used for different task types. Some AI models can only be trained by supervised learning, some AI models can only be trained by unsupervised learning, and some AI models can be trained by both supervised learning and unsupervised learning. The trained AI model can be used to complete a specific task. Generally speaking, AI models in machine learning need to be trained by supervised learning. Training AI models by supervised learning can enable AI models to learn the associations between training data and corresponding annotations in the training data set with annotations more specifically, so that the trained AI model has a higher accuracy rate when used to predict other input reasoning data.
[0040] The loss function is a function used to measure the degree to which the AI model has been trained. In the process of training the AI model, because we hope that the output of the AI model is as close as possible to the value we really want to predict, we can compare the predicted value of the current AI model based on the input data with the target value we really want, and then update the parameters in the AI model based on the difference between the two. Each training uses the loss function to determine the difference between the value predicted by the current AI model and the actual target value, and updates the parameters of the AI model until the AI model can predict the target value we really want or a value very close to the target value we really want, that is, the loss function is less than the threshold and is relatively stable, then the AI model is considered to be trained.
[0041] Gradient: A vector of partial derivatives of a function. During the training of an AI model, the model parameters need to be adjusted to make the loss function of the next iteration smaller. The gradient descent method is often used to update the model parameters. Therefore, in each iteration, the gradient of the loss function corresponding to the training data needs to be calculated, and then the parameters of the AI model are updated according to the gradient.
[0042] Distributed training: Split the training into multiple independent computing nodes for independent calculations, and then periodically summarize and redistribute the results, thereby accelerating the training process of the AI model. Distributed training can include data parallelism, model parallelism, hybrid parallelism, pipeline parallelism and other parallel methods. Distributed training is one of the commonly used acceleration methods in the AI model training process.
[0043] AI platform: a platform that provides a convenient AI development environment and convenient development tools for AI developers and users. The AI platform has built-in pre-trained AI models or AI sub-models for solving different problems. The AI platform can search and establish applicable AI models according to user needs. Users only need to determine their needs in the AI platform and prepare the training data set according to the prompts and upload it to the AI platform. The AI platform can train an AI model for the user that can be used to achieve the user's needs. Alternatively, the user prepares his own algorithm (also called the initial AI model) and training data set according to the prompts and uploads them to the AI platform. The AI platform can train an AI model that can be used to achieve the user's needs based on the user's own algorithm and training data set. Users can use the trained AI model to complete their specific tasks. It should be understood that the AI model before being trained by the AI platform in the embodiment of the present application (for example, the algorithm uploaded by the user, the algorithm preset by the AI platform, or the pre-trained model) is called the initial AI model.
[0044] Training job: A set of programs that need to be executed to complete the training of an initial AI model. The completion of a training job means that the training of an initial AI model is completed and a trained AI model is obtained.
[0045] Training task: A single program in a set of programs corresponding to a training job; that is, a task logic instance submitted by a user, which identifies the difference between tasks. For example, a training task for an initial AI model is used to perform multiple rounds of iterative training on the initial AI model. Among them, a training job includes at least one training task.
[0046] Training process: is a process of a training task executed on a computing node. One training process corresponds to one training task, and one computing node can execute one or more training tasks, so there are one or more training processes on one computing node.
[0047] Training framework: It is a toolkit or function package that is needed during the AI model training process. It is a running program framework that each training task in the training job needs to rely on. In the initial stage of deep learning, each deep learning researcher needs to write a lot of repetitive code. In order to improve work efficiency, researchers write these codes into a framework and put it online for all researchers to use. This framework is also called the training framework. Currently, the most popular deep learning frameworks in the world are Tensorflow, Caffe, Theano, MXNet, Torch, and PyTorch.
[0048] Optimizer: Used to update the parameters in the AI model.
[0049] Scheduler: Used to adjust the learning rate according to a predefined strategy.
[0050] Collective Communications is a global communication operation in which all processes in a process group participate. The most basic operations include send, receive, copy, intra-group process barrier synchronization Barrier, and inter-node process synchronization (signal+wait). These basic operations are combined to form a set of communication templates, also called communication primitives, such as: 1-to-many broadcast, many-to-1 gather, many-to-many all-gather, 1-to-many scatter, many-to-1 reduce, many-to-many all-reduce, combined reduce and divergent reduce-scatter, many-to-many all-to-all, etc.
[0051] See also Figure 1Schematic diagram of the scene structure applicable to the parameter processing method of the embodiment of the present application, including: a master node and several computing nodes, and the master node can be provided with an AI platform. The master node can schedule computing nodes through the AI platform for AI model training. Each computing node can be a computing device (such as a server) or a computing card (such as a GPU).
[0052] For example, Figure 2 The schematic diagram of the structure of an electronic device 100 that can be used as a master node is shown. The electronic device 100 may include a processor 110, a memory 120, a communication interface 130, and the like.
[0053] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0054] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0055] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.
[0056] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0057] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0058] The memory 120 can be used to store computer executable program codes, which include instructions. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as a phone book, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 120, and / or instructions stored in a memory provided in the processor.
[0059] The communication interface 130 is used to implement communication between the electronic device 100 and other devices or communication networks. For example, the communication interface 130 can be used to communicate with other computing nodes.
[0060] It should be noted that when the computing node is implemented by an electronic device, its implementation structure can also refer to Figure 2 The structure of the electronic device shown.
[0061] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
[0062] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 1Taking the electronic device with the structure shown as an example, the method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.
[0063] Figure 3 is a flow chart of a parameter processing method according to an embodiment of the present application. Figure 3 As shown, the method may include:
[0064] Step 301: When the master node determines to perform parameter backup for the first training job, it sends a first parameter export command to computing node 1.
[0065] Optionally, the triggering time for performing parameter backup of the first training job may be determined by the master node or specified by the user. For example, it may be that the user performs parameter backup for the first training job in the user interface (UI) provided by the AI platform, or the number of iterations reaches a preset threshold.
[0066] The first parameter export command may be used to instruct computing node 1 to export parameters of a training task of a first training job.
[0067] Optionally, if an association relationship between a training task and a training job is set in computing node 1, the first parameter export command may include: an identifier of the first training job. Accordingly, computing node 1 may obtain the training task in computing node 1 that belongs to the first training job based on the identifier of the first training job.
[0068] Optionally, if the association relationship between the training task and the training job is not set in computing node 1, the first parameter export command may include: an identifier of the training task. Accordingly, computing node 1 may obtain the training task in computing node 1 that belongs to the first training job based on the identifier of the training task. The training task in computing node 1 that belongs to the first training job may be one or more, and therefore, the identifier of the training task included in the first parameter export command may be an identifier of one training task or identifiers of multiple training tasks.
[0069] In the subsequent step 302, it is taken as an example that the training task 1 in the computing node 1 belongs to the first training job.
[0070] The computing node 1 in this step may be implemented differently in different distributed training architectures. For example, in a distributed training architecture of parameter server (PS) mode, computing node 1 may be a parameter server (PS), and in a distributed training architecture of collective communication (CC) mode, computing node 1 may be a distributed node in CC mode.
[0071] Step 302: The master node receives the parameter value of the training task 1 sent by the computing node 1 in response to the parameter export command.
[0072] In this step, the parameter value of training task 1 sent by computing node 1 may be the real-time parameter value of the first training task, that is, the latest parameter value, in other words, the parameter value obtained after the most recent parameter iteration.
[0073] Step 303: The master node stores the parameter value of training task 1 as the backup parameter value of training task 1.
[0074] Optionally, when the first training job includes multiple training tasks, the backup parameter values of each training task can be uniformly stored in one file as the backup parameter file of the first training job, thereby achieving unified management of the backup parameter values of the first training job in the master node.
[0075] Step 304: When the master node determines to resume the breakpoint training of the first training job, it sends the backup parameter value of training task 1 to computing node 1, which is used as the parameter value for the breakpoint resumption training of training task 1.
[0076] Figure 3 In the method shown, the master node implements the backup and recovery of the parameter values of the training task 1 of the first training job in the computing node 1, and implements the endpoint continued training of the training task 1 of the first training job in the computing node 1.
[0077] It should be noted that Figure 3 The computing node 1 in the method shown can be extended to other computing nodes that perform the training tasks of the first training job, so that the master node can perform with each computing node corresponding to the first training job (that is, each computing node that performs the training tasks of the first training job). Figure 3 The method shown achieves unified backup and recovery of the parameter values of the first training job by backing up and restoring the parameter values of the training tasks of the first training job in each computing node corresponding to the first training job.
[0078] In another embodiment provided by the present application, if there are multiple training tasks belonging to the first training job executed in computing node 1, taking the execution of two training tasks, training task 1 and training task 2, as an example, then,
[0079] Step 302 may also include: receiving a parameter value of training task 2 sent by computing node 1 in response to a parameter export command;
[0080] Step 303 may also include: storing the parameter value of training task 2 as a backup parameter value of training task 2;
[0081] Step 304 may also include: when determining to resume the breakpoint training of the first training job, sending the backup parameter value of the training task 2 to the computing node 1, which is used as the parameter value for the breakpoint resumption training of the training task 2.
[0082] The embodiment of the present application implements the backup and recovery of the parameter values of training task 1 and training task 2 of the first training job in computing node 1.
[0083] It should be noted that in the embodiment of the present application, the training tasks belonging to the first training job executed in the computing node 1 can also be expanded to more than 3, that is, the master node can uniformly back up and restore the parameter values of all training tasks belonging to the first training job executed in the computing node 1.
[0084] Figure 4 is another flow chart of the parameter processing method of the embodiment of the present application, such as Figure 4 As shown, the method may include:
[0085] Step 401: computing node 1 receives a first parameter export command for a first training job sent by a master node; the first parameter export command is sent by the master node when determining to perform parameter backup for the first training job.
[0086] Step 402: Computing node 1 sends parameter values of training task 1 of the first training job to the master node in response to the first parameter export command, so that the master node stores the parameter values of training task 1 as backup parameter values of training task 1 of the first training job.
[0087] Step 403: Computing node 1 receives the backup parameter value of training task 1 sent by the master node, which is used as the parameter value for breakpoint resumption of training task 1.
[0088] Figure 4 In the method shown, computing node 1 cooperates with the master node to implement the backup and recovery of parameters in computing node 1, saving training time and training resources for distributed training.
[0089] It should be noted that Figure 4 The computing node 1 in the method shown can be extended to other computing nodes that perform the training tasks of the first training job, such as computing node 2, computing node 3, etc., so that each computing node corresponding to the first training job (that is, each computing node that performs the training tasks of the first training job) can cooperate with the main node to perform Figure 4 The method shown achieves unified backup and recovery of the parameter values of the first training job by backing up and restoring the parameter values of the training tasks of the first training job in each computing node corresponding to the first training job.
[0090] The following uses the distributed training architecture of the CC mode and the distributed training architecture of the PS mode as examples to illustrate the implementation of the parameter processing method of the embodiment of the present application.
[0091] See also Figure 5 , is a schematic diagram of the distributed training architecture of the CC mode provided in the embodiment of the present application. Figure 5 As shown, it includes: a master node 501, k distributed nodes 502, and k is a natural number greater than 1.
[0092] The master node 501 and each distributed node 502 may communicate with each other through their respective communication interfaces, and the communication may be wired communication or wireless communication.
[0093] The master node 501 may be provided with an AI platform, and the distributed node 502 may serve as the computing node.
[0094] See also Figure 6 , is the embodiment of this application based on Figure 5 A schematic diagram of a parameter processing method provided by the scenario shown. Figure 6 The distributed node shown in can be any distributed node corresponding to the first training job, that is, any distributed node that executes the training task of the first training job.
[0095] like Figure 6 As shown, the method may include: an initialization phase, a breakpoint resume training export phase, a training stop phase, and a breakpoint resume training recovery phase.
[0096] The initialization phase includes the following steps 601 to 602.
[0097] Step 601: The master node sends the training tasks and parameter initialization data of the first training job to the distributed nodes corresponding to the first training job respectively.
[0098] Training tasks may include: model data, training data, etc.
[0099] The distributed nodes corresponding to the first training job are the distributed nodes that need to perform the training tasks of the first training job. The distributed nodes corresponding to the first training job may be part or all of the distributed nodes in the distributed training architecture.
[0100] It should be noted that, based on the different parallel methods used in distributed training, the model data and training data included in the training tasks of distributed nodes can be different. For example, in the data parallel method, each distributed node can load the same initial AI model, but use different training data for training; in the model parallel method, each distributed node can load part of the model data of the initial AI model and use the same training data for training.
[0101] The parameter initialization data may include: parameter name, parameter initialization value or initialization rule. The above initialization rule is used by the distributed node to determine the initial value of the parameter.
[0102] It should be noted that the number of training tasks of the first training job sent by the master node to a distributed node may be one or more.
[0103] Optionally, when the master node sends the training tasks and parameter initialization data of the first training job to each distributed node, it may simultaneously send: the identifier of the first training job and the identifier of the training task, so that the distributed nodes know the training job to which the training task belongs.
[0104] Step 602: Each distributed node performs the training task according to the training task and parameter initialization data.
[0105] Each distributed node can calculate the initial values of the parameters of the AI model based on the parameter initialization data; the forward propagation algorithm of each distributed node can calculate the predicted value on a small part of the training data based on the initial values of the parameters of the AI model, and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and updates the initial value of the parameter of the AI model to the new parameter value based on the gradient, thereby completing the first round of iteration;
[0106] Similar to the first round of iterations mentioned above, in each subsequent round of iterations, the forward propagation algorithm of each distributed node can calculate the predicted value on a small part of the training data based on the current parameter value of the AI model parameters, and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and updates the current parameter value of the AI model parameters to the new parameter value according to the gradient.
[0107] When each distributed node performs parameter iteration, it can report to the master node every time one round of parameter iteration or multiple rounds of parameter iteration are completed, so that the master node is informed of the rounds of parameter iteration performed by the distributed nodes.
[0108] The breakpoint-resume training export stage includes the following steps 603 to 605.
[0109] Step 603: The master node detects the parameter backup triggering condition of the first training job, and sends a parameter export command for the first training job to the distributed node corresponding to the first training job.
[0110] In one instance, the parameter backup can be triggered periodically, that is, the parameter backup is performed once at a certain interval, so that the backed-up parameter values are relatively new parameter values, so that they can be restored to relatively new parameter values in the subsequent breakpoint training recovery phase, saving training time and training resources for distributed training.
[0111] Optionally, the parameter backup triggering condition may include: the number of parameter iteration rounds reaches a preset threshold. The value of the above threshold is not limited in the embodiment of the present application.
[0112] For example, the above threshold value can be 100 times. Every time the master node detects that the parameter iteration rounds of the first training job reach 100 times, it determines that the parameter backup trigger condition of the first training job is detected. That is, when the master node detects that the parameter iteration rounds of the first training job are 100 times, 200 times, 300 times... and other iteration rounds, it determines that the parameter backup trigger condition of the first training job is detected.
[0113] In another example, the triggering of parameter backup can be triggered by the user. For example, the master node can provide a UI interface for the user through the AI platform. The UI interface provides the user with an interface for triggering parameter backup for the first training job. The user can perform specified operations in the UI interface to trigger the parameter backup of the first training job. Accordingly, when the master node detects the user's triggering operation for the parameter backup of the first training job in the UI interface, it determines that the parameter backup triggering condition for the first training job is detected.
[0114] In one example, the parameter export command may include: an identifier of the first training job; in this case, the distributed node may determine the training tasks belonging to the first training job among the training tasks executed by itself according to the identifier of the first training job.
[0115] In another instance, the parameter export command may also include: an identifier of the first training job, and an identifier of the training task of the first training job executed in the distributed node; in this case, the master node specifies the training task for which parameter export is required in the distributed node, and the distributed node determines the training task based on the identifier of the training task.
[0116] Step 604: The distributed node receives the parameter export command and sends the latest parameter value of the training task belonging to the first training job to the master node.
[0117] It should be noted that the latest parameter value of the training task here can be the value of the parameter updated after the most recent parameter iteration. Continuing with the example of the first threshold of 100 times in step 603, if the master node sends a parameter export command to each distributed node when the round of parameter iteration is 200, then the latest value of the parameter in the distributed node can be the value of the parameter obtained by updating the parameter when the iteration round is 200.
[0118] If a distributed node includes more than two training tasks belonging to the first training job, the distributed node may send the latest parameter values of the more than two training tasks belonging to the first training job to the master node.
[0119] Step 605: The master node stores the latest parameter values of the training tasks belonging to the first training job sent by each distributed node as backup parameter values of the training tasks of the first training job.
[0120] Optionally, the master node may store the latest parameter values of the training tasks belonging to the first training job as a backup file. The save path and file name of the backup file may be set by default by the master node or by the user, and are not limited in this embodiment of the present application.
[0121] In a possible implementation, the master node can be compatible with the file path and / or file name configuration of frameworks such as TF / PT, as shown below: saver.save(sess, 'my-model', global_step=0) ==> filename: 'my-model-0'.
[0122] It should be noted that steps 603 to 605 may be triggered once or multiple times during the AI model training process. When steps 603 to 605 are triggered and executed multiple times, a backup file may be generated each time, thereby obtaining multiple backup files. For example, the following takes the generation of three backup files as an example:
[0123] 1: saver.save(sess,'my-model',global_step=0)==>filename:'my-model-0'
[0124] 2: …
[0125] 3: saver.save(sess,'my-model',global_step=1000)==>filename:'my-model-1000'
[0126] It should be noted that these backup files can be stored according to the backup time or number of iterations, so that when breakpoint resume training needs to be started later, the user can independently select the backup parameter values used for breakpoint resume training, that is, independently select the starting point of breakpoint resume training.
[0127] Optionally, the backup file may include: the identifier of the first training job, the identifier of the distributed node, the parameter name, and the parameter value. The format of the backup file is as follows:
[0128] {
[0129] "distributedNodeNum":2,
[0130] “distributed-node-0”:[
[0131] {
[0132] "variable-xx": "aa",
[0133] } ]
[0135] “distributed-node-1”:[
[0136] {
[0137] "variable-yy":"bb",
[0138] } ]
[0140] The training stop phase includes the following step 606 .
[0141] Step 606: When the master node detects that at least one computing node corresponding to the first training job fails, it sends a stop instruction to the computing nodes corresponding to the first training job that have not failed, and the computing nodes that have not failed stop the training tasks of the first training job.
[0142] An exemplary method for the master node to detect whether each computing node corresponding to the first training job has a fault is described as follows:
[0143] The computing node can monitor whether it has a fault. If it detects that it has a fault, it can report the fault to the master node, so that the master node can be informed that a computing node has a fault; or,
[0144] The master node may monitor the communication with each computing node, and if no feedback is received from a computing node within a preset time, it is determined that the computing node has failed.
[0145] The breakpoint resume training recovery phase includes the following steps 607 to 609.
[0146] Step 607: The master node detects a breakpoint resume training trigger operation for the first training job, and sends a training resume command for the first training job to the distributed node corresponding to the first training job.
[0147] Optionally, the user can select the breakpoint-resume training control of the first training job in the user interface provided by the AI platform, and accordingly, the master node detects a breakpoint-resume training trigger operation for the first training job; or, if the master node detects that the fault of the computing node that had previously failed and corresponding to the first training job is resolved, accordingly, the master node detects a breakpoint-resume training trigger operation for the first training job.
[0148] Optionally, if the master node has backed up the parameter values for the first training job multiple times, that is, multiple backup files of the first training job are stored in the master node, the AI platform can provide a UI interface for the user, displaying the backup time, number of iterations, etc. of each backup file in the UI interface, so that the user can independently select the backup file for breakpoint resumption, that is, select the starting point for training recovery in breakpoint resumption.
[0149] Optionally, each distributed node may feed back a confirmation message to the master node upon receiving the training resumption command, so as to indicate to the master node that it is able to execute the first training job.
[0150] Step 608: The master node resends the training task of the first training job and the backup parameter value of the training task to the distributed nodes.
[0151] Optionally, the master node may send the training task of the first training job and the latest backup parameter value to the original computing node corresponding to the training task, or may allocate a new computing node for the training task, which is not limited in the embodiments of the present application.
[0152] For example, training task 1 of the first training job is assigned to computing node 1 for execution during step 606. In this step, the master node can resend training task 1 of the first training job to computing node 1, or to other computing nodes such as computing node 2.
[0153] Optionally, if the user has not specified a backup file for breakpoint resume when the master node detects a breakpoint resume trigger operation for the first training job in step 607, the master node may send a backup parameter value based on the latest backup of the first training job to the distributed nodes in this step; if the user has specified a backup file for breakpoint resume when the master node detects a breakpoint resume trigger operation for the first training job in step 607, the master node may send a backup parameter value based on the backup file of the first training job specified by the user to the distributed nodes in this step.
[0154] Step 609: Each distributed node executes the training task according to the backup parameter value to perform AI model training for the first training job.
[0155] When the distributed nodes perform training tasks based on the received backup parameter values, they can use the received backup parameter values as the parameter values used in the first round of parameter iteration after breakpoint resumption. Thus, the distributed nodes can use the forward propagation algorithm to calculate the predicted value on a small part of the training data based on the received backup parameter values, and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and updates the backup parameter value of the parameter to the new parameter value based on the gradient, thus completing the first round of parameter iteration;
[0156] Afterwards, in each subsequent round of parameter iteration, the forward propagation algorithm of the distributed nodes can calculate the predicted value on a small part of the training data based on the current parameter value of the AI model's parameters (that is, the parameter value obtained in the previous round of parameter iteration), and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and updates the current parameter value of the parameter to the new parameter value according to the gradient.
[0157] Figure 6 The method shown implements the unified backup and recovery of parameter values in each distributed node by the master node under the distributed training architecture of the CC mode.
[0158] See also Figure 7 , is a schematic diagram of a distributed training architecture of the PS mode applicable to the parameter processing method of the embodiment of the present application. Figure 7 As shown, including:
[0159] Master node, k ps, x training nodes (workers), k is a natural number, x is a natural number.
[0160] The master node and each PS can communicate through a communication interface, and the master node and each worker can communicate through a communication interface. PS and workers with associated relationships can communicate through a communication interface. The above communication can be wired communication or wireless communication.
[0161] In the distributed training framework of the PS mode, the worker is responsible for completing data reading, obtaining parameter values from the PS, forward calculation, reverse gradient calculation, and other processing, and uploading the calculated gradient to the corresponding PS; after receiving the gradient from the worker, the PS performs gradient aggregation and parameter iteration based on the gradient from the worker to obtain the updated parameter value for the worker to perform the next round of AI model training. The worker and the PS for which the parameters are iterated and updated are the PS and worker with the above-mentioned associated relationship.
[0162] It should be noted that the parameters of a worker can be iteratively updated in multiple PSs, and one PS can also handle the iterative update of parameters of multiple workers, which is not limited in the embodiments of the present application.
[0163] For example Figure 7 In the example, worker1, worker2 and ps1 are associated with each other, that is, worker1 and worker2 can obtain the parameter values of the training tasks of the first training job from ps1; workerx and psk are associated with each other, that is, workerx can obtain the parameter values of the training tasks of the first training job from psk.
[0164] See also Figure 8 , is the embodiment of this application based on Figure 7 A schematic diagram of a parameter processing method provided by the scenario shown. Figure 8 The worker shown in can be any worker corresponding to the first training job, that is, any worker that performs the training task of the first training job; Figure 8 The ps shown in is the ps associated with the worker, that is, the ps for performing parameter iteration on the training task of the first training job executed by the worker.
[0165] like Figure 8 As shown, the method may include: an initialization phase, a breakpoint resume training export phase, a training stop phase, and a breakpoint resume training recovery phase.
[0166] The initialization phase includes the following steps 801 to 802.
[0167] Step 801: The master node sends a training task of the first training job to each worker, and sends initialization data of parameters of the training task to the PS corresponding to each worker.
[0168] Step 802: The woker sends a parameter acquisition request to the corresponding PS. The PS determines the initial value of the parameter according to the initialization data of the parameter, and sends the initial value of the parameter to the corresponding worker. The worker trains the model based on the received initial value of the parameter.
[0169] The parameter acquisition request is used to request the initial values of the parameters of the training task executed by the worker.
[0170] Optionally, the parameter acquisition request may include: the first training job identifier, the identifier of the training task performed by the worker; so that the ps can obtain the parameters requested by the worker.
[0171] Afterwards, the worker's forward propagation algorithm can calculate the predicted value on a small part of the training data based on the initial value of the parameter, and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and sends a parameter acquisition request to the ps corresponding to the worker. The parameter acquisition request can carry the gradient of the parameter; the ps updates the initial value of the parameter as the new parameter value based on the gradient, and sends the new parameter value to the worker for the worker to perform the training task and conduct the next round of AI model training, thus completing the first round of parameter iteration;
[0172] Similar to the first round of iterations mentioned above, in each subsequent round of parameter iterations, the forward propagation algorithm of each worker can calculate the predicted value on a small part of the training data based on the current parameter value of the AI model parameter (that is, the parameter value obtained in the previous round of parameter iteration), and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and carries the gradient of the parameter in the parameter acquisition request to the corresponding ps. The ps updates the current parameter value of the parameter as the new parameter value according to the gradient, and sends the new parameter value of the parameter to the worker.
[0173] The process of exporting a node by resuming training at a breakpoint includes the following steps 803 to 805 .
[0174] Step 803: The master node detects the parameter backup triggering condition of the first training job, and sends a parameter export command for the first training job to the PS corresponding to the first training job.
[0175] The master node detects the parameter backup trigger condition of the first training job, and the corresponding description in step 603 may be referred to, which will not be repeated here.
[0176] The ps corresponding to the first training job is also the ps that executes the parameter iteration of the training task in the first training job. In other words, it is the ps associated with the worker that executes the training task in the first training job.
[0177] Step 804: Each PS sends the latest parameter value of the training task belonging to the first training job to the master node in response to the parameter export command.
[0178] The implementation of this step may refer to the corresponding description in step 604, the main difference being that the computing node is replaced by ps in this step.
[0179] Step 805: The master node stores the latest parameter values of the training tasks belonging to the first training job sent by each ps as backup parameter values of the training tasks of the first training job.
[0180] The implementation of this step can refer to the corresponding description in step 605, and the difference lies in the adaptive change of the information included in the backup file. For example, the backup file in the master node in this step may include: the identifier of the first training job, the ps identifier, the worker identifier, the parameter name, and the parameter value. Specific examples are as follows:
[0181]
[0182]
[0183] The training stop phase includes the following step 806 .
[0184] Step 806: When the master node detects that at least one worker or PS corresponding to the first training job fails, it sends a training stop instruction to the workers that have not failed, and the workers that have not failed stop the training tasks of the first training job. It also sends a parameter update stop instruction to the PS that has not failed, and the PS that has not failed stops the parameter update of the training tasks of the first training job.
[0185] In one example, the training stop instruction may include: an identifier of the first training job.
[0186] In another example, the training stop instruction may include: an identifier of the first training job, and an identifier of a training task of the first training job executed by the worker.
[0187] In one example, the parameter update stop indication may include: an identifier of the first training job.
[0188] In another example, the parameter update stop indication may include: an identifier of the first training job, and an identifier of a training task for which ps performs parameter iteration.
[0189] The breakpoint resume training recovery phase includes the following steps 807 to 809.
[0190] Step 807: The master node detects the breakpoint resume training trigger operation for the first training job, and sends a training resume command for the first training job to the ps and the worker respectively.
[0191] In this step, the master node detects the implementation of the breakpoint resumption trigger operation for the first training job, which can be referred to step 607 and will not be repeated here.
[0192] Optionally, the ps and worker nodes may feedback a confirmation message to the master node upon receiving the training resume command, so as to indicate to the master node that they are able to execute the first training job.
[0193] Step 808: The master node sends the training task of the first training job to the worker, and sends the backup parameter value of the training task of the first training job to the PS associated with the worker.
[0194] Optionally, if the user has not specified a backup file for breakpoint resume when the master node detects a breakpoint resume trigger operation for the first training job in step 807, the master node may send a backup parameter value based on the latest backup of the first training job to ps in this step; if the user has specified a backup file for breakpoint resume when the master node detects a breakpoint resume trigger operation for the first training job in step 807, the master node may send a backup parameter value based on the backup file of the first training job specified by the user to ps in this step.
[0195] Step 809: the worker sends a parameter acquisition request to the associated PS, the PS sends the backup parameter value to the worker according to the parameter acquisition request, and the worker performs the training task based on the received backup parameter value.
[0196] When the worker performs a training task based on the received backup parameter value, the received backup parameter value can be used as the parameter value used in the first round of parameter iteration after breakpoint resumption. Thus, the worker can use the forward propagation algorithm to calculate the predicted value on a small part of the training data based on the received backup parameter value, and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and sends a parameter acquisition request to the ps corresponding to the worker. The parameter acquisition request can carry the gradient of the parameter; the ps updates the backup parameter value of the parameter as the new parameter value based on the gradient, and sends the new parameter value to the worker for the worker to perform the training task and conduct the next round of AI model training, thus completing the first round of parameter iteration;
[0197] Afterwards, in each subsequent round of parameter iteration, the worker's forward propagation algorithm can calculate the predicted value on a small part of the training data based on the current parameter value of the AI model's parameter (that is, the parameter value obtained in the previous round of parameter iteration), and then the back propagation algorithm calculates the gradient of the parameter based on the predicted value and the loss function, and carries the gradient of the parameter in the parameter acquisition request to the corresponding ps. The ps updates the current parameter value of the parameter as the new parameter value according to the gradient, and sends the new parameter value of the parameter to the worker.
[0198] Figure 8 The method shown implements the unified backup and recovery of parameter values in each distributed node by the master node under the distributed training architecture of the PS mode.
[0199] The parameter processing method provided in the embodiment of the present application supports the unified backup and recovery of the parameters of the training job by the master node in the distributed training scenario; moreover, the master node can back up and restore the parameters of each computing node, so that a unified backup and recovery strategy can be formulated through the master node to complete the unified import and export of parameters on the computing node; moreover, the parameters backed up and restored in the embodiment of the present application can support generalization, without limiting the specific parameters and business scenarios, so that it is not limited to the variable parameters used in the optimizer or initializer, and can be freely expanded and named independently, and can be parsed and processed in the computing node; in addition, the backup and recovery of the training framework of the parameter processing method in the embodiment of the present application can be uniformly presented to the outside world with an application programming interface (API), which is used in conjunction with the API of standard frameworks such as TF / PT, and the external interfaces of distributed and stand-alone scenarios are consistent, so that the consistent parameter import and export experience of stand-alone and distributed training can be achieved without modifying the user's training script, which is convenient for user operation.
[0200] An embodiment of the present application also provides a master node, including: a processor; a memory; the processor is used to execute the method provided in the embodiment of the present application.
[0201] An embodiment of the present application also provides a computing node, including: a processor; a memory; the processor is used to execute the method provided in the embodiment of the present application.
[0202] The embodiment of the present application also provides a parameter processing system, including the master node provided by the embodiment of the present application and at least one computing node provided by the embodiment of the present application.
[0203] The embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided by the embodiment of the present application.
[0204] The embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, it enables the computer to execute the method provided by the embodiment of the present application.
[0205] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.
[0206] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0207] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0208] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk and other media that can store program codes.
[0209] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A parameter processing method, characterized in that: include: When it is determined to perform parameter backup of the first training job, a first parameter export command is sent to the first computing node; receiving a parameter value of a first training task sent by the first computing node in response to the parameter export command; The first training task is a training task of the first training job; storing the parameter value of the first training task as a backup parameter value of the first training task; When it is determined to perform breakpoint resume training of the first training job, the backup parameter value of the first training task is sent to the first computing node to be used as the parameter value for breakpoint resume training of the first training task.
2. The method according to claim 1, characterized in that Also includes: When it is determined to perform parameter backup of the first training job, a second parameter export command is sent to the second computing node; receiving a parameter value of a second training task sent by the second computing node in response to the parameter export command; the second training task is a training task of the first training job; storing the parameter value of the second training task as a backup parameter value of the second training task; When it is determined to resume the breakpoint training of the first training job, the backup parameter value of the second training task is sent to the second computing node to be used as the parameter value for the breakpoint resumption training of the second training task.
3. The method according to claim 1 or 2, characterized in that: Also includes: When the number of iterations of the first training job reaches a preset first threshold, it is determined to perform parameter backup of the first training job.
4. The method according to claim 1 or 2, characterized in that: Also includes: When it is detected that at least one computing node corresponding to the first training job fails, the computing nodes corresponding to the first training job that have not failed are instructed to stop the training tasks of the first training job.
5. The method according to claim 1 or 2, characterized in that: The determining to resume the breakpoint training of the first training operation includes: A breakpoint-resume training recovery request for the first training job is received from the user, and it is determined to perform breakpoint-resume training recovery for the first training job.
6. A parameter processing method, characterized in that: include: Receive a first parameter export command for a first training job sent by a master node; The first parameter export command is sent by the master node when determining to perform parameter backup of the first training job; In response to the first parameter export command, sending the parameter value of the first training task to the master node, so that the master node stores the parameter value of the first training task as a backup parameter value of the first training task; The first training task is a training task of the first training job; Receive the backup parameter value of the first training task sent by the master node, and use it as the parameter value for breakpoint resumption of training of the first training task.
7. The method according to claim 6, characterized in that Before receiving the backup parameter value of the first training task sent by the master node, the method further includes: receiving a first stop instruction for the first training operation; In response to the first stop instruction, the first training task is stopped or parameter update of the first training task is stopped.
8. A master node, characterized in that: include: one or more processors; Memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, enable the master node to perform the method according to any one of claims 1 to 5.
9. A computing node, characterized in that: include: one or more processors; Memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, cause the computing node to perform the method of claim 6 or 7.
10. A parameter processing system, characterized in that: It comprises the master node as described in claim 8 and at least one computing node as described in claim 9.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 7.