Backup method and device thereof

Abstract

The present application provides a backup method and a device thereof. A management node sends backup policy information to a computing node, the backup policy information being used for instructing the computing node to store parameter information of a first target layer in the first target layer in a process of training a model; and, on the basis of the backup policy information, the computing node stores the parameter information of the first target layer during model training of a first target round. In this way, the training progress of the model can be prevented from being affected due to loss of the parameter information caused by the computing node failing.

Description

Backup method and apparatus thereof

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese Patent Application No. 202410424292.5, filed on April 9, 2024, and entitled "Backup method and apparatus thereof", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of communication technology, and in particular to a backup method and apparatus thereof. BACKGROUND

[0004] In order to improve the intelligent and automated level of the network, artificial intelligence (AI) and machine learning (ML) technologies are being widely applied. In the AI and ML technologies, the training of a neural network model is involved. In the model training process, due to the possible interface damage, card drop, power failure, overheating and other fault problems of the network and nodes, the loss of parameter information generated in the training process will be caused, and the training progress of the model will be affected. SUMMARY

[0005] Embodiments of the present application provide a backup method and apparatus thereof for backing up parameter information in a training process.

[0006] In a first aspect, the present application provides a backup method, which can be executed by a computing node, or by other devices including the functions of the computing node, or by a chip system (which can also be replaced by a chip) or other functional modules, which can realize the functions of the computing node, and the chip system or functional modules are arranged in the computing node, for example. Take the method executed by the computing node as an example: the computing node receives first backup strategy information from a first management node, the first backup strategy information being used to instruct the computing node to store parameter information of a first target layer in a first target round in a process of training a model; and the computing node stores the parameter information of the first target layer in the model training of the first target round based on the first backup strategy information.

[0007] In this embodiment, the first management node sends backup strategy information to any computing node participating in model training, for informing the computing node to store parameter information of which layer or layers in which round or rounds of model training, and the computing node stores parameter information of the corresponding layer in the corresponding round of model training based on the received backup strategy information. In this way, the loss of parameter information caused by failure of the computing node can be avoided, thereby affecting the training progress of the model. The rounds indicated in the backup strategy information sent by the first management node to each computing node can be completely different or partially different; the layers indicated in the backup strategy information sent by the first management node to each computing node can be completely different or partially different.

[0008] In a possible implementation, the first target round is part of all rounds of the model training, and / or the first target layer is part of all layers of the model.

[0009] In this implementation, the computing node does not need to store parameter information of all layers in all rounds, and the training delay caused by the storage process can be reduced.

[0010] In a possible implementation, the computing node can also receive the distributed training manner and the first training subtask of the model from the first management node; the computing node trains the model based on the distributed training manner and the first training subtask; wherein the first training subtask includes a task for training parameter information of the first target layer, and optionally, can also include a task for training parameter information of layers other than the first target layer; that is, the backup strategy information is associated with the training subtask, and the computing node wants to store parameter information of a layer, on the premise that the computing node trains parameter information of the layer; the distributed training manner includes one or more of the following: data parallel training manner, tensor parallel training manner, and pipeline parallel training manner. Optionally, the training subtask of the computing node is related to the resource information of the computing node, and the training subtask of the computing node is determined based on the resource information of the computing node, so that the resources can be reasonably utilized.

[0011] In a possible implementation, the computing node can also send failure indication information to the first management node, the failure indication information being used to indicate that the computing node fails in the process of training the model; then the computing node receives model parameter recovery strategy information from the first management node; the model parameter recovery strategy information is used to indicate to load parameter information of the second target layer stored last time, the second target layer being part of or all of the first target layer; after the failure is repaired, the computing node loads parameter information of the second target layer stored last time based on the model parameter recovery strategy information, and performs recovery training on the model to obtain a training result.

[0012] In this implementation, the computing node reports to the first management node after a failure occurs, and can obtain model parameter recovery strategy information. The computing node loads parameter information of a corresponding layer based on the model parameter recovery strategy information to recover the model and continue training, which can avoid retraining after a failure and shorten the model training time.

[0013] In a possible implementation, the first management node is a cross-domain management system NMS entity, and the computing node is a domain management system EMS entity; or the first management node is a domain management system EMS entity, and the computing node is a network element NE.

[0014] In a second aspect, the present application provides a backup method, which can be executed by a first management node, or by other equipment including a function of the first management node, or by a chip system (which can also be replaced by a chip) or other functional modules capable of implementing the function of the first management node, for example, the chip system or functional modules are arranged in the first management node. Taking the example of the method executed by the first management node, the first management node determines first backup strategy information corresponding to a first computing node participating in model training, the first backup strategy information is used to instruct the first computing node to store parameter information of a first target layer in a first target round in the process of training the model, and the first computing node is any one of the nodes participating in the model training; and the first management node sends the first backup strategy information to the first computing node.

[0015] In this embodiment, the first management node sends backup strategy information to any computing node participating in model training, which is used to inform the computing node to store parameter information of which layer or layers in which round or rounds of model training, and the computing node stores parameter information of a corresponding layer in the process of model training in a corresponding round based on the received backup strategy information. In this way, the loss of parameter information caused by a failure can be avoided to affect the training progress of the model. The rounds indicated in the backup strategy information sent by the first management node to each computing node can be completely different or partially different; the layers indicated in the backup strategy information sent by the first management node to each computing node can be completely different or partially different.

[0016] In a possible implementation, the first target round is part of all rounds of the model training, and / or the first target layer is part of all layers of the model.

[0017] In this implementation, the computing node does not need to store parameter information of all layers in all rounds, which can reduce the training delay caused by the storage process.

[0018] In a possible implementation, the first management node can further acquire resource information of the first computing node; based on the resource information of the first computing node and the training task of the model, the first management node can send a distributed training manner of the model and a first training subtask to the first computing node, where the first training subtask belongs to the training task of the model; the distributed training manner can include one or more of the following: a data parallel training manner, a tensor parallel training manner, and a pipeline parallel training manner; the first training subtask can include a task for training parameter information of the first target layer, and optionally, can further include a task for training parameter information of a layer other than the first target layer; that is, the backup strategy information is associated with the training subtask, and a computing node wants to store parameter information of a layer, the precondition is that the computing node trains the parameter information of the layer.

[0019] In this implementation, the training subtask of a computing node is related to the resource information of the computing node, and the training subtask of the computing node is determined based on the resource information of the computing node, so that the resources can be reasonably utilized.

[0020] In a possible implementation, the first management node can further acquire resource information of a second computing node, where the second computing node is any one of the computing nodes for training the model except the first computing node; based on the resource information of the first computing node, the resource information of the second computing node, and the training task of the model, the first management node can send the distributed training manner and the first training subtask to the first computing node.

[0021] In this implementation, the training subtask determined by the first management node for any computing node is not only related to the resource information of the computing node, but also related to the resource information of other computing nodes. The first management node determines the training subtask for a single computing node by considering the total resources of all computing nodes and the resource information of the single computing node, so that the resource information of each computing node can be reasonably utilized.

[0022] In a possible implementation, the first management node can further send, based on the resource information of the first computing node, the resource information of the second computing node, and the training task of the model, the distributed training manner and a second training subtask to the second computing node, where the second training subtask belongs to the training task of the model.

[0023] In a possible implementation, the first management node can further receive failure indication information from the first computing node, the failure indication information being used to indicate that the first computing node fails during training of the model; and the first management node can further send, according to the failure indication information and the first backup strategy information, model parameter recovery strategy information to the first computing node, the model parameter recovery strategy information being used to indicate loading of parameter information in a second target layer that is the last time stored, the second target layer being part of or all of the first target layer.

[0024] In this implementation, the computing node reports to the first management node after failure, and can obtain model parameter recovery strategy information. The computing node loads parameter information of a corresponding layer based on the model parameter recovery strategy information, to recover the model and continue training, which can avoid retraining after failure and shorten the model training time.

[0025] In a possible implementation, the first management node can be a cross-domain management system NMS entity, and the first computing node can be a domain management system EMS entity.

[0026] In a possible implementation, the first management node can determine the first backup strategy information according to second backup strategy information, the second backup strategy information being used to indicate storage of parameter information in a third target layer in a third target round during training of the model; the first target round belongs to the third target round, and the first target layer belongs to the third target layer.

[0027] In a possible implementation, the first management node can further receive the second backup strategy information from a second management node.

[0028] In a possible implementation, the second management node can be a cross-domain management system NMS entity, the first management node can be a domain management system EMS entity, and the first computing node can be a network element NE.

[0029] In a third aspect, a communication apparatus is provided, which can be the computing node of the first aspect. The communication apparatus has the functions of the computing node. The communication apparatus can be a functional module in the computing node, such as a baseband device or a chip system. Alternatively, the communication apparatus can be the first management node of the second aspect. The communication apparatus has the functions of the first management node. The communication apparatus can be a functional module in the first management node, such as a baseband device or a chip system.

[0030] In an optional implementation, the communication apparatus includes a baseband apparatus and a radio frequency apparatus. In another optional implementation, the communication apparatus includes a processing unit (also referred to as a processing module) and a transceiver unit (also referred to as a transceiver module). The transceiver unit can implement a transmitting function and a receiving function. When the transceiver unit implements the transmitting function, it can be referred to as a transmitting unit (also referred to as a transmitting module). When the transceiver unit implements the receiving function, it can be referred to as a receiving unit (also referred to as a receiving module). The transmitting unit and the receiving unit can be the same functional module, which is referred to as a transceiver unit and can implement the transmitting function and the receiving function. Alternatively, the transmitting unit and the receiving unit can be different functional modules, and the transceiver unit refers to these functional modules in general.

[0031] In a possible implementation, the communication apparatus further includes a storage unit (also referred to as a storage module), and the processing unit is configured to be coupled with the storage unit and execute programs or instructions in the storage unit, so that the communication apparatus can perform the functions of the computing node according to the first aspect or perform the functions of the first management node according to the second aspect.

[0032] When the communication apparatus is the computing node according to the first aspect, the following at least one of the possible implementations is included:

[0033] In a possible implementation, the transceiver unit is configured to receive first backup policy information from the first management node, where the first backup policy information is used to instruct the communication apparatus to store parameter information of a first target layer in a first target round during training of a model. The processing unit is configured to store the parameter information of the first target layer in the model training of the first target round based on the first backup policy information.

[0034] In a possible implementation, the transceiver unit is further configured to receive a distributed training manner and a first training subtask of the model from the first management node. The first training subtask includes at least a task of training the parameter information of the first target layer. The distributed training manner includes one or more of the following: a data parallel training manner, a tensor parallel training manner, and a pipeline parallel training manner. The processing unit is further configured to train the model based on the distributed training manner and the first training subtask.

[0035] In a possible implementation, the transceiver is further configured to: send fault indication information to the first management node, the fault indication information being used to indicate that the communication device has a fault in the process of training the model; and receive model parameter recovery strategy information from the first management node; the model parameter recovery strategy information is used to indicate that parameter information in a second target layer that is stored last is loaded, the second target layer being part of or all of the first target layer; and the processing unit is further configured to: after the fault is repaired, load the parameter information in the second target layer that is stored last based on the model parameter recovery strategy information, and perform recovery training on the model to obtain a training result.

[0036] In a possible implementation, the first management node is a cross-domain management system NMS entity, and the communication device is a domain management system EMS entity; or the first management node is a domain management system EMS entity, and the communication device is a network element NE.

[0037] When the communication device is the first management node according to the second aspect, the following implementations are possible.

[0038] In a possible implementation, the processing unit is configured to: determine first backup strategy information corresponding to a first computing node participating in model training, the first backup strategy information being used to indicate that the first computing node stores parameter information of a first target layer in a first target round in the process of training the model, the first computing node being any one of the nodes participating in the model training; and the transceiver is configured to: send the first backup strategy information to the first computing node.

[0039] In a possible implementation, the transceiver is further configured to: obtain resource information of the first computing node; and based on the resource information of the first computing node and a training task of the model, send a distributed training mode of the model and a first training subtask to the first computing node, the first training subtask belonging to the training task of the model; the first training subtask at least includes a task of training parameter information of the first target layer; and the distributed training mode includes one or more of the following: a data parallel training mode, a tensor parallel training mode, and a pipeline parallel training mode.

[0040] In a possible implementation, the transceiving unit is further configured to: acquire resource information of a second computing node, the second computing node being any one of the computing nodes for training the model except the first computing node; and send the distributed training manner and the first training subtask to the first computing node based on the resource information of the first computing node, the resource information of the second computing node, and the training task of the model.

[0041] In a possible implementation, the transceiving unit is further configured to: send the distributed training manner and a second training subtask to the second computing node based on the resource information of the first computing node, the resource information of the second computing node, and the training task of the model, the second training subtask belonging to the training task of the model.

[0042] In a possible implementation, the transceiving unit is further configured to: receive failure indication information from the first computing node, the failure indication information being used to indicate that the first computing node has failed in the process of training the model; and send model parameter recovery strategy information to the first computing node according to the failure indication information and the first backup strategy information, the model parameter recovery strategy information being used to indicate loading of parameter information in a second target layer that is stored last time, the second target layer being part of or all of the first target layer.

[0043] In a possible implementation, the communication device is a cross-domain management system NMS entity, and the first computing node is a domain management system EMS entity.

[0044] In a possible implementation, the processing unit is specifically configured to: determine the first backup strategy information according to second backup strategy information, the second backup strategy information being used to indicate storage of parameter information in a third target layer in a third target round in the process of training the model; the first target round belonging to the third target round, and the first target layer belonging to the third target layer.

[0045] In a possible implementation, the transceiving unit is further configured to: receive the second backup strategy information from a second management node.

[0046] In a possible implementation, the second management node is a cross-domain management system NMS entity, the communication device is a domain management system EMS entity, and the first computing node is a network element NE.

[0047] In a fourth aspect, a communication apparatus is provided, which includes an interface circuit and a processor, and optionally, a memory. The memory is configured to store a computer program, and the processor is coupled to the memory and the interface circuit. When the processor reads the computer program or instructions, the communication apparatus is caused to perform the method performed by the computing node in the first aspect or the method performed by the first management node in the second aspect. The interface circuit is configured to receive a signal from another communication apparatus and transmit the signal to the processor or send a signal from the processor to another communication apparatus. The processor is configured to implement the method performed by the computing node in the first aspect or the method performed by the first management node in the second aspect by means of logic circuit or execution of code instructions.

[0048] In a possible implementation, the communication apparatus is a chip or a chip system.

[0049] In a fifth aspect, a communication apparatus is provided, which includes a processor and optionally, a memory. The processor and the memory are coupled. The memory is configured to store a computer program or instructions. The processor is configured to execute part or all of the computer program or instructions in the memory, which when executed, is configured to implement the function of the computing node in the first aspect or the function of the first management node in the second aspect.

[0050] In a possible implementation, the apparatus can further include a transceiver, which is configured to transmit a signal processed by the processor or receive a signal input to the processor. The transceiver can perform the transmitting action or the receiving action performed by the computing node in the first aspect or the first management node in the second aspect.

[0051] In a possible implementation, the processing unit in the fourth aspect and the fifth aspect can be implemented by means of the processor, the storage unit in the third aspect can be implemented by means of the memory, and the transceiving unit in the third aspect can be implemented by means of the transceiver.

[0052] In a possible implementation, the communication apparatus is a chip or a chip system.

[0053] In a sixth aspect, a communication system is provided, which includes a computing node and a first management node. The computing node is configured to perform the method performed by the computing node in the first aspect, and the first management node is configured to perform the method performed by the first management node in the second aspect. For example, the computing node and the first management node can be implemented by means of the communication apparatus in the third aspect.

[0054] In a seventh aspect, a computer-readable storage medium is provided, which is configured to store a computer program or instructions, which, when executed, cause the method in the first aspect or the second aspect to be implemented.

[0055] In an eighth aspect, a computer program product is provided, which comprises instructions, which, when executed on a computer, cause the method in the first aspect or the second aspect to be implemented. BRIEF DESCRIPTION OF DRAWINGS

[0056] FIG. 1 is a schematic diagram of an architecture of a communication system;

[0057] FIG. 2 is a schematic diagram of a neural network model;

[0058] FIG. 3 is a schematic diagram of a communication method provided by the present application;

[0059] FIG. 4 is a schematic diagram of a communication method provided by the present application;

[0060] FIG. 5 is a schematic diagram of a communication method provided by the present application;

[0061] FIG. 6 is a structural diagram of a communication device provided by the present application;

[0062] FIG. 7 is a structural diagram of a communication device provided by the present application. DETAILED DESCRIPTION

[0063] The technical solution of the present application can be applied to a terrestrial network (TN) and a non-terrestrial network (NTN), such as a satellite network. The technical solution of the present application can be applied to various wireless communication systems, such as a fourth generation (4G) system (also referred to as a long term evolution (LTE) system), a fifth generation (5G) system (also referred to as a new radio (NR) system), or a next generation mobile communication system or other similar communication system (such as a sixth generation (6G) system), without limitation. In addition, the technical solution of the present application can be applied to a device-to-device (D2D) scenario, such as an NR-D2D scenario, or a V2X scenario, such as an NR-V2X scenario. The technical solution of the present application can also be applied to the fields of intelligent driving, assisted driving, intelligent connected vehicles, and factory manufacturing scenarios.

[0064] As shown in FIG. 1, a schematic diagram of a communication system architecture is introduced. The communication system includes a network management system (NMS), an element management system (EMS), and a network element (NE). The NMS can manage multiple EMSs, and the EMS can manage multiple NEs. The NMS can be referred to as a cross-domain management function unit. The EMS can be referred to as a domain management function network element or a radio access network (RAN) domain management function network element. Among them, the network element generally refers to a RAN device or a function network element of a core network (CN). The EMS and the NE can be deployed in one device, for example, the EMS and the NE can be two functional modules in one device; the EMS and the NE can also be deployed in different devices, and the embodiments of the present application do not make specific limitations on this.

[0065] The (R)AN device is a device that provides wireless communication functions for terminal devices, and the (R)AN device is also referred to as an access network device. The RAN device includes but is not limited to: a g nodeB (gNB) in 5G, an evolved node B (eNB), a radio network controller (RNC), a node B (NB), a base station controller (BSC), a base transceiver station (BTS), a home base station (for example, a home evolved nodeB or a home node B, HNB), a baseBand unit (BBU), a transmitting and receiving point (TRP), a transmitting point (TP), a mobile switching center, and the like. In systems using different wireless access technologies, the names of devices with base station functions may be different, for example, in a 5th generation (5G) system, it is referred to as a RAN or a gNB (5G NodeB); in an LTE system, it is referred to as an evolved NodeB (eNB or eNodeB); in a 3rd generation (3G) system, it is referred to as a Node B (Node B), and the like.

[0066] The core network network element includes one or more of the following network elements:

[0067] The access management network element (may also be referred to as a mobility management network element) is a control plane network element provided by an operator network, responsible for access control and mobility management of terminal devices accessing the operator network, for example, including mobile state management, allocation of user temporary identity, authentication and user functions. In the 5G communication system, the access management network element can be an access and mobility management function (AMF) network element. In future communication systems, the access management network element can still be an AMF network element, or it can also have other names, which are not limited by the present application.

[0068] The session management network element is mainly responsible for session management in a mobile network, such as session establishment, modification, and release. Specific functions such as allocating IP addresses for users and selecting user plane network elements that provide message forwarding functions. In the 5G communication system, the session management network element can be a session management function (SMF) network element. In future communication systems, the session management network element can still be an SMF network element, or it can also have other names, which are not limited by the present application.

[0069] The user plane network element is responsible for the forwarding and receiving of user data in terminal devices. It can receive user data from a data network and transmit it to terminal devices through access network devices; the user plane network element can also receive user data from terminal devices through access network devices and forward it to the data network. The transmission resources and scheduling functions provided by the user plane network element for terminal devices are managed and controlled by the SMF network element. In the 5G communication system, the user plane network element can be a user plane function (UPF) network element. In future communication systems, the user plane network element can still be a UPF network element, or it can also have other names, which are not limited by the present application.

[0070] The data management network element is used to generate authentication credentials, user identity processing (such as storing and managing user permanent identities), access control, and subscription data management. In the 5G communication system, the data management network element can be a unified data management (UDM) network element. In future communication systems, the unified data management can still be a UDM network element, or it can also have other names, which are not limited by the present application.

[0071] Data storage network element, responsible for storing structured data information, including subscription information, policy information, and network data or service data with standard format definition. In the 5G communication system, the data storage network element can be a unified data repository (UDR). In future communication systems, the network exposure function network element can still be a UDR network element, or it can also have other names, which are not limited in this application.

[0072] Policy control network element, mainly supporting to provide a unified policy framework to control network behavior, providing policy rules to control layer network functions, and being responsible for obtaining user subscription information related to policy decision. In the 4G communication system, the policy control network element can be a policy and charging rules function (PCRF) network element. In the 5G communication system, the policy control network element can be a policy control function (PCF) network element. In future communication systems, the policy control network element can still be a PCF network element, or it can also have other names, which are not limited in this application.

[0073] Network storage network element, which can be used to provide network element discovery function, and provide network element information corresponding to the network element type based on the request of other network elements. The NRF also provides network element management services, such as network element registration, update, deregistration, and network element state subscription and push. In the 5G communication system, the network storage network element can be a network repository function (NRF) network element. In future communication systems, the network storage network element can still be a NRF network element, or it can also have other names, which are not limited in this application.

[0074] Network exposure function network element, which can be used to provide services and capabilities provided by 3GPP network function devices to the outside in a secure manner. In the 5G communication system, the network exposure function network element can be a network exposure function (NEF) network element. In future communication systems, the network exposure function network element can still be a NEF network element, or it can also have other names, which are not limited in this application.

[0075] Network slice selection function network element, which can be used to select a suitable network slice for the service of the terminal. In the 5G communication system, the network slice selection network element can be a network slice selection function (NSSF) network element. In future communication systems, the network exposure function network element can still be a NSSF network element, or it can also have other names, which are not limited in this application.

[0076] The network data analytics function network element can collect data from various network functions (NFs), such as a policy control network element, a session management network element, a user plane network element, an access management network element, and an application function network element (through a network capability exposure function network element), and perform analysis and prediction. In a 5G communication system, the network data analytics network element can be a network data analytics function (NWDAF). In future communication systems, the network data analytics function network element can still be an NWDAF network element, or can also have other names, which are not limited in the present application.

[0077] It can be understood that the above network elements or functions can be network elements in a hardware device, or software functions running on a dedicated hardware, or virtualized functions instantiated on a platform (for example, a cloud platform). In a possible implementation, the above network elements or functions can be implemented by one device, or can be implemented by multiple devices together, or can be a functional module in one device, and the embodiments of the present application do not make specific limitations.

[0078] In the following, some terms or concepts in the embodiments of the present application are explained and described to facilitate understanding by those skilled in the art.

[0079] (1), neural network model:

[0080] The neural network model is a network model that simulates the behavior characteristics of animal neural networks. This network model adjusts the connection relationship between a large number of neurons in the internal network, so as to achieve the purpose of processing information. FIG. 2 is a schematic block diagram of a neural network model 100. The neural network model 100 includes n layers of neurons, each of the n layers of neurons includes one or more neurons, and all neurons in each layer are connected to all neurons in the next layer. Taking the neural network model 100 in FIG. 2 as an example, the first layer includes two neurons, each of the second to (n-1)th layers includes three neurons, and the nth layer includes one neuron, where n is an integer greater than or equal to 2, and i is an integer less than or equal to n and greater than or equal to 1.

[0081] The process of training the neural network, i.e., the process of learning the weights of each neuron in each layer of the neural network model, aims to obtain optimal weights of each neuron in each layer. In the following, one round of model training in the process of training the neural network model 100 is described in detail in combination with FIG. 2. Training data is obtained from a training data set, and the training data is taken as the input of the first layer of the neural network model 100. After the input of the first layer passes through multiple neurons in the first layer to the nth layer, a prediction result is output from the nth layer. Specifically, each neuron has its corresponding weight. The training data is input into the first layer of neurons, and the first layer of neurons outputs an output value of the first layer of neurons based on the corresponding weight. The output value of the first layer of neurons is taken as the input of the second layer of neurons, and the second layer of neurons outputs an output value of the second layer of neurons based on the corresponding weight. In this way, a prediction result is finally output from the nth layer.

[0082] In the process of training the neural network model 100, it is desired that the prediction result output from the nth layer of the neural network model 100 is as close as possible to the prior knowledge of the training data, which is also referred to as the ground truth and generally includes the true result of the training data provided by a person. Therefore, the corresponding weights of each layer of neurons in the neural network model 100 can be updated according to the difference between the current prediction result and the prior knowledge. Specifically, a loss function is calculated according to the prediction result and the prior knowledge. According to the loss function, the gradient corresponding to the optimization parameter (such as the first-order momentum and the second-order momentum) is calculated along the direction from the nth layer to the first layer, and the gradient of the corresponding weight of each layer of neurons is calculated based on the optimization parameter. After obtaining the gradient of the corresponding weight of each layer of neurons, the corresponding weight of each layer of neurons is updated according to the gradient, and one round of model training is completed. It should be noted that the calculation of the gradient of the corresponding weight of each layer of neurons is performed layer by layer in the order from the nth layer to the first layer. After the gradient of the corresponding weight of the ith layer of neurons is calculated, the gradient of the corresponding weight of the i-1th layer of neurons is calculated. The gradient can be obtained by taking the derivative of the loss function with respect to the weight.

[0083] In the process of multiple rounds of model training, the corresponding weights of each layer of neurons are constantly updated, so that the prediction result output from the neural network model 100 is as close as possible to the prior knowledge of the training data.

[0084] (2) Distributed training mode:

[0085] The model training of the present application can be distributed training, which involves multiple computing nodes in model training. The model training of the present application can also be centralized training, which involves one computing node in model training. The distributed training mode includes the following multiple modes:

[0086] 1) Data parallel training mode: the same neural network model parameters are copied to multiple computing nodes, and a training data set is divided for each computing node to perform training calculation, and each computing device is responsible for the training process of part of the training data independently to achieve parallel acceleration.

[0087] 2) Tensor parallel training mode: the parameter matrix of each layer of the neural network model is cut, each computing node is responsible for the training process of part of the parameters, and multiple computing nodes share the memory demand at the cost of communication between nodes.

[0088] 3) Pipeline parallel training mode: the neural network model is split according to the layer structure, each computing node is responsible for the training process of part of the parameters in the model, so that multiple computing nodes can process data and features of different layers in the model at the same time, and multiple computing nodes share the memory demand at the cost of communication between nodes.

[0089] In the model training process, due to network, hardware, node interface damage, card drop, power failure, overheating and other fault problems, the loss of parameter information generated during training will affect the model training process. In order to ensure that the model training can be restored after fault repair, the parameter information generated during training can be backed up during training, and the backup content is loaded to realize fast recovery of model training after fault repair. One possible backup method is: after each round of training, the parameter information generated in the round of training is backed up. If the size of the model is large, the content to be stored in one backup process is more, the backup time will be longer, and the time to load the backup content will also be longer. Frequent backup, a large number of memory read and write bring a large time overhead, which will prolong the total training time. Based on this, the application further provides a method for backing up based on strategy backup information.

[0090] In order to better introduce the embodiments of the application, the method provided by the embodiments of the application will be introduced below in combination with the drawings. In the following, unless otherwise specified, the steps represented by dashed lines in the drawings corresponding to each embodiment of the application are optional steps.

[0091] FIG. 3 is a flow diagram of a backup method provided by an embodiment of the present application, which is introduced by taking the interaction between a first management node and a first computing node as an example. The first computing node is any computing node participating in model training. Optionally, the computing nodes participating in model training also include one or more computing nodes such as a second computing node and a third computing node. The interaction process between the first management node and each computing node participating in model training is similar. When applied to the communication system shown in FIG. 1, the first management node is a cross-domain management system NMS entity, and the computing node is an element management system EMS entity, or a network element NE, or a combination of EMS and NE.

[0092] FIG. 3 includes at least the following steps:

[0093] Step 301: The first management node determines first backup strategy information corresponding to the first computing node, which is used to indicate that the first computing node stores parameter information of a first target layer in a first target round during the process of training the model.

[0094] The first management node determines backup strategy information corresponding to each computing node participating in model training. For the convenience of description, the backup strategy information determined for the first computing node is referred to as first backup strategy information.

[0095] The first backup strategy information includes information of the first target round (for example, the index of the first target round) and information of the first target layer (for example, the index of the first target layer). The first computing node can know which round or rounds to store the parameter information through the information of the first target round, and the first computing node can know which layer or layers to store the parameter information through the information of the first target layer.

[0096] In a possible example, the first target round is part of all rounds of the model training. For example, the model training is 1,000 rounds, and the first target round is from the 200th round to the 500th round. It should be noted that the first computing node participates in the 1,000 rounds of model training, and when saving the parameter information, the parameter information in the 200th to 500th round is saved. This does not mean that the first computing node does not participate in the model training process other than the 200th to 500th round.

[0097] In addition, the first target round can include one round or multiple rounds. The multiple rounds can be consecutive rounds, for example, from the 100th round to the 120th round. The multiple rounds can also be non-consecutive rounds, for example, the 100th round, the 110th round, and the 120th round. The multiple rounds can also be partially continuous and partially discontinuous rounds, for example, from the 100th round to the 110th round, and from the 150th round to the 160th round.

[0098] In one possible example, the first target layer is part of all layers of the model. For example, the model includes 7 layers, and the first target layer includes the 2nd layer, the 4th layer, and the 6th layer.

[0099] In addition, the first target layer can include one layer, or can include multiple layers; the multiple layers can be consecutive layers, such as the 2nd layer to the 5th layer; the multiple layers can also be non-consecutive layers, such as the 2nd layer and the 4th layer; the multiple layers can also be partially consecutive and partially non-consecutive layers, such as the 2nd layer, the 4th layer, and the 5th layer.

[0100] Further possible cases are that, in different first target round model training, the first target layer can be the same, or can be different. Taking the same example, the first target round is from the 100th round to the 120th round, and the first target layer is the 2nd layer and the 4th layer, that is, in the model training from the 100th round to the 120th round, the parameter information of the 2nd layer and the 4th layer is stored. Taking the different example, the first target round is from the 100th round to the 120th round, in the 100th round to the 110th round, the first target layer is the 2nd layer and the 4th layer; in the 110th round to the 120th round, the first target layer is the 2nd layer and the 3rd layer.

[0101] The computing node does not need to store the parameter information of all layers in all rounds, which can reduce the training delay caused by the storage process.

[0102] Step 302: The first management node sends the first backup strategy information to the first computing node.

[0103] Correspondingly, the first computing node receives the first backup strategy information from the first management node.

[0104] Step 303: The first computing node stores the parameter information of the first target layer in the model training of the first target round based on the first backup strategy information.

[0105] The parameter information (the parameter information can also be referred to as training state information) includes but is not limited to one or more of the following: weight, gradient, and optimization parameters corresponding to the gradient (such as first-order momentum and second-order momentum).

[0106] Optionally, if there is a second computing node participating in the model training, the first management node can also determine backup strategy information corresponding to the second computing node. For the convenience of description, the backup strategy information determined for the second computing node is referred to as fourth backup strategy information, and the fourth backup strategy information is used to instruct the second computing node to store parameter information of a fourth target layer in model training of a fourth target round in the process of training the model. For example, the fourth backup strategy information includes information of the fourth target round (such as an index of the fourth target round) and information of the fourth target layer (such as an index of the fourth target layer). The first management node sends the fourth backup strategy information to the second computing node, and correspondingly, the second computing node receives the fourth backup strategy information from the first management node. The second computing node stores the parameter information of the fourth target layer in the model training of the fourth target round based on the fourth backup strategy information.

[0107] In a possible example, the fourth target round is part of all rounds of the model training. The first target round is completely different from the fourth target round, or partially the same and partially different. In a possible example, the fourth target layer is part of all layers of the model. The first target layer is completely different from the fourth target layer, or partially the same and partially different. That is, the backup strategy information determined by the first management node for the first computing node and the second computing node can be completely different, or partially the same and partially different. Optionally, the layers indicated in the backup strategy information determined by the first management node for all computing nodes are all layers of the model; and the rounds indicated in the backup strategy information determined by the first management node for all computing nodes are all rounds or part of the rounds of the model training.

[0108] In this embodiment, the first management node sends the backup strategy information to the computing nodes participating in the model training, to inform the computing nodes which layer or layers of parameter information to store in which round or rounds of the model training, and the computing nodes store the parameter information of the corresponding layer in the model training of the corresponding round based on the received backup strategy information.

[0109] In the embodiments of the present application, the first management node can also determine training sub-tasks for each computing node participating in the model training, and send the training sub-tasks to each computing node respectively. Further, the computing nodes can train the model based on the training sub-tasks. The training sub-task can be understood as training which layer or layers of parameter information (the computing node participates in the training of all rounds). The training sub-tasks corresponding to different computing nodes can be completely different, or partially different and partially the same. The set of training sub-tasks of each computing node is the training task of the model.

[0110] The following describes an example of the first management node determining the training sub-tasks corresponding to each computing node:

[0111] Example 1: For any computing node, the first management node determines the training subtask corresponding to the computing node based on the resource information of the computing node. That is, the training subtask determined by the first management node for a certain computing node is associated with the resource information of the computing node.

[0112] The resource information includes but is not limited to one or more of the following: computing resources, storage resources, computing power resources. The "resource information of the computing node" in the embodiment of the application refers to the resource information of the computing node that can be used for model training, which is simply referred to as the resource information of the computing node hereinafter.

[0113] For any computing node, the resource information of the computing node can complete the training subtask determined for the computing node.

[0114] Example 2: For any computing node, the first management node determines the training subtask corresponding to the computing node based on the resource information of the computing node and the resource information of other computing nodes participating in model training. That is: on the basis of example 1, further optionally, when determining the training subtask of a certain computing node, the first management node will also consider the resource information of other computing nodes participating in model training.

[0115] For example, the computing nodes participating in model training include a first computing node, a second computing node and a third computing node; the first management node determines the training subtask corresponding to the first computing node based on the resource information of the first computing node and the resource information of the second computing node and the third computing node. For example, the first management node determines the training subtask corresponding to the first computing node based on the proportion of the resource information of the first computing node in the total of the resource information of all computing nodes participating in model training (i.e. the first computing node, the second computing node and the third computing node). For example, the difference between the first proportion and the second proportion is less than a set threshold, wherein the first proportion is the proportion of the resource information of the first computing node in the total of the resource information of all computing nodes participating in model training, and the second proportion is the proportion of the training subtask of the first computing node in the training task of the model.

[0116] Similarly, the first management node determines the training subtask corresponding to the second computing node based on the resource information of the second computing node and the total of the resource information of all computing nodes participating in model training. The first management node determines the training subtask corresponding to the third computing node based on the resource information of the third computing node and the total of the resource information of all computing nodes participating in model training.

[0117] When determining the training subtask of a certain computing node, not only the resource information of the computing node is considered, but also the resource information of other computing nodes, so that the training subtasks are balanced for each computing node, and the problem that the resources cannot be reasonably utilized due to the small amount of training subtasks determined by the computing node with more resources and the large amount of training subtasks determined by the computing node with less resources is avoided.

[0118] The step 301 introduces that the first management node determines the first backup strategy information corresponding to the first computing node. For example, the first management node can determine the first backup strategy information corresponding to the first computing node based on the training subtask of the first computing node. Similarly, the first management node can determine the fourth backup strategy information corresponding to the second computing node based on the training subtask of the second computing node. That is, there is an association relationship between the training subtask and the backup strategy information determined by the first management node for any computing node. In theory, the computing node wants to store the parameter information of a certain layer, the premise is that the computing node trains the parameter information of the layer. When the first management node determines the backup strategy information of a certain computing node, all layers in the training subtask determined for the computing node can be determined as the target layer in the backup strategy information, or part of the layers in all layers can be determined as the target layer in the backup strategy information.

[0119] The association relationship between the training subtask and the backup strategy information is introduced by taking the first computing node as an example. For convenience of description, the training subtask determined for the first computing node is referred to as the first training subtask. The association relationship between the first backup strategy information determined for the first computing node and the first training subtask is that the first training subtask at least includes a task for training the parameter information of the first target layer (the first target layer is the first target layer indicated by the first backup strategy information). For example, the model includes 7 layers, the first target layer indicated in the first backup strategy information of the first computing node includes the 2nd layer and the 4th layer, and the first training subtask of the first computing node can include a task for training the parameter information of the 2nd layer and the 4th layer. Optionally, the first training subtask can also include a task for training the parameter information of the 3rd layer.

[0120] In the embodiment of the present application, if the distributed training mode is used to train the model, the first management node can also determine the specific distributed training mode used for model training, and send the specific distributed training mode to each computing node, so that the computing node can train the model based on the specific distributed training mode.

[0121] The specific distributed training manner includes one or more of the following: data parallel training manner, tensor parallel training manner, and pipeline parallel training manner. The specific content of these distributed training manners has been described above and will not be repeated here. Distributed training is performed in cooperation between different devices, such as transmitting gradients, weights, and the like. Distributed training manner can enable a computing node to determine which parameters to transmit to which devices.

[0122] During the training of the model, the computing node can fail, for example, interface damage, card drop, power failure, overheating, and the like. After the computing node fails, the computing node can send failure indication information to the first management node, and correspondingly, the first management node receives the failure indication information. The failure indication information is used to indicate that the computing node fails during the training of the model. Further optionally, the failure indication information can also indicate the type of failure, such as power failure, overheating, and the like.

[0123] After receiving the failure indication information, the first management node can determine the model parameter recovery strategy information for the computing node that fails. The model parameter recovery strategy information is used to indicate which layer or layers of the parameter information stored last time is loaded. The first management node sends the model parameter recovery strategy information to the computing node that fails, and correspondingly, the computing node that fails receives the model parameter recovery strategy information. After the failure is repaired, the computing node that fails loads the parameter information of the corresponding layer stored last time based on the model parameter recovery strategy information.

[0124] For any computing node that fails, the first management node determines the model parameter recovery strategy information corresponding to the computing node based on the backup strategy information corresponding to the computing node. That is, the model parameter recovery strategy information corresponding to the computing node that fails is associated with the backup strategy information. In theory, the computing node wants to load the parameter information of which layer or layers, and the prerequisite is to store the parameter information of the layer or layers. When the first management node determines the model parameter recovery strategy information, it can determine all layers in the backup strategy information determined for the computing node as layers in the model parameter recovery strategy information, or determine part of the layers in all layers as layers in the model parameter recovery strategy information. The layer in the model parameter recovery strategy information can be one layer or multiple layers.

[0125] Take the first computing node failure as an example: the first management node sends the model parameter recovery strategy information to the first computing node based on the first backup strategy information, and the model parameter recovery strategy information is used to indicate loading the parameter information in the second target layer stored last time, and the second target layer is part or all of the first target layer (the first target layer is the first target layer indicated in the first backup strategy information). The first computing node receives the model parameter recovery strategy information from the first management node; after the failure is repaired, the first computing node loads the parameter information in the second target layer stored last time based on the model parameter recovery strategy information, performs recovery training on the model, and then obtains the training result.

[0126] Among them, the model parameter recovery strategy information can specifically include the information of the target layer (such as the index of the target layer), for example, the model parameter recovery strategy information determined for the first computing node includes the information of the second target layer. The computing node that fails (such as the first computing node) can know which layer or layers of parameter information to load through the information of the target layer (such as the second target layer).

[0127] Among them, the parameter information in the second target layer stored last time refers to the parameter information in the second target layer stored last time in a complete round. If the computing node is storing the parameter information of a certain layer in a certain training round, and a failure occurs at this time, the computing node does not complete a whole round of training, which is not counted as the last time storage, but looks back one round. For example, the first backup strategy information of the first computing node indicates to store the parameter information of the 2nd layer and the 4th layer, and the model fails when training to the 100th round. The first computing node has stored the parameter information of the 4th layer in the 100th round, and has not yet stored the parameter information of the 2nd layer. After the failure is repaired, the first computing node loads the parameter information of the 4th layer and the 2nd layer in the 99th round, instead of the parameter information of the 4th layer in the 100th round.

[0128] In addition, the failure repair method is different for different failures, for example, the failure is power failure, and the failure repair method is power on; for example, the failure is overheating, and the failure repair method is heat dissipation. It should be noted that the order of repairing the failure of the computing node and receiving the model parameter recovery strategy information is not limited.

[0129] After the computing node trains the model to obtain a training result, the computing node can send the training result to the first management node. After the first management node receives the training results from the computing nodes participating in the model training, the first management node can aggregate the training results, and optimize the model based on the aggregation result. Optionally, when the computing node sends the training result to the first management node, the computing node can also send an indication information, which is used to indicate that the training result is obtained after the backup strategy information determined for the computing node is used. Taking the first computing node as an example, when the first computing node sends the training result to the first management node, the first computing node can also send the first indication information, which is used to indicate that the training result is obtained after the first backup strategy information is used. Optionally, before the computing node sends the indication information to the first management node, the computing node can also determine whether a failure occurs in the computing node; if no failure occurs in the computing node, the indication information can not be sent. The first management node can analyze the model training process and result based on the indication information, so as to better train the model in the future.

[0130] In combination with the example of FIG. 3 and the foregoing introduction of the determination of the training subtask by the first management node and the failure of the computing node, as shown in FIG. 4, a flowchart of a communication method is further introduced, which includes the following steps.

[0131] It should be noted that only one computing node is drawn in FIG. 4. If multiple computing nodes participate in the model training in the example, the interaction process between the other undrawn computing nodes and the first management node is similar to the interaction process between the drawn computing node and the first management node.

[0132] Step 401: The first management node receives a model training request.

[0133] The request can come from an operator, an access network element, a core network element, or the like. The model training request can include model and training data information. The model information includes the number of layers of the model, the number of neurons of each layer, the connection relationship between the neurons, the initial weight of each neuron, and the like.

[0134] Step 402: The first management node queries a plurality of computing nodes for resource information available for model training in the computing nodes.

[0135] For example, the resources include but are not limited to computing resources, storage resources, and computing power resources. The resource information can include the size of the resources, and optionally, the running state information of the resources. For example, the running state information includes all resource information of the computing node, occupied resource information, and remaining resource information available for model training. For example, the computing node has a total of 80G of display memory, 30G of which is occupied, and 50G of which is available for model training.

[0136] Step 403: Each queried computing node sends resource information available for model training in the computing node (resource information available for model training in the computing node, hereinafter referred to as resource information of the computing node) to the first management node.

[0137] Correspondingly, the first management node receives the resource information from each computing node.

[0138] Step 402 and step 403 can also be located before step 401, but the execution time of step 401 and step 402 cannot differ too much, otherwise the obtained resource information is not accurate enough.

[0139] Step 404: The first management node determines whether to use a distributed training manner for model training.

[0140] In an example, the device or network element sending the training request to the first management node can inform the first management node whether to use a distributed training manner for model training. For example, in step 401, the training request can also include indication information indicating whether to use a distributed training manner for model training or not. Optionally, if the indication information indicates to use a distributed training manner for model training, the indication information can also indicate which specific distributed training manner to use for model training. The indication information can be carried in the training request of step 401, or can be carried in other messages other than the training request. In this example, the order of step 404, step 402 and step 403 is not limited.

[0141] In another example, the first management node determines whether to use a distributed training manner for model training according to the model size (which can be understood as the number of layers of the model and the number of neurons per layer) obtained in step 401, the data volume of the training data, and the resource information of each computing node obtained in step 403. For example, the first management node determines the training task of the model according to the model size and the data volume of the training data, and determines the resource information required for this model training based on the training task of the model, and determines whether the resource information required for this model training is greater than the resource information available for model training in any of the computing nodes. If yes, one computing node cannot complete the model training, and a distributed training manner is used for model training.

[0142] Step 405: The first management node determines the following contents according to the training task of the model and the resource information available for model training in each computing node: each computing node participating in model training, a distributed training manner, training data corresponding to each computing node participating in model training, training subtasks, and backup strategy information.

[0143] The first management node can determine all the computing nodes that send the resource information to the first management node in step 403 as the computing nodes participating in the model training, or determine part of the computing nodes as the computing nodes participating in the model training. The following introduces examples of determining the computing nodes participating in the model training:

[0144] For example, the first management node can determine the computing nodes participating in the model training based on the size of the resource amount. For example, the computing nodes with a resource amount greater than a set threshold, or the first few computing nodes with a large resource amount, are determined as the computing nodes participating in the model training.

[0145] For another example, the first management node can determine the computing nodes participating in the model training based on the proportion of the resource available for the model training in the total resource. For example, the computing nodes with a proportion greater than a set threshold, or the first few computing nodes with a large proportion, are determined as the computing nodes participating in the model training.

[0146] It can be understood that no matter how the computing nodes participating in the model training are determined, the resources of these computing nodes should be able to complete the training task of the model.

[0147] The training subtask can be understood as training the parameter information of which layer or layers (the computing node participates in the training of all rounds). The training subtasks corresponding to different computing nodes can be completely different, or partially different and partially same. The set of the training subtasks of each computing node participating in the model training is the training task of the model. The process of determining the training subtask corresponding to each computing node participating in the model training by the first management node can refer to the examples 1 or 2 introduced in the foregoing, which will not be repeated here.

[0148] For any computing node, the first management node can determine the backup strategy information corresponding to the computing node according to the training subtask corresponding to the computing node. For example, all the layers in the training subtask determined for the computing node can be determined as the target layers in the backup strategy information, or part of the layers can be determined as the target layers in the backup strategy information. The related content has been introduced in the foregoing, which will not be repeated here.

[0149] Step 406: The first management node sends the following content corresponding to each computing node to the computing node: the distributed training mode, the training subtask, the backup strategy information, the model information, and the training data.

[0150] Correspondingly, the computing node receives the information.

[0151] The distributed training mode received by all the computing nodes is the same, for example, a data parallel training mode, or a tensor parallel training mode, or a pipeline parallel training mode. The specific content of these distributed training modes has been introduced in the foregoing and will not be repeated here. The distributed training mode can enable the computing node to determine which parameters to transfer to which other devices.

[0152] The model information received by all the computing nodes is the same, for example, the model information includes the number of layers of the model, the number of neurons of each layer, the connection relationship between the neurons, the initial weight of each neuron, etc.

[0153] The training data received by different computing nodes can be completely different, or partially the same and partially different.

[0154] For any computing node, the information in step 406 can be carried in a message sent to the computing node, or can be carried in different messages sent to the computing node.

[0155] Step 407: The computing node performs model training according to the model information, the training data, the training subtask, and the distributed training mode received in step 406.

[0156] Step 408: During the model training, the computing node stores the parameter information of the corresponding layer in the corresponding round of model training according to the backup strategy information received in step 406.

[0157] The parameter information can be referred to as training state information, and the parameter information includes but is not limited to: weight, gradient, and optimization parameter corresponding to the gradient (for example, first-order momentum and second-order momentum).

[0158] Step 409: During the model training, a computing node fails, and the computing node sends failure indication information to the first management node.

[0159] Correspondingly, the first management node receives the failure indication information from the computing node that has failed.

[0160] The failure indication information is used to indicate that the computing node has failed during the model training.

[0161] Step 410: The first management node determines model parameter recovery strategy information based on the backup strategy information sent to the computing node that has failed in step 406.

[0162] The model parameter recovery strategy information is used to indicate which layer or layers of the parameter information stored last time to load.

[0163] The first management node can determine all layers in the backup strategy information determined for the computing node as the layers in the model parameter recovery strategy information, or determine part of the layers as the layers in the model parameter recovery strategy information. The layers in the model parameter recovery strategy information can be one layer or multiple layers.

[0164] The related content of the model parameter recovery strategy information has been introduced above, and will not be repeated here.

[0165] Step 411: The first management node sends the model parameter recovery strategy information to the failed computing node.

[0166] Correspondingly, the failed computing node receives the model parameter recovery strategy information.

[0167] It should be noted that the failed computing node is only used to represent the computing node that receives the model parameter recovery strategy information, and does not mean that the computing node is still in a failure state when receiving the model parameter recovery strategy information.

[0168] Step 412: After repairing the failure, the failed computing node loads the parameter information of the corresponding layer stored in step 408 according to the model parameter recovery strategy information, and continues to train.

[0169] During the model training, one or more computing nodes may fail, or no computing node may fail. In addition, for any computing node, it may fail only once, or it may fail multiple times. Overall, steps 409 to 412 may not occur, may occur once, or may occur multiple times.

[0170] Step 413: After the model training is completed, each computing node participating in the model training sends the training result to the first management node.

[0171] Correspondingly, the first management node receives the training result from each computing node participating in the model training.

[0172] Optionally, in the case of reloading the parameter information after the computing node fails (i.e., performing steps 409 to 412), the computing node can also send indication information to the first management node, the indication information indicating that the training result uses the backup strategy information received in step 406 to back up the parameter information. Correspondingly, the first management node receives the indication information. The first management node can analyze the training result based on the indication information to better perform model training subsequently.

[0173] Optionally, step 414: the first management node aggregates the training results from the respective computing nodes, and optimizes the model based on the aggregated results.

[0174] If the first management node is the central node of the distributed training, the first management node performs step 414; if the first management node is not the central node of the distributed training, the first management node can send the training results from the respective computing nodes to the central node, and the central node aggregates the training results to optimize the model. The central node is, for example, the device or network element that sends the training request to the second management node in step 401.

[0175] In the example of FIG. 4, in one possible example, the first management node is the NMS, and the computing node is the combination of the EMS and the NE; in another possible example, the first management node is the NMS, and the computing node is the NE, and the interaction between the NMS and the NE is transparently passed through the EMS, for example, step.

[0176] The foregoing FIG. 3 and FIG. 4 introduce the first management node and the computing node participating in the model training, and in another possible scenario, the second management node also participates in the model training, and the second management node can manage the first management node, and the first management node can manage the computing node. When applied to the communication system of FIG. 1, the first management node can be the EMS, the second management node can be the NMS, and the computing node can be the NE.

[0177] The second management node can determine the training sub-tasks and the backup strategy information corresponding to the respective first management nodes, and this process is similar to the process of the first management node determining the training sub-tasks and the backup strategy information corresponding to the respective computing nodes, and the difference is only that the second management node replaces the first management node, and the first management node replaces the computing node, and this process will not be described in detail here.

[0178] The second management node sends the respective training sub-tasks and the backup strategy information corresponding to the respective first management nodes. Correspondingly, the first management node receives the training sub-tasks and the backup strategy information from the second management node.

[0179] The first management node can determine the training sub-tasks of the respective computing nodes managed by the first management node participating in the model training according to the training sub-tasks received from the second management node. This process is similar to the process of the first management node determining the training sub-tasks corresponding to the respective computing nodes, and the difference is only that the training task of the model is changed to the training sub-tasks received by the first management node, that is, the first management node can regard the training sub-tasks received by itself as the training task of the model, and this process will not be described in detail here.

[0180] The first management node can determine backup strategy information of each computing node participating in model training managed by the first management node according to the received backup strategy information from the second management node. The round in the backup strategy information of any computing node belongs to the round in the backup strategy information received by the first management node, and the layer in the backup strategy information of any computing node belongs to the layer in the backup strategy information received by the first management node. Taking the first computing node as an example: the first management node determines the first backup strategy information according to the second backup strategy information; wherein the second backup strategy information is used to indicate the parameter information of the third target layer in the third target round in the process of training the model; the first target round belongs to the third target round, for example, the first target round is part of the rounds or all rounds of the third target round; the first target layer belongs to the third target layer, and the first target layer is part of the layers or all layers in the third target layer. For any first management node, the set of backup strategy information of each computing node participating in model training managed by the first management node is the second backup strategy information corresponding to the first management node.

[0181] In addition, only for any computing node, the first management node can determine the backup strategy information of the computing node based on the training subtask of the computing node. This process is the same as the process of determining the first backup strategy information corresponding to the first computing node based on the training subtask of the first computing node as introduced in the foregoing, and this process will not be described in detail here.

[0182] As shown in FIG. 5, a flowchart of a communication method is introduced, including the following steps:

[0183] It should be noted that only one first management node is drawn in FIG. 5, and if multiple first management nodes participate in this example, the interaction processes of other undrawn first management nodes with the second management node and the computing node are similar to those of the drawn first management node. Similarly, only one computing node is drawn in FIG. 5, and if multiple computing nodes participate in this example, the interaction processes of other undrawn computing nodes with the first management node are similar to those of the drawn computing node with the first management node.

[0184] Step 501: The second management node receives a model training request.

[0185] The request can come from an operator, an access network or a core network element, etc. The model training request can contain model and training data information, etc. The model information includes the number of layers of the model, the number of neurons in each layer, the connection relationship between neurons, the initial weight of each neuron, etc.

[0186] Step 502a: The second management node queries each first management node participating in model training for resource information available for model training in the first management node.

[0187] Step 502b: The first management node queries each computing node managed by the first management node for resource information available for model training in the computing node.

[0188] For example, the resources include but are not limited to computing resources, storage resources, and computing power resources. The resource information can include the size of the resources, and optionally, the running state information of the resources. For example, the running state information includes all resource information of the computing node, occupied resource information, and remaining resource information available for model training. For example, the computing node has a total of 80G of display memory, 30G of which is occupied by other tasks, and the remaining 50G is available for model training.

[0189] Step 503a: Each computing node sends resource information available for model training (resource information available for model training in the computing node, hereinafter referred to as resource information of the computing node) to the first management node.

[0190] Correspondingly, the first management node receives the resource information from each computing node.

[0191] Step 503b: The first management node sends the resource information available for model training in the first management node or the resource information received from each computing node in step 503a to the second management node.

[0192] The first management node can integrate the resource information of each computing node managed by the first management node and then send the resource information available for model training in the first management node to the second management node, or can not integrate the resource information of each computing node and directly send the resource information of each computing node to the second management node.

[0193] The resource information available for model training in the first management node is the sum of the resource information available for model training in each computing node received by the first management node in step 503a.

[0194] Step 504: The second management node determines whether to use a distributed training mode for model training.

[0195] Step 504 can refer to the description of step 404, and the execution subject is replaced by the second management node.

[0196] In an example, the device or network element sending the training request to the second management node can inform the second management node whether to use the distributed training manner for model training. For example, in step 501, the training request can further include indication information indicating whether to use the distributed training manner for model training or not. Optionally, if the indication information indicates to use the distributed training manner for model training, the indication information can further indicate which specific distributed training manner to use for model training. The indication information can be carried in the training request in step 501, or in other messages other than the training request. In this example, the order of steps 504, 502a, 502b, 503a and 503b is not limited.

[0197] In another example, the second management node determines whether to use the distributed training manner for model training according to the model size (which can be understood as the number of layers of the model and the number of neurons per layer) obtained in step 501, the data volume of the training data, and the resource information of each computing node obtained in step 503b. For example, the second management node determines the training task of the model according to the model size and the data volume of the training data, determines the resource information required for this model training based on the training task of the model, and determines whether the resource information required for this model training is greater than the resource information available for model training in any of the computing nodes. If so, one computing node cannot complete the model training, and it can be determined that multiple computing nodes are used for model training, i.e., the distributed training manner is used for model training.

[0198] Step 505: The second management node determines the following content according to the training task of the model and the resource information available for model training of each first management node: each first management node participating in model training, the distributed training manner, the second training data corresponding to each first management node participating in model training, the second training subtask, and the second backup strategy information.

[0199] This step 505 can refer to the introduction of step 405. The difference between step 505 and step 405 includes that the first management node in step 405 is replaced by the second management node in step 505, and in addition, the computing node in step 405 is replaced by the first management node in step 505. Other specific details are not repeated.

[0200] Step 506a: The second management node sends the following content corresponding to each first management node to each first management node: the distributed training manner, the second training subtask, the second backup strategy information, the model information, the second training data, etc.

[0201] Correspondingly, the first management node receives these information.

[0202] Step 506b: The first management node determines the following according to the second training subtask, the second backup strategy information, and the resource information of each computing node participating in model training managed by the first management node and available for model training: each computing node participating in model training, a distributed training manner, training data corresponding to each computing node participating in model training, a training subtask, and backup strategy information.

[0203] Step 506b can refer to the description of step 405. The difference between step 506b and step 405 includes that the training task of the model is replaced by the second training subtask received by the first management node. In addition, the computing node in step 405 is replaced by the first management node in step 506b, and other specific details are not repeated.

[0204] It should be noted that when determining the backup strategy information of each computing node, it needs to be determined based on the second backup strategy information received by the first management node. The round in the backup strategy information of any computing node belongs to the round in the backup strategy information received by the first management node. The layer in the backup strategy information of any computing node belongs to the layer in the backup strategy information received by the first management node. For any first management node, the set of backup strategy information of each computing node participating in model training managed by the first management node is the second backup strategy information corresponding to the first management node. This part has been described in detail in the foregoing, and will not be repeated here.

[0205] Step 506c: The first management node sends the following to each computing node participating in model training managed by it: a distributed training manner, a training subtask, backup strategy information, model information, and training data.

[0206] Correspondingly, the computing node receives this information.

[0207] Step 506c can refer to step 406, which will not be repeated.

[0208] Step 507: The computing node performs model training according to the model information, training data, training subtask, and distributed training manner received in step 506c.

[0209] Step 508: During the model training process, the computing node stores the parameter information of the corresponding layer according to the backup strategy information received in step 506c.

[0210] The parameter information can be referred to as training state information, and the parameter information includes but is not limited to: weight, gradient, and optimization parameters (such as first-order momentum and second-order momentum) corresponding to the gradient.

[0211] Step 509a: In the model training process, if a computing node fails, the computing node sends failure indication information to the first management node.

[0212] Correspondingly, the first management node receives the failure indication information from the failed computing node.

[0213] The failure indication information is used to indicate that the computing node fails in the model training process.

[0214] Optionally, the failed computing node can also send backup strategy information of the failed computing node to the first management node.

[0215] Step 509b: The first management node sends the failure indication information and the backup strategy information received in step 509a to the second management node.

[0216] If the first management node receives the backup strategy information in step 509a, the first management node can directly forward it to the second management node; if the first management node does not receive the backup strategy information in step 509a, the first management node can find the backup strategy information determined for the failed computing node in step 506b and send it to the second management node in step 509b.

[0217] Step 510: The second management node determines model parameter recovery strategy information based on the backup strategy information corresponding to the failed computing node.

[0218] The model parameter recovery strategy information is used to indicate which layer or layers of the latest stored parameter information to load.

[0219] When the second management node determines the model parameter recovery strategy information, it can determine all the layers in the backup strategy information determined for the computing node as the layers in the model parameter recovery strategy information, or determine part of the layers as the layers in the model parameter recovery strategy information. The layers in the model parameter recovery strategy information can be one layer or multiple layers.

[0220] The related content of the model parameter recovery strategy information has been introduced in the foregoing, and will not be repeated here.

[0221] This step 510 can refer to step 410, and the difference between step 510 and step 410 is that the first management node in step 410 is replaced by the second management node in step 510.

[0222] Step 511a: The second management node sends the model parameter recovery strategy information to the first management node corresponding to the failed computing node.

[0223] Correspondingly, the first management node receives the model parameter recovery strategy information.

[0224] Step 511b: The first management node sends the model parameter recovery strategy information to the failed computing node.

[0225] Correspondingly, the computing node receives the model parameter recovery strategy information.

[0226] It should be noted that the failed computing node is only used to represent the computing node that receives the model parameter recovery strategy information, and does not mean that the computing node is still in a failure state when receiving the model parameter recovery strategy information.

[0227] Step 512: After the failure is repaired, the failed computing node loads the parameter information of the corresponding layer stored in step 508 according to the model parameter recovery strategy information, and continues to train.

[0228] During the model training process, one or more computing nodes may fail, or no computing node may fail. In addition, for any computing node, it may only fail once, or it may fail multiple times. Overall, steps 509a to 512 may not occur, may occur once, or may occur multiple times.

[0229] Step 513a: After the model training is completed, each computing node participating in the model training sends the training result to the first management node.

[0230] Correspondingly, the first management node receives the training result from each computing node participating in the model training.

[0231] Optionally, in the case of reloading the parameter information after the computing node fails (i.e., performing steps 509 to 512), the computing node can also send indication information to the first management node, the indication information indicating that the training result uses the backup strategy information received in step 506c to back up the parameter information. Correspondingly, the first management node receives the indication information.

[0232] Step 513b: The first management node sends the training result to the second management node.

[0233] Correspondingly, the second management node receives the training result from each first management node.

[0234] The training result can be an aggregated result of the training results from each computing node managed by the first management node participating in the model training, or the training result from the computing node can not be aggregated, but directly forwarded to the second management node.

[0235] Optionally, if the first management node receives the indication information in step 513a, the first management node can also send indication information to the second management node in step 513b, the indication information being used to indicate that the training result uses the backup strategy information received in step 506c to backup the parameter information. The second management node can analyze the training result based on the indication information, so as to better perform model training subsequently.

[0236] Optionally, step 514: the second management node can aggregate the training results from the respective first management nodes, and optimize the model based on the aggregation result.

[0237] If the second management node is the center node of the distributed training, the second management node performs step 514; if the second management node is not the center node of the distributed training, the second management node can send the training result from each computing node to the center node, and the center node aggregates the training result to optimize the model. The center node is, for example, the device or network element that sends the training request to the second management node in step 501.

[0238] After the model is used for a period of time, the model can be fine-tuned, and the fine-tuning can also be understood as model training. The process of fine-tuning is the same as the process of model training described above, and is not repeated here. Fine-tuning can be initiated by a computing node, a first management node, a second management node, or any device using the model.

[0239] It can be understood that, in order to implement the functions in the above embodiments, the computing node and the management node include the corresponding hardware structure and / or software module for performing each function. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application scenario and design constraints of the technical solution.

[0240] FIGS. 6 and 7 are structural schematic diagrams of possible communication apparatuses provided by embodiments of the present application. These communication apparatuses can be used to implement the functions of the computing node and the management node in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments.

[0241] As shown in FIG. 6, the communication apparatus 600 includes a processing unit 610 and a transceiver unit 620.

[0242] For example, the communication apparatus 600 is configured to implement the functions of the first management node, the second management node, the computing node in the method embodiments of FIG. 3, FIG. 4, and FIG. 5. The transceiver 620 is configured to perform the receiving actions and the sending actions performed by the first management node, the second management node, and the computing node in the method embodiments. The processing unit 610 is configured to perform the actions performed by the first management node, the second management node, and the computing node in the method embodiments, except for the sending actions and the receiving actions.

[0243] For example, when the communication apparatus 600 is configured to implement the functions of the first management node in the method embodiment of FIG. 3, the transceiver 620 is configured to send the first backup policy information. The processing unit 610 is configured to determine the first backup policy information.

[0244] For example, when the communication apparatus 600 is configured to implement the functions of the first computing node in the method embodiment of FIG. 3, the transceiver 620 is configured to receive the first backup policy information. The processing unit 610 is configured to store the parameter information of the first target layer in the first target round based on the first backup policy information.

[0245] The detailed description of the processing unit 610 and the transceiver 620 can be directly obtained by referring to the related description in the method embodiments of FIG. 3 to FIG. 5, which will not be repeated here. The processing unit 610 can be implemented by a processor, and the transceiver 620 can be implemented by a transceiver.

[0246] As shown in FIG. 7, the communication apparatus 700 includes a processor 710 and an interface circuit 720. The processor 710 and the interface circuit 720 are coupled with each other. It can be understood that the interface circuit 720 can be a transceiver or an input / output interface. Optionally, the communication apparatus 700 can further include a memory 730 for storing instructions executed by the processor 710 or storing input data required by the processor 710 to execute instructions or storing data generated after the processor 710 executes instructions. Sometimes, the interface circuit 720 can also be understood as a part of the processor 710, and the communication apparatus 700 includes the processor 710 at this time.

[0247] When the communication apparatus 700 is configured to implement the method embodiments of FIG. 3 to FIG. 5, the processor 710 is configured to implement the functions of the processing unit 610, and the interface circuit 720 is configured to implement the functions of the transceiver 620.

[0248] When the communication apparatus is a chip applied to a terminal device, the terminal device chip implements the functions of the terminal device in the method embodiments. The terminal device chip receives information from a network device, which can be understood as the information being received by other modules (such as a radio frequency module or an antenna) in the terminal device first, and then being sent to the terminal device chip by the modules. The terminal device chip sends information to the network device, which can be understood as the information being sent to other modules (such as a radio frequency module or an antenna) in the terminal device first, and then being sent to the network device by the modules.

[0249] When the communication apparatus is a chip applied to a network device, the network device chip implements the functions of the network device in the method embodiments. The network device chip receives information from a terminal device, which can be understood as the information being received by other modules (such as a radio frequency module or an antenna) in the network device first, and then being sent to the network device chip by the modules. The network device chip sends information to the terminal device, which can be understood as the information being sent to other modules (such as a radio frequency module or an antenna) in the network device first, and then being sent to the terminal device by the modules. The network device module here can be a baseband chip of the network device, or a DU or other module, and the DU here can be a DU under the O-RAN architecture.

[0250] In this application, entity A sending information to entity B can be A sending directly to B, or A sending indirectly to B through other entities. Similarly, entity B receiving information from entity A can be entity B receiving the information sent by entity A directly, or entity B receiving the information sent by entity A indirectly through other entities. The entity A and B here can be a network device or a terminal device, or a module inside the network device or a module inside the terminal device. The sending and receiving of information can be the information interaction between a network device and a terminal device, or the information interaction between two network devices, such as the information interaction between a CU and a DU; or the information interaction between different modules in one apparatus, such as the information interaction between a terminal device chip and other modules in the terminal device, or the information interaction between a network device chip and other modules in the network device.

[0251] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0252] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program can make a computer execute the communication method when the computer program is executed by the computer. In other words, the computer program includes instructions for implementing the communication.

[0253] The embodiments of the present application further provide a computer program product, which includes computer program codes. When the computer program codes are executed on a computer, the computer can execute the communication method provided by the embodiments of the present application.

[0254] The embodiments of the present application further provide a communication system, which includes a computing node and a first management node for executing the communication method.

[0255] The method steps in the embodiments of the present application can be implemented by means of hardware, or by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a compact disc read-only memory (CD-ROM) (also known as a read-only optical disc), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a base station or a terminal. Of course, the processor and the storage medium can also exist as discrete components in the base station or the terminal.

[0256] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments are performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a first control plane network element, user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer programs or instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired or wireless manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; an optical medium, for example, a digital video disc; and a semiconductor medium, for example, a solid-state disk. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile storage media.

[0257] In various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0258] In the embodiments of the present application, the number of nouns, unless otherwise specified, means "singular or plural", that is, "one or more". "At least one" means one or more, and "multiple" means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. For example, A / B means A or B. "At least one of the following" or "one or more of the following" and the like means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c, or one or more of a, b or c, means a, b, c, a and b, a and c, b and c, or a and b and c. Each of a, b and c can be single or multiple.

[0259] The terms "first", "second", and the like in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the size, content, order, time sequence, priority, or importance of the plurality of objects. Moreover, such names do not represent the difference in the content, the sending / receiving end, the sending order, the size, the application scenario, the priority, or the importance, and the like included in the two pieces of information. In addition, the numbering of the steps in each embodiment introduced in the present application is only for distinguishing different steps, and is not used to limit the order between the steps.

Claims

1. A backup method, characterized in that: Applied to compute nodes, including: Receive first backup strategy information from the first management node, where the first backup strategy information is used to instruct the computing node to store parameter information of the first target layer in the first target round during training of the model; Based on the first backup strategy information, parameter information of the first target layer is stored in the model training of the first target round.

2. The method according to claim 1, wherein The first target round is a portion of all rounds of the model training, and / or the first target layer is a portion of all layers of the model.

3. The method according to claim 1 or 2, wherein: Also includes: Receiving a distributed training mode and a first training subtask of the model from the first management node; The first training subtask includes at least: a task for training parameter information of the first target layer; the distributed training method includes one or more of the following: data parallel training method, tensor parallel training method, and pipeline parallel training method; The model is trained based on the distributed training method and the first training subtask.

4. The method according to any one of claims 1 to 3, wherein Also includes: Sending fault indication information to the first management node, where the fault indication information is used to indicate that a fault has occurred in the computing node during training of the model; receiving model parameter recovery strategy information from the first management node; The model parameter recovery strategy information is used to instruct to load the parameter information in the second target layer stored most recently, where the second target layer is a part of or all of the first target layer; After the fault is repaired, the parameter information in the second target layer that was most recently stored is loaded based on the model parameter recovery strategy information, and recovery training is performed on the model to obtain a training result.

5. The method according to any one of claims 1 to 4, characterized in that The first management node is a cross-domain management system NMS entity, and the computing node is a domain management system EMS entity; or, The first management node is a domain management system EMS entity, and the computing node is a network element NE.

6. A backup method, characterized in that: Applied to the first management node, including: Determining first backup strategy information corresponding to a first computing node participating in model training, where the first backup strategy information is used to instruct the first computing node to store parameter information of a first target layer in a first target round during training of the model, the first computing node being any node participating in the model training; Send the first backup policy information to the first computing node.

7. The method according to claim 6, wherein The first target round is a portion of all rounds of the model training, and / or the first target layer is a portion of all layers of the model.

8. The method according to claim 6 or 7, wherein: Also includes: Obtaining resource information of the first computing node; Sending, based on the resource information of the first computing node and the training task of the model, a distributed training mode of the model and a first training subtask to the first computing node, where the first training subtask belongs to the training task of the model; The first training subtask includes at least: a task for training parameter information of the first target layer; The distributed training method includes one or more of the following: data parallel training method, tensor parallel training method, and pipeline parallel training method.

9. The method according to claim 8, wherein Also includes: Obtain resource information of a second computing node, where the second computing node is any one of the computing nodes for model training except the first computing node; The sending, based on the resource information of the first computing node and the training task of the model, the distributed training mode of the model and the first training subtask to the first computing node includes: Based on the resource information of the first computing node, the resource information of the second computing node and the training task of the model, the distributed training mode and the first training subtask are sent to the first computing node.

10. The method according to claim 9, wherein Also includes: Based on the resource information of the first computing node, the resource information of the second computing node and the training task of the model, the distributed training mode and the second training subtask are sent to the second computing node, where the second training subtask belongs to the training task of the model.

11. The method according to any one of claims 6 to 10, characterized in that Also includes: receiving fault indication information from the first computing node, the fault indication information being used to indicate that a fault occurs in the first computing node during training of the model; Based on the fault indication information and the first backup strategy information, model parameter recovery strategy information is sent to the first computing node, and the model parameter recovery strategy information is used to indicate the loading of the parameter information in the second target layer that was most recently stored, where the second target layer is part of or all of the first target layer.

12. The method according to any one of claims 6 to 11, wherein: The first management node is a cross-domain management system NMS entity, and the first computing node is a domain management system EMS entity.

13. The method according to claim 6 or 7, wherein: The determining of first backup strategy information corresponding to the first computing node participating in model training includes: Determine the first backup strategy information based on the second backup strategy information; wherein, the second backup strategy information is used to indicate that parameter information of the third target layer in the third target round is stored during training of the model; the first target round belongs to the third target round, and the first target layer belongs to the third target layer.

14. The method according to claim 13, wherein Also includes: Receive the second backup strategy information from the second management node.

15. The method according to claim 13 or 14, characterized in that The second management node is a cross-domain management system NMS entity, the first management node is a domain management system EMS entity, and the first computing node is a network element NE.

16. A communication device, characterized in that: The method comprises a module for executing the method according to any one of claims 1 to 15.

17. A communication device, characterized in that: comprising a processor coupled to a memory; The memory is used to store computer programs or instructions; The processor is configured to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, is configured to implement the method according to any one of claims 1 to 15.

18. A communication device, characterized in that: including processor and memory; The memory is used to store computer programs or instructions; The processor is configured to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, is configured to implement the method according to any one of claims 1 to 15.

19. A communication chip system, characterized in that: comprising a processor and an interface circuit, wherein the processor is coupled to the memory; The memory is used to store computer programs or instructions; The processor is configured to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, is configured to implement the method according to any one of claims 1 to 15.

20. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instruction. When the computer program or instruction is executed by the communication device, the method according to any one of claims 1 to 15 is implemented.

21. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed on a computer, the method according to any one of claims 1 to 15 is implemented.

Images