Artificial intelligence model training method, model parameter synchronization method and first entity

Abstract

The invention discloses a training method of an artificial intelligence model, a model parameter synchronization method and a first entity, and the training method of the artificial intelligence model comprises the steps that the first entity carries out communication addressing among parameter synchronization services of each training partition in a plurality of training partitions, the parameter synchronization service carries out model parameter synchronization among the plurality of training partitions; wherein the plurality of training partitions are obtained by splitting an artificial intelligence model training task, and the plurality of training partitions are deployed in different clouds.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly relates to a method for training an artificial intelligence model, a method for synchronizing model parameters, a device, a device, a first entity, a storage medium, and a computer program product. Background Art

[0002] At present, the training process of an artificial intelligence (AI) model has a high demand for computing power. Currently, building a distributed artificial intelligence model training framework based on serverless computing can further improve the manageability of computing power resources, but it cannot guarantee high-frequency and large-scale cross-node communication. Summary of the Invention

[0003] To solve the related technical problems, the embodiments of this application provide a method for training an artificial intelligence model, a method for synchronizing model parameters, a device, a device, a first entity, a storage medium, and a computer program product.

[0004] The technical solution of the embodiments of this application is implemented as follows:

[0005] The embodiments of this application provide a method for training an artificial intelligence model, which is applied to a first entity. The method includes:

[0006] Performing communication addressing among the parameter synchronization services of each training partition in multiple training partitions, so that the parameter synchronization services synchronize model parameters among the multiple training partitions; where

[0007] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0008] In the above solution, before performing communication addressing among the parameter synchronization services of each training partition in multiple training partitions, the method further includes:

[0009] Splitting an artificial intelligence model training task into the multiple training partitions, and / or deploying the multiple training partitions in the different clouds.

[0010] In the above solution, the first entity supports a distributed communication management service. Before performing communication addressing among the parameter synchronization services of each training partition in multiple training partitions, the method further includes:

[0011] Assigning a corresponding first identifier to the parameter synchronization service of each training partition in the multiple training partitions, where the first identifier is used to uniquely map the communication address of the corresponding parameter synchronization service in the wide area network.

[0012] In the above solution, there is a mapping relationship between the first identifier and the internal communication address of the corresponding parameter synchronization service in the training partition to which it belongs. Communicating and addressing between the parameter synchronization services of each training partition among the multiple training partitions includes:

[0013] Based on the first identifier corresponding to each parameter synchronization service and the corresponding mapping relationship, perform communication addressing between the parameter synchronization services of each training partition among the multiple training partitions.

[0014] An embodiment of the present application also provides a model parameter synchronization method, which is applied to a first entity. The method includes:

[0015] In the case where the communication addressing of the parameter synchronization services of each training partition among the multiple training partitions is completed, perform model parameter synchronization between the multiple training partitions based on the parameter synchronization services of each training partition; wherein,

[0016] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0017] Wherein, in the above solution, before performing model parameter synchronization between the multiple training partitions based on the parameter synchronization services of each training partition, the method further includes:

[0018] Split the artificial intelligence model training task into the multiple training partitions, and / or deploy the multiple training partitions in the different clouds.

[0019] In the above solution, the first entity supports a parameter synchronization service. Performing model parameter synchronization between the multiple training partitions based on the parameter synchronization services of each training partition includes:

[0020] The first parameter synchronization service in the first training partition obtains one or more gradients, where each gradient represents the gradient of the model parameters determined each time the model parameters in the first training partition are updated;

[0021] When the set condition is met, the first parameter synchronization service sends one or more first gradients to the second parameter synchronization services of each second training partition in one or more second training partitions. The one or more first gradients are determined based on first information, and the first information represents all the gradients obtained by the first parameter synchronization service after the last model parameter synchronization.

[0022] In the above solution, the set condition includes:

[0023] Meet the set synchronization frequency, where the synchronization frequency characterizes the frequency of model parameter synchronization, and the synchronization frequency is used to adjust the computing power resource allocation strategy and function trigger strategy for each training partition among the multiple training partitions.

[0024] In the above solution, the method further includes:

[0025] The first parameter synchronization service in the first training partition updates the model parameters of the model in the first training partition based on one or more second gradients sent by the third parameter synchronization service in the third training partition; where

[0026] The one or more second gradients are determined based on second information, and the second information characterizes all gradients obtained by the third parameter synchronization service after the last model parameter synchronization, where each gradient characterizes the gradient of the model parameters determined each time the model in the third training partition is updated.

[0027] In the above solution, the first entity supports a training management service, where the training management service is used to provide processing requests and response results in real time during the process of updating the model parameters of the model, and to provide a function routing mechanism.

[0028] In the above solution, among the one or more first gradients, the first gradients with absolute values less than a set threshold account for a first percentage of the total number of first gradients.

[0029] In the above solution, the first percentage is negatively correlated with the convergence degree of the model.

[0030] In the above solution, the first percentage is determined based on one or more evaluation performances of the model on a validation set.

[0031] An embodiment of this application also provides a training device for an artificial intelligence model, including:

[0032] A communication addressing unit, configured to perform communication addressing between parameter synchronization services of each training partition among multiple training partitions, so that the parameter synchronization services perform model parameter synchronization among the multiple training partitions; where

[0033] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0034] An embodiment of this application also provides a model parameter synchronization device, including:

[0035] A parameter synchronization unit, configured to perform model parameter synchronization among the multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition in the multiple training partitions is completed; wherein,

[0036] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0037] An embodiment of the present application further provides a device, including: a first processor and a first communication interface; wherein,

[0038] The first communication interface is configured to perform communication addressing among the parameter synchronization services of each training partition in the multiple training partitions, so that the parameter synchronization service performs model parameter synchronization among the multiple training partitions; wherein,

[0039] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0040] An embodiment of the present application further provides a device, including: a second processor and a second communication interface; wherein,

[0041] The second communication interface is configured to perform model parameter synchronization among the multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition in the multiple training partitions is completed; wherein,

[0042] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0043] An embodiment of the present application further provides a device, including: a processor and a memory for storing a computer program that can run on the processor,

[0044] Wherein, when the processor is used to run the computer program, it executes the steps of any of the above-mentioned artificial intelligence model training methods, or executes the steps of any of the above-mentioned model parameter synchronization methods.

[0045] An embodiment of the present application further provides a first entity, characterized in that the first entity includes:

[0046] A communication addressing unit, configured to perform communication addressing among the parameter synchronization services of each training partition in the multiple training partitions; and / or,

[0047] A parameter synchronization unit is used to perform model parameter synchronization among the multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service for each training partition in the multiple training partitions is completed; wherein,

[0048] The multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0049] An embodiment of the present application also provides a storage medium, on which a computer program is stored. The computer program, when executed by a processor, implements the steps of any one of the above-mentioned artificial intelligence model training methods or implements the steps of any one of the above-mentioned model parameter synchronization methods.

[0050] An embodiment of the present application also provides a computer program product, including a computer program. The computer program, when executed by a processor, implements the steps of any one of the above-mentioned artificial intelligence model training methods or implements the steps of any one of the above-mentioned model parameter synchronization methods.

[0051] In an embodiment of the present application, an artificial intelligence model training task is split into multiple training partitions deployed in different clouds, and communication addressing is performed among the parameter synchronization services of each training partition in the multiple training partitions to effectively ensure high-frequency and large-scale cross-node communication. On the basis of completing the communication addressing, during the training process of the artificial intelligence model, cross-cloud model parameter synchronization is performed among the multiple training partitions, thereby effectively utilizing the computing power resources of multiple clouds, improving the training efficiency of the artificial intelligence model and reducing the training cost. Description of the Drawings

[0052] Figure 1 It is a schematic flowchart of the implementation of the artificial intelligence model training method according to an embodiment of the present application;

[0053] Figure 2 It is a schematic flowchart of the implementation of the model parameter synchronization method according to an embodiment of the present application;

[0054] Figure 3 It is an example diagram of the implementation process of the model parameter synchronization method according to an embodiment of the present application;

[0055] Figure 4 It is an example diagram of the first entity architecture according to an embodiment of the present application;

[0056] Figure 5 It is a schematic structural diagram of the artificial intelligence model training device according to an embodiment of the present application;

[0057] Figure 6 It is a schematic structural diagram of the model parameter synchronization device according to an embodiment of the present application;

[0058] Figure 7Schematic diagram of a device structure according to an embodiment of the present application;

[0059] Figure 8 Another schematic diagram of a device structure according to an embodiment of the present application. Detailed implementation manners

[0060] The training of an artificial intelligence model involves a large amount of data processing and highly complex mathematical operations, thus posing relatively high requirements for computing power. To meet the growing computing power requirements for artificial intelligence model training, the training framework of artificial intelligence models has gradually evolved from a single-machine architecture to a multi-machine distributed architecture, which can effectively combine multiple single-machine resources, improving the flexibility and scalability of the training framework. Serverless computing is a new cloud computing paradigm that manages the life cycle at the function granularity, featuring high elasticity and fine granularity, and can further improve the manageability of computing power resources, but it cannot guarantee high-frequency and large-scale cross-node communication.

[0061] Based on this, the embodiments of the present application split the artificial intelligence model training task into multiple training partitions deployed in different clouds, and perform communication addressing between the parameter synchronization services of each training partition in the multiple training partitions to effectively guarantee high-frequency and large-scale cross-node communication. On the basis of completing the communication addressing, during the training process of the artificial intelligence model, cross-cloud model parameter synchronization is performed between the multiple training partitions, thereby effectively utilizing the computing power resources of multiple clouds, improving the training efficiency of the artificial intelligence model, and reducing the training cost.

[0062] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0063] First, the first entity involved in the embodiments of the present application will be described.

[0064] In the embodiments of the present application, the first entity can be understood as a cross-cloud artificial intelligence model training platform based on Serverless, that is, a cross-cloud distributed training framework for artificial intelligence models based on Serverless. The first entity is constructed among multiple clouds to utilize the computing power of multiple clouds to provide cross-cloud artificial intelligence model training services, thereby supporting the realization of requirements such as distributed training data in different locations, collaborative computing power in different locations, and low-cost training, and ultimately realizing artificial intelligence model training with unified management of multiple clouds and invisible underlying computing power.

[0065] The embodiments of the present application provide a method for training an artificial intelligence model, which is applied to the first entity. Referring to Figure 1 , the method includes:

[0066] Step 101: Perform communication addressing among the parameter synchronization services of each training partition in multiple training partitions, so that the parameter synchronization services synchronize model parameters among multiple training partitions.

[0067] Among them, the multiple training partitions are obtained by splitting the artificial intelligence model training task, and the multiple training partitions are deployed in different clouds

[0068] In the embodiment of the present application, the training task of the artificial intelligence model is split into multiple training partitions, that is to say, the artificial intelligence model is trained through multiple training partitions. In practical applications, these multiple training partitions are deployed in different clouds, so as to realize the cross-cloud distributed artificial intelligence model training process. When completing the training of the artificial intelligence model based on multiple training partitions, the multiple training partitions can communicate with each other to synchronize model parameters.

[0069] In one embodiment, before performing communication addressing among the parameter synchronization services of each training partition in multiple training partitions, the method further includes:

[0070] Split the artificial intelligence model training task into the multiple training partitions, and / or deploy the multiple training partitions in different clouds.

[0071] Here, the first entity can support splitting the artificial intelligence model training task into multiple training partitions and deploying the multiple training partitions in different clouds to realize cross-cloud distributed artificial intelligence model training. In practical applications, one training partition can be deployed on each of multiple clouds.

[0072] In the cross-cloud distributed artificial intelligence model training process, each training partition performs the training work of the artificial intelligence model respectively. Therefore, model parameter synchronization is required among multiple training partitions to ensure the completion of the artificial intelligence model training task. In the embodiment of the present application, a parameter synchronization service is deployed on each training partition, and the first entity supports a distributed communication management service. The distributed communication management service performs communication addressing among the parameter synchronization services of these training partitions, so that communication can be realized among the parameter synchronization services of different training partitions, and the transmission of model parameters can be completed. On this basis, model parameter synchronization among multiple training partitions can be realized.

[0073] In practical applications, the parameter synchronization service can be deployed on the corresponding training partition in the form of a parameter server (PS, Parameter Server).

[0074] In the implementation of this application, a corresponding communication addressing mechanism is proposed, that is, a distributed communication management service, which is used to implement model parameter synchronization. The first entity supports this distributed communication management service. And corresponding to this distributed communication management service, in one embodiment, before performing communication addressing between the parameter synchronization services of each training partition among multiple training partitions, the method further includes:

[0075] Assign a corresponding first identifier to the parameter synchronization service of each training partition among multiple training partitions, and this first identifier is used to uniquely map the communication address of the corresponding parameter synchronization service in the wide area network.

[0076] Here, the parameter synchronization service of each training partition is deployed in a local serverless operating environment, and each parameter synchronization service is configured with a unique wide area network communication address to support communication between the parameter synchronization services of different training partitions and achieve synchronization of model parameters.

[0077] In addition, the parameter synchronization service of each training partition also holds a specific internal communication address in the local serverless operating environment for access by cloud function instances on local worker nodes. Based on this, in one embodiment, there is a mapping relationship between the first identifier and the internal communication address of the corresponding parameter synchronization service in the training partition to which it belongs. Correspondingly, performing communication addressing between the parameter synchronization services of each training partition among multiple training partitions includes:

[0078] Perform communication addressing between the parameter synchronization services of each training partition among multiple training partitions based on the first identifier corresponding to each parameter synchronization service and the corresponding mapping relationship.

[0079] Corresponding to the cross-cloud scenario, that is, the situation where multiple training partitions are deployed in different clouds. In the cross-cloud distributed training workflow, each parameter synchronization service holds a specific internal communication address in the local serverless operating environment for access by cloud function instances of local worker nodes. The setting of this internal communication address is to ensure that cloud function instances of local worker nodes can conveniently access and obtain the required model parameters, so that they can efficiently obtain model parameters from the parameter synchronization service to further support the synchronization of model parameters during the training process.

[0080] On this basis, in the face of the cross-cloud scenario, in addition to supporting local access, the parameter synchronization service also needs to support cross-cloud communication between different parameter synchronization services. Therefore, by introducing the first identifier, the accuracy and conflict-freeness of cross-cloud communication between different parameter synchronization services can be ensured.

[0081] In a cross-cloud distributed training workflow, first wait for the parameter synchronization services in each cloud to be ready. Once the parameter synchronization services are ready, assign a first identifier to each parameter synchronization service. This first identifier is used to uniquely map or indicate the communication address of the parameter synchronization service in the wide area network, including the IP address and port in the wide area network. And by mapping the internal communication address of the parameter synchronization service in the serverless running environment to the IP address and port in the wide area network, a corresponding mapping relationship is established. Through this mapping relationship, communication addressing can be achieved between different parameter synchronization services. One parameter synchronization service can obtain the wide area network communication address of another parameter synchronization service. In this way, the parameter synchronization services from different clouds can communicate with each other and share information.

[0082] Based on the training method of the artificial intelligence model provided in the above embodiment, in order to improve the synchronization efficiency of model parameters during training, an embodiment of the present application also provides a model parameter synchronization method, which is applied to a first entity. Refer to Figure 2 , the method includes:

[0083] Step 201: In the case where the communication addressing of the parameter synchronization services for each training partition among multiple training partitions is completed, perform model parameter synchronization among the multiple training partitions based on the parameter synchronization services for each training partition.

[0084] Among them, the multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0085] In an embodiment, before performing model parameter synchronization among the multiple training partitions based on the parameter synchronization services for each training partition, the method further includes:

[0086] Split the artificial intelligence model training task into multiple training partitions, and / or deploy the multiple training partitions in different clouds.

[0087] Here, the first entity can support splitting the artificial intelligence model training task into multiple training partitions and deploying the multiple training partitions in different clouds to achieve cross-cloud distributed training of the artificial intelligence model. In actual application, one training partition can be deployed on each of multiple clouds.

[0088] During the training process of an artificial intelligence model with multiple training partitions, each training partition conducts the training work of the artificial intelligence model independently. Therefore, model parameter synchronization is required among multiple training partitions to ensure the completion of the artificial intelligence model training task. In the embodiments of the present application, a parameter synchronization service is deployed on each training partition. By performing communication addressing among the parameter synchronization services of these training partitions, communication can be achieved among the parameter synchronization services of different training partitions, and the transmission of model parameters can be completed, thereby enabling model parameter synchronization among multiple training partitions.

[0089] In the embodiments of the present application, the synchronization of model parameters is implemented based on the Asynchronous Stochastic Gradient Descent with Gradient Accumulation (ASGD-GA) mechanism. The ASGD-GA mechanism provides an asynchronous coordination strategy for different training partitions. Based on this, in one embodiment, the first entity supports the parameter synchronization service. Correspondingly, based on the parameter synchronization service of each training partition, model parameter synchronization among multiple training partitions includes:

[0090] The first parameter synchronization service in the first training partition obtains one or more gradients, where each gradient represents the gradient of the model parameters determined each time the model parameters of the first training partition are updated.

[0091] When the set conditions are met, the first parameter synchronization service sends one or more first gradients to the second parameter synchronization service of each of the one or more second training partitions. The one or more first gradients are determined based on the first information, and the first information represents all the gradients obtained by the first parameter synchronization service after the last model parameter synchronization.

[0092] Here, during the model parameter synchronization process, the first parameter synchronization service acts as the sender, and the second parameter synchronization service acts as the receiver. Combining Figure 3 , the specific model parameter synchronization process is as follows:

[0093] 1. The cloud function instance of the worker node in Training Partition 1 pulls the latest model parameters from the parameter server, calculates the gradient of the model parameters, and pushes the gradient to the parameter server;

[0094] 2. The parameter server updates the artificial intelligence model according to the received first gradient, that is, updates the model parameters of the artificial intelligence model;

[0095] 3. The parameter server determines whether cross-cloud model parameter synchronization is required based on set conditions. If cross-cloud model parameter synchronization is not required, this round of training iteration is completed, and the gradients are accumulated. If cross-cloud model parameter synchronization is required, step 4 is executed;

[0096] 4. The parameter server sends one or more gradients to the parameter server of training partition 2 based on the ASGD-GA synchronization policy and the communication addressing result. Here, the gradients sent refer to the gradients of all the obtained model parameters after the parameter server of training partition 1 last updated the model parameters;

[0097] 5. The parameter server of training partition 2 updates the model parameters of the artificial intelligence model according to the ASGD-GA synchronization policy.

[0098] In one embodiment, the set conditions include:

[0099] Meeting the set synchronization frequency, where the synchronization frequency characterizes the frequency of model parameter synchronization of the model, and this synchronization frequency is used to adjust the computing power resource allocation policy and function trigger policy of each training partition in multiple training partitions.

[0100] Exemplarily, the frequency of model parameter synchronization between different training partitions can be increased by increasing the synchronization frequency, so as to reduce the difference in computing power resources between different training partitions. Or, the trigger frequency of the cloud function instances of the worker nodes can be increased by increasing the synchronization frequency.

[0101] Here, the ASGD-GA synchronization policy synchronizes model parameters by transmitting gradients, and the synchronization condition, that is, the set condition, is defined as the synchronization frequency in this embodiment. When this synchronization frequency is met, the corresponding parameter synchronization service sends the accumulated gradients of this training partition to the parameter synchronization services of other training partitions. When the receiving parameter synchronization service receives this accumulated gradient, it will use the stochastic gradient descent algorithm to update the model parameters; if this synchronization frequency is not met, the gradients of each iteration of the artificial intelligence model within this training partition will be accumulated. Here, the accumulated gradient is all the gradients obtained by the parameter synchronization service of this training partition after the last model parameter synchronization within a training partition.

[0102] In the above model parameter synchronization process, the synchronization of model parameters and the update of model parameters within this training partition are asynchronous. Therefore, the synchronization of model parameters does not hinder the training process of this training partition, and the receiving parameter synchronization service does not need to wait for the sending parameter synchronization service to complete this round of training.

[0103] In practical applications, a series of optimization schemes can be adopted based on the ASGD-GA synchronization strategy to reduce the communication overhead of model parameter synchronization and improve communication efficiency. Generally speaking, the gradient update of model parameters is positively skewed, that is, as the artificial intelligence model converges during training, more and more gradient values gradually tend to 0. These small enough gradient values have far less impact on the convergence speed of the model than large gradient values. Therefore, it is not necessary to transmit these small gradients in each round of training. However, when the model has not yet converged, discarding too many gradient values will lose some features and affect the final accuracy of the model. Therefore, in this embodiment, gradient compression technology is used to reduce the amount of data for gradient transmission, and gradient sparsification is achieved by discarding a part of the "unimportant" gradients. Specifically, the minimum R% sparsity algorithm can be adopted. By selecting a threshold, the number of gradients with absolute values less than this threshold accounts for R% of all gradient numbers, and then this part of the gradients with absolute values less than the threshold is discarded. The sparsified part is accumulated through error compensation.

[0104] Based on this, in one embodiment, among one or more first gradients, the first gradients with absolute values less than the set threshold account for the first percentage of the total number of first gradients.

[0105] Gradient compression technology may cause loss of model accuracy and needs to be balanced between training performance and model accuracy when used. Due to the diversity of models and training environments, it is crucial to select an appropriate threshold R% for a specific model and training environment during actual training. Feasible selection strategies include:

[0106] Selection strategy one: Adaptive threshold adjustment.

[0107] The core of this selection strategy is to dynamically adjust the threshold R% of gradient sparsification to meet the needs of the model at different training stages. At the initial stage of training, a larger R% can be selected to retain more gradient information, which helps the model capture more feature information in the initial stage and thus accelerate the convergence speed. As the training process progresses and the model begins to gradually converge, R% can be gradually reduced at this time to reduce communication overhead without excessive loss of model accuracy. This adaptive adjustment can be achieved by monitoring the convergence speed of the model and the changes in gradients.

[0108] That is to say, the first percentage is negatively correlated with the convergence degree of the model.

[0109] For example, when it is monitored that the gradient update begins to become flat, R% can be appropriately reduced.

[0110] Selection strategy two: Performance metric feedback.

[0111] The core of this selection strategy lies in evaluating the performance of the model on the validation set, such as accuracy, loss function value, etc., after each training cycle, so as to adjust the gradient according to the performance metrics of model training. If the performance metrics show a decrease in model accuracy, which may be due to information loss caused by excessive gradient compression, then R% can be appropriately increased to retain more gradient information. On the contrary, if the model performance is stable or improving, and the communication overhead is within an acceptable range, R% can be maintained or further reduced.

[0112] That is to say, the first percentage is determined based on one or more evaluation performances of the model on the validation set.

[0113] In practical applications, selection strategy one and selection strategy two can be combined with each other to form a closed-loop optimization system. In this way, not only can R% be adjusted according to real-time performance feedback, but also future trends can be predicted, so as to adjust R% in advance to maintain the efficiency and stability of model training.

[0114] For the case where the first parameter synchronization service is the receiver and the third parameter synchronization service is the sender during the model parameter synchronization process, in one embodiment, the method further includes:

[0115] The first parameter synchronization service in the first training partition updates the model parameters of the model in the first training node of the first training partition based on one or more second gradients sent by the third parameter synchronization service in the third training partition; wherein,

[0116] The one or more second gradients are determined based on second information, and the second information represents all gradients sent by the third training nodes in the third training partition received by the third parameter synchronization service after the last model parameter synchronization of the model.

[0117] Here, the process of updating the model parameters of the model is implemented based on the training management service, wherein the training management service is used to provide processing requests and response results in real time during the process of updating the model parameters of the model, and provide a function routing mechanism.

[0118] Based on the communication addressing between cross-cloud parameter synchronization services, the above model parameter synchronization method can perform cross-cloud model parameter synchronization between multiple training partitions during the training process of an artificial intelligence model, thereby effectively utilizing the computing power resources of multiple clouds, improving the training efficiency of the artificial intelligence model and reducing the training cost.

[0119] Based on the above embodiments, Figure 4 shows an architecture example corresponding to the first entity in the embodiments of the present application. Refer to Figure 4, the first entity provided by the embodiments of the present application includes a training task management layer and a training layer, thus simplifying the management logic of training tasks in a multi-cloud scenario. Among them, the task scheduler of the training task management layer is responsible for generating a deployment strategy for computing resources, and the communication management unit is responsible for cross-cloud communication between each training partition in the training layer, that is, for supporting the implementation of a cross-cloud communication addressing mechanism. In actual application, a distributed communication management service can be deployed on the communication management unit; the training layer includes multiple training partitions, and these multiple training partitions can be deployed on different clouds. Each training partition is responsible for the training work of a specific artificial intelligence model in its affiliated cloud area, and for synchronizing the model parameters of the artificial intelligence model between different clouds. Specifically, a working node is deployed on each training partition, and a cloud function instance runs on the working node for training an artificial intelligence model based on a local data set. At the same time, a parameter synchronization service is deployed corresponding to each training partition. The parameter server is responsible for managing the model state of the artificial intelligence model, updating the model parameters of the model, and communicating with the parameter synchronization services of other training partitions to achieve model parameter synchronization.

[0120] To implement the training method of the artificial intelligence model in the embodiments of the present application, the embodiments of the present application also provide a training device for an artificial intelligence model, which can be deployed in the first entity. Specifically, it can be deployed in Figure 4 the communication management unit. As Figure 5 shown, the device includes:

[0121] A communication addressing unit 501, configured to perform communication addressing between the parameter synchronization services of each training partition among multiple training partitions, so that the parameter synchronization services synchronize model parameters among multiple training partitions.

[0122] Among them, the multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0123] Among them, in one embodiment, the first entity supports a distributed communication management service, and the device further includes:

[0124] An allocation unit, configured to allocate a corresponding first identifier to the parameter synchronization service of each training partition among multiple training partitions, and the first identifier is used to uniquely map the communication address of the corresponding parameter synchronization service in the wide area network.

[0125] In one embodiment, there is a mapping relationship between the first identifier and the internal communication address of the corresponding parameter synchronization service in its affiliated training partition. Correspondingly, the communication addressing unit 501 is configured to:

[0126] Based on the first identifier corresponding to each parameter synchronization service and the corresponding mapping relationship, perform communication addressing between the parameter synchronization services of each training partition among multiple training partitions.

[0127] In actual application, the communication addressing unit 501 can be implemented by the communication interface in the training device of the artificial intelligence model, and the allocation unit can be implemented by the processor in the training device of the artificial intelligence model.

[0128] It should be noted that: when the training device of the artificial intelligence model provided in the above embodiment trains the artificial intelligence model, only the division of the above program modules is used for illustration. In actual application, the above processing can be allocated to different program modules according to needs, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the training device of the artificial intelligence model provided in the above embodiment and the embodiment of the artificial intelligence model training method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be elaborated here.

[0129] To implement the model parameter synchronization method in the embodiments of the present application, the embodiments of the present application also provide a model parameter synchronization device, which can be deployed in the first entity. Specifically, it can be deployed in Figure 4 the parameter server of. As Figure 6 shown, the device includes:

[0130] The parameter synchronization unit 601 is used to perform model parameter synchronization among multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition in multiple training partitions is completed.

[0131] Among them, the multiple training partitions are obtained by splitting the artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0132] In one embodiment, the first entity supports the parameter synchronization service, and the first parameter synchronization service in the first training partition obtains one or more gradients, where each gradient represents the gradient of the model parameters determined when the model parameters of the first training partition are updated each time.

[0133] When the set condition is met, the first parameter synchronization service sends one or more first gradients to the second parameter synchronization service of each second training partition in one or more second training partitions, and the one or more first gradients are determined based on the first information, and the first information represents all the gradients obtained by the first parameter synchronization service after the last model parameter synchronization.

[0134] In one embodiment, the set condition includes:

[0135] Meet the set synchronization frequency, which represents the frequency of model parameter synchronization, and the synchronization frequency is used to adjust the computing power resource allocation strategy and function trigger strategy of each training partition in multiple training partitions.

[0136] In one embodiment, the first parameter synchronization service in the first training partition updates the model parameters of the model in the first training partition based on one or more second gradients sent by the third parameter synchronization service in the third training partition; wherein,

[0137] The one or more second gradients are determined based on second information, and the second information represents all gradients obtained by the third parameter synchronization service after the last model parameter synchronization, wherein each gradient represents the gradient of the model parameters determined when the model in the third training partition updates the model parameters each time.

[0138] In one embodiment, the first entity supports a training management service, wherein the training management service is used to provide processing requests and response results in real time during the process of updating the model parameters of the model, and provide a function routing mechanism.

[0139] In one embodiment, among the one or more first gradients, the first gradients with absolute values less than a set threshold account for a first percentage of the total number of first gradients.

[0140] In one embodiment, the first percentage is negatively correlated with the convergence degree of the model.

[0141] In one embodiment, the first percentage is determined based on one or more evaluation performances of the model on a validation set.

[0142] In practical applications, the parameter synchronization unit 601 can be implemented by a communication interface in the model parameter synchronization device.

[0143] It should be noted that: when the model parameter synchronization device provided in the above embodiment performs model parameter synchronization, only the above-mentioned division of each program module is used for illustration. In practical applications, the above-mentioned processing can be allocated to different program modules according to needs, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-mentioned processing. In addition, the model parameter synchronization device provided in the above embodiment and the model parameter synchronization method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0144] Based on the hardware implementation of the above program module, and in order to implement the method on the device side of the embodiments of the present application, the embodiments of the present application also provide a device, as Figure 7 shown, the device 700 includes:

[0145] A first communication interface 701 capable of interacting with other network nodes;

[0146] The first processor 702 is connected to the first communication interface 701 to implement information exchange with other network nodes and is used to execute the method provided by one or more technical solutions of the above device side when running the computer program. The computer program is stored in the first memory 703.

[0147] Specifically, the first communication interface 701 is used to perform communication addressing between the parameter synchronization services of each training partition in the multiple training partitions, so that the parameter synchronization service performs model parameter synchronization between the multiple training partitions.

[0148] Among them, multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and multiple training partitions are deployed in different clouds.

[0149] In one embodiment, the first entity supports a distributed communication management service, and the first processor 702 is used to:

[0150] A corresponding first identifier is allocated to the parameter synchronization service of each training partition in the plurality of training partitions, and the first identifier is used to uniquely map the communication address of the corresponding parameter synchronization service in the wide area network.

[0151] In one embodiment, a mapping relationship is established between the first identifier and the corresponding parameter synchronization service in the internal communication address of the training partition to which it belongs. Correspondingly, the first communication interface 701 is used to perform communication addressing between the parameter synchronization services of each training partition in multiple training partitions based on the first identifier corresponding to each parameter synchronization service and the corresponding mapping relationship.

[0152] It should be noted that the specific processing process of the first processor 702 and the first communication interface 701 can be understood by referring to the above method.

[0153] Of course, in actual application, the various components in the device 700 are coupled together through the bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .

[0154] The first memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the device 700. Examples of such data include: any computer program used to operate on the device 700.

[0155] The method disclosed in the embodiments of the present application above can be applied to or implemented by the first processor 702. The first processor 702 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in hardware or instructions in software form in the first processor 702. The above first processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 702 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the method disclosed in the embodiments of the present application, it can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, and this storage medium is located in the first memory 703. The first processor 702 reads the information in the first memory 703 and combines its hardware to complete the steps of the foregoing method.

[0156] In an exemplary embodiment, the device 700 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontroller units (MCUs), microprocessors, or other electronic components, and is used to execute the foregoing method.

[0157] Based on the hardware implementation of the above program module, and in order to implement the method on the device side in the embodiments of the present application, the embodiments of the present application also provide a device, as Figure 8 shown, the device 800 includes:

[0158] A second communication interface 801, capable of interacting with other network nodes for information;

[0159] A second processor 802, connected to the second communication interface 801 to achieve information interaction with other network nodes, and when used to run a computer program, execute the method provided by one or more of the above technical solutions on the device side. The computer program is stored on the second memory 803.

[0160] Specifically, the second communication interface 801 is used to perform model parameter synchronization among multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition in multiple training partitions is completed.

[0161] Among them, the multiple training partitions are obtained by splitting an artificial intelligence model training task, and the multiple training partitions are deployed in different clouds.

[0162] In an embodiment, the first entity supports the parameter synchronization service, and the first parameter synchronization service in the first training partition obtains one or more gradients, where each gradient represents the gradient of the model parameters determined each time the model parameters in the first training partition are updated;

[0163] When the set conditions are met, the first parameter synchronization service sends one or more first gradients to the second parameter synchronization service of each second training partition in one or more second training partitions. The one or more first gradients are determined based on the first information, and the first information represents all the gradients obtained by the first parameter synchronization service after the last model parameter synchronization.

[0164] In an embodiment, the set conditions include:

[0165] Meet the set synchronization frequency, which represents the frequency of model parameter synchronization. The synchronization frequency is used to adjust the computing power resource allocation strategy and function trigger strategy of each training partition in multiple training partitions.

[0166] In an embodiment, the first parameter synchronization service in the first training partition updates the model parameters of the model in the first training partition based on one or more second gradients sent by the third parameter synchronization service in the third training partition; among them,

[0167] The one or more second gradients are determined based on the second information, and the second information represents all the gradients obtained by the third parameter synchronization service after the last model parameter synchronization. Each gradient represents the gradient of the model parameters determined each time the model parameters in the third training partition are updated.

[0168] In an embodiment, the first entity supports the training management service, where the training management service is used to provide processing requests and response results in real time during the process of updating the model parameters of the model, and provide a function routing mechanism.

[0169] In an embodiment, among the one or more first gradients, the first gradients with absolute values less than the set threshold account for the first percentage of the total number of first gradients.

[0170] In one embodiment, the first percentage is negatively correlated with the convergence degree of the model.

[0171] In one embodiment, the first percentage is determined based on one or more evaluation performances of the model on the validation set.

[0172] It should be noted that: The specific processing procedures of the second processor 802 and the second communication interface 801 can be understood with reference to the above method.

[0173] Of course, in actual application, the various components in the device 800 are coupled together through the bus system 804. It can be understood that the bus system 804 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 8 all kinds of buses are labeled as the bus system 804.

[0174] The second memory 803 in the embodiments of the present application is used to store various types of data to support the operation of the device 800. Examples of these data include: any computer program for operating on the device 800.

[0175] The method disclosed in the above embodiments of the present application can be applied to the second processor 802 or implemented by the second processor 802. The second processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the second processor 802 or by instructions in software form. The above-mentioned second processor 802 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The second processor 802 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the method disclosed in the embodiments of the present application, it can be directly embodied as being executed and completed by the hardware decoding processor, or by a combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, and this storage medium is located in the second memory 803. The second processor 802 reads the information in the second memory 803 and combines its hardware to complete the steps of the foregoing method.

[0176] In an exemplary embodiment, the device 800 can be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, Microprocessors, or other electronic components for executing the foregoing method.

[0177] It can be understood that the memories (the first memory 703 and the second memory 803) in the embodiments of the present application can be volatile memories or non-volatile memories, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory), an electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), a ferromagnetic random access memory (FRAM, ferromagnetic random access memory), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM, Compact Disc Read-Only Memory); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM, Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as a static random access memory (SRAM, Static Random Access Memory), a synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory), a dynamic random access memory (DRAM, Dynamic Random Access Memory), a synchronous dynamic random access memory (SDRAM, Synchronous Dynamic Random Access Memory), a double data rate synchronous dynamic random access memory (DDR SDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), an enhanced synchronous dynamic random access memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), a synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and a direct rambus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0178] In an exemplary embodiment, the embodiments of the present application further provide a storage medium, namely a computer storage medium, specifically a computer-readable storage medium. For example, it includes a first memory 703 that stores a computer program, and the above computer program can be executed by a first processor 702 of the device 700 to complete the foregoing method steps on the device side. Another example is a second memory 803 that stores a computer program, and the above computer program can be executed by a second processor 802 of the device 800 to complete the foregoing method steps on the device side. The computer-readable storage medium can be a FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0179] Exemplarily, the embodiments of the present application further provide a computer program product, including a computer program that can be executed by a first processor 702 of the device 700 to complete the foregoing method steps on the device side. Or, the computer program can be executed by a second processor 802 of the device 800 to complete the foregoing method steps on the device side.

[0180] It should be noted that: "first", "second", etc. are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence.

[0181] In this article, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "one or more" in this article represents any combination of any one or at least two of multiple ones. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C.

[0182] In addition, the technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

[0183] The above is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A training method for an artificial intelligence model, characterized in that: Applied to a first entity, the method comprises: Perform communication addressing between parameter synchronization services of each training partition in a plurality of training partitions, so that the parameter synchronization service performs model parameter synchronization between the plurality of training partitions; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

2. The method according to claim 1, characterized in that Before performing communication addressing between parameter synchronization services of each training partition in the plurality of training partitions, the method further includes: The artificial intelligence model training task is split into the multiple training partitions, and / or the multiple training partitions are deployed in the different clouds.

3. The method according to claim 1 or 2, characterized in that: The first entity supports a distributed communication management service, and before communication addressing is performed between parameter synchronization services of each training partition in the plurality of training partitions, the method further comprises: A corresponding first identifier is allocated to the parameter synchronization service of each training partition in the multiple training partitions, where the first identifier is used to uniquely map the communication address of the corresponding parameter synchronization service in the wide area network.

4. The method according to claim 3, characterized in that The first identifier and the corresponding parameter synchronization service have a mapping relationship between the internal communication address of the training partition to which it belongs, and communication addressing is performed between the parameter synchronization services of each training partition in the multiple training partitions, including: Based on the first identifier corresponding to each parameter synchronization service and the corresponding mapping relationship, communication addressing is performed between the parameter synchronization services of each training partition in the multiple training partitions.

5. A model parameter synchronization method, characterized in that: Applied to a first entity, the method comprises: When the communication addressing of the parameter synchronization service of each training partition in the plurality of training partitions is completed, model parameter synchronization is performed between the plurality of training partitions based on the parameter synchronization service of each training partition; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

6. The method according to claim 5, characterized in that In the parameter synchronization service based on each training partition, before performing model parameter synchronization between the multiple training partitions, the method further includes: The artificial intelligence model training task is split into the multiple training partitions, and / or the multiple training partitions are deployed in the different clouds.

7. The method according to claim 5, characterized in that The first entity supports a parameter synchronization service, wherein the parameter synchronization service based on each training partition performs model parameter synchronization between the multiple training partitions, including: The first parameter synchronization service in the first training partition acquires one or more gradients, wherein each gradient represents a gradient of a model parameter determined each time the first training partition updates a model parameter of the model; When the set conditions are met, the first parameter synchronization service sends one or more first gradients to the second parameter synchronization service of each second training partition in one or more second training partitions, and the one or more first gradients are determined based on first information, and the first information represents all gradients acquired by the first parameter synchronization service after the last model parameter synchronization.

8. The method according to claim 7, characterized in that The setting conditions include: The set synchronization frequency is met, where the synchronization frequency represents the frequency of model parameter synchronization, and the synchronization frequency is used to adjust the computing resource allocation strategy and function triggering strategy of each training partition in the multiple training partitions.

9. The method according to claim 5, characterized in that The method further comprises: The first parameter synchronization service in the first training partition updates the model parameters of the model of the first training partition based on one or more second gradients sent by the third parameter synchronization service in the third training partition; wherein, The one or more second gradients are determined based on second information, where the second information represents all gradients acquired by the third parameter synchronization service after the last model parameter synchronization, wherein each gradient represents a gradient of a model parameter determined each time the third training partition updates a model parameter of the model.

10. The method according to claim 7 or 9, characterized in that: The first entity supports a training management service, wherein the training management service is used to provide processing requests and response results in real time during the process of updating model parameters of the model, and to provide a function routing mechanism.

11. The method according to claim 7, characterized in that Among the one or more first gradients, the first gradients whose absolute values ​​are smaller than a set threshold account for a first percentage of the total number of first gradients.

12. The method according to claim 11, characterized in that The first percentage is negatively correlated with a degree of convergence of the model.

13. The method according to claim 7, characterized in that The first percentage is determined based on one or more evaluation performances of the model on a validation set.

14. A training device for an artificial intelligence model, characterized in that: include: A communication addressing unit is used to perform communication addressing between parameter synchronization services of each training partition in a plurality of training partitions, so that the parameter synchronization service performs model parameter synchronization between the plurality of training partitions; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

15. A model parameter synchronization device, characterized in that: include: A parameter synchronization unit is used to perform model parameter synchronization between the multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition is completed; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

16. A device, characterized in that include: A first processor and a first communication interface; wherein, The first communication interface is used to perform communication addressing between parameter synchronization services of each training partition in a plurality of training partitions, so that the parameter synchronization service performs model parameter synchronization between the plurality of training partitions; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

17. A device, characterized in that include: A second processor and a second communication interface; wherein, The second communication interface is used to perform model parameter synchronization between the multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition is completed; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

18. A device, characterized in that include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 4, or executes the steps of the method described in any one of claims 5 to 13.

19. A first entity, characterized in that: The first entity comprises: a communication addressing unit, configured to perform communication addressing between parameter synchronization services of each training partition in a plurality of training partitions; and / or, A parameter synchronization unit is used to perform model parameter synchronization between the multiple training partitions based on the parameter synchronization service of each training partition when the communication addressing of the parameter synchronization service of each training partition is completed; wherein, The multiple training partitions are obtained by splitting the artificial intelligence model training tasks, and the multiple training partitions are deployed in different clouds.

20. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 4, or implements the steps of the method according to any one of claims 5 to 13.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 4, or implements the steps of the method according to any one of claims 5 to 13.

Images