A distributed model training optimization method, system, medium and product

Abstract

A distributed model training optimization method, system, medium and product, which relate to the field of model training. By adopting the above technical solution, the training optimization system divides the terminal devices into multiple device groups and sets relay nodes in each device group. When the preset synchronization conditions are met, it is no longer necessary to send them to the terminal devices one by one, but only to the relay nodes, thereby realizing the hierarchical transmission of model parameters. After calculating the parameter difference, the training optimization system screens the parameters to be updated, and at the same time, according to the synchronization frequency of the parameters to be updated, determines the high-frequency synchronization parameter set and the low-frequency synchronization parameter set, so as to send different parameters to be updated at different times. This differentiated parameter synchronization strategy not only ensures the timely update of important parameters, but also avoids the performance bottleneck caused by the frequent transmission of all parameters, significantly reduces the network communication overhead, and thus improves the overall efficiency of distributed training.

Description

Technical Field

[0001] The present application relates to the field of model training, and in particular to a distributed model training optimization method, system, medium and product. Background Art

[0002] As an important deep reinforcement learning algorithm, Deep Deterministic Policy Gradient (DDPG) achieves policy optimization in continuous action space through the Actor-Critic architecture. In practical applications, distributed training has become a mainstream training method to improve the training effect and generalization ability of the model.

[0003] During the training process of the DDPG model, the terminal device determines its own first device state and generates an action corresponding to the first device state through the Actor first online network. The terminal device stores the policy experience corresponding to the first device state in the experience pool. The policy experience includes the first device state, the action, the second device state after the action is executed, and the immediate reward of the action. The management device samples the experience pool, and after obtaining the policy sample, adjusts the model parameters of the Actor second online network through the Critic network according to the policy sample. When the preset synchronization conditions are met, the management device synchronizes the adjusted Actor second online network to each Actor first online network, so that each synchronized Actor first online network has the same parameters as the adjusted Actor second online network.

[0004] However, the relevant technology still has certain defects. Since the parameters of the entire Actor second online network need to be transmitted to all terminal devices every time the preset synchronization conditions are met, when the number of terminal devices is large, frequent model synchronization will generate a large amount of network communication overhead, thus causing performance bottlenecks and affecting training efficiency. Summary of the invention

[0005] The present application provides a distributed model training optimization method, system, medium and product for reducing communication overhead and improving transmission efficiency.

[0006] In a first aspect, the present application provides a distributed model training optimization method, which is applied to a training optimization system of a deep deterministic policy gradient model, the training optimization system comprising a management device and multiple terminal devices, the deep deterministic policy gradient model comprising an evaluation network and an action network, the action network comprising a first online network and a second online network, the terminal device is deployed with the first online network, and the management device is deployed with the evaluation network and the second online network, the method comprising: dividing all terminal devices into multiple device groups according to the network topology structure, and determining a relay node in each device group, the terminal device is used to perform model training, and the relay node is used to forward model parameters; calculating the parameter difference between the second online network in the management device and the first online network in each terminal device, generating a parameter change amount set, the parameter change amount set comprising multiple parameter differences; screening parameter changes Quantize the parameter differences in the set that exceed the preset difference threshold to obtain a set of parameters to be updated, which includes multiple parameters to be updated; based on the weight coefficients of the parameters to be updated and the weight coefficient-synchronization frequency correspondence table, determine the high-frequency synchronization parameter set and the low-frequency synchronization parameter set, and the weight coefficient is positively correlated with the synchronization frequency; quantize and compress the parameters to be updated in the high-frequency synchronization parameter set to obtain a first compression parameter, and quantize and compress the parameters to be updated in the low-frequency synchronization parameter set to obtain a second compression parameter, and the quantization compression is used to convert floating-point numbers into low-order fixed-point numbers; transmit the first compression parameter to the terminal device in the device group through the relay node at a first moment, and transmit the second compression parameter to the terminal device in the device group through the relay node at a second moment; when the terminal device receives the first compression parameter and / or the second compression parameter, update the parameters.

[0007] By adopting the above technical solution, the training optimization system divides the terminal devices into multiple device groups and sets relay nodes in each device group. When the preset synchronization conditions are met, it is no longer necessary to send them to the terminal devices one by one, but only to the relay nodes, thus realizing the hierarchical transmission of model parameters. After calculating the parameter difference, the training optimization system selects the parameters to be updated, and at the same time, according to the synchronization frequency of the parameters to be updated, determines the high-frequency synchronization parameter set and the low-frequency synchronization parameter set, so as to send different parameters to be updated at different times. This differentiated parameter synchronization strategy not only ensures the timely update of important parameters, but also avoids the performance bottleneck caused by the frequent transmission of all parameters, significantly reduces network communication overhead, and thus improves the overall efficiency of distributed training.

[0008] In combination with some embodiments of the first aspect, in some embodiments, all terminal devices are divided into multiple device groups according to the network topology structure, and the relay nodes in each device group are determined, specifically including: constructing a network topology graph according to the network topology structure between the terminal devices, the network topology graph including multiple nodes and edges connecting the nodes, the nodes are used to represent the terminal devices, and the edges are used to represent the network connections between the terminal devices; based on the network topology graph, calculating the degree centrality and betweenness centrality of each node, the degree centrality is used to represent the number of edges directly connected to the node, and the betweenness centrality is used to represent the number of shortest paths passing through the node; clustering the nodes according to a preset clustering algorithm to obtain multiple node clusters, the node clusters corresponding to the device groups; calculating the weighted values ​​of the degree centrality and betweenness centrality of each node in each node cluster; determining the node with the largest weighted value in each node cluster as a relay node.

[0009] By adopting the above technical solution, the training optimization system constructs a network topology diagram between terminal devices, and uses a preset clustering algorithm to form node clusters on this basis to obtain device groups, thus achieving a more scientific and reasonable device grouping. The training optimization system selects the node with the largest weighted value of degree centrality and betweenness centrality in the node cluster as the relay node, ensuring the optimal communication efficiency within each device group. This grouping method based on network topology characteristics makes the network transmission path shorter and the bandwidth more fully utilized during parameter synchronization, thereby further reducing communication overhead.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the parameter difference between the second online network in the management device and the first online network in each terminal device is calculated to generate a parameter change set, which includes multiple parameter difference values, specifically including: obtaining the current parameters of the second online network in the management device and the historical parameters of the first online network in each terminal device; calculating the difference between the current parameters and the corresponding historical parameters to obtain the parameter difference; and adding the multiple parameter differences to the parameter change set.

[0011] By adopting the above technical solution, the training optimization system compares the current parameters of the second online network in the management device with the historical parameters of the first online network in the terminal device, calculates the parameter difference and generates a set of parameter changes. This screening mechanism based on the degree of parameter change can accurately identify the parameters that have changed significantly during the model training process. Compared with the method of synchronizing all parameters uniformly, this differentiated update strategy not only ensures the timely synchronization of important parameters, but also avoids redundant transmission of parameters that have not changed much, making the entire parameter synchronization process more targeted and efficient.

[0012] In combination with some embodiments of the first aspect, in some embodiments, based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, a high-frequency synchronization parameter set and a low-frequency synchronization parameter set are determined, and the weight coefficient is positively correlated with the synchronization frequency, specifically including: based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, determining the synchronization frequency of the parameter to be updated; dividing the parameter to be updated into a high-frequency synchronization parameter set and a low-frequency synchronization parameter set according to a preset frequency threshold and the synchronization frequency of the parameter to be updated.

[0013] By adopting the above technical solution, the training optimization system determines the synchronization frequency based on the weight coefficient-synchronization frequency correspondence table and the weight coefficient of the parameter to be updated, thereby realizing adaptive adjustment of the parameter update frequency. The parameters to be updated are divided into two categories: a high-frequency synchronization parameter set and a low-frequency synchronization parameter set according to the preset frequency threshold, so that the parameters to be updated with high importance can be synchronized more frequently, while the parameters to be updated with lower importance use a lower synchronization frequency. This differentiated synchronization mechanism based on the importance of parameters not only ensures the timely update of key parameters, but also avoids excessive synchronization of secondary parameters, thereby significantly reducing communication overhead while ensuring training results.

[0014] In combination with some embodiments of the first aspect, in some embodiments, before the step of transmitting the first compression parameter to the terminal device in the device group through the relay node at the first moment and transmitting the second compression parameter to the terminal device in the device group through the relay node at the second moment, the method also includes: determining the moment after the reference moment passes through a preset high-frequency synchronization period as the first moment, and the preset high-frequency synchronization period is the synchronization period of the high-frequency synchronization parameter set; determining the moment after the reference moment passes through a preset low-frequency synchronization period as the second moment, and the preset low-frequency synchronization period is the synchronization period of the low-frequency synchronization parameter set.

[0015] By adopting the above technical solution, the training optimization system sets the reference time and calculates the first time of high-frequency synchronization and the second time of low-frequency synchronization respectively, thus realizing the timing management of parameter synchronization. This timing arrangement based on different synchronization cycles not only avoids time conflicts in the synchronization process, but also can flexibly adjust the time interval of parameter updates according to resource conditions and training requirements, thus improving the adaptability of the distributed model training optimization method.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the terminal device receives the first compression parameter and / or the second compression parameter, after performing the parameter updating step, the method further includes: obtaining a training loss value of the first online network in the terminal device; when the training loss value is greater than a preset loss threshold, calculating a gradient value of the training loss value relative to each parameter to be updated; based on the gradient value, adjusting the initial weight coefficient of the parameter to be updated to obtain a final weight coefficient; and updating the final weight coefficient to a weight coefficient-synchronization frequency correspondence table.

[0017] By adopting the above technical solution, the training optimization system monitors the training loss value and dynamically adjusts the weight coefficient of the parameter to be updated, so that the importance evaluation of the parameter to be updated can change dynamically with the training process. This closed-loop feedback design not only improves the accuracy of parameter importance evaluation, but also enables the entire training process to automatically adjust the synchronization strategy according to the actual effect, thereby achieving better training results.

[0018] In combination with some embodiments of the first aspect, in some embodiments, based on the gradient value, the initial weight coefficient of the parameter to be updated is adjusted to obtain the final weight coefficient, specifically including: taking the ratio of the absolute value of the gradient value to a preset reference value as the adjustment coefficient of the initial weight coefficient; multiplying the initial weight coefficient by the adjustment coefficient to obtain the final weight coefficient.

[0019] By adopting the above technical solution, the training optimization system calculates the ratio of the absolute value of the gradient value to the preset benchmark value as the adjustment coefficient, and multiplies it with the initial weight coefficient to obtain the final weight coefficient, thereby realizing the quantitative evaluation of the importance of parameters. This makes the importance evaluation of different parameters have a unified measurement standard, which not only retains the historical importance information of the parameters, but also can timely reflect the current influence of the parameters, thereby realizing the accurate quantification and dynamic adjustment of the importance of the parameters.

[0020] In a second aspect, an embodiment of the present application provides a training optimization system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the training optimization system to perform the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a training optimization system, enables the training optimization system to perform the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a training optimization system, the training optimization system executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] It can be understood that the training optimization system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] 1. By adopting the above technical solution, the training optimization system divides the terminal devices into multiple device groups and sets relay nodes in each device group. When the preset synchronization conditions are met, it is no longer necessary to send them to the terminal devices one by one, but only to the relay nodes, thus realizing the hierarchical transmission of model parameters. After calculating the parameter difference, the training optimization system selects the parameters to be updated, and at the same time, according to the synchronization frequency of the parameters to be updated, determines the high-frequency synchronization parameter set and the low-frequency synchronization parameter set, so as to send different parameters to be updated at different times. This differentiated parameter synchronization strategy not only ensures the timely update of important parameters, but also avoids the performance bottleneck caused by the frequent transmission of all parameters, significantly reduces network communication overhead, and thus improves the overall efficiency of distributed training.

[0026] 2. By adopting the above technical solution, the training optimization system constructs a network topology diagram between terminal devices, and uses a preset clustering algorithm to form node clusters on this basis to obtain device groups, thus achieving more scientific and reasonable device grouping. The training optimization system selects the node with the largest weighted value of degree centrality and betweenness centrality in the node cluster as the relay node, ensuring the optimal communication efficiency within each device group. This grouping method based on network topology characteristics makes the network transmission path shorter and the bandwidth more fully utilized during parameter synchronization, thereby further reducing communication overhead.

[0027] 3. By adopting the above technical solution, the training optimization system determines the synchronization frequency based on the weight coefficient-synchronization frequency correspondence table and the weight coefficient of the parameter to be updated, thereby realizing adaptive adjustment of the parameter update frequency. The parameters to be updated are divided into two categories: a high-frequency synchronization parameter set and a low-frequency synchronization parameter set according to the preset frequency threshold, so that the parameters to be updated with high importance can be synchronized more frequently, while the parameters to be updated with lower importance use a lower synchronization frequency. This differentiated synchronization mechanism based on the importance of parameters not only ensures the timely update of key parameters, but also avoids excessive synchronization of secondary parameters, thereby significantly reducing communication overhead while ensuring training results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural diagram of the distributed model training optimization method in the embodiment of the present application;

[0029] Figure 2 It is a flow chart of the distributed model training optimization method in the embodiment of the present application;

[0030] Figure 3 This is another flowchart of the distributed model training optimization method in the embodiment of the present application;

[0031] Figure 4 It is a schematic diagram of the structure of a physical device of the training optimization system in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0033] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0034] The following is a structural description of the method provided by this implementation. Figure 1 , which is a structural diagram of the distributed model training optimization method in an embodiment of the present application.

[0035] The training optimization system includes a management device and multiple terminal devices. The DDPG model includes a Critic Network and an Actor Network. The Actor Network includes a first online network and a second online network. Each terminal device is deployed with the first online network as an intelligent agent in the reinforcement learning model, and the management device is deployed with the Critic Network and the second online network.

[0036] The terminal device is used to perform the training of the DDPG model. It can generate corresponding actions based on its current state to overcome the defects of the centralized scheduling method. The management device can be a server that communicates with multiple terminal devices. For example, a Web server, an FTP server, a data processing server, etc.

[0037] Since the parameters of the entire Actor second online network need to be transmitted to all terminal devices every time the preset synchronization conditions are met, when the number of terminal devices is large, frequent model synchronization will generate a large amount of network communication overhead, thereby causing performance bottlenecks and affecting training efficiency. Therefore, this application proposes a distributed model training optimization method, that is, grouping all terminal devices to obtain multiple device groups, and setting a relay node for each device group. In this way, when the preset synchronization conditions are met, it is no longer necessary to send them to the terminal devices one by one, but only to the relay nodes, thereby realizing the hierarchical transmission of model parameters.

[0038] The following is a description of the process of the method provided by this implementation. Figure 2 , which is a flow chart of the distributed model training optimization method in an embodiment of the present application.

[0039] S201, dividing all terminal devices into multiple device groups according to the network topology structure, and determining the relay nodes in each device group, the terminal devices are used to perform model training, and the relay nodes are used to forward model parameters;

[0040] Among them, the network topology structure is used to represent the network connection relationship between terminal devices; the device group refers to a collection of terminal devices with similar network connection relationships; the relay node refers to the core node in the device group responsible for receiving and forwarding model parameters, which is used to achieve efficient distribution of parameters within the device group; the model parameters are used to represent the weights and bias values ​​of each network layer in the DDPG model.

[0041] Specifically, first, the training optimization system collects the network connection information of all terminal devices and constructs a network topology diagram that represents the network connection relationship between terminal devices. Then, based on the network topology diagram, the training optimization system calculates the degree centrality (the number of other terminal devices directly connected to the terminal device) and betweenness centrality (the number of shortest paths passing through the terminal device) of each terminal device. Next, the training optimization system uses a preset clustering algorithm (such as K-means) to cluster terminal devices with similar network characteristics to form device groups. Finally, the training optimization system selects the terminal device with the highest weighted value of degree centrality and betweenness centrality in each device group as a relay node.

[0042] Optionally, in general, all terminal devices are divided into multiple device groups according to the network topology structure, and the relay nodes in each device group are determined in the following manner, which is not limited here: construct a network topology graph according to the network topology structure between the terminal devices, and the network topology graph includes multiple nodes and edges connecting the nodes, the node is used to represent the terminal device, and the edge is used to represent the network connection between the terminal devices; based on the network topology graph, the degree centrality and betweenness centrality of each node are calculated, and the degree centrality is used to represent the number of edges directly connected to the node, and the betweenness centrality is used to represent the number of shortest paths passing through the node; the nodes are clustered according to a preset clustering algorithm to obtain multiple node clusters, and the node clusters correspond to the device groups; the weighted values ​​of the degree centrality and betweenness centrality of each node in each node cluster are calculated; and the node with the largest weighted value in each node cluster is determined as a relay node.

[0043] The following is an example to illustrate. Suppose there are 8 terminal devices participating in the DDPG model training, namely D1 to D8.

[0044] The network topology is:

[0045] D1 is connected to D2 and D3;

[0046] D2 is connected to D1, D3, and D4;

[0047] D3 is connected to D1, D2, D4, and D5;

[0048] D4 is connected to D2, D3, and D5;

[0049] D5 is connected to D3, D4, D6, and D7;

[0050] D6 is connected to D5, D7, and D8;

[0051] D7 is connected to D5, D6, and D8;

[0052] D8 is connected to D6 and D7;

[0053] Calculate degree centrality (the number of other terminal devices directly connected to this terminal device):

[0054] D3 is the highest (4 connections);

[0055] D5 is next (4 connections, because the sequence number of D3 is before that of D5, so D3 is the highest, and D5 is the second);

[0056] D2, D4, D6, and D7 each have 3 connections;

[0057] D1 and D8 each have 2 connections;

[0058] Calculate betweenness centrality (the number of shortest paths through this terminal):

[0059] D3 and D5 are the highest (because they are located in the center of the network);

[0060] D2, D4, D6, and D7 are next;

[0061] D1 and D8 are the lowest (because they are at the edge of the network);

[0062] Device grouping (based on network features, the K-means algorithm is used to divide terminal devices into two groups):

[0063] The first device group: D1, D2, D3, D4 (left side of the network);

[0064] The second device group: D5, D6, D7, D8 (right side of the network);

[0065] Select a relay node:

[0066] The first device group selects D3 as the relay node because D3 has the highest degree centrality (4 connections), the highest betweenness centrality (located on the critical path), and can communicate efficiently with other terminal devices in the group;

[0067] The second device group selects D5 as the relay node because D5 has a higher degree centrality (4 connections) and a higher betweenness centrality, which can connect the two groups;

[0068] Parameter passing process:

[0069] D3 is responsible for collecting and distributing the model parameters of the first device group (D1, D2, D4);

[0070] D5 is responsible for collecting and distributing the model parameters of the second group of devices (D6, D7, D8).

[0071] This grouping and relay node selection can reduce network communication overhead, improve parameter synchronization efficiency, balance network load, and improve overall training efficiency.

[0072] S202, calculating the parameter difference between the second online network in the management device and the first online network in each terminal device, and generating a parameter change amount set, the parameter change amount set including a plurality of parameter difference values;

[0073] The parameter difference refers to the numerical difference at the corresponding parameter position; the parameter change amount set is used to store all the changed parameter differences.

[0074] Specifically, during the model training process, when the management device completes a round of parameter optimization, it needs to calculate the update amount to determine which parameters need to be synchronized. First, the training optimization system obtains the latest parameter value (current parameter) of the second online network in the management device, and simultaneously obtains the old parameter value (historical parameter) of the first online network in each terminal device. Then, for each parameter position, the training optimization system calculates the difference between the value of the current parameter and the value of the corresponding historical parameter. Finally, the training optimization system adds all calculated parameter differences to the parameter change amount set to prepare for subsequent parameter screening.

[0075] Optionally, in general, the parameter difference between the second online network in the management device and the first online network in each terminal device is calculated to generate a parameter change set, and the parameter change set includes multiple parameter differences. This can be achieved in the following way, which is not limited here: obtaining the current parameters of the second online network in the management device and the historical parameters of the first online network in each terminal device; calculating the difference between the current parameters and the corresponding historical parameters to obtain the parameter difference; and adding the multiple parameter differences to the parameter change set.

[0076] Assume that in DDPG model training, the network structure is simplified as follows:

[0077] First layer: 4 parameters;

[0078] Second layer: 3 parameters;

[0079] Output layer: 2 parameters;

[0080] Manage the current parameters of the second online network in the device:

[0081] First layer: [0.5, 0.3, 0.8, 0.2];

[0082] Second layer: [0.6, 0.4, 0.7];

[0083] Output layer: [0.3, 0.5];

[0084] Historical parameters of the first online network in the terminal device:

[0085] Terminal device 1:

[0086] First layer: [0.4, 0.3, 0.7, 0.2];

[0087] Second layer: [0.5, 0.4, 0.6];

[0088] Output layer: [0.2, 0.4];

[0089] Terminal device 2:

[0090] First layer: [0.3, 0.2, 0.8, 0.1];

[0091] Second layer: [0.4, 0.3, 0.7];

[0092] Output layer: [0.3, 0.4];

[0093] Calculate the parameter difference:

[0094] For terminal device 1:

[0095] First layer difference: [0.1, 0.0, 0.1, 0.0];

[0096] Second layer difference: [0.1, 0.0, 0.1];

[0097] Output layer difference: [0.1, 0.1];

[0098] For terminal device 2:

[0099] First layer difference: [0.2, 0.1, 0.0, 0.1];

[0100] Second layer difference: [0.2, 0.1, 0.0];

[0101] Output layer difference: [0.0, 0.1];

[0102] Collect all non-zero differences into a parameter variation set to generate a parameter variation set:

[0103] Parameter change of terminal device 1:

[0104] "layer1_parameter1": 0.1,

[0105] "layer1_parameter3": 0.1,

[0106] "layer2_parameter1": 0.1,

[0107] "layer2_parameter3": 0.1,

[0108] 'layer3_parameter1': 0.1,

[0109] "layer3_parameter2":0.1};

[0110] Parameter change of terminal device 2:

[0111] "layer1_parameter1": 0.2,

[0112] "layer1_parameter2": 0.1,

[0113] "layer1_parameter4": 0.1,

[0114] "layer2_parameter1": 0.2,

[0115] "layer2_parameter2": 0.1,

[0116] "layer3_parameter2":0.1}.

[0117] S203, screening the parameter difference in the parameter variation set that exceeds a preset difference threshold, and obtaining a set of parameters to be updated, wherein the set of parameters to be updated includes a plurality of parameters to be updated;

[0118] Among them, the preset difference threshold refers to the numerical standard for determining whether the parameter needs to be updated; the parameter to be updated refers to the parameter whose difference exceeds the preset difference threshold and needs to be synchronized; the parameter set to be updated is used to store all parameters to be updated that need to be synchronized and updated.

[0119] Specifically, in order to avoid unnecessary synchronization of parameters with small changes, the training optimization system needs to screen the parameter changes. For each parameter difference in the parameter change set, it is compared with the preset difference threshold. If the absolute value of a parameter difference exceeds the preset difference threshold, it means that the parameter has changed significantly and needs to be synchronized and updated. The training optimization system will collect these parameters with significant changes into the parameter set to be updated.

[0120] Continuing with the parameter change amount set of step S202, it is assumed that the preset difference threshold is set to 0.15.

[0121] Parameter screening of terminal device 1:

[0122] The original parameter variation set:

[0123] "layer1_parameter1": 0.1,

[0124] "layer1_parameter3": 0.1,

[0125] "layer2_parameter1": 0.1,

[0126] "layer2_parameter3": 0.1,

[0127] "layer3_parameter1": 0.1,

[0128] "layer3_parameter2":0.1};

[0129] Screening results: All parameter differences are less than the threshold of 0.15, no update is required, and the parameter set to be updated is empty;

[0130] Parameter screening of terminal device 2:

[0131] The original parameter variation set:

[0132] "layer1_parameter1": 0.2,

[0133] "layer1_parameter2": 0.1,

[0134] "layer1_parameter4": 0.1,

[0135] "layer2_parameter1": 0.2,

[0136] "layer2_parameter2": 0.1,

[0137] "layer3_parameter2":0.1}.

[0138] Filter results:

[0139] Parameter set to be updated: {"layer 1_parameter 1" "0.2, "layer 2_parameter 1": 0.2}.

[0140] S204, based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, determine a high-frequency synchronization parameter set and a low-frequency synchronization parameter set, where the weight coefficient is positively correlated with the synchronization frequency;

[0141] Among them, the weight coefficient refers to the importance of the parameter to be updated in the model; the weight coefficient-synchronization frequency correspondence table refers to the mapping relationship between the weight coefficient of the recorded parameter and the synchronization frequency; the synchronization frequency is used to indicate the time interval for parameter update; the high-frequency synchronization parameter set refers to the important parameter set that requires a higher update frequency; the low-frequency synchronization parameter set refers to the secondary parameter set that can be updated at a lower frequency; the positive correlation means that the larger the weight coefficient, the higher the corresponding synchronization frequency.

[0142] Specifically, first, the training optimization system queries the weight coefficient-synchronization frequency correspondence table to obtain the synchronization frequency corresponding to the weight coefficient of each parameter to be updated. Then, the training optimization system uses the preset frequency threshold as the dividing point, and divides the parameters to be updated whose synchronization frequency is higher than the preset frequency threshold into the high-frequency synchronization parameter set, and divides the parameters to be updated whose synchronization frequency is lower than the preset frequency threshold into the low-frequency synchronization parameter set. This differentiated synchronization strategy based on importance ensures the timely update of key parameters and avoids excessive synchronization of secondary parameters.

[0143] Optionally, under normal circumstances, based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, a high-frequency synchronization parameter set and a low-frequency synchronization parameter set are determined, and the positive correlation between the weight coefficient and the synchronization frequency can be achieved in the following ways, which are not limited here: based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, the synchronization frequency of the parameter to be updated is determined; according to the preset frequency threshold and the synchronization frequency of the parameter to be updated, the parameter to be updated is divided into a high-frequency synchronization parameter set and a low-frequency synchronization parameter set.

[0144] S205, quantizing and compressing the parameters to be updated of the high-frequency synchronization parameter set to obtain a first compressed parameter, and quantizing and compressing the parameters to be updated of the low-frequency synchronization parameter set to obtain a second compressed parameter, wherein the quantization and compression is used to convert a floating-point number into a low-bit fixed-point number;

[0145] Among them, quantization compression refers to the process of reducing the numerical precision of parameters; floating-point numbers usually use 32-bit or 64-bit floating-point representation; low-order fixed-point numbers are represented by fixed-point numbers with fewer bits; the first compression parameter refers to the result of compressing the parameters to be updated of the high-frequency synchronization parameter set; the second compression parameter refers to the result of compressing the parameters to be updated of the low-frequency synchronization parameter set.

[0146] Specifically, in order to further reduce the communication bandwidth occupation, the training optimization system needs to compress the parameters to be updated. For the parameters to be updated in the high-frequency synchronization parameter set, the training optimization system uses a relatively high-precision quantization scheme (such as 8-bit fixed-point numbers) for compression to obtain the first compressed parameters; for the parameters to be updated in the low-frequency synchronization parameter set, a lower-precision quantization scheme (such as 4-bit fixed-point numbers) can be used for compression to obtain the second compressed parameters. The compression process maps the original floating-point number to the fixed-point number range of the corresponding precision, significantly reducing the amount of transmitted data while ensuring the necessary accuracy.

[0147] S206, transmitting the first compression parameter to the terminal device in the device group through the relay node at the first moment, and transmitting the second compression parameter to the terminal device in the device group through the relay node at the second moment;

[0148] Among them, the first moment refers to the update time point of the parameters to be updated in the high-frequency synchronization parameter set; the second moment refers to the update time point of the parameters to be updated in the low-frequency synchronization parameter set; the transmission process refers to the process of sending the compressed parameters from the management device to the terminal device via the relay node; the terminal device in the device group refers to all devices in the same group that need to receive parameter updates.

[0149] Specifically, the training optimization system sends the first compression parameter to the relay node of each device group at the first moment (usually after a short fixed time interval), and the relay node immediately forwards the first compression parameter to other terminal devices in the group after receiving it. The training optimization system sends the second compression parameter to the relay node of each device group in the same way at the second moment (usually after a longer fixed time interval), and the relay node immediately forwards the second compression parameter to other terminal devices in the group after receiving it. This time-sharing transmission strategy avoids network congestion and improves the efficiency of parameter synchronization.

[0150] S207: After receiving the first compression parameter and / or the second compression parameter, the terminal device updates the parameters.

[0151] Among them, reception refers to the process of the terminal device obtaining compression parameters from the relay node; parameter update refers to replacing the old parameters at the corresponding position in the local model of the terminal device with the received new parameters; and / or indicates that only one of the compression parameters may be received, or both may be received.

[0152] Specifically, when the terminal device receives the compressed parameters forwarded from the relay node, it first needs to perform a decompression operation to convert the low-order fixed-point number back to a floating-point number, and then update the decompressed new parameter value to the corresponding position in the first online network according to the position of the parameter to be updated. If the terminal device receives the first compressed parameter and the second compressed parameter at the same time, they are updated separately. After the update is completed, the terminal device can use the updated model parameters to continue the training task.

[0153] By adopting the above technical solution, the training optimization system divides the terminal devices into multiple device groups and sets relay nodes in each device group. When the preset synchronization conditions are met, it is no longer necessary to send them to the terminal devices one by one, but only to the relay nodes, thus realizing the hierarchical transmission of model parameters. After calculating the parameter difference, the training optimization system selects the parameters to be updated, and at the same time, according to the synchronization frequency of the parameters to be updated, determines the high-frequency synchronization parameter set and the low-frequency synchronization parameter set, so as to send different parameters to be updated at different times. This differentiated parameter synchronization strategy not only ensures the timely update of important parameters, but also avoids the performance bottleneck caused by the frequent transmission of all parameters, significantly reduces network communication overhead, and thus improves the overall efficiency of distributed training.

[0154] The following is a more detailed description of the process of the method provided by this implementation. Figure 3 , is another flow chart of the distributed model training optimization method in an embodiment of the present application.

[0155] After step S205, the following steps may or may not be performed, which is not limited here:

[0156] S301, determining a moment after a preset high-frequency synchronization period of a reference moment as a first moment, where the preset high-frequency synchronization period is a synchronization period of the high-frequency synchronization parameter set;

[0157] Among them, the reference time refers to the initial time point when parameter synchronization starts; the preset high-frequency synchronization period refers to the fixed time interval for updating the parameters to be updated in the high-frequency synchronization parameter set, which is usually short; the first time represents the specific time point when the parameters to be updated in the high-frequency synchronization parameter set are updated.

[0158] Specifically, first, the training optimization system sets a reference time as the starting point of the synchronization time, and then accumulates the preset high-frequency synchronization period (for example, every 100 milliseconds) to obtain the first time as the synchronization time point of the parameters to be updated in the high-frequency synchronization parameter set. This setting ensures that important parameters can be updated in a timely manner at a short fixed time interval, maintaining the real-time and accuracy of model training.

[0159] S302: Determine a moment after a preset low-frequency synchronization period of the reference moment as a second moment, where the preset low-frequency synchronization period is a synchronization period of the low-frequency synchronization parameter set;

[0160] The preset low-frequency synchronization period refers to a fixed time interval for updating the parameters to be updated in the low-frequency synchronization parameter set, which is usually short; the second moment indicates a specific time point for updating the parameters to be updated in the low-frequency synchronization parameter set.

[0161] Specifically, the training optimization system uses the same reference time as the starting point of the synchronization time, and then accumulates the preset low-frequency synchronization period (for example, every 500 milliseconds) to obtain the second time as the synchronization time point of the parameters to be updated in the low-frequency synchronization parameter set. This setting allows the secondary parameters to be updated at a longer time interval, which reduces the communication overhead while ensuring the model performance and improves the overall efficiency.

[0162] S303, transmitting the first compression parameter to the terminal device in the device group through the relay node at a first moment, and transmitting the second compression parameter to the terminal device in the device group through the relay node at a second moment;

[0163] For details, please refer to step S206, which will not be described in detail here.

[0164] S304: After receiving the first compression parameter and / or the second compression parameter, the terminal device updates the parameters;

[0165] For details, please refer to step S207, which will not be described in detail here.

[0166] S305, obtaining a training loss value of a first online network in a terminal device;

[0167] The training loss value refers to the performance evaluation indicator in the model training process, which is used to indicate the degree of deviation between the prediction result and the target value.

[0168] Specifically, the training optimization system collects the real-time training data of the first online network through the monitoring module of the terminal device and calculates the training loss value of the current batch. This training loss value reflects the training effect of the model under the current parameter configuration and provides a basis for subsequent parameter adjustments. For example, if the mean square error (MES) is used as the loss function, the mean of the sum of the squares of the difference between the predicted value and the true value is calculated.

[0169] S306. When the training loss value is greater than the preset loss threshold, calculate the gradient value of the training loss value relative to each parameter to be updated;

[0170] Among them, the preset loss threshold refers to the standard value for judging whether the training effect needs to be optimized and adjusted; the training loss value refers to the actual loss value measured in the current training process; the gradient value refers to the partial derivative of the training loss value with respect to the parameter to be updated, which is used to indicate the sensitivity of the training loss value to changes in the parameters.

[0171] Specifically, the training optimization system compares the obtained training loss value with the preset loss threshold. If the training loss value exceeds the preset loss threshold, it means that the current parameter configuration may not be optimized enough. At this time, the training optimization system will use the back propagation algorithm to calculate the gradient value of the training loss value for each parameter to be updated to quantify the influence of each parameter to be updated on the training effect. For example, if the gradient value of a parameter to be updated is large, it means that the parameter to be updated has a greater influence on the training effect.

[0172] S307, taking the ratio of the absolute value of the gradient value to the preset reference value as an adjustment coefficient of the initial weight coefficient;

[0173] Among them, the absolute value is used to indicate the size of the gradient value while ignoring the direction; the preset benchmark value refers to the reference value used to normalize the gradient value; the initial weight coefficient refers to the original importance weight of the parameter to be updated; the adjustment coefficient refers to the proportional factor used to correct the initial weight coefficient; the ratio refers to the result of the division operation of two values.

[0174] Specifically, first, the training optimization system calculates the absolute value of each gradient value to eliminate the influence of the positive and negative directions, and then the training optimization system divides the absolute value by the preset reference value (usually an appropriate constant such as 1.0) to obtain a normalized ratio. This ratio reflects the relative influence of the parameter to be updated on the training effect, and the training optimization system uses it as an adjustment coefficient to correct the initial weight coefficient of the parameter to be updated. For example, if the absolute value of the gradient value of a parameter to be updated is twice the preset reference value, then its adjustment coefficient is 2, indicating that the importance of the parameter to be updated needs to be increased accordingly.

[0175] S308, multiplying the initial weight coefficient by the adjustment coefficient to obtain a final weight coefficient;

[0176] Among them, multiplication refers to the multiplication operation of two values; the final weight coefficient refers to the importance weight of the parameter to be updated after adjustment.

[0177] Specifically, the training optimization system multiplies the original initial weight coefficient of the parameter to be updated with the newly calculated adjustment coefficient to obtain the final weight coefficient that reflects the new importance of the parameter to be updated. This adjustment mechanism combines the historical importance of the parameter to be updated (initial weight coefficient) and the feedback of the current training effect (adjustment coefficient), making the importance assessment of the parameter to be updated more accurate. For example, if the initial weight coefficient of a parameter to be updated is 0.5 and the adjustment coefficient is 2.0, its final weight coefficient will be updated to 1.0, indicating that the importance of the parameter to be updated has increased.

[0178] S309: Update the final weight coefficient to the weight coefficient-synchronization frequency correspondence table.

[0179] Specifically, the training optimization system writes the final weight coefficient of each parameter to be updated into the weight coefficient-synchronization frequency correspondence table, overwriting the original weight coefficient record. This weight coefficient-synchronization frequency correspondence table maintains the mapping relationship between the importance of the parameter and its synchronization frequency. The training optimization system will automatically adjust the synchronization frequency of the parameter according to the weight coefficient after the parameter is updated. For example, if the final weight coefficient of a parameter is significantly improved, the training optimization system will increase the synchronization frequency of the parameter accordingly to ensure that more important parameters can be updated more frequently. This dynamic adjustment mechanism enables the training optimization system to adaptively optimize the parameter synchronization strategy, improving training efficiency while ensuring training results.

[0180] The following describes the training optimization system in the embodiment of the present invention from the perspective of hardware processing. Figure 4 , which is a schematic diagram of a physical device structure of a training optimization system in an embodiment of the present application.

[0181] It should be noted that Figure 4 The structure of the training optimization system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0182] like Figure 4 As shown, the training optimization system includes a CPU 401, which can perform various appropriate actions and processes according to the program stored in the ROM 402 or the program loaded from the storage part 408 into the RAM 403, such as executing the method described in the above embodiment. In the RAM 403, various programs and data required for system operation are also stored. The CPU 401, the ROM 402 and the RAM 403 are connected to each other via a bus 404. An I / O interface 405 is also connected to the bus 404.

[0183] The following components are connected to the I / O interface 405: an input section 406 including an audio input device, a button switch, etc.; an output section 407 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0184] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the CPU 401, various functions defined in the present invention are executed.

[0185] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0187] Specifically, the training optimization system of this embodiment includes a processor and a memory. A computer program is stored in the memory. When the computer program is executed by the processor, the distributed model training optimization method provided in the above embodiment is implemented.

[0188] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the training optimization system described in the above embodiment; or may exist independently without being assembled into the training optimization system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the training optimization system, the training optimization system implements the distributed model training optimization method provided in the above embodiment.

[0189] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0190] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0191] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A distributed model training optimization method, characterized in that: A training optimization system applied to a deep deterministic policy gradient model, the training optimization system comprising a management device and a plurality of terminal devices, the deep deterministic policy gradient model comprising a judgment network and an action network, the action network comprising a first online network and a second online network, the terminal device is deployed with the first online network, the management device is deployed with the judgment network and the second online network, and the method comprises: Dividing all the terminal devices into a plurality of device groups according to the network topology structure, and determining a relay node in each of the device groups, wherein the terminal devices are used to perform model training, and the relay nodes are used to forward model parameters; Calculating a parameter difference between the second online network in the management device and the first online network in each of the terminal devices to generate a parameter change amount set, wherein the parameter change amount set includes a plurality of parameter difference values; Filtering the parameter difference values ​​exceeding a preset difference threshold value in the parameter variation set to obtain a parameter set to be updated, wherein the parameter set to be updated includes a plurality of parameters to be updated; Based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, a high-frequency synchronization parameter set and a low-frequency synchronization parameter set are determined, and the weight coefficient is positively correlated with the synchronization frequency; Quantize and compress the parameters to be updated of the high-frequency synchronization parameter set to obtain a first compressed parameter, and quantize and compress the parameters to be updated of the low-frequency synchronization parameter set to obtain a second compressed parameter, wherein the quantization and compression is used to convert a floating-point number into a low-bit fixed-point number; Transmitting the first compression parameter to the terminal device in the device group through the relay node at a first moment, and transmitting the second compression parameter to the terminal device in the device group through the relay node at a second moment; After the terminal device receives the first compression parameter and / or the second compression parameter, it updates the parameters.

2. The method according to claim 1, characterized in that The dividing all the terminal devices into a plurality of device groups according to the network topology structure and determining the relay nodes in each of the device groups specifically includes: According to the network topology structure between the terminal devices, a network topology graph is constructed, wherein the network topology graph includes a plurality of nodes and edges connecting the nodes, wherein the nodes are used to represent the terminal devices, and the edges are used to represent the network connections between the terminal devices; Based on the network topology graph, calculating the degree centrality and betweenness centrality of each of the nodes, wherein the degree centrality is used to indicate the number of edges directly connected to the node, and the betweenness centrality is used to indicate the number of shortest paths passing through the node; Clustering the nodes according to a preset clustering algorithm to obtain a plurality of node clusters, wherein the node clusters correspond to the device groups; Calculating weighted values ​​of the degree centrality and the betweenness centrality of each node in each of the node clusters; In each of the node clusters, the node with the largest weighted value is determined as the relay node.

3. The method according to claim 1, characterized in that The calculating of the parameter difference between the second online network in the management device and the first online network in each of the terminal devices to generate a parameter change amount set, wherein the parameter change amount set includes a plurality of parameter differences, specifically includes: Acquire current parameters of the second online network in the management device and historical parameters of the first online network in each of the terminal devices; Calculating the difference between the current parameter and the corresponding historical parameter to obtain the parameter difference; Add a plurality of the parameter differences to the parameter variation set.

4. The method according to claim 1, characterized in that: The step of determining a high-frequency synchronization parameter set and a low-frequency synchronization parameter set based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table, wherein the weight coefficient is positively correlated with the synchronization frequency, specifically includes: Determining the synchronization frequency of the parameter to be updated based on the weight coefficient of the parameter to be updated and the weight coefficient-synchronization frequency correspondence table; According to a preset frequency threshold and the synchronization frequency of the parameters to be updated, the parameters to be updated are divided into the high-frequency synchronization parameter set and the low-frequency synchronization parameter set.

5. The method according to claim 1, characterized in that Before the step of transmitting the first compression parameter to the terminal device in the device group through the relay node at a first moment, and transmitting the second compression parameter to the terminal device in the device group through the relay node at a second moment, the method further includes: Determine a moment after a preset high-frequency synchronization period of the reference moment as a first moment, wherein the preset high-frequency synchronization period is a synchronization period of the high-frequency synchronization parameter set; A moment after a preset low-frequency synchronization period of the reference moment is determined as a second moment, where the preset low-frequency synchronization period is a synchronization period of the low-frequency synchronization parameter set.

6. The method according to claim 1, characterized in that After the step of updating the parameters after the terminal device receives the first compression parameter and / or the second compression parameter, the method further includes: Obtaining a training loss value of a first online network in the terminal device; When the training loss value is greater than a preset loss threshold, calculating the gradient value of the training loss value relative to each of the parameters to be updated; Based on the gradient value, adjusting the initial weight coefficient of the parameter to be updated to obtain a final weight coefficient; The final weight coefficient is updated to the weight coefficient-synchronization frequency correspondence table.

7. The method according to claim 6, characterized in that The step of adjusting the initial weight coefficient of the parameter to be updated based on the gradient value to obtain the final weight coefficient specifically includes: The ratio of the absolute value of the gradient value to the preset reference value is used as the adjustment coefficient of the initial weight coefficient; The initial weight coefficient is multiplied by the adjustment coefficient to obtain the final weight coefficient.

8. A training optimization system, characterized in that: The training optimization system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the training optimization system to execute the method described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a training optimization system, the training optimization system is caused to execute the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a training optimization system, the training optimization system is caused to perform the method according to any one of claims 1 to 7.

Images