Model processing method and device, equipment, medium and product

Abstract

The invention provides a model processing method and apparatus, a device, a medium and a product. The method comprises the steps of obtaining a first model parameter of a first model of a reference edge device; determining a first encoding model of the first model by using a first encoding module of the auto-encoder based on the first model parameter; determining a first decoding model of the first encoding model by using a first decoding module of the auto-encoder based on the first encoding model; training the auto-encoder based on the first model and the first decoding model to obtain a first pre-trained auto-encoder; and determining a first global model of the distributed system by using the first pre-training auto-encoder. According to the method, the local model is utilized to train the auto-encoder for model compression transmission, so that dynamic compression of the local model can be realized when the auto-encoder is utilized to compress the local model, and therefore, when the encoder is utilized to compress and transmit the local model to determine the global model of the distributed system, the dynamic compression of the local model can be realized. The accuracy of the global model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a model processing method, device, equipment, medium and product. Background Art

[0002] Distributed learning is a machine learning method that involves distributing learning tasks across multiple physically or logically separated computing units. This method can handle large data sets, improve computing efficiency, and enhance the scalability and fault tolerance of the system. In the distributed learning stage, multiple computing nodes can be used to perform machine learning tasks, and the final learning results are sent to a central server for aggregation. However, the existing distributed learning has a large amount of data transmission. In each round of iteration, the edge node needs to upload its locally trained model update to the central server to participate in the determination of the global model. These locally trained model updates often have many parameters, resulting in a huge amount of data transmission.

[0003] In order to reduce the amount of data transmission in the prior art, compression technologies such as sparsity and quantization are usually used to reduce the amount of information in the model update to reduce the transmission cost. However, the compression methods of these technologies for local models are usually fixed, which makes it impossible to dynamically adjust the above compression methods when the local model changes, resulting in possible errors in the determined new global model. At the same time, the compression of the local model is not extreme, resulting in a certain waste of resources. Summary of the invention

[0004] The present disclosure provides a model processing method, device, equipment, medium and product to solve the problems in the related technology, realize dynamic compression of local models, improve the accuracy when determining the global model, and reduce resource waste.

[0005] The first aspect embodiment of the present disclosure proposes a model processing method, which is applied to a central server of a distributed system, including: obtaining first model parameters of a first model of a reference edge device, where the first model is a local uncompressed model of the reference edge device; based on the first model parameters, using a first encoding module of an autoencoder, determining a first encoding model of the first model; based on the first encoding model, using a first decoding module of the autoencoder, determining a first decoding model of the first encoding model; based on the first model and the first decoding model, training the autoencoder to obtain a first pre-trained autoencoder; using the first pre-trained autoencoder, determining a first global model of the distributed system.

[0006] In some embodiments of the present disclosure, determining the first decoding model of the first coding model using the first decoding module of the autoencoder includes: decoding the first coding model using the first decoding module to obtain the first decoding model; or, decoding the first coding model using the first decoding module with reference to the first model parameters to obtain the first decoding model.

[0007] In some embodiments of the present disclosure, training an autoencoder based on a first model and a first decoding model to obtain a first pre-trained autoencoder includes: iteratively training the autoencoder based on a second model parameter of the first encoding model and a third model parameter of the first decoding model; when the number of iterative training reaches a number threshold and / or the residual value between the third model parameter described by the objective function obtained based on the iterative training and the second model parameter reaches a minimum threshold, determining the trained autoencoder as a second pre-trained autoencoder; determining a model compression rate based on the second pre-trained autoencoder and the first model parameters; and adjusting the second pre-trained autoencoder based on the model compression rate to obtain the first pre-trained model.

[0008] In some embodiments of the present disclosure, iterative training of the autoencoder based on the second model parameters of the first encoder and the third model parameters of the first decoder includes: calculating the residual value between the second model parameters and the third model parameters using the objective function; and updating the parameters of the autoencoder based on the residual value using the back propagation algorithm to achieve iterative training of the autoencoder.

[0009] In some embodiments of the present disclosure, using a first pre-trained autoencoder, determining a first global model of a distributed system includes: sending a second encoding module of the first pre-trained autoencoder to a first edge device; receiving a second encoding model sent by the edge device, the second encoding model being obtained by the first edge device encoding a local model of the first edge device using the second encoding module; decoding the second encoding model using the second decoding module of the first pre-trained model to obtain a second decoding model; and determining the first global model based on the second decoding model.

[0010] In some embodiments of the present disclosure, the method also includes: sending the first global model uncompressed to the first edge device; or, compressing the first global model using an encoding module, and sending a second decoding module and the compressed first global model to the edge device, so that the edge device obtains the first global model.

[0011] The second aspect embodiment of the present disclosure proposes a model processing device, which includes: an acquisition unit, used to acquire first model parameters of a first model of a reference edge device, where the first model is a local uncompressed model of the reference edge device; a first determination unit, used to determine a first encoding model of the first model based on the first model parameters using a first encoding module of the autoencoder; a second determination unit, used to determine a first decoding model of the first encoding model based on the first encoding model using a first decoding module of the autoencoder; a training unit, used to train the autoencoder based on the first model and the first decoding model to obtain a first pre-trained autoencoder; and a third determination unit, used to determine a first global model of the distributed system using the first pre-trained autoencoder.

[0012] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, executes the method described in the first aspect embodiment of the present disclosure.

[0013] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0014] The fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect embodiment of the present disclosure when executed by a processor.

[0015] In summary, according to a model processing method proposed in the present disclosure, the method includes: obtaining first model parameters of a first model of a reference edge device, the first model being a local uncompressed model of the reference edge device; based on the first model parameters, using a first encoding module of an autoencoder, determining a first encoding model of the first model; based on the first encoding model, using a first decoding module of the autoencoder, determining a first decoding model of the first encoding model; based on the first model and the first decoding model, training the autoencoder to obtain a first pre-trained autoencoder; using the first pre-trained autoencoder to determine a first global model of the distributed system. The method of the present disclosure trains an autoencoder for model compression transmission using a local model, so that when the local model is compressed using the autoencoder, dynamic compression of the local model can be achieved, thereby improving the accuracy of the global model when the local model is compressed and transmitted using the encoder to determine the global model of the distributed system.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0018] Figure 1 An example diagram of the working principle of an autoencoder provided in an embodiment of the present disclosure;

[0019] Figure 2 A flow chart of a model processing method provided by an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of another model processing method provided by an embodiment of the present disclosure;

[0021] Figure 4 A flowchart of an autoencoder training method provided in an embodiment of the present disclosure;

[0022] Figure 5 A flowchart of another autoencoder training method provided in an embodiment of the present disclosure;

[0023] Figure 6 An example diagram of distributed system compression transmission provided by an embodiment of the present disclosure;

[0024] Figure 7 Another example diagram of distributed system compression transmission provided by an embodiment of the present disclosure;

[0025] Figure 8 A schematic diagram of the structure of a model processing device provided by an embodiment of the present disclosure;

[0026] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0028] The following is a brief introduction to the background technology involved in this disclosure:

[0029] 1. Distributed learning is a machine learning method that involves distributing learning tasks to multiple physically or logically separated computing units. This method can handle large data sets, improve computing efficiency, and enhance the scalability and fault tolerance of the system. In the distributed learning stage, multiple computing nodes can be used to perform machine learning tasks, and the final learning results can be sent to the central server for aggregation. However, the existing distributed learning has a large amount of data transmission. In each round of iteration, the edge node needs to upload its locally trained model update to the central server to participate in the determination of the global model, and these locally trained model updates often have many parameters, resulting in a huge amount of data transmission. For example, a model using the classic convolutional neural network architecture VGGNet architecture has about 138 million parameters. Even under ideal network conditions, it takes about one minute to transmit such a model using a 5G network. This large-scale data transmission not only consumes a lot of communication resources, but also increases the energy consumption and latency of the system.

[0030] A typical distributed learning is Federated Learning (FL). FL allows multiple devices or servers to collaboratively train models without centralizing data in a single location, thereby reducing the risk of data leakage. Federated learning involves large-scale data transmission, which not only consumes a lot of communication resources, but also increases the system's energy consumption and latency.

[0031] In order to reduce the amount of data transmission in the prior art, compression technologies such as sparsity and quantization are usually used to reduce the amount of information in the model update to reduce the transmission cost. However, the compression methods of these technologies for local models are usually fixed, which makes it impossible to dynamically adjust the above compression methods when the local model changes, resulting in possible errors in the determined new global model. At the same time, the compression of the local model is not extreme, resulting in a certain waste of resources.

[0032] 2. As a deep generative model, the autoencoder achieves data compression and reconstruction by learning the potential representation of the data. The autoencoder consists of two parts: the encoder and the decoder. The encoder maps the input data to the potential representation in the latent space, while the decoder restores the potential representation to the original data. Its structure is as follows: Figure 1 shown.

[0033] like Figure 1 The autoencoder shown in the figure, as a neural network, has the same input and output. It uses the idea of ​​sparse coding to reconstruct itself by recombining some high-order features. The autoencoder first compresses the vector in the high-dimensional space into a low-dimensional vector (i.e., latent vector) through the encoding layer, and then decompresses the low-dimensional vector through the decoding layer to reconstruct the original sample.

[0034] Because the data features of different user devices in the distributed learning system are redundant. Taking federated learning as an example, feature redundancy is divided into two cases: horizontal federated learning HFL and vertical federated learning VFL. For example, for horizontal federated learning HFL, if several institutions with similar businesses cooperate, the features that need to be learned and predicted are similar, but the users and samples are different. For example, different banks in a region have their own customers, but the data features that need to be analyzed are deposit amounts, loan amounts, etc., then the data between them are independent and identically distributed, and there will be redundancy between the training features; for example, for vertical federated learning VFL, some data features in different nodes may be the same, and both Party A and Party B hold part of the model, and there is also redundancy in the training features. Based on these prerequisites, it is possible to effectively improve the model compression rate in this solution.

[0035] By learning the probability distribution of data, the autoencoder can efficiently extract the key features of the data, remove redundant components, and compress it into a low-dimensional latent space representation (the features of the low-dimensional representation of the inherent characteristics of the autoencoder are mutually orthogonal, and the learned low-dimensional representation usually contains less redundant information), which can maximize the compression rate of the model. At the same time, the autoencoder has strong flexibility in the decompression process and can generate diverse reconstruction results based on the latent space representation. The loss function proposed in this scheme can guide the autoencoder to select the optimal reconstruction result and improve the accuracy of the model decompression recovery.

[0036] Secondly, the application scenarios of the method of the present disclosure are exemplified. It should be understood that the scenarios described in the following embodiments should not limit the present disclosure, and the present disclosure does not limit the application scenarios.

[0037] In some optional embodiments, a distributed system generally includes multiple edge devices, and the network environment and local models between each edge device are different. Therefore, when determining the global model, it is necessary to specify the compression method during compression transmission according to the actual parameters of each local model. Therefore, the central server can obtain the first model parameters of the first model in the reference edge device, and use the first model parameters to train the autoencoder, so that the trained autoencoder can fully compress and decompress the local models of each edge device in the distributed system, laying the foundation for determining the global model.

[0038] In some optional embodiments, the network environment of each edge device of the distributed system and the local model of each edge device will usually change. Therefore, when the configuration environment of the edge device changes, the global model of the distributed system should also change. At this time, if the compression method before the change is still used to compress and transmit the first model after the change, it may cause resource waste due to insufficient model compression. At the same time, using the decompression method before the change to decompress the compressed model after the change may also cause the decompressed model to deviate greatly from the original model, thereby causing the updated global model to deviate from the expected global model. Therefore, the central server can re-acquire the first model parameters of the first model in the reference edge device, so as to use the first model parameters to train the autoencoder, so that the trained autoencoder can still fully compress and decompress the local models of each edge device in the distributed system, laying the foundation for determining the global model after the edge device configuration is changed.

[0039] The present disclosure is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Figure 1 A flow chart of a model processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the method is executed by a central server in a distributed system, and includes steps 101-105.

[0041] Step 101: Obtain first model parameters of a first model of a reference edge device.

[0042] In some embodiments, the first model is a local uncompressed model in the reference edge device (ie, a local model of the distributed system deployed in the reference edge device).

[0043] In some embodiments, the reference edge device may be all edge devices in the distributed system, or may be only some edge devices in the distributed system with good network status or good performance, which is not limited in this disclosure.

[0044] In other words, the reference edge device is the edge device used when training the autoencoder.

[0045] In some embodiments, the present disclosure does not limit the method for obtaining the first model parameters, which may be, for example, obtaining the first model parameters by receiving the first model directly uploaded by the edge device, or, for example, obtaining the first model parameters by connecting to a database storing the first model parameters.

[0046] In some embodiments, the first model parameters may include a local parameter set of the first model, and may also include a model weight of the first model parameters when participating in autoencoder training.

[0047] In some embodiments, before obtaining the first model parameters, the central server may also initialize the parameters of the server. For example, the parameters of the central server may include the following parameters: a preset window size q, a current number of iterations t, a compression rate C, a reference edge device set, and a compression error threshold. The role of the window size q is to consider the errors in the past q iterations or time windows when evaluating the error of the autoencoder, rather than just the error based on a single iteration. The window size q acts as a buffer here, allowing the system to observe and evaluate the performance of the autoencoder within a time window. If the average error within this time window exceeds a preset threshold, the system can adjust the compression rate or increase the number of reference edge devices. The compression rate C can be a dynamic or fixed value.

[0048] Step 102: Based on the first model parameters, a first encoding module of the autoencoder is used to determine a first encoding model of the first model.

[0049] In some embodiments, the first model may be encoded by the following formula to obtain a first encoded model:

[0050]

[0051] in Yes i,t The latent dimension and compressed version, w i,t represents the model weight of user device i at global iteration t, w i,t The subscript i in the example represents the index of the edge device or client. Each edge device has its own local data set. The subscript i is used to distinguish these different edge devices. The subscript t represents the number of global iterations. During the training process of distributed learning, the model weights can be continuously updated in multiple global iterations. en Represents an encoding module.

[0052] In some embodiments, the autoencoder includes a first encoding module and a first decoding module, and the first encoding module is used to encode (ie, compress) the first model.

[0053] In some optional embodiments, the edge device uses the encoding module f of the autoencoder en to compress the first model, and convert the compressed first encoding model into Sent to the central server.

[0054] Step 103: Based on the first encoding model, a first decoding model of the first encoding model is determined using a first decoding module of the autoencoder.

[0055] In some embodiments, the first coding model may be decoded (decompressed) by the following formula to obtain a first decoding model:

[0056]

[0057] in, represents the parameters of the first decoding model after decoding, f de Represents the decoding module.

[0058] In some optional embodiments, the central server uses the decoding module f of the automatic encoder de Based on the transmitted latent vector Determine the original value of the local model of the approximate edge node i To calculate a new global model based on the two.

[0059] Step 104: train the autoencoder based on the first model and the first decoding model to obtain a first pre-trained autoencoder.

[0060] In some embodiments, the central server can determine the parameter loss of the autoencoder when encoding and decoding the model based on the first model and the first decoding model, and then iteratively train the autoencoder based on the parameter loss to obtain a first pre-trained autoencoder that meets the requirements of the distributed system.

[0061] In some embodiments, when training the autoencoder, the input data itself can be used as supervision based on the back-propagation algorithm and the optimization method (such as the gradient descent method) to guide the neural network to try to learn a mapping relationship, thereby obtaining an autoencoder model with updated reconstruction parameters and entering the next round of iteration process, but not limited to this.

[0062] Step 105: Determine a first global model of the distributed system using a first pre-trained autoencoder.

[0063] In some optional embodiments, the central server may send the second encoding module of the first pre-trained autoencoder to the first edge device, so that the first edge device can compress its own local model according to the second encoding module, and the edge device sends the compressed generated second encoding model to the central server, so that the central server determines the first global model according to the second encoding model, wherein the first edge device refers to an edge device used to determine the global model.

[0064] In summary, according to the model processing method proposed in the present disclosure, the method includes: obtaining the first model parameters of the first model of the reference edge device; based on the first model parameters, using the first encoding module of the autoencoder, determining the first encoding model of the first model; based on the first encoding model, using the first decoding module of the autoencoder, determining the first decoding model of the first encoding model; based on the first model and the first decoding model, training the autoencoder to obtain a first pre-trained autoencoder; using the first pre-trained autoencoder, determining the first global model of the distributed system. This method uses a local model to train an autoencoder for model compression transmission, so that when the local model is compressed by the autoencoder, dynamic compression of the local model can be achieved, so that when the local model is compressed and transmitted by the encoder to determine the global model of the distributed system, the accuracy of the global model can be improved.

[0065] Figure 3 A flow chart of a model processing method proposed in the present disclosure is further shown. Figure 2 The embodiment shown, Figure 3 The following steps are included.

[0066] Step 201: Obtain a first model parameter of a first model of a reference edge device.

[0067] In some embodiments, the principle of step 201 is the same as that of the embodiment shown in step 101, and reference may be made to the above related description, which will not be repeated here.

[0068] Step 202: use a first decoding module to decode the first coding model to obtain a first decoding model; or use a first decoding module to decode the first coding model with reference to the first model parameter to obtain a first decoding model.

[0069] In some embodiments, when training the autoencoder, the first encoding module in the autoencoder can be directly used to decode the first encoding model; the first model parameters (i.e., the parameters of the uncompressed model) can also be introduced as a reference to provide richer training signals.

[0070] In some embodiments, when the first model parameter is used as a reference and the first decoding module is used to decode the first coding model to obtain the first decoding model, the following formula may be used for model decoding:

[0071]

[0072] in, represents the first model parameter, that is, the uncompressed local model involved in training, i.e. k is the number of different first model parameters involved in training.

[0073] Step 203: iteratively train the autoencoder based on the second model parameters of the first encoding model and the third model parameters of the first decoding model.

[0074] Specifically, the objective function can be used to calculate the residual value between the second model parameter and the third model parameter; then based on the residual value, the back propagation algorithm is used to update the parameters of the autoencoder to achieve iterative training of the autoencoder.

[0075] It should be understood that when different methods are used to obtain the first decoding model in step 202, different objective functions are used in the iterative training in step 203.

[0076] Optionally, when the first decoding module is used to decode the first coding model, the objective function used in iterative training may be the following formula:

[0077]

[0078] Optionally, when the first model parameter is used as a reference and the first decoding module is used to decode the first coding model, the objective function used in iterative training may be the following formula:

[0079]

[0080] Among them, L2 represents the objective function, and the other parameters have the same meanings as those in the above formula and will not be repeated here.

[0081] In particular, the loss function of the present disclosure uses the L2-norm (i.e., the Euclidean distance) to calculate the difference between the first model and the first decoding model. The loss function is used as the optimization target to guide how the weights of the autoencoder are adjusted to minimize the error in the compression and recovery process. By minimizing the loss function, the autoencoder learns how to compress and restore the model more effectively.

[0082] Optionally, in some embodiments, the back propagation algorithm is used to update the parameters of the autoencoder, wherein the back propagation algorithm is, for example, a gradient descent method, and the input data itself (the first model parameter, the second model parameter, and the third model parameter) is used as supervision to guide the neural network in the autoencoder to try to learn a mapping relationship, thereby obtaining an autoencoder model with updated reconstruction parameters and entering the next round of iteration. The back propagation step includes calculating the gradient of the loss function with respect to the autoencoder weights, starting from the error of the output layer, and passing it back to each layer of the network, and using gradient descent or its variants (such as the Adam optimizer) to update the weights and biases of the autoencoder.

[0083] Step 204, when the number of iterative training reaches a number threshold and / or the residual value between the third model parameter described by the objective function obtained based on the iterative training and the second model parameter reaches a minimum threshold, the trained autoencoder is determined as the second pre-trained autoencoder.

[0084] In some embodiments, when the number of iterative training reaches a threshold number, the autoencoder iteratively obtained when the threshold is reached can be determined as the second pre-trained autoencoder; or when the residual value between the third model parameter described by the objective function obtained by iterative training and the second model parameter reaches a minimum threshold, the obtained autoencoder is determined as the second pre-trained autoencoder.

[0085] Step 205: determine a model compression rate based on the second pre-trained autoencoder and the first model parameters.

[0086] In some embodiments, when the distributed system compresses and transmits the model, it not only considers the error between the decompressed model and the original model, but also considers the compression rate when compressing the original model.

[0087] In some embodiments, the first model can be compressed by the encoding module of the second pre-trained autoencoder, and the encoding module can be decompressed by the decoding module, and finally the model compression rate is determined by comparing the parameters of the decompressed model with the parameters of the first model.

[0088] In some embodiments, the compression ratio may be defined as Among them, |w i,t | represents the total number of parameters of the first model, and represents the decompressed parameters of the first model after being compressed by the encoder part of the second pre-trained autoencoder. However, the compression ratio can also be defined as the ratio of the dimension of the original model parameters to the dimension of the compressed model parameters. Among them, d i,t is the number of original model parameters, and is the number of model parameters after compression. For example, if an original model has 1000 parameters and the autoencoder compresses it to only 100 parameters, then the compression ratio C is 10.

[0089] Step 206: Based on the model compression rate, adjust the second pre-trained autoencoder to obtain the first pre-trained model.

[0090] In some embodiments, the compression rate is for the performance of the autoencoder. Under the error conditions that meet the requirements, the lower the compression rate, the better the performance. The key point is to find a balance between the compression rate and the autoencoder performance. Increasing the compression rate can reduce communication costs, but too large a compression rate may cause the model performance to degrade. In fact, it is the size of the hidden layer of the autoencoder that determines the compression rate, and the hidden layer can be adjusted according to the compression target (the purpose of model training). Increasing the compression rate will increase information loss and reduce the ability of the model to accurately reconstruct the input.

[0091] In some embodiments, the second pre-trained autoencoder can be adjusted by changing the output dimension in the autoencoder, and the adjusted second pre-trained autoencoder can be determined as the first pre-trained encoder.

[0092] For example, if the determined model compression rate is small, the output dimension (i.e., intermediate representation) of the second pre-trained autoencoder can be adjusted to improve the model compression rate. For example, assuming that the first model has 1000 parameters and the intermediate representation set by the second pre-trained autoencoder is 100 parameters, then the compression rate = 1000 / 100 = 10. When it is desired to improve the compression rate of the second pre-trained autoencoder, the intermediate representation can be set to 80 parameters, and the compression rate is 12.5 at this time.

[0093] Step 207: Send the second encoding module of the first pre-trained autoencoder to the first edge device.

[0094] In some embodiments, when the central server completes the training of the autoencoder and obtains the first pre-trained autoencoder, the central server sends the second encoding module of the first pre-trained autoencoder to the first edge device, so that the first edge device can use the second encoding module to compress its own local model and transmit it to the central server.

[0095] The first edge device refers to an edge device in the distributed system used to determine the global model.

[0096] Step 208: Receive a second encoding model sent by the first edge device.

[0097] In some embodiments, the central server receives the second encoding model sent by the edge device to determine the first global model of the distributed system by using the second encoding model.

[0098] In other words, the first edge device compresses and transmits its own local model using the received second coding model.

[0099] Step 209: Decode the second encoding model using the second decoding module of the first pre-trained model to obtain a second decoding model.

[0100] Step 210: Determine a first global model, a global model, based on the second decoding model.

[0101] In some embodiments, the central server determines the real-time parameters of the local models of each first edge device based on the second decoding model, thereby integrating the real-time parameters of the local models of each first edge device to generate a first global model.

[0102] Step 211, send the first global model to the first edge device without compression; or, compress the first global model using the encoding module, and send the second decoding module and the compressed first global model to the first edge device, so that the first edge device obtains the first global model.

[0103] In some embodiments, after the central server determines the first global model, it sends the first global model to each edge device of the distributed system so that the edge device obtains the determined global model.

[0104] When the central server sends the first global model to the edge device, if the current network bandwidth of the distributed system is large, the first global model can be directly sent to the edge device without compression. If the current network is relatively tight, the first global model can be compressed using the encoding module, and the second decoding module and the compressed first global model can be sent to the edge device, so that the edge device can use the second decoding module to decompress the compressed model, thereby obtaining the first global model.

[0105] In summary, the model processing method proposed in the present disclosure includes: obtaining first model parameters of a first model with reference to an edge device, where the first model is a local uncompressed model of the reference edge device; based on the first model parameters, using the first encoding module of the autoencoder, determining the first encoding model of the first model; using the first decoding module, decoding the first encoding model to obtain the first decoding model; or, using the first model parameters as a reference, using the first decoding module to decode the first encoding model to obtain the first decoding model; iteratively training the autoencoder based on the second model parameters of the first encoding model and the third model parameters of the first decoding model; when the number of iterative training reaches a threshold number and / or the residual value between the third model parameter described by the objective function obtained based on the iterative training and the second model parameter reaches a minimum threshold When the value is reached, the trained autoencoder is determined as the second pre-trained encoder; based on the second pre-trained autoencoder and the first model parameters, the model compression rate is determined; based on the model compression rate, the second pre-trained autoencoder is adjusted to obtain the first pre-trained model; the second encoding module of the first pre-trained autoencoder is sent to the first edge device; the second encoding model sent by the first edge device is received; the second encoding model is decoded by the second decoding module of the first pre-trained model to obtain the second decoding model; based on the second decoding model, the first global model is determined; the first global model is sent to the first edge device without compression; or, the first global model is compressed by the encoding module, and the second decoding module and the compressed first global model are sent to the first edge device, so that the first edge device obtains the first global model. The method disclosed in the present invention trains the autoencoder for model compression transmission by using the local model, so that the local model can be dynamically compressed when the autoencoder is used to compress the local model, so that when the encoder is used to compress and transmit the local model to determine the global model of the distributed system, the accuracy of the global model can be improved.

[0106] The following is an exemplary description of the disclosed method:

[0107] In the training state, the central server receives uncompressed local model parameters from multiple edge devices and uses these parameters to train the autoencoder to learn how to compress and reconstruct model parameters. For example, for 5 edge nodes, a total of 5*20=100 original models from 20 rounds of training are retained as training data, and the original models are used in turn for model training based on the initialization parameters. The specific steps are as follows: Figure 4 and Figure 5 shown.

[0108] Embodiment 1, as Figure 4As shown in the figure, in the training state, based on the back propagation algorithm and the optimization method (such as the gradient descent method), the input data itself is used as supervision to guide the neural network to try to learn a mapping relationship, thereby obtaining an autoencoder model with updated reconstruction parameters and entering the next round of iteration. Among them, the back propagation step calculates the gradient of the loss function with respect to the autoencoder weights, starting from the error of the output layer, and propagates backwards to each layer of the network, using gradient descent or its variants (such as the Adam optimizer) to update the weights and biases of the autoencoder.

[0109] To train the autoencoder on the server, the distributed learning (e.g., FL) system starts as usual, with the edge devices transmitting the actual local weight values ​​to the server without compression, and the server uses these values ​​to train, validate, and test the autoencoder. The autoencoder is then sent to the edge nodes, instructing them to use the encoder f de Compress the local model before transmission. Use a fully connected autoencoder with one hidden layer and ReLU activation function, whose size is adjusted according to the size of the global model, i.e. |w i,t |.

[0110] The compression ratio is defined as Among them, |w i,t | represents the total number of parameters of the original model on the user device, and Represents the model parameters after compression by the encoder part of the autoencoder. The compression rate is used to evaluate the performance of the model. Under the required error conditions, the lower the compression rate, the better the performance. The key point is to find a balance between the compression rate and model performance. Increasing the compression rate can reduce communication costs, but too large a compression rate may cause the model performance to degrade. In fact, it is the size of the hidden layer of the autoencoder that determines the compression rate, and the hidden layer can be adjusted according to the compression target (the purpose of model training). Increasing the compression rate will increase information loss and reduce the model's ability to accurately reconstruct the input.

[0111] In addition, the compression ratio can also be defined as the ratio of the dimension of the original model parameters to the dimension of the compressed model parameters. Among them, d i,t is the number of original model parameters, and is the number of model parameters after compression. For example, if an original model has 1000 parameters and the autoencoder compresses it to only 100 parameters, then the compression ratio C is 10. If you want to adjust the compression ratio, this proposal can change the dimension of the encoder output, that is, the size of the intermediate representation. For example, suppose the original model has 1000 parameters, and this proposal hopes to compress it to different degrees through the autoencoder. If you set the intermediate representation to 100 parameters, then the compression ratio = 1000 / 100 = 10.

[0112] Embodiment 2, as Figure 5 As shown, in order to further improve the accuracy of the autoencoder, this proposal proposes a better way. The server is connected to a part of edge devices (referred to as reference edge devices). These edge devices (generally devices with good hardware performance and stable network connection performance with the server) do not need to be compressed during distributed learning (such as FL) and send local models. Accordingly, before training the autoencoder, it is necessary to detect the network status or hardware performance of the edge device, and use the edge nodes with good network status or good performance as reference edge devices. The original models of these reference edge nodes are added to the training of the autoencoder.

[0113] The uncompressed model is added to the training as a reference. In the autoencoder training, using both uncompressed and compressed models as input can provide richer training signals and help the autoencoder learn better compressed representations. The models of some accurate users are not compressed during the training process.

[0114] After the autoencoder is trained, the server can continue to use the uncompressed local model of the reference edge device to reconstruct the local models of other users during the decoding process. These accurate user models can be used as references to help the autoencoder more accurately reconstruct the compressed models of other users. As mentioned earlier, the data of different user devices are correlated, and this correlation leads to redundancy between model parameters. By using both uncompressed and compressed models, the autoencoder can learn this redundant information and use it to improve compression efficiency.

[0115] By introducing an uncompressed model, the training of the autoencoder can adjust the compression rate more flexibly. During the training process, the compression rate can be adjusted according to the error tolerance and compression target to achieve the best compression effect. As mentioned above, the iterative error correction characteristics of the FL system can tolerate a certain degree of compression error. By using an uncompressed model to assist training, the autoencoder can better learn how to maximize the compression rate without sacrificing FL convergence.

[0116] With this accurate information, the server can compute the global model, and for the architecture of the autoencoder with accurate information, combine the uncompressed model with the compressed models of other users in the hidden layer, referring to the edge device so that the autoencoder can better track and keep up with the FL process.

[0117] To alleviate the error loss problem caused by the compression process, this proposal selects a part of edge devices. These reference edge devices do not compress their local models, and their model updates are sent directly to the server. Since the reference edge devices provide uncompressed and more accurate model updates, the server can use this information to better train the decoder part of the autoencoder.

[0118] In this way, even if the model updates of other edge devices have been compressed, the server can use the model updates of the reference edge device to reconstruct a global model that is closer to the original state. This process enables the autoencoder to track the state of the global model more accurately and keep in sync with the optimal model in each round of federated learning iteration. The presence of the reference edge device improves the performance of the autoencoder during compression and decompression, allowing the entire federated learning system to maintain high model accuracy and convergence while reducing communication costs.

[0119] In this proposal, the training of the autoencoder is based on the L2 loss function, which measures the difference between the original model parameters and the model parameters reconstructed by the decoder. The conditions for judging the end of the autoencoder training may include the following: Preset number of iterations: A fixed number of iterations can be set for training. When this number is reached, the training process ends. Error tolerance: An error tolerance threshold can be set. When the L2 loss of the autoencoder is lower than this threshold, the training is considered to be good enough and can be ended. Convergence judgment: Determine whether the autoencoder has converged by monitoring the change of the L2 loss function. If the loss function changes little or no longer decreases significantly in multiple consecutive iterations, this may indicate that the autoencoder has reached a convergence state. Validation set performance: During the training process, an independent validation set can be used to evaluate the performance of the autoencoder. When the performance on the validation set (for example, measured by accuracy, precision, or other indicators) no longer improves or reaches the expected goal, the training can be ended.

[0120] The present disclosure also provides two methods for directly performing compressed transmission between two central servers and edge devices.

[0121] Method 1: Figure 6 As shown, the distributed system only compresses the model for upstream transmission and sends the encoder part of the autoencoder to the edge nodes so that they can use their encoding part to compress the local model for the next round of iteration. At the same time, the server uses the decoding part of the autoencoder to decode the compressed local model updated by the edge nodes and calculate the global model.

[0122] Method 2: Figure 7 As shown, the model is compressed in both upstream and downstream transmissions of the distributed system. The server needs to send the entire autoencoder to the edge node so that the edge node can use the autoencoder to compress the local model, update the iterative local model and send it to the server. At the same time, the server uses the decoding part of the autoencoder to decode the compressed user-updated local model and calculate the global model. The server uses the autoencoder to compress the iterative global model and then sends it to the edge node. The edge node uses the autoencoder to decompress the compressed global model.

[0123] Therefore, this solution has the following beneficial effects:

[0124] This method uses a local model to train an autoencoder for model compression transmission, so that when the local model is compressed by the autoencoder, dynamic compression of the local model can be achieved, thereby improving the accuracy of the global model when the encoder is used to determine the global model of the distributed system.

[0125] Figure 8 FIG. 8 is a schematic diagram of a model processing device 800 provided in an embodiment of the present disclosure. Figure 8 As shown, the model processing device includes:

[0126] An acquiring unit 810 is configured to acquire a first model parameter of a first model of a reference edge device, where the first model is a local uncompressed model of the reference edge device;

[0127] A first determining unit 820 is configured to determine a first encoding model of the first model based on the first model parameters using a first encoding module of the autoencoder;

[0128] A second determining unit 830 is configured to determine a first decoding model of the first encoding model based on the first encoding model using a first decoding module of the autoencoder;

[0129] A training unit 840 is used to train the autoencoder based on the first model and the first decoding model to obtain a first pre-trained autoencoder;

[0130] The third determination unit 850 is used to determine a first global model of the distributed system by using the first pre-trained autoencoder.

[0131] In some embodiments of the present disclosure, the second determination unit 830 is further used to use the first decoding module to decode the first coding model to obtain the first decoding model; or, use the first decoding module to decode the first coding model with reference to the first model parameters to obtain the first decoding model.

[0132] In some embodiments of the present disclosure, the training unit 840 is further used to iteratively train the autoencoder based on the second model parameters of the first encoding model and the third model parameters of the first decoding model; when the number of iterative training reaches a threshold number and / or the residual value between the third model parameter described by the objective function obtained based on the iterative training and the second model parameter reaches a minimum threshold, the trained autoencoder is determined as the second pre-trained encoder; based on the second pre-trained autoencoder and the first model parameters, the model compression rate is determined; based on the model compression rate, the second pre-trained autoencoder is adjusted to obtain the first pre-trained model.

[0133] In some embodiments of the present disclosure, the training unit 840 is further used to calculate the residual value between the second model parameter and the third model parameter using the objective function; based on the residual value, the back propagation algorithm is used to update the parameters of the autoencoder to achieve iterative training of the autoencoder.

[0134] In some embodiments of the present disclosure, the third determination unit 850 is also used to send the second encoding module of the first pre-trained autoencoder to the first edge device; receive the second encoding model sent by the first edge device, where the second encoding model is obtained by the first edge device using the second encoding module to the local model of the first edge device; decode the second encoding model using the second decoding module of the first pre-trained model to obtain a second decoding model; and determine the first global model based on the second decoding model.

[0135] In some embodiments of the present disclosure, the third determination unit 850 is further used to send the first global model to the edge device without compression; or, to compress the first global model using the encoding module, and to send the second decoding module and the compressed first global model to the first edge device, so that the first edge device obtains the first global model.

[0136] In summary, the model processing device proposed in the present disclosure includes: an acquisition unit for acquiring a first model parameter of a first model of a reference edge device, wherein the first model is a local uncompressed model of the reference edge device; a first determination unit for determining a first encoding model of the first model based on the first model parameter using a first encoding module of an autoencoder; a second determination unit for determining a first decoding model of the first encoding model based on the first encoding model using a first decoding module of the autoencoder; a training unit for training the autoencoder based on the first model and the first decoding model to obtain a first pre-trained autoencoder; and a third determination unit for determining a first global model of a distributed system using the first pre-trained autoencoder. The device of the present disclosure trains an autoencoder for model compression transmission using a local model, so that dynamic compression of the local model can be achieved when the local model is compressed using the autoencoder, thereby improving the accuracy of the global model when the local model is compressed and transmitted using the encoder to determine the global model of the distributed system.

[0137] It should be noted that: when the model processing device provided in the above embodiment performs model processing, it only uses the division of the above-mentioned program modules as an example. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the model processing device is divided into different program modules to complete all or part of the processing described above.

[0138] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0139] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0140] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Fig. 9 As shown, the electronic device 900 includes at least one processor 902; and a memory 901 communicatively connected to the at least one processor 902; wherein the memory 901 stores instructions executable by the at least one processor 902, and the instructions are executed by the at least one processor 902 to implement the steps of the model processing method described in the embodiment of the present disclosure; or, the instructions are executed by the at least one processor 902 to implement the steps of the model processing method described in the embodiment of the present disclosure.

[0141] It is understood that the electronic device also includes a communication interface. The various components in the electronic device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0142] It can be understood that the memory 901 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).The memory 901 described in this embodiment of the present solution is intended to include, but is not limited to, these and any other suitable types of memories.

[0143] The method disclosed in the above embodiment of the present disclosure may be applied to the processor 902, or implemented by the processor 902. The processor 902 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 902 or an instruction in the form of software.

[0144] The processor 902 described above may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 902 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this solution. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of this solution, it may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium, which is located in the memory 901, and the processor 902 reads the information in the memory 901 and completes the steps of the aforementioned method in combination with its hardware.

[0145] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), FPGA, general purpose processor, controller, MCU, microprocessor, or other electronic components to execute the aforementioned method.

[0146] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the steps of the model processing method described in the embodiment of the present solution when the computer instructions are executed.

[0147] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0148] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present solution. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0149] Any process or method description in the flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiment of the present scheme includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present scheme belong.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0151] It should be understood that the various parts of the implementation of the present solution can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned implementation, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another implementation, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0152] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0153] In addition, each functional unit in each embodiment of the present solution can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0154] Although the implementation methods of the present scheme have been shown and described above, it can be understood that the above implementation methods are exemplary and cannot be understood as limitations on the present scheme. Ordinary technicians in this field can change, modify, replace and modify the above implementation methods within the scope of the present scheme.

Claims

1. A model processing method, characterized in that: The method is applied to a central server of a distributed system, comprising: Obtaining first model parameters of a first model of a reference edge device, where the first model is a local uncompressed model of the reference edge device; Determine a first encoding model of the first model based on the first model parameters using a first encoding module of the autoencoder; Based on the first encoding model, using the first decoding module of the autoencoder, determining a first decoding model of the first encoding model; Based on the first model and the first decoding model, training the autoencoder to obtain a first pre-trained autoencoder; A first global model of the distributed system is determined using the first pre-trained autoencoder.

2. The method according to claim 1, characterized in that The determining, based on the first encoding model and using the first decoding module of the autoencoder, a first decoding model of the first encoding model comprises: Decoding the first coding model using the first decoding module to obtain the first decoding model; or, The first encoding model is decoded using the first decoding module with the first model parameters as a reference to obtain the first decoding model.

3. The method according to claim 1, characterized in that The step of training the autoencoder based on the first model and the first decoding model to obtain a first pre-trained autoencoder includes: Iteratively training the autoencoder based on the second model parameters of the first encoding model and the third model parameters of the first decoding model; When the number of iterative training times reaches a threshold number and / or the residual value between the third model parameter described by the objective function obtained by iterative training and the second model parameter reaches a minimum threshold, determining the trained autoencoder as the second pre-trained autoencoder; Determining a model compression rate based on the second pre-trained autoencoder and the first model parameters; Based on the model compression rate, the second pre-trained autoencoder is adjusted to obtain the first pre-trained model.

4. The method according to claim 3, characterized in that: The iterative training of the autoencoder based on the second model parameter of the first encoder and the third model parameter of the first decoder includes: Calculating a residual value between the second model parameter and the third model parameter using an objective function; Based on the residual value, a back propagation algorithm is used to update the parameters of the autoencoder to achieve iterative training of the autoencoder.

5. The method according to claim 1, characterized in that Determining the first global model of the distributed system by using the first pre-trained autoencoder includes: Sending a second encoding module of the first pre-trained autoencoder to a first edge device; Receive a second coding model sent by the edge device, where the second coding model is obtained by the first edge device encoding a local model of the first edge device using the second coding module; Decoding the second encoding model using a second decoding module of the first pre-trained model to obtain a second decoding model; Based on the second decoding model, a first global model is determined.

6. The method according to claim 1, characterized in that The method further comprises: sending the first global model to the first edge device without compression; Or, the first global model is compressed by using the encoding module, and the second decoding module and the compressed first global model are sent to the first edge device, so that the first edge device obtains the first global model.

7. A model processing device, characterized in that: The device comprises: An acquiring unit, configured to acquire a first model parameter of a first model of a reference edge device, where the first model is a local uncompressed model of the reference edge device; A first determining unit, configured to determine a first encoding model of the first model by using a first encoding module of an autoencoder based on the first model parameters; A second determining unit, configured to determine a first decoding model of the first encoding model based on the first encoding model and using a first decoding module of the autoencoder; A training unit, configured to train the autoencoder based on the first model and the first decoding model to obtain a first pre-trained autoencoder; The third determining unit is used to determine a first global model of the distributed system by using the first pre-trained autoencoder.

8. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the method described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images