Quantization coding method, device and system

Abstract

The invention provides a quantization coding method, which comprises the following steps of: performing quantization processing on a pruned cumulative gradient, and coding a quantized parameter. The quantization processing and the coding processing are based on parameters after layer pruning, and are combined with a layer pruning technology, so that the calculation amount of local training and the communication traffic of model aggregation can be reduced, the calculation and communication loads in system interaction are further reduced, the system efficiency of federal learning is optimized, and the system performance is improved. Therefore, the requirement on the equipment configuration performance in the actual deployment federated learning is reduced.

Description

Technical Field

[0001] The present application relates to the field of federated learning technology, and specifically to a quantization coding method, device and system. Background Art

[0002] Federated learning is a distributed machine learning framework that allows training data to be distributed on mobile devices and learns shared models by aggregating updates from local computations. Federated learning is considered for deployment in many applications due to its simple process, protection of distributed data privacy, and high communication and computation efficiency.

[0003] In the federated learning system, communication and computing are deeply coupled and interact with each other. This dual limitation jointly restricts the communication efficiency and algorithm performance of the federated learning system. In the prior art, the model compression technology of traditional centralized deep learning is usually borrowed to improve the communication and computing efficiency of the system. For example, the model structure is compressed and simplified through model pruning, and the specific parameters are compressed through parameter quantization, thereby realizing the model compression of the neural network.

[0004] The current quantization coding scheme needs to have good coding efficiency when the model parameters are extremely large and meet the sparsity conditions. It is mainly suitable for the quantization coding of neural networks with a single model. However, the new pruning strategy under federated learning has weak matching and cannot achieve high coding efficiency. The quantization coding mechanism for layer pruning still needs to be improved. Summary of the invention

[0005] The present application provides a quantization encoding method that can optimize the system efficiency of federated learning, thereby reducing the requirements for device configuration performance in the actual deployment of federated learning.

[0006] In a first aspect, a quantization coding method is provided, which can be executed by a user equipment, or by a chip or circuit configured in the user equipment, or by a logic module or software capable of implementing all or part of the functions of the user equipment. This application does not limit this.

[0007] The method includes: determining a first neural model, the first neural model is a first training model after hierarchical pruning, the first neural model corresponds to a first accumulated gradient set, the first accumulated gradient set includes the accumulated gradients of each retained layer after hierarchical pruning; quantizing the accumulated gradients in each retained layer in the first accumulated gradient set to obtain a second accumulated gradient set, the second accumulated gradient set includes the accumulated gradients of each retained layer after quantization; encoding the second accumulated gradient set to obtain a first coding sequence set, the first coding sequence set includes the quantized coding sequences of each retained layer; and sending the first coding sequence set.

[0008] Based on this technical solution, the accumulated gradient after pruning is quantized and the quantized parameters are encoded. Both quantization and encoding are based on the parameters after layer pruning. Combined with layer pruning technology, it can reduce the amount of computation for local training and the amount of communication for model aggregation, further reduce the computation and communication load in system interaction, optimize the system efficiency of federated learning, and thus reduce the requirements for device configuration performance in the actual deployment of federated learning.

[0009] In combination with the first aspect, in some implementations of the first aspect, the first neural model is obtained from a network device, the first neural model is hierarchically pruned by the network device, and the network device is a network device in the distributed learning system.

[0010] This solution is suitable for heterogeneous model scenarios. In this scenario, the models deployed on each distributed user are different. The network equipment can allocate initialized pruned sub-models of different complexity to each user through even different numbers of layers. The user equipment locally trains the pruned sub-model to obtain the cumulative gradient of the parameters of the pruned sub-model.

[0011] For scenarios with heterogeneous distributed user models, by matching layer pruning sub-models of different complexities and combining them with parameter quantization methods, both the computational workload of local training and the communication workload of model transmission can be effectively reduced.

[0012] It can be understood that in the homogeneous model scenario, there are no restrictions on the configuration and training process of the local model of the user device, and distributed users can perform layer pruning on the accumulated gradients of the local model parameters after the local model training is completed.

[0013] For scenarios with distributed user model homogeneity, a joint layer pruning and quantization method is used to significantly reduce the amount of communication during model upload at the expense of a small and controllable global model performance degradation.

[0014] In combination with the first aspect, in some implementations of the first aspect, the first neural model is a hierarchical pruned neural model determined by the network device according to the hierarchical complexity of the user equipment.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the accumulated gradients of each retained layer in the first accumulated gradient set are quantized to obtain a second accumulated gradient set, and the method includes: determining a quantization value and a modulus of the accumulated gradients of each retained layer; determining a quantization interval of each accumulated gradient of each retained layer according to the quantization value and the modulus of the accumulated gradients of each retained layer; and representing the second accumulated gradient set according to the quantization value, the modulus of the accumulated gradients of each retained layer, the quantization interval of each accumulated gradient, and a sign function.

[0016] In this technical solution, quantization processing is based on the parameter gradient of the retained layer after layer pruning, which can reduce the computational workload of local training and the communication workload of model aggregation, and improve the communication and computing efficiency of the system.

[0017] In combination with the first aspect, in some implementations of the first aspect, the quantized value is represented by b, and the magnitude of the accumulated gradient of each retained layer is represented by Indicates that the quantization interval is numbered using express, represents a symbol function, wherein l is the lth layer of the reserved layer, k is the user equipment, and the second accumulated gradient set can be expressed by the following formula:

[0018]

[0019] in,

[0020] Wherein, s is the number of the quantization interval,

[0021] in, is the probability of random quantization,

[0022] In combination with the first aspect, in certain implementations of the first aspect, the second accumulated gradient set is encoded to obtain a first encoded sequence set, and the method includes: encoding and splicing the quantization value and the modulus of the accumulated gradients of each retained layer; for each accumulated gradient of each retained layer, respectively encoding and splicing the quantization interval and the sign function, the quantization interval and the sign function are located at the end of the codeword.

[0023] In combination with the first aspect, in some implementations of the first aspect, cyclic Elias encoding is performed on the quantized value.

[0024] For the accumulated gradient after layer pruning quantization, a corresponding encoding scheme is designed based on Elias code, which has a lower upper bound of the asymptotic code length under the setting of layer pruning. At the same time, the encoding results of the gradients of each layer show that the quantization encoding scheme for uploading layer pruning in the model requires fewer bits.

[0025] In a second aspect, a quantization coding method is provided, which can be executed by a network device, or by a chip or circuit configured in the network device, or by a logic module or software that can implement all or part of the network device functions. This application does not limit this.

[0026] The method includes: receiving multiple first coding sequence sets from multiple user devices, the first coding sequence sets include coding sequences after quantization of retained layers corresponding to a first neural model, the multiple user devices correspond one-to-one to the multiple first coding sequence sets, and the multiple user devices are user devices in the distributed learning system; performing layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model, and the second neural model is used for local training of the multiple users.

[0027] Based on this technical solution, the network device aggregates the first coding sequence sets of multiple users, wherein the first coding sequence set is the parameter gradient after quantization encoding for layer pruning. Both the quantization processing and the encoding processing are based on the parameters after layer pruning. Combined with the layer pruning technology, it can reduce the computational workload of local training and the communication volume of model aggregation, further reduce the computational and communication loads in system interaction, optimize the system efficiency of federated learning, and thus reduce the requirements for device configuration performance in the actual deployment of federated learning.

[0028] In combination with the second aspect, in some implementations of the second aspect, a first neural model is determined based on hierarchical pruning; and the first neural model is sent to the multiple user devices.

[0029] In combination with the second aspect, in certain implementations of the second aspect, the first neural model is a hierarchical pruned neural model determined by the multiple user devices according to their respective hierarchical complexities.

[0030] In a third aspect, a quantization coding device is provided, which may be a user equipment, or a chip or circuit configured in the user equipment, or a logic module or software capable of implementing all or part of the functions of the user equipment. This application does not limit this.

[0031] The device includes: a processing unit, used to determine a first neural model, the first neural model is a local training model after hierarchical pruning, the first neural model corresponds to a first accumulated gradient set, the first accumulated gradient set includes the accumulated gradients of each retained layer after hierarchical pruning; the processing unit is also used to quantize the accumulated gradients in each retained layer in the first accumulated gradient set to obtain a second accumulated gradient set, the second accumulated gradient set includes the accumulated gradients of each retained layer after quantization; the processing unit is also used to encode the second accumulated gradient set to obtain a first coding sequence set, the first coding sequence set includes the quantized coding sequences of each retained layer; a transceiver unit is used to send the first coding sequence set.

[0032] In combination with the third aspect, in certain implementations of the third aspect, the transceiver unit is also used to obtain the first neural model from a network device, where the first neural model is hierarchically pruned by the network device, and the network device is a network device in the distributed learning system.

[0033] In combination with the third aspect, in certain implementations of the third aspect, the first neural model is a hierarchical pruned neural model determined by the network device according to the hierarchical complexity of the user equipment.

[0034] In combination with the third aspect, in certain implementations of the third aspect, the processing unit is specifically used to determine a quantization value and a modulus of the accumulated gradients of each retained layer; determine a quantization interval of each accumulated gradient of each retained layer according to the quantization value and the modulus of the accumulated gradients of each retained layer; and represent the second accumulated gradient set according to the quantization value, the modulus of the accumulated gradients of each retained layer, the quantization interval of each accumulated gradient, and a sign function.

[0035] In conjunction with the third aspect, in some implementations of the third aspect, the quantized value is represented by b, and the magnitude of the accumulated gradient of each retained layer is represented by Indicates that the quantization interval is numbered using express, represents a symbol function, wherein l is the lth layer of the reserved layer, k is the user equipment, and the second accumulated gradient set can be expressed by the following formula:

[0036]

[0037] in,

[0038] Wherein, s is the number of the quantization interval,

[0039] in, is the probability of random quantization,

[0040] In combination with the third aspect, in certain implementations of the third aspect, the processing unit is also used to encode and splice the quantization value and the modulus of the accumulated gradient of each retained layer; for each accumulated gradient of each retained layer, the quantization interval and the sign function are respectively encoded and spliced, and the quantization interval and the sign function are located at the end of the codeword.

[0041] In combination with the third aspect, in certain implementations of the third aspect, the processing unit is further configured to perform cyclic Elias encoding on the quantized value.

[0042] It should be understood that the specific process and possible implementation method of the transceiver unit and the processing unit of the device to perform the corresponding steps have been described in detail in the above-mentioned first aspect. The beneficial effects of this aspect can be referred to the first aspect. For the sake of brevity, they will not be repeated here.

[0043] In a fourth aspect, a quantization coding device is provided, which may be a network device, or a chip or circuit configured in the network device, or a logic module or software capable of implementing all or part of the functions of the network device. This application does not limit this.

[0044] The device includes: a transceiver unit, used to receive multiple first coding sequence sets from multiple user devices, the first coding sequence sets include coding sequences after quantization of a reserved layer corresponding to a first neural model, the multiple user devices correspond one-to-one to the multiple first coding sequence sets, and the multiple user devices are user devices in the distributed learning system; a processing unit, used to perform layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model, and the second neural model is used for local training of the multiple users.

[0045] In combination with the fourth aspect, in certain implementations of the fourth aspect, the processing unit is further used to determine a first neural model based on hierarchical pruning; and the transceiver unit is further used to send the first neural model to the multiple user devices.

[0046] In combination with the fourth aspect, in certain implementations of the fourth aspect, the first neural model is a hierarchical pruned neural model determined by the multiple user devices according to their respective hierarchical complexities.

[0047] It should be understood that the specific process and possible implementation method of the transceiver unit and the processing unit of the device to perform the corresponding steps have been described in detail in the above-mentioned second aspect. The beneficial effects of this aspect can be referred to the second aspect. For the sake of brevity, they will not be repeated here.

[0048] In a fifth aspect, the present application provides a quantization encoding device, including a processor, for executing the methods provided in the above aspects.

[0049] For the operations such as sending and acquiring / receiving involved in the processor, unless otherwise specified, or unless they conflict with their actual function or internal logic in the relevant description, they can be understood as operations such as processor output, reception, input, etc., or as sending and receiving operations performed by the radio frequency circuit and antenna, and this application does not limit this.

[0050] In a sixth aspect, a quantization coding system is provided, characterized in that it includes multiple user devices and a network device, each of the multiple user devices is used to determine a first neural model, the first neural model is a local training model after hierarchical pruning, the first neural model corresponds to a first cumulative gradient set, and the first cumulative gradient set includes the cumulative gradients of each retained layer after hierarchical pruning; each user device is also used to quantize the cumulative gradients in each retained layer in the first cumulative gradient set to obtain a second cumulative gradient set, and the second cumulative gradient set includes the cumulative gradients after the cumulative gradients of each retained layer are quantized; each user device is also used to encode the second cumulative gradient set to obtain a first coding sequence set, and the first coding sequence set includes the coding sequences after the quantization of each retained layer; each user device is also used to send the first coding sequence set; the network device is used to receive multiple first coding sequence sets from the multiple user devices, and the multiple user devices correspond to the multiple first coding sequence sets one by one; the network device is used to perform layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model, and the second neural model is used for local training of the multiple users.

[0051] In a seventh aspect, the present application provides a device comprising: a memory for storing programs; and at least one processor for executing computer programs or instructions stored in the memory to execute the method provided by any one of the above aspects or its implementation.

[0052] In an eighth aspect, the present application provides a computer-readable storage medium, which stores a computer program or instructions. When the computer program or instructions are run on a computer, the method provided by any one of the above aspects or its implementation is executed.

[0053] In a ninth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the method provided by any one of the above aspects or its implementation method to be executed.

[0054] In a tenth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the processor reads instructions stored in a memory through the communication interface and executes a method provided by any one of the above aspects or its implementation.

[0055] Optionally, as an implementation method, the chip also includes a memory, in which a computer program or instructions are stored, and the processor is used to execute the computer program or instructions stored in the memory. When the computer program or instructions are executed, the processor is used to execute the method provided by any one of the above aspects or its implementation methods.

[0056] The chip may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a schematic diagram of a communication system 100 applicable to an embodiment of the present application.

[0058] Figure 2 It is a schematic diagram of a system architecture 200 applicable to an embodiment of the present application.

[0059] Figure 3 It is a schematic diagram of a regional federation architecture 00 applicable to an embodiment of the present application.

[0060] Figure 4 It is a schematic diagram of a system architecture 400 of a federated learning layer pruning quantization coding applicable to an embodiment of the present application.

[0061] Figure 5 It is a schematic flow chart of a quantization encoding method applicable to an embodiment of the present application.

[0062] Figure 6 It is a quantization coding flow chart applicable to the embodiment of the present application.

[0063] Figure 7 It is another quantization coding flow chart applicable to the embodiment of the present application.

[0064] Figure 8 It is a quantitative flow chart applicable to the embodiment of the present application.

[0065] Fig. 9 It is a coding flow chart applicable to the embodiment of the present application.

[0066] Fig.10 It is a schematic diagram of a codeword structure for layer-by-layer encoding applicable to an embodiment of the present application.

[0067] Fig.11 It is a schematic diagram of the variation of the coding length with the parameter quantity applicable to the embodiment of the present application.

[0068] Fig.12 It is a schematic diagram of the number of bits required for encoding each layer of the model applicable to the embodiment of the present application.

[0069] Fig.13 It is a structural schematic diagram of a quantization encoding device applicable to an embodiment of the present application.

[0070] Fig.14 It is a structural diagram of a quantization coding architecture applicable to an embodiment of the present application. DETAILED DESCRIPTION

[0071] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0072] The technical solution provided in this application can be applied to various communication systems, such as: fifth generation (5th generation, 5G) or new radio (new radio, NR) system, long term evolution (long term evolution, LTE) system, LTE frequency division duplex (frequency division duplex, FDD) system, LTE time division duplex (time division duplex, TDD) system, etc. The technical solution provided in this application can also be applied to future communication systems, such as the sixth generation mobile communication system. The technical solution provided in this application can also be applied to device to device (D2D) communication, vehicle to everything (V2X) communication, machine to machine (M2M) communication, machine type communication (MTC), and Internet of things (IoT) communication system or other communication systems.

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] Figure 1 A schematic diagram of a communication system 100 applied in an embodiment of the present application is shown. The communication system 100 may include a core network device 110, an access network device 120 and at least one terminal device (such as Figure 1 The maximum device 130 and the terminal device 140 in it).

[0075] The core network device 110 and the access network device 120 can be connected wirelessly or wired. The core network device 110 and the access network device 120 can be independent and different physical devices, or the logical functions can be integrated on the same physical device, or the functions of some core network devices and some access network devices can be integrated on one physical device.

[0076] The access network device 120 may be a device that communicates with a terminal device, such as a base station or a base station controller. Each access network device 120 may provide communication coverage for a specific geographical area, and may communicate with a terminal device located in the coverage area (cell). The access network device 120 may be an evolved NodeB (eNB or eNodeB) in an LTE system, or a wireless controller in a cloud radio access network (CRAN) scenario, or the network device may be a relay station, an access point, a vehicle-mounted device, a wearable device, and a network device in a 5G network, for example, a base station (gNodeB or gNB) or a transmission receiving point / transmission reception point (TRP) in a new radio (NR), or the access network device 120 may also be a network device in a future evolved public land mobile network (PLMN) network, etc., and the embodiments of the present application are not limited thereto.

[0077] It should be understood that the access network device 120 in the wireless communication system can be any device with wireless transceiver functions. The device includes but is not limited to: evolved Node B (eNB), base transceiver station (BTS), home base station (e.g., Home evolved NodeB, or Home Node B, HNB), baseband unit (BBU), access point (AP) in wireless fidelity (WIFI) system, wireless relay node, wireless backhaul node, transmission point (TP) or transmission and reception point (TRP), etc., and can also be a gNB in ​​5G, such as NR, system, or a transmission point (TRP or TP), one or a group of antenna panels (including multiple antenna panels) of a base station in a 5G system, or a network node constituting a gNB or a transmission point, such as a baseband unit (BBU), or a distributed unit (DU), etc.

[0078] In some deployments, the gNB may include a centralized unit (CU) and a DU. The gNB may also include an active antenna unit (AAU). The CU implements some functions of the gNB, and the DU implements some functions of the gNB. For example, the CU is responsible for processing non-real-time protocols and services, and implementing the functions of the radio resource control (RRC) and packet data convergence protocol (PDCP) layers. The DU is responsible for processing physical layer protocols and real-time services, and implementing the functions of the radio link control (RLC) layer, the media access control (MAC) layer, and the physical (PHY) layer. The AAU implements some physical layer processing functions, radio frequency processing, and related functions of active antennas. Since the information of the RRC layer will eventually become the information of the PHY layer, or be converted from the information of the PHY layer, under this architecture, high-level signaling, such as RRC layer signaling, can also be considered to be sent by the DU, or by the DU+AAU. It can be understood that the network device can be a device including one or more of a CU node, a DU node, and an AAU node. In addition, the CU may be divided into a network device in an access network (radio access network, RAN), or the CU may be divided into a network device in a core network (core network, CN), and this application does not limit this.

[0079] The wireless communication system 100 also includes one or more terminal devices located within the coverage area of ​​the access network device 120, such as Figure 1130 and 140 are shown in FIG. The terminal device 130 or the terminal device 140 may be mobile or fixed. The terminal device 130 or the terminal device 140 may refer to an access terminal, a user equipment (UE), a user unit, a user station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent or a user device. The access terminal may be a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, a terminal device in a future 5G network, or a terminal device in a future evolved public land mobile network (PLMN), etc. The terminal device in the embodiment of the present application may also be a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the application scenario.

[0080] In an embodiment of the present application, a terminal device or a network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. In addition, the embodiment of the present application does not specifically limit the specific structure of the execution subject of the method provided in the embodiment of the present application, as long as it can communicate according to the method provided in the embodiment of the present application by running a program that records the code of the method provided in the embodiment of the present application, for example, the execution subject of the method provided in the embodiment of the present application can be a terminal device or a network device, or a functional module in a terminal device or a network device that can call a program and execute a program.

[0081] In addition, various aspects or features of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0082] Figure 1 A core network device, an access network device and two terminal devices are shown exemplarily. Optionally, the communication system 100 may include multiple network devices and the coverage of a network device may include other numbers of terminal devices, which is not limited in the embodiments of the present application.

[0083] Optionally, the wireless communication system 100 may also include other network entities such as a network controller and a mobility management entity, but the embodiments of the present application are not limited thereto.

[0084] The embodiments of the present application may be applicable to downlink signal transmission, uplink signal transmission, and device-to-device (D2D) signal transmission. For downlink signal transmission, the transmitting device is a wireless access network device, and the corresponding receiving device is a terminal device. For uplink signal transmission, the transmitting device is a terminal device, and the corresponding receiving device is a wireless access network device. For D2D signal transmission, the transmitting device is a terminal device, and the corresponding receiving device is also a terminal device. In the embodiments of the present application, there is no limitation on the transmission direction of the signal.

[0085] The access network device and the terminal device, as well as the terminal device and the terminal device, can communicate through the licensed spectrum, or can communicate through the unlicensed spectrum, or can communicate through both the licensed spectrum and the unlicensed spectrum. The access network device and the terminal device, as well as the terminal device and the terminal device, can communicate through the spectrum below 6G, or can communicate through the spectrum above 6G, or can communicate through the spectrum below 6G and the spectrum above 6G at the same time. The embodiments of the present application do not limit the spectrum resources used between the access network device and the terminal device.

[0086] Figure 2 A schematic diagram of a system architecture 200 provided in the present application is shown, and the system architecture 200 can be used to implement the method for training a neural network in the present application. Figure 2 In the embodiment, the data collection device 260 is used to collect training data.

[0087] Taking the training of the local model of the user device in the present application as an example, the training data may include real results (e.g., real images) and categories corresponding to the real results (e.g., category labels of real images), wherein the categories corresponding to the real results (e.g., category labels of real images) may be manually pre-labeled.

[0088] After collecting the training data, the data collection device 260 stores the training data in the database 230 , and the training device 220 obtains the target model / rule 201 through training based on the training data maintained in the database 230 .

[0089] The following describes how the training device 220 obtains the target model / rule 201 based on the training data. The training device 220 performs a task (e.g., image processing) based on the category of the real result, and compares the execution result (e.g., the generated image) with the real result until the difference between the execution result output by the training device 220 and the real result is less than a certain threshold, thereby completing the training of the target model / rule 201.

[0090] The target model / rule 201 in the present application can specifically be a convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 230 does not necessarily all come from the data acquisition device 260, but may also be received from other devices. It should also be noted that the training device 220 does not necessarily train the target model / rule 201 entirely based on the training data maintained by the database 230, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0091] The target model / rule 201 trained by the training device 220 can be applied to different systems or devices, such as Figure 2 The execution device 210 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 2 In the embodiment, the execution device 210 is configured with an input / output (I / O) interface 212 for data interaction with an external device. A user may input data to the I / O interface 212 through a client device 240. The input data may include an image to be processed input by the client device in the embodiment of the present application.

[0092] Preprocessing module 213 and preprocessing module 214 are used to perform preprocessing according to the input data (such as the image to be processed) received by I / O interface 212. In the present application, preprocessing module 213 and preprocessing module 214 may be omitted (or only one of the preprocessing modules may be present), and the computing module 211 may be used directly to process the input data.

[0093] When the execution device 210 preprocesses the input data, or when the computing module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0094] Finally, the I / O interface 212 returns the processing result (eg, the generated image) to the client device 240 for providing to the user.

[0095] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different goals or different tasks. The corresponding target models / rules 201 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0096] exist Figure 2 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 212. In another case, the client device 240 can automatically send input data to the I / O interface 212. If the client device 240 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 212 and the output results of the output I / O interface 212 as new sample data, and store them in the database 230. Of course, it is also possible not to collect through the client device 240, but the I / O interface 212 directly stores the input data of the input I / O interface 212 and the output results of the output I / O interface 212 as new sample data in the database 230.

[0097] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 210. In other cases, the data storage system 150 may also be placed in the execution device 210.

[0098] like Figure 2 As shown, the target model / rule 201 is obtained through training according to the training device 220. The target model / rule 201 in the embodiment of the present application can be the multi-task neural network in the present application. Specifically, the multi-task neural network in the present application can be CNN, DNN, MLP, etc.

[0099] In order to facilitate understanding of the embodiments of the present application, some terms appearing in the embodiments of the present application are explained below.

[0100] (1) Neural Network

[0101] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0102]

[0103] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0104] (2) Deep Neural Networks

[0105] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0106] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0107] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0108] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0109] (3) Convolutional Neural Network

[0110] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract data information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0111] (4) Local Model

[0112] The local model is a model trained by each distributed user based on local data.

[0113] (5) Global Model

[0114] The global model is a model that is obtained by the central server integrating multiple local models for distribution.

[0115] (6) Model aggregation

[0116] Model aggregation is the process of integrating the model parameters uploaded by distributed users during the federation update round to obtain the global model parameters.

[0117] (7) Gradient Module

[0118] The norm of the gradient parameter gradient is obtained by backpropagation during model training.

[0119] The following is an introduction to the federated learning model involved in this application.

[0120] As an emerging distributed learning framework, federated learning is generally considered to be a type of distributed system with high communication efficiency due to its concise model training paradigm and parameter transfer rules. In addition, federated learning can be widely used in many fields because of its protection of distributed data privacy.

[0121] At the same time, the federated learning model is a model of deep coupling of communication and computing: on the one hand, the communication network serves distributed computing and provides the basic function of information transmission; on the other hand, intelligent algorithms can also empower the communication network and optimize system performance.

[0122] Therefore, distributed network computing systems represented by federated learning are also considered to be a core function of future communication networks. Figure 3 As shown, it can be summarized into two basic steps: local distributed training and central periodic federation integration:

[0123] (1) Distributed users train local models based on local data;

[0124] (2) In the federated update round, distributed users upload local model parameters to the central server. After the central server aggregates the model, it transmits the new global model back to each distributed user for subsequent local training.

[0125] Figure 3 A schematic diagram of a regional federation architecture 00 applicable to an embodiment of the present application is shown, the architecture includes at least one central server and multiple distributed user devices, for example, Figure 3 The base station central server and k user equipments are shown in FIG. The central server provides centralized services such as federated updates and overall system scheduling for distributed user equipments in the area.

[0126] Among them, the central server can be deployed on the regional base station, and can also be called a base station central server. The central server can also be called a regional central node, a parameter server, and other similar terms. The embodiments of the present application do not limit this.

[0127] Among them, the user equipment can also be called user terminal equipment, can also be called terminal equipment, or can also be directly called user or terminal, which is not limited in the embodiments of the present application.

[0128] The typical federated learning framework can be summarized into two main steps: 1) Each distributed user uses local data to train a local model and periodically uploads the model parameters or gradients to the central server; 2) The central server aggregates the local models to obtain a global model and sends the updated global model to each distributed user for the next round of local training.

[0129] Federated learning reduces redundant data transmission in distributed training by using models as the basic element of interaction, physically isolates training data, and effectively utilizes distributed computing devices and data.

[0130] For example, in the FedAvg framework, in the tth global cycle, the parameter server selects a set of K participating users for model aggregation, and the global model The update formula can be expressed as:

[0131]

[0132] Among them, g k,t is the accumulated gradient of local training of user k in this round, η k Local training learning rate for user k ， ω k is the aggregation weight of user k and is generally set to the proportion of the user's local data volume, that is, D k is the local data of user k.

[0133] One of the main challenges facing federated learning is the dual bottlenecks of communication and computing, especially in the deployment of federated learning tasks based on neural network models. On the one hand, the heterogeneity of user devices and configurations leads to large differences in the computing power of local users, which seriously affects the complexity and latency of the system's computing process. On the other hand, in some mobile applications with many distributed users, bandwidth resources are limited, channel quality also varies, and the communication cost of parameter sharing may be very high. In addition, due to the deep coupling and interaction between communication and computing in the federated learning system, this dual limitation jointly restricts the system efficiency and algorithm performance of federated learning. Therefore, how to improve the communication and computing efficiency of the system has become an urgent problem to be solved in the actual deployment of federated learning systems.

[0134] To solve this problem, most of the common technologies currently used are drawn from traditional centralized deep learning model compression technologies, represented by model pruning and parameter quantization. Model pruning divides the original neural network into smaller sub-models, thereby reducing the total number of model parameters, optimizing the amount of local training computation and the amount of communication for model interaction; parameter quantization is to quantize the parameters of the neural network, reduce the number of bits required to represent the parameter values, and then combine the encoding process to reduce the encoding length of the model communication. At the same time, pruning and quantization belong to two different stages of model compression. The former compresses and simplifies the model structure, and the latter compresses and represents specific parameters. The two are generally combined to achieve neural network model compression.

[0135] Pruning technology in federated learning has undergone many evolutions, from the initial strategy similar to random discarding to focusing on the importance of parameter positions in the model, and deriving block and layer-based pruning technology. At the same time, parameter quantization and corresponding encoding schemes mostly use processing technologies under a single model, without targeted consideration of pruning strategies in federated learning, and lack of parameter quantization and encoding scheme design that matches pruning.

[0136] Based on the above background, the present invention focuses on the parameter quantization and coding optimization method of federated learning based on layer pruning. According to the layered pruning strategy, the corresponding quantization mechanism and coding scheme are designed, so as to further improve the communication and computing efficiency of the system while controlling the degradation of algorithm performance. At the same time, the coding length during parameter transmission is reduced, thereby lowering the performance threshold of federated learning for equipment requirements in actual system deployment.

[0137] In view of this, the present application proposes a quantization coding method for distributed learning models, which can quantize and encode federated learning parameters based on layer pruning. The targeted quantization coding scheme can reduce the computational and communication load in system interaction and optimize the system efficiency of federated learning.

[0138] The solution of this application is described in detail below.

[0139] The communication method provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings. The embodiment provided by the present application can be applied to the above Figure 1 The communication system shown is not limited.

[0140] Figure 4 A schematic diagram of a system architecture 400 for federated learning layer pruning quantization coding provided in an embodiment of the present application.

[0141] The system is a federated learning system with multiple distributed users, which can include 1 parameter server and N distributed users. In each global round t, K users are selected according to the rules to form a participating user set P. tParticipate in this round of model aggregation. Figure 4 As shown in the figure, in a global round, each user first trains his local model based on local data, and then accumulates the gradient {g k,t} Perform layer pruning to obtain the accumulated local gradient after pruning in is the complete L-layer model gradient, is the cumulative gradient after layer pruning, l represents the set of retained layers L k All the numbers in . Then, we get The quantized accumulated gradient is transmitted to the parameter server through encoding. The parameter server performs model aggregation after decoding. At this time, the model aggregation is modified to be performed layer by layer:

[0142]

[0143] in is an indicator function of whether the lth layer of the local model of user k is retained. Through layer-by-layer aggregation, the parameter server updates the global model and sends it to each user to start the next round of training.

[0144] The quantization coding scheme proposed in this application is described in detail below.

[0145] Figure 5 A schematic flow chart of a quantization coding method for a distributed learning model provided in an embodiment of the present application. The method 500 is applicable to the above-mentioned architecture 300 and architecture 400. For the convenience of description below, the method 500 is exemplified by taking the interaction between the execution subject as a user device and a network device as an example. It can be understood that the user device can be a component of the terminal device (such as a chip or circuit), or the network device can also be a component of the network device (such as a chip or circuit), without limitation.

[0146] The user device may be understood as one of multiple user devices in the distributed learning system, and the network device may be understood as a network device in the distributed learning system, or the network device is a parameter server in the distributed learning system, for example, the network device may be a base station or a network node such as a regional center node. This application does not limit this.

[0147] Figure 5 The method 500 shown may include S510 to S560 , and each step in the method 500 is described in detail below.

[0148] S510: The user equipment determines a first neural model.

[0149] The first neural model is the first training model after layered pruning. The first neural model corresponds to a first accumulated gradient set, and the first accumulated gradient set includes accumulated gradients of each retained layer after layered pruning.

[0150] In one possible implementation, the user device performs hierarchical pruning on the trained local model to obtain a first neural model.

[0151] For example, in a federated learning scenario of homogeneous models, there are no restrictions on the configuration and training process of the local model of the user device. After the local model training is completed, the distributed user can perform layer pruning on the accumulated gradients of the local model parameters to obtain the first neural model.

[0152] After layer pruning is performed on the accumulated gradients of the local model parameters, the accumulated gradients of each retained layer constitute a first accumulated gradient set, that is, the first neural model corresponds to the first accumulated gradient set.

[0153] Among them, the user equipment can train the local model based on the local training configuration sent by the network device to obtain the cumulative gradient of the parameters of the local model.

[0154] Among them, the user equipment can perform probability-based layer pruning on the accumulated gradient of the local model parameters, and the embodiment of the present application does not limit the specific layer pruning scheme.

[0155] In another possible implementation, the network device performs global configuration, configures an initialization layer pruned sub-model for each user, and the user device performs local training on the initialization layer pruned sub-model to obtain a first neural model.

[0156] For example, in a federated learning scenario of heterogeneous models, due to differences in factors such as computing power of each user, the models deployed on distributed users are different. The network device can assign initialized pruned sub-models of different complexities to each user by setting a different number of layers. The user device locally trains the pruned sub-model to obtain the cumulative gradient of the parameters of the pruned sub-model, that is, the first neural model.

[0157] Among them, after the user equipment locally trains the pruned sub-model, the cumulative gradient of the pruned sub-model parameters is the cumulative gradient of the retained layer after layered pruning, and the cumulative gradients of each retained layer constitute a first cumulative gradient set, that is, the first neural model corresponds to the first cumulative gradient set.

[0158] Among them, the network device can allocate a layer-pruned sub-model to each user according to the layer complexity, wherein the allocation of layer pruning complexity comprehensively considers factors such as the user's local computing device, data volume, and transmission rate.

[0159] Among them, the user equipment can train the local model based on the local training configuration sent by the network device to obtain the cumulative gradient of the sub-model parameters after pruning.

[0160] It can be understood that in the above two implementation methods, no matter whether the user device performs layer pruning on the trained local model or the user device performs local training on the layer pruned model assigned by the network device, the obtained neural model (first neural model) is the cumulative gradient (first cumulative gradient set) of each retained layer after layer pruning.

[0161] In the present application, the quantization coding scheme is performed on the accumulated gradients of each retained layer after pruning of the layer.

[0162] S520: The user equipment quantizes the accumulated gradients in each reserved layer in the first accumulated gradient set to obtain a second accumulated gradient set.

[0163] In one possible implementation, the user equipment determines a quantization value and a modulus of the accumulated gradients of each retained layer; determines a quantization interval of each accumulated gradient of each retained layer according to the quantization value and the modulus of the accumulated gradients of each retained layer; and represents a second accumulated gradient set according to the quantization value, the modulus of the accumulated gradients of each retained layer, the quantization interval of each accumulated gradient, and a sign function.

[0164] The quantization value may be configured by the network device, or may be determined by the user device according to its own needs. This embodiment of the present application does not limit this.

[0165] Among them, it is necessary to quantify the accumulated gradients of all parameters in each layer and output the quantized results.

[0166] Exemplarily, the quantized value is represented by b, and the magnitude of the accumulated gradient of each retained layer is represented by Indicates that the quantization interval is numbered using express, represents a symbol function, wherein l is the lth layer of the reserved layer, k is the user equipment, and the second accumulated gradient set can be expressed by the following formula:

[0167]

[0168] in,

[0169] Wherein, s is the number of the quantization interval,

[0170] in, is the probability of random quantization,

[0171] The specific quantitative scheme is shown below. Figure 8 The scheme shown.

[0172] S530: The user equipment performs encoding processing on the second accumulated gradient set to obtain a first encoding sequence set.

[0173] In a possible implementation, the quantization value and the modulus of the accumulated gradient of each reserved layer are encoded and concatenated; for each accumulated gradient of each reserved layer, the quantization interval and the sign function are encoded and concatenated respectively, wherein the quantization interval and the sign function are located at the end of the codeword.

[0174] Exemplarily, the modulus of the accumulated gradient of each retained layer can be represented by Float32, occupying 32 bits, and the quantized value is cyclically Elias encoded.

[0175] It is determined that the encoding of all accumulated gradients is completed, and then a first encoding sequence set is obtained.

[0176] The specific encoding scheme is shown below Figure 8 The scheme shown.

[0177] S540: The user equipment sends a first coding sequence set to the network equipment.

[0178] S550: The network device receives a plurality of first coding sequence sets from a plurality of user equipments.

[0179] Each user equipment uploads its own first coding sequence set to the network device.

[0180] S560, the network device performs layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model.

[0181] After decoding the multiple first coding sequence sets, the network device merges the accumulated gradients layer by layer to obtain a new global model, that is, a second neural model.

[0182] The embodiments of the present application do not limit the specific scheme of layer-by-layer polymerization.

[0183] In the present application, each user device can download the second neural model to update the local model, that is, the second neural model is used as the local model. When the specified global round is completed, the training is terminated, and when the specified global round is not completed, the next round of federated cycle training is started.

[0184] Based on the above technical solution, for hierarchical pruning and aggregation, each layer is quantized, and the quantized parameters are encoded using Elias code as the basic coding mode. This quantization coding scheme has good adaptability to model parameter upload in hierarchical pruning mode. Combined with layer pruning technology, it can further reduce the computational workload of local training and the communication workload of model aggregation, and improve the communication and computing efficiency of the federated learning system.

[0185] Next, the quantization encoding process provided in the embodiment of the present application is described in detail.

[0186] Figure 6 A quantization coding flow chart provided in an embodiment of the present application is shown.

[0187] The solution of the flowchart 600 is applicable to a scenario of federated learning of homogeneous models, wherein the network device takes a parameter server as an example.

[0188] The specific steps are as follows:

[0189] S610, the parameter server performs global configuration, including system settings and model initialization.

[0190] S620, start the global round, first the parameter server distributes local training configuration and layer pruning probability to participating distributed users

[0191] S630, each participating user performs local training to obtain the cumulative gradient of the local model parameters:

[0192]

[0193] in Represents the cumulative gradient of the parameters of the lth layer in the neural network model.

[0194] S640, each participating user according to probability Perform layer pruning on the accumulated gradient to obtain the pruned local accumulated gradient:

[0195]

[0196] Where L k is the set of reserved layer numbers, the characteristic function reserved for the lth layer of the model of user k satisfy:

[0197]

[0198] S650, each participating user randomly quantizes the accumulated gradient after layer pruning to obtain

[0199] S660: Each participating user quantizes the accumulated gradient based on the Elias code Encode and upload to the parameter server;

[0200] S670: After decoding, the parameter server aggregates the received accumulated gradients layer by layer to obtain a new global model. The aggregation expressions of each layer are:

[0201]

[0202] S680, each user downloads the global model to update the local model;

[0203] S690, determine whether the specified global round is completed, if completed, end, otherwise jump to S620 to start the next round of federation cycle training.

[0204] Figure 7 Another quantization coding flow chart provided in an embodiment of the present application is shown.

[0205] The solution of the flowchart 700 is applicable to the scenario of federated learning of heterogeneous models, wherein the network device takes a parameter server as an example.

[0206] The specific steps are as follows:

[0207] S710, the parameter server performs system global configuration, according to the level complexity {L k}A pruned sub-model is allocated to each user, where the allocation of layer complexity takes into account factors such as the user's local computing device, data volume, and transmission rate.

[0208] S720, the parameter server distributes the training settings of the current global round.

[0209] S730, each user trains the local sub-model to obtain the accumulated gradient of the pruned sub-model

[0210] S740, each participating user quantifies the accumulated gradient to obtain

[0211] S750: Participating users quantize the accumulated gradient based on Elias code Encode and upload to the parameter server.

[0212] S760, the parameter server decodes the quantized accumulated gradient and aggregates the model layer by layer to obtain a new global model The aggregation expressions of each layer are:

[0213]

[0214] S770, the user downloads the corresponding layer of the global model to update the local model;

[0215] S780, determine whether the specified global round is completed, if completed, end, otherwise jump to S720 to start the next round of federation cycle training.

[0216] Next, the process of stochastic gradient quantization in S650 and S740 in the above flow is described in detail.

[0217] Figure 8 A quantization flow chart provided in an embodiment of the present application is shown.

[0218] The specific steps are as follows:

[0219] S810, input the accumulated gradient of the reserved layer of each user and quantization settings b.

[0220] S820, accumulate gradients according to the input retention layer and quantization setting b, the model of the accumulated gradient of the calculation layer

[0221] S830, for each parameter gradient, calculate the quantization interval number in which it is located

[0222] S840, calculate the random quantization probability according to the following formula:

[0223]

[0224] The random quantization probability is the distance from the value to the left end point of the quantization interval, and then the interval number is randomly set:

[0225]

[0226] S850, obtain the random quantization representation of each parameter gradient:

[0227]

[0228] where sgn(·) is the sign function;

[0229] S860, determine whether to quantize all parameter gradients in the layer, if yes, output the quantization result, otherwise jump to S830 to continue execution.

[0230] S870, output layer gradient Quantitative representation of

[0231] Available through b. To express.

[0232] Next, the encoding process in S660 and S750 in the above flow is described in detail.

[0233] Fig. 9 A coding flow chart provided in an embodiment of the present application is shown.

[0234] The specific steps are as follows:

[0235] S910, quantized representation of layer gradient after input pruning b.

[0236] S920, encode and concatenate the modulus and quantization setting of the layer gradient, use Float32 to represent the modulus of the gradient, occupy 32 bits, and perform cyclic Elias encoding on the integer b.

[0237] S930 , for each parameter gradient, encode and concatenate its random quantization interval number and sign bit respectively.

[0238] S940, concatenating the encoded result to the end of the codeword.

[0239] S950, determining whether the encoding of all accumulated gradients is completed, if yes, the process ends; otherwise, the process jumps to step S930 to continue.

[0240] S960: Output the encoding result of the lth layer quantization gradient.

[0241] In the above encoding process, the integers of the quantization setting and the interval number are cyclically Elias encoded.

[0242] The specific steps are:

[0243] Elias encoding input: integer N to be encoded;

[0244] Elias coding step 1: set 0 at the end of the codeword;

[0245] Elias encoding step 2: If N←1, stop and complete the encoding;

[0246] Elias coding step 3: put the binary representation of N in the codeword prefix;

[0247] Elias coding step 4: Let N equal the number of prefix codeword bits obtained in the previous step minus 1;

[0248] Elias encoding step 5: Return to step 2 and encode the prefix in a loop.

[0249] Fig.10 A schematic diagram of a codeword structure for layer-by-layer encoding provided in an embodiment of the present application is shown.

[0250] First, the module of the accumulated gradient of the reserved layer is represented by 32 bits, and the quantization setting parameter b is an integer, which is then represented by the Elias code, and then the interval number of the quantized accumulated gradient of each parameter in the layer is encoded in the form of the integer number of the Elias code and a 1-bit sign bit. Thus, the codeword structure designed by the present invention is formed.

[0251] In steps S670 and S760, the parameter server decodes the received codeword and recovers the quantized parameter gradient.

[0252] Exemplarily, the decoding process can be performed according to Fig.10 The codeword structure shown is performed sequentially, wherein the specific steps for decoding the cyclic Elias code are:

[0253] Elias decoding input: Elias codeword;

[0254] Elias decoding step 1: Initialize integer variable N←1 and prepare to read codewords from front to back;

[0255] Elias decoding step 2: If the next bit is 0, stop and output the current N to complete the decoding;

[0256] Elias decoding step 3: If the next bit is 1, read it plus the next N more bits, and use the integer value represented by the binary as the new N value, and return to step 2 to loop decoding.

[0257] Based on the above scheme, whether in the scenario of federated learning of homogeneous models or heterogeneous models, the accumulated gradients of parameters after layered pruning are quantized and encoded and uploaded to the parameter server; after decoding on the parameter server, the global model is updated through layer aggregation and sent to each distributed user for subsequent training.

[0258] Exemplarily, Table 1 shows the numerical results of layer pruning quantization coding in the isomorphic model scenario in this application.

[0259] The results are the image classification experimental results under the CIFAR-10 dataset.

[0260] Table 1

[0261]

[0262] As can be seen from Table 1, by setting different layer pruning probabilities and quantization parameters b, the amount of model parameters uploaded can be significantly reduced. The accuracy drop of the global model is small and controllable.

[0263] Exemplarily, Table 2 shows the numerical results of layer pruning quantization coding in the heterogeneous model scenario in this application.

[0264] The results are the image classification experimental results under the CIFAR-10 dataset.

[0265] Table 2

[0266]

[0267] As can be seen from Table 2, the layer pruning quantization coding scheme in the heterogeneous mode can significantly reduce the communication volume of model interaction and the amount of computation during local training. At the same time, under different quantization settings, the accuracy of the global model decreases less than that without quantization. In the scenario of heterogeneous user models, the quantization coding method based on layer pruning can simultaneously reduce the communication load and computational complexity of the system, and the algorithm performance loss brought about is controllable, effectively lowering the system performance threshold to support the deployment of federated learning applications.

[0268] Fig.11 A schematic diagram showing the variation of coding length with parameter quantity is shown. Fig.12 The figure shows the number of bits required for encoding each layer of the model. By comparing the quantization coding schemes, it can be seen that in the parameter scale of each layer of the general model, the layer pruning quantization coding scheme has a lower asymptotic code length upper limit than the original QSGD coding scheme, and the number of bits required for encoding the parameters of each layer when uploading the model is also lower. Based on this, the layer pruning quantization coding can better match the layered pruning mode and achieve lower communication overhead when uploading the model.

[0269] In this application, a corresponding quantization method is designed based on the layer pruning technology in federated learning, which can optimize the communication and computing efficiency of the system; for scenarios with isomorphic distributed user models, a joint layer pruning quantization method is used to exchange a small and controllable global model performance degradation for a significant reduction in the amount of communication when uploading the model; for scenarios with heterogeneous distributed user models, by matching layer pruning sub-models of different complexities and combining parameter quantization methods, the amount of computation for local training and the amount of communication for model transmission can be effectively reduced at the same time; for the accumulated gradient after layer pruning quantization, a corresponding encoding scheme is designed based on Elias code, which has a lower asymptotic code length upper bound under the setting of layer pruning, and the encoding results of the gradients of each layer show that the quantization encoding scheme for layer pruning when uploading the model requires fewer bits.

[0270] Based on the technical solution provided in this application, through a targeted quantization coding scheme, the computing and communication loads in system interactions are further reduced, the system efficiency of federated learning is optimized, and the requirements for device configuration performance in the actual deployment of federated learning are reduced.

[0271] Above, combined Figures 4 to 12 The method provided by the embodiment of the present application is described in detail. Figure 13 to Figure 14 The device provided in the embodiment of the present application is described in detail. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment, so the contents not described in detail can be referred to the method embodiment above, and for the sake of brevity, they will not be repeated here.

[0272] The device is used to implement the above-mentioned embodiments and related implementation methods, and the descriptions have been made no further. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0273] Fig.13 It is a structural diagram of a quantization coding device provided in an embodiment of the present application.

[0274] The device 1300 includes a transceiver unit 1310 and a processing unit 1320, wherein the transceiver unit 1310 can be used to implement corresponding communication functions, and the processing unit 1320 can be used to perform data processing.

[0275] Optionally, the transceiver unit 1310 may also be referred to as a communication interface or a communication unit, including a sending unit and / or a receiving unit. The transceiver unit 1310 may be a transceiver (including a transmitter and / or a receiver), an input / output interface (including an input and / or output interface), a pin or a circuit, etc. The transceiver unit 1310 may be used to perform the steps of sending and / or receiving in the above method embodiment.

[0276] Optionally, the processing unit 1320 may be a processor (may include one or more), a processing circuit with a processor function, etc., and may be used to execute other steps except sending and receiving in the above method embodiment.

[0277] Optionally, the device 1300 further includes a storage unit, which may be a memory, an internal storage unit (e.g., a register, a cache, etc.), an external storage unit (e.g., a read-only memory, a random access memory, etc.), etc. The storage unit is used to store instructions, and the processing unit 1320 executes the instructions stored in the storage unit so that the communication device executes the above method.

[0278] In one design, the apparatus 1300 may be used to perform the actions performed by the user equipment in the above method embodiments, such as the apparatus 1300 may be used to perform the actions performed by the user equipment in the above method 500. In this case, the apparatus 1300 may be a component of the user equipment, the transceiver unit 1310 is used to perform the transceiver-related operations on the user equipment side in the above method embodiments, and the processing unit 1320 is used to perform the processing-related operations of the user equipment in the above method embodiments.

[0279] For example, the processing unit 1320 is used to determine a first neural model, which is a local training model after hierarchical pruning. The first neural model corresponds to a first cumulative gradient set, and the first cumulative gradient set includes the cumulative gradients of each retained layer after hierarchical pruning; the processing unit 1320 is also used to quantize the cumulative gradients in each retained layer in the first cumulative gradient set to obtain a second cumulative gradient set, and the second cumulative gradient set includes the cumulative gradients after quantizing the cumulative gradients of each retained layer; the processing unit 1320 is also used to encode the second cumulative gradient set to obtain a first coding sequence set, and the first coding sequence set includes the quantized coding sequences of each retained layer; the transceiver unit 1310 is used to send the first coding sequence set.

[0280] It should be understood that the transceiver unit 1310 may also perform other operations performed by the user equipment in any of the methods 500 described above, which will not be described in detail here.

[0281] In one design, the device 1300 may be used to execute the actions executed by the network device in each of the above method embodiments, such as the device 1300 may be used to execute the actions executed by the network device in the above method 500. In this case, the device 1300 may be a component of the network device, the transceiver unit 1310 is used to execute the operations related to transceiving on the network device side in the above method embodiments, and the processing unit 1320 is used to execute the operations related to processing of the network device in the above method embodiments.

[0282] For example, the transceiver unit 1310 is used to receive multiple first coding sequence sets from multiple user devices, the first coding sequence sets include coding sequences after quantization of the retained layers corresponding to the first neural model, the multiple user devices correspond one-to-one to the multiple first coding sequence sets, and the multiple user devices are user devices in the distributed learning system; the processing unit 1320 is used to perform layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model, and the second neural model is used for local training of the multiple users.

[0283] It should be understood that the transceiver unit 1310 and the processing unit 1320 may also perform other operations performed by the network device in any of the above methods 500, which will not be described in detail here.

[0284] It should also be understood that the device 1300 here is embodied in the form of a functional unit. The term "unit" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit and / or other suitable components that support the described functions. In an optional example, those skilled in the art can understand that the device 1300 can be specifically a network device in the above-mentioned embodiments, and can be used to execute the various processes and / or steps corresponding to the network device in the above-mentioned method embodiments. To avoid repetition, it will not be repeated here.

[0285] The apparatus 1300 of each of the above-mentioned schemes has the function of implementing the corresponding steps executed by the device in the above-mentioned method, or the apparatus 1300 of each of the above-mentioned schemes has the function of implementing the corresponding steps executed by the access network device in the above-mentioned method. The function can be implemented by hardware, or the corresponding software can be implemented by hardware. The hardware or software includes one or more modules corresponding to the above-mentioned functions; for example, the transceiver unit can be replaced by a transceiver (for example, the sending unit in the transceiver unit can be replaced by a transmitter, and the receiving unit in the transceiver unit can be replaced by a receiver), and other units, such as the processing unit, can be replaced by a processor to respectively perform the sending and receiving operations and related processing operations in each method embodiment.

[0286] In addition, the transceiver unit 1310 may also be a transceiver circuit (for example, may include a receiving circuit and a sending circuit), and the processing unit may be a processing circuit.

[0287] It should be pointed out that Fig.13 The device in the embodiment may be a network element or device in the foregoing embodiment, or may be a chip or a chip system, such as a system on chip (SoC). The transceiver unit may be an input / output circuit or a communication interface; the processing unit may be a processor or a microprocessor or an integrated circuit integrated on the chip. This is not limited here.

[0288] Fig.14 It is a schematic diagram of a quantization coding architecture provided in an embodiment of the present application. Fig.14 The quantization coding device 1400 shown includes: a processor 1410, a memory 1420 and a transceiver 1430. The processor 1410 is coupled to the memory 1420 and is used to execute instructions stored in the memory 1420 to control the transceiver 1430 to send and / or receive signals.

[0289] It should be understood that the processor 1410 and the memory 1420 can be combined into a processing device, and the processor 1410 is used to execute the program code stored in the memory 1420 to implement the above functions. In specific implementation, the memory 1420 can also be integrated into the processor 1410, or independent of the processor 1410. It should be understood that the processor 1410 can also correspond to each processing unit in the previous quantization encoding device, and the transceiver 1430 can correspond to each receiving unit and sending unit in the previous quantization encoding device.

[0290] It should also be understood that the transceiver 1430 may include a receiver (or receiver) and a transmitter (or transmitter). The transceiver may further include an antenna, and the number of antennas may be one or more. The transceiver may also be a communication interface or interface circuit.

[0291] Specifically, the quantization coding device 1400 may correspond to the user equipment or the network device in the method 500 according to the embodiment of the present application. The communication device 1400 may include a unit of the method performed by the user equipment or a unit of the method performed by the network device in the method 500. It should be understood that the specific process of each unit performing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0292] When the communication device 1400 is a chip, the chip includes an interface unit and a processing unit, wherein the interface unit may be an input / output circuit or a communication interface; and the processing unit may be a processor or a microprocessor or an integrated circuit integrated on the chip.

[0293] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0294] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0295] The present application also provides a computer-readable medium on which a computer program is stored. When the computer program is executed by a computer, the functions of any of the above method embodiments are implemented.

[0296] The present application also provides a computer program product, which implements the functions of any of the above method embodiments when executed by a computer.

[0297] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0298] In the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0299] It should be understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0300] It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The names of all nodes and messages in this application are merely names set by this application for the convenience of description. The names in the actual network may be different. It should not be understood that this application limits the names of various nodes and messages. On the contrary, any name with the same or similar function as the node or message used in this application is regarded as the method or equivalent replacement of this application, and is within the scope of protection of this application.

[0301] It should also be understood that in the present application, "when", "if" and "if" all mean that the UE or base station will take corresponding actions under certain objective circumstances. It is not a time limit, and it does not require the UE or base station to make judgments when implementing it, nor does it mean that there are other limitations.

[0302] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0303] The term "at least one of..." or "at least one of..." herein refers to all or any combination of the listed items. For example, "at least one of A, B, and C" may refer to the following six situations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time. "At least one" herein refers to one or more. "More than one" refers to two or more.

[0304] It should be understood that the terms "includes", "comprising", "having" and their variations all mean "including but not limited to", unless specifically emphasized otherwise.

[0305] It should be understood that in various embodiments of the present application, the first, second and various digital numbers are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application. For example, to distinguish different information.

[0306] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0307] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the description of the corresponding processes and beneficial effects in the aforementioned method embodiments, and will not be repeated here.

[0308] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0309] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0310] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0311] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0312] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A quantization encoding method, characterized in that: Applied to a user device, the user device being one of a plurality of user devices in a distributed learning system, comprising: Determine a first neural model, where the first neural model is a first training model after hierarchical pruning, the first neural model corresponds to a first accumulated gradient set, and the first accumulated gradient set includes accumulated gradients of each retained layer after hierarchical pruning; quantizing the accumulated gradients in each reserved layer in the first accumulated gradient set to obtain a second accumulated gradient set, wherein the second accumulated gradient set includes the accumulated gradients after the accumulated gradients of each reserved layer are quantized; Performing encoding processing on the second accumulated gradient set to obtain a first encoding sequence set, wherein the first encoding sequence set includes encoding sequences obtained after quantization of each reserved layer; The first set of coded sequences is sent.

2. The method according to claim 1, characterized in that The determining of the first neural model comprises: The first neural model is obtained from a network device, where the first neural model is layered pruned by the network device, and the network device is a network device in the distributed learning system.

3. The method according to claim 2, characterized in that The first neural model is a hierarchical pruned neural model determined by the network device according to the hierarchical complexity of the user equipment.

4. The method according to any one of claims 1 to 3, characterized in that The step of quantizing the accumulated gradients of each reserved layer in the first accumulated gradient set to obtain a second accumulated gradient set includes: Determining the quantized value and the modulus of the accumulated gradient of each retained layer; Determining a quantization interval of each accumulated gradient of each reserved layer according to the quantization value and the modulus of the accumulated gradient of each reserved layer; The second accumulated gradient set is represented according to the quantization value, the modulus of the accumulated gradients of each retained layer, the quantization interval of each accumulated gradient, and a sign function.

5. The method according to claim 4, characterized in that The quantized value is represented by b, and the magnitude of the accumulated gradient of each retained layer is represented by Indicates that the quantization interval is numbered using express, represents a symbol function, wherein l is the lth layer of the reserved layer, k is the user equipment, and the second accumulated gradient set can be expressed by the following formula: in, Wherein, s is the number of the quantization interval, in, is the probability of random quantization, 6. The method according to any one of claims 1 to 5, characterized in that The second accumulated gradient set is encoded to obtain a first encoding sequence set, including: Performing encoding and splicing on the quantized value and the modulus of the accumulated gradient of each retained layer; For each accumulated gradient of each reserved layer, the quantization interval and the sign function are respectively encoded and concatenated, and the quantization interval and the sign function are located at the end of the codeword.

7. The method according to claim 6, characterized in that Circular Elias encoding is performed on the quantized value.

8. A quantization encoding method, characterized in that: Applied to a network device, the network device is a network device in a distributed learning system, the method comprises: Receiving a plurality of first coding sequence sets from a plurality of user devices, the first coding sequence sets comprising coding sequences quantized from a reserved layer corresponding to a first neural model, the plurality of user devices corresponding one-to-one to the plurality of first coding sequence sets, and the plurality of user devices being user devices in the distributed learning system; A second neural model is obtained by performing layer-by-layer aggregation according to the multiple first coding sequence sets, and the second neural model is used for local training of the multiple users.

9. The method according to claim 8, characterized in that The method further comprises: determining a first neural model according to layer-wise pruning; The first neural model is sent to the plurality of user devices.

10. The method according to claim 8, characterized in that The first neural model is a hierarchical pruned neural model determined by the multiple user devices according to their respective hierarchical complexities.

11. A quantization encoding device, characterized in that: include: A processing unit, configured to determine a first neural model, wherein the first neural model is a local training model after hierarchical pruning, the first neural model corresponds to a first accumulated gradient set, and the first accumulated gradient set includes accumulated gradients of each retained layer after hierarchical pruning; The processing unit is further configured to perform quantization processing on the accumulated gradients in each reserved layer in the first accumulated gradient set to obtain a second accumulated gradient set, wherein the second accumulated gradient set includes the accumulated gradients after the accumulated gradients of each reserved layer are quantized; The processing unit is further configured to perform encoding processing on the second accumulated gradient set to obtain a first encoding sequence set, wherein the first encoding sequence set includes quantized encoding sequences of each reserved layer; A transceiver unit is used to send the first coding sequence set.

12. The device according to claim 11, characterized in that The transceiver unit is further used to: The first neural model is obtained from a network device, where the first neural model is layered pruned by the network device, and the network device is a network device in the distributed learning system.

13. The device according to claim 12, characterized in that The first neural model is a hierarchical pruned neural model determined by the network device according to the hierarchical complexity of the user equipment.

14. The device according to any one of claims 11 to 13, characterized in that The processing unit is specifically used for: Determining the quantized value and the modulus of the accumulated gradient of each retained layer; Determining a quantization interval of each accumulated gradient of each reserved layer according to the quantization value and the modulus of the accumulated gradient of each reserved layer; The second accumulated gradient set is represented according to the quantization value, the modulus of the accumulated gradients of each retained layer, the quantization interval of each accumulated gradient, and a sign function.

15. The device according to claim 14, characterized in that The quantized value is represented by b, and the magnitude of the accumulated gradient of each retained layer is represented by Indicates that the quantization interval is numbered using express, represents a symbol function, wherein l is the lth layer of the reserved layer, k is the user equipment, and the second accumulated gradient set can be expressed by the following formula: in, Wherein, s is the number of the quantization interval, in, is the probability of random quantization, 16. The device according to any one of claims 11 to 15, characterized in that The processing unit is further configured to: Performing encoding and splicing on the quantized value and the modulus of the accumulated gradient of each retained layer; For each accumulated gradient of each reserved layer, the quantization interval and the sign function are respectively encoded and concatenated, and the quantization interval and the sign function are located at the end of the codeword.

17. The device according to claim 16, characterized in that The processing unit is further configured to perform cyclic Elias encoding on the quantized value.

18. A quantization encoding device, characterized in that: include: A transceiver unit, configured to receive a plurality of first coding sequence sets from a plurality of user devices, wherein the first coding sequence sets include coding sequences quantized from a reserved layer corresponding to a first neural model, the plurality of user devices correspond one-to-one to the plurality of first coding sequence sets, and the plurality of user devices are user devices in the distributed learning system; A processing unit is used to perform layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model, where the second neural model is used for local training of the multiple users.

19. The device according to claim 18, characterized in that The processing unit is further used to determine a first neural model according to hierarchical pruning; and the transceiver unit is further used to send the first neural model to the multiple user equipments.

20. The device according to claim 18, characterized in that The first neural model is a hierarchical pruned neural model determined by the multiple user devices according to their respective hierarchical complexities.

21. A system for distributed learning models, characterized in that: Including multiple user devices and network devices, Each of the plurality of user devices is used to determine a first neural model, where the first neural model is a local training model after hierarchical pruning, the first neural model corresponds to a first accumulated gradient set, and the first accumulated gradient set includes accumulated gradients of each retained layer after hierarchical pruning; Each user equipment is further configured to quantize the accumulated gradients in each reserved layer in the first accumulated gradient set to obtain a second accumulated gradient set, wherein the second accumulated gradient set includes the accumulated gradients obtained by quantizing the accumulated gradients of each reserved layer; Each user equipment is further configured to perform encoding processing on the second accumulated gradient set to obtain a first encoding sequence set, where the first encoding sequence set includes quantized encoding sequences of each reserved layer; Each user equipment is further configured to send the first coding sequence set; The network device is used to receive a plurality of first coding sequence sets from the plurality of user equipments, and the plurality of user equipments correspond to the plurality of first coding sequence sets one by one; The network device is used to perform layer-by-layer aggregation according to the multiple first coding sequence sets to obtain a second neural model, and the second neural model is used for local training of the multiple users.

22. A quantization encoding device, characterized in that: The device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the device executes the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 10.

24. A chip system, characterized in that: The chip system includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method as claimed in any one of claims 1 to 10.

Images