Communication method and communication device

Abstract

The embodiment of the invention provides a communication method and device. The method comprises the following steps: a base station determines to calculate a first importance weight, wherein the first importance weight is the importance weight of each category of data samples in a plurality of categories of data samples corresponding to terminal equipment; the base station sends first indication information to the terminal equipment, the first indication information is used for indicating the terminal equipment to send a first parameter, and the first parameter is used for the base station to calculate a first importance weight; the base station receives a first parameter sent by the terminal equipment; the base station calculates a first importance weight according to the first parameter; and the base station sends the first importance weight to the terminal equipment. According to the method, the problems of poor model precision, slow convergence and the like caused by non-independent identically distributed local data of each terminal device participating in federated training are solved.

Description

A communication method and a communication device

[0001] The embodiments of the present application relate to the field of communications, and in particular, to a communication method and a communication device.

[0002] As an emerging distributed learning framework, federated learning is generally considered to be a type of distributed system with high communication efficiency due to its concise model training paradigm and parameter transfer rules. The core idea of ​​federated learning is to build a global model based on virtual fusion data by performing distributed model training between multiple data sources with local data, without the need to exchange local individual or sample data, and only by exchanging model parameters or intermediate results.

[0003] However, in relevant theoretical, experimental research and engineering practice, non-independent and identically distributed local user data is considered to be a key challenge faced by federated learning. The non-independent and identically distributed characteristics of local data at the distribution level are mainly manifested in two aspects. First, the imbalance of training sample categories in local data. Second, the distribution of local data categories among distributed users (terminal devices) is inconsistent. Among them, the former affects the training effect of the local model, and the unbalanced sample category distribution easily leads to overfitting, poor generalization performance on the test set and other problems. The latter makes the local models of each distributed user show different training trends and tendencies, resulting in poor accuracy and convergence of the global model after federation integration.

[0004]

[0005] The embodiment of the present application provides a communication method and a communication device, which solves the problems of poor model accuracy and slow convergence caused by the non-independent and identically distributed local data of each terminal device participating in the federated training by introducing the importance weight sampling of sample data of each category of the terminal device and the interaction mechanism between the base station and the terminal device. The importance weight derived based on the relevant theory not only takes into account the uneven distribution of samples of different categories, but also takes into account the gradient Lipschitz coefficient that affects the convergence of the model, so that the pattern features of samples of different categories are learned by the local model while also ensuring that the deviation between local models is controllable.

[0006] In a first aspect, a communication method is provided, wherein a base station determines that a first importance weight needs to be calculated, the first importance weight being the importance weight of each category of data samples in multiple categories corresponding to a terminal device; the base station sends a first indication message to the terminal device, the first indication message is used to instruct the terminal device to send a first parameter, and the first parameter is used by the base station to calculate the first importance weight; the base station receives the first parameter sent by the terminal device; the base station calculates the first importance weight based on the first parameter; and the base station sends the first importance weight to the terminal device.

[0007] The communication method provided in the embodiment of the present application solves the problems of poor model accuracy and slow convergence caused by the non-independent and identically distributed local data of each terminal device participating in the federated training by introducing importance weighted sampling of sample data of each category of the terminal device and the interaction mechanism between the base station and the terminal device. The importance weight derived based on relevant theories not only takes into account the uneven distribution of samples of different categories, but also takes into account the gradient Lipschitz coefficient that affects the convergence of the model, so that the pattern features of samples of different categories are learned by the local model while also ensuring that the deviation between local models is controllable. In addition, the update of the importance weight is deployed in the base station, which can make full use of the computing power advantage of the base station and liberate the computing power of the terminal device.

[0008] In combination with the first aspect, in some implementations of the first aspect, the first parameter includes distribution of local data of the terminal device.

[0009] In combination with the first aspect, in some implementations of the first aspect, the first parameter includes distribution of local data of the terminal device and a gradient corresponding to the local data.

[0010] In combination with the first aspect, in some implementations of the first aspect, the base station determines a distribution of global data, where the distribution of the global data is a distribution of local data corresponding to a plurality of terminal devices associated with the base station when performing federated training;

[0011] The base station calculates a first importance weight according to the first parameter, including:

[0012] The base station calculates a first importance weight according to the first parameter and the distribution of the global data.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the first importance weight satisfies the following conditions:

[0014]

[0015] Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0016] In a second aspect, a communication method is provided, including: a terminal device receives first indication information sent by a base station, the first indication information is used to indicate that the first indication information is used to indicate that the terminal device sends a first parameter, and the first parameter is used by the base station to calculate a first importance weight; the terminal device sends the first parameter to the base station according to the first indication information; the terminal device receives the first importance weight sent by the base station; the terminal device trains the next local model according to the first importance weight.

[0017] The communication method provided in the embodiment of the present application solves the problems of poor model accuracy and slow convergence caused by the non-independent and identically distributed local data of each terminal device participating in the federated training by introducing importance weighted sampling of sample data of each category of the terminal device and the interaction mechanism between the base station and the terminal device. The importance weight derived based on relevant theories not only takes into account the uneven distribution of samples of different categories, but also takes into account the gradient Lipschitz coefficient that affects the convergence of the model, so that the pattern features of samples of different categories are learned by the local model while also ensuring that the deviation between local models is controllable. In addition, the update of the importance weight is deployed in the base station, which can make full use of the computing power advantage of the base station and liberate the computing power of the terminal device.

[0018] In combination with the second aspect, in some implementations of the second aspect, the first parameter includes distribution of local data of the terminal device.

[0019] In combination with the second aspect, in some implementations of the second aspect, the first parameter includes distribution of local data of the terminal device and a gradient corresponding to the local data.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the first importance weight satisfies the following conditions:

[0021]

[0022] Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0023] According to a third aspect, a communication method is provided, comprising: a base station sends second indication information to a terminal device, the second indication information instructing the terminal device to send a second parameter, the second parameter being used to calculate a second importance weight, the second importance weight being the importance weight of each category of data samples in multiple categories corresponding to the terminal device; the base station receives the second parameter sent by the terminal device; the base station determines that the second importance weight is calculated by the terminal device; the base station sends third indication information to the terminal device, the third indication information instructing the terminal device to calculate the second importance weight.

[0024] The communication method provided in the embodiment of the present application can locally update the importance weight for terminal devices with surplus computing power or for which the federal update timeliness is not strong, thereby reducing the pressure of system scheduling and further improving the overall efficiency.

[0025] In combination with the third aspect, in some implementations of the third aspect, the second parameter includes distribution of local data of the terminal device.

[0026] In combination with the third aspect, in certain implementations of the third aspect, the second parameter includes the distribution of local data of the terminal device and a gradient corresponding to the local data.

[0027] In combination with the third aspect, in certain implementations of the third aspect, the base station determines the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federal training; the base station sends the distribution of global data to the terminal device.

[0028] In conjunction with the third aspect, in certain implementations of the third aspect, the second importance weight satisfies the following conditions:

[0029]

[0030] Among them, p N,n represents the distribution of global data, which is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of multiple terminal devices, n represents the terminal device is the nth terminal device, α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0031] In a fourth aspect, a communication method is provided, including: a terminal device receives second indication information sent by a base station, the second indication information instructs the terminal device to send a second parameter to the base station, the second parameter is used to calculate a second importance weight, and the second importance weight is the importance weight of each category of data samples in multiple categories corresponding to the terminal device; the terminal device sends the second parameter to the base station according to the second indication information; the terminal device receives third indication information sent by the base station, the third indication information instructs the terminal device to calculate the second importance weight; the terminal device calculates the second importance weight according to the second parameter; the terminal device trains the next local model according to the second importance weight.

[0032] The communication method provided in the embodiment of the present application can locally update the importance weight for terminal devices with surplus computing power or for which the federal update timeliness is not strong, thereby reducing the pressure of system scheduling and further improving the overall efficiency.

[0033] In combination with the fourth aspect, in certain implementations of the fourth aspect, the second parameter includes distribution of local data of the terminal device.

[0034] In combination with the fourth aspect, in certain implementations of the fourth aspect, the second parameter includes the distribution of local data of the terminal device and a gradient corresponding to the local data.

[0035] In combination with the fourth aspect, in certain implementations of the fourth aspect, the base station determines the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federal training; the base station sends the distribution of global data to the terminal device.

[0036] In conjunction with the fourth aspect, in certain implementations of the fourth aspect, the second importance weight satisfies the following conditions:

[0037]

[0038] Among them, p N,n represents the distribution of global data, which is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of multiple terminal devices, n represents the terminal device is the nth terminal device, α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0039] In a fifth aspect, a method for calculating importance weights is provided, including: a first device calculates a gradient Lipschitz coefficient of each category of data samples in a plurality of categories of data samples in a terminal device according to the distribution of local data of the terminal device; the first device calculates an importance weight of each category of data samples in a plurality of categories of data samples according to the gradient Lipschitz coefficient of each category of data samples in a plurality of categories of data samples, and the importance weights of the plurality of categories of data samples are used for the terminal device to train a local model for the next time.

[0040] In combination with the fifth aspect, in some implementations of the fifth aspect, the first device is a base station.

[0041] In combination with the fifth aspect, in some implementations of the fifth aspect, the first device is a terminal device.

[0042] In conjunction with the fifth aspect, in certain implementations of the fifth aspect, the importance weight satisfies the following conditions:

[0043]

[0044] Among them, p N,n represents the distribution of global data, which is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of multiple terminal devices, n represents the terminal device is the nth terminal device, α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0045] In a sixth aspect, a communication device is provided, comprising a unit for executing each step of the communication method in the first aspect or the third aspect and its implementation method.

[0046] In one design, the communication device is a communication chip, which may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0047] In another design, the communication apparatus is a communication device (eg, a base station, etc.), and the communication chip may include a transmitter for sending information and a receiver for receiving information or data.

[0048] In a seventh aspect, a communication device is provided, comprising a unit for executing each step of the communication method in the second aspect or the fourth aspect and its implementation method.

[0049] In one design, the communication device is a communication chip, which may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0050] In another design, the communication apparatus is a communication device (eg, a terminal device, etc.), and the communication chip may include a transmitter for sending information and a receiver for receiving information or data.

[0051] In an eighth aspect, a communication device is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the communication device executes the communication method in the above-mentioned first to fourth aspects and their respective implementation methods.

[0052] Optionally, the number of the processors is one or more, and the number of the memories is one or more.

[0053] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0054] Optionally, the communication device also includes a transmitter (transmitter) and a receiver (receiver).

[0055] In the ninth aspect, a computer program product is provided, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the communication method in the above-mentioned first to fourth aspects and their respective implementation methods.

[0056] In the tenth aspect, a computer-readable medium is provided, which stores a computer program (also referred to as code, or instruction) which, when executed on a computer, enables the computer to execute the communication method in the above-mentioned first to fourth aspects and their respective implementation methods.

[0057] In the eleventh aspect, a chip system is provided, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a communication device equipped with the chip system executes the communication methods in the above-mentioned first to fourth aspects and their respective implementation methods.

[0058] Among them, the chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0059] Figure 1 is a schematic diagram of the traditional federated learning framework.

[0060] FIG. 2 is a schematic diagram of a system architecture provided in an embodiment of the present application.

[0061] FIG3 is an example of a communication method provided in an embodiment of the present application.

[0062] FIG4 is a method for a base station to calculate a first importance weight.

[0063] FIG. 5 is another example of a communication method provided in an embodiment of the present application.

[0064] FIG6 is another example of a communication method provided in an embodiment of the present application.

[0065] FIG. 7 is a schematic diagram of a process for a base station to calculate the importance weight of a terminal device

[0066] FIG8 is another example of a communication method provided in an embodiment of the present application.

[0067] FIG. 9 is an example of a schematic diagram of the effect of the communication method provided in an embodiment of the present application.

[0068] FIG. 10 is another example of a schematic diagram of the effect of the communication method provided in an embodiment of the present application.

[0069] FIG. 11 is an example of a communication device provided in an embodiment of the present application.

[0070] FIG. 12 is another example of a communication device provided in an embodiment of the present application.

[0071] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0073] In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically limited. The reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0074] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0075] The technical solutions of the embodiments of the present application can be applied to various communication systems, for example: Global System of Mobile communication (GSM) system, Code Division Multiple Access (CDMA) system, Wideband Code Division Multiple Access (WCDMA) system, General Packet Radio Service (GPRS), Long Term Evolution (LTE) system, LTE Frequency Division Duplex (FDD) system, LTE Time Division Duplex (TDD), Universal Mobile Telecommunication System (UMTS), Worldwide Interoperability for Microwave Access (WiMAX) communication system, fifth generation (5G) system or new radio (NR), etc.

[0076] The terminal device in the embodiments of the present application may refer to a user device, an access terminal, a user unit, a user station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent or a user device. The terminal device may also be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, a terminal device in a 5G network or a terminal device in an evolved Public Land Mobile Network (PLMN), etc., and the embodiments of the present application are not limited to this.

[0077] The key terms involved in this application are introduced below.

[0078] It should be noted that in the introduction of key terms, the distributed users may be terminal devices, and the central server may be a base station.

[0079] 1. Local model:

[0080] The local model is a model trained by each distributed user based on local data.

[0081] 2. Global model:

[0082] The global model is a model that is obtained by the central server integrating multiple local models for distribution.

[0083] 3. Model aggregation:

[0084] Model aggregation is the process of integrating the model parameters uploaded by distributed users during the federation update round to obtain the global model parameters.

[0085] 4. Gradient modulus:

[0086] The norm of the gradient parameter gradient is obtained by backpropagation during model training.

[0087] 5. Gradient Lipschitz coefficient of the local model:

[0088] For local models and the model trained centrally using all the data and any training sample z of class i on user k k,i , so that The minimum coefficient L k,i, where l(·) is the loss function of the deep learning task.

[0089] The following is an introduction to the federated learning model involved in this application.

[0090] As an emerging distributed learning framework, federated learning is generally considered to be a type of distributed system with high communication efficiency due to its concise model training paradigm and parameter transfer rules.

[0091] At the same time, the federated learning model is a model of deep coupling of communication and computing: on the one hand, the communication network serves distributed computing and provides the basic function of information transmission; on the other hand, intelligent algorithms can also empower the communication network and optimize system performance.

[0092] Therefore, distributed network computing systems represented by federated learning are also considered to be a core function of future communication networks. The basic framework of traditional federated learning is shown in Figure 1, which can be summarized into two basic steps: local distributed training and central periodic federated integration:

[0093] (1) Distributed users train local models based on local data;

[0094] (2) In the federated update round, distributed users upload local model parameters to the central server. After the central server aggregates the model, it transmits the new global model back to each distributed user for subsequent local training.

[0095] FIG1 shows the structure of a regional federation, in which the central server can be deployed on a regional base station to provide centralized services such as federation updates and overall system scheduling for distributed user terminal devices in the region.

[0096] Since model aggregation in common federated learning algorithms is integration at the parameter level, the training effect of the local model has an important impact on the performance and convergence speed of the global model. As a data-driven algorithm, the training data of deep learning models represented by neural networks largely determines the final training effect. Therefore, in the federated learning scenario, local data indirectly affects the global model, which in turn affects the communication and computing efficiency of the system. Non-independent and identically distributed local user data is considered to be a key challenge facing federated learning. The following will introduce several methods to combat the non-independent and identically distributed characteristics of local data.

[0097] 1. Data enhancement.

[0098] From the perspective of data enhancement, the non-independent and identically distributed characteristics of local data can be combated. For example, a smaller enhanced data, i.e., distributed data, can be shared with distributed users through a central server to combat the non-independent and identically distributed characteristics of local data.

[0099] However, data enhancement solutions require additional data transmission, which increases communication costs and may also have limitations such as data privacy.

[0100] 2. Model level.

[0101] At the model level, we can combat the non-IID characteristics of local data. For example, model distillation can modify the loss function of distributed user local model training to make the output of the local model similar to that of the global model, thereby reducing the difference between local models.

[0102] However, the solution for model distillation requires distributed users to additionally calculate the target model output locally, which increases the amount of computation on the local side of the distributed users.

[0103] 3. Selection of distributed users or data.

[0104] Add a selection module for distributed users or data under the federated learning framework.

[0105] In terms of distributed users, by selecting distributed users with better model performance to participate in model aggregation, the deviation brought by poor local models to the global model can be reduced. For example, the base station can solve the problem of non-independent and identically distributed to a certain extent through local data integration.

[0106] In terms of data, each distributed user can choose local data of better quality during training, which helps improve the performance of the local model.

[0107] 4. Resampling of data samples.

[0108] From the perspective of distributed user local training sample sampling, there is a federated learning method based on local training sample gradient dynamic sampling. This method defines a sample importance weight. In the tth training cycle, for the kth user and the training sample z in its local data, k,i , the calculation method of its importance weight is as follows:

[0109]

[0110] in is the input and output of the Lth layer of the neural network model in this back propagation, l(z k,i ; θt) is the neural network model θ of the training cycle t For sample z k,i loss function, such as cross entropy in classification problems or mean squared error in regression problems.

[0111] This method is performed at the beginning of local training of each distributed user. After each round of federated update, each participating distributed user uses the downloaded global model to calculate the importance weight of each sample in the local data according to the above formula. In the subsequent local training cycle, the data samples are sampled according to the importance weight for training and updating the local model.

[0112] However, this federated learning method based on dynamic sampling of local training sample gradients has many shortcomings.

[0113] First, the importance weights in this method are defined from an intuitive perspective. Under the global model updated according to the importance weights, samples with larger moduli of the gradient in back propagation are given higher importance weights, without considering the performance and convergence speed of the global model after resampling. Although this method has achieved certain performance improvements in actual experiments, it lacks theoretical support.

[0114] Second, the calculation of importance weights in this method depends on the local sample data of each group of distributed users. Each distributed user needs to download the updated global model and perform backpropagation calculations sample by sample. However, in some actual federated learning systems, especially lightweight mobile scenarios, the computing resources of distributed users are prioritized, and calculations on the distributed user side require more computing time, thereby reducing the overall efficiency of the system.

[0115] Third, the federated learning architecture corresponding to this method lacks interaction between distributed users and central servers, and fails to fully utilize global information or the computing power support provided by the central server.

[0116] FIG2 is a schematic diagram of the system architecture provided by the embodiment of the present application. As shown in FIG2, a multi-user collaborative federated learning system has a central server (e.g., a regional base station) and N (N>1) distributed users participating in the training. The training data is divided into M (M>1) categories according to the label. The distribution of local data of each distributed user is {p N,n}, the distribution of all data is {p n}.

[0117] The federated learning system shown in Figure 2 mainly operates in three stages. First, within the federated update round, in addition to aggregating the collected local models, the central server also assists in estimating the gradient Lipschitz coefficient of each category of samples based on the local models of each distributed user and the new global model obtained in this federated round, and calculates the importance weights; second, distributed users receive new importance weights while downloading the new global model; third, distributed users sample training data samples according to the importance weights when training local models.

[0118] FIG3 shows an example of a communication method provided by an embodiment of the present application. In the method, the base station may correspond to a central server, and the terminal device may correspond to a distributed user. As shown in FIG3 , the method includes:

[0119] S310: The base station determines to calculate a first importance weight.

[0120] Specifically, the first importance weight is the importance weight of each category of data samples in multiple categories of data samples corresponding to the terminal device.

[0121] It should be noted that the base station can calculate the first importance weight of the terminal device, and can also calculate the first importance weights of other terminal devices in the federated learning system related to the base station.

[0122] Optionally, the base station may calculate the first importance weights of all terminal devices in the federated learning system associated with the base station.

[0123] Optionally, the base station may calculate the first importance weights of some terminal devices in the federated learning system related to the base station. For example, the base station may calculate the first importance weights of some terminal devices by itself, and the base station may instruct other terminal devices to calculate their own first importance weights.

[0124] Optionally, the base station may determine whether to calculate the first importance weight of the terminal device based on one or more factors including the distribution of the base station's global data, system scheduling, communication link quality, the computing power of the base station, the computing power of the terminal device, etc.

[0125] The distribution of global data is the distribution of local data corresponding to multiple terminal devices related to the base station during federated training. In other words, the base station collects the distribution of local data corresponding to all terminal devices related to the base station during federated training, integrates them, and obtains the distribution of global data.

[0126] S320, the base station sends first indication information to the terminal device, where the first indication information is used to instruct the terminal device to send a first parameter, and the first parameter is used by the base station to calculate a first importance weight.

[0127] Correspondingly, the terminal device receives the first indication information.

[0128] Optionally, the first parameter includes distribution of local data of the terminal device.

[0129] Optionally, the first parameter also includes a gradient corresponding to local data of the terminal device. The gradient is calculated by the terminal device based on the local data.

[0130] S330, the terminal device sends a first parameter to the base station according to the first indication information.

[0131] Correspondingly, the base station receives the first parameter sent by the terminal device.

[0132] S340: The base station calculates a first importance weight according to the first parameter.

[0133] Optionally, the base station determines the distribution of global data according to the distribution of local data of multiple terminal devices associated with the base station during federal training.

[0134] Optionally, the base station calculates a first importance weight based on the first parameter and the global data.

[0135] FIG4 is a method for a base station to calculate a first importance weight. As shown in FIG4, the method includes:

[0136] S410, the base station calculates the gradient Lipschitz coefficient of each category of data samples in multiple categories of data samples in the terminal device according to the distribution of local data of the terminal device.

[0137] S420: The base station calculates the importance weight of each category of data samples among the multiple categories of data samples according to the gradient Lipschitz coefficient of the data samples of each category among the multiple categories of data samples.

[0138] It should be noted that in the communication method shown in Figure 5, the step of the terminal device calculating the second importance weight is similar to the step shown in Figure 4, and it is only necessary to replace the base station with the terminal device. It will not be repeated in the following description.

[0139] In one embodiment, the first importance weight satisfies the following conditions:

[0140]

[0141] Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0142] in, The gradient Hiplitz deviation for each category is Said represents the global model, Represents a local model.

[0143] S350, the base station sends the first importance weight to the terminal device.

[0144] Correspondingly, the terminal device receives the first importance weight sent by the base station.

[0145] S360: The terminal device trains the next local model according to the first importance weight.

[0146] The communication method provided in the embodiment of the present application solves the problems of poor model accuracy and slow convergence caused by the non-independent and identically distributed local data of each terminal device participating in the federated training by introducing importance weighted sampling of sample data of each category of the terminal device and the interaction mechanism between the base station and the terminal device. The importance weight derived based on relevant theories not only takes into account the uneven distribution of samples of different categories, but also takes into account the gradient Lipschitz coefficient that affects the convergence of the model, so that the pattern features of samples of different categories are learned by the local model while also ensuring that the deviation between local models is controllable. In addition, the update of the importance weight is deployed in the base station, which can make full use of the computing power advantage of the base station and liberate the computing power of the terminal device.

[0147] FIG5 shows another example of a communication method provided in an embodiment of the present application. As shown in FIG5 , the method includes:

[0148] S510, the base station sends second indication information to the terminal device, where the second indication information instructs the terminal device to send a second parameter, where the second parameter is used to calculate a second importance weight.

[0149] Correspondingly, the terminal device receives the second indication information sent by the base station.

[0150] Specifically, the second importance weight is the importance weight of each category of data samples in multiple categories of data samples corresponding to the terminal device.

[0151] Optionally, the second parameter includes distribution of local data of the terminal device.

[0152] Optionally, the second parameter also includes a gradient corresponding to local data of the terminal device. The gradient is calculated by the terminal device based on the local data.

[0153] S520, the terminal device sends a second parameter to the base station according to the second indication information.

[0154] Correspondingly, the base station receives the second parameter sent by the terminal device.

[0155] Optionally, the base station determines the distribution of global data according to the second parameter. The distribution of the global data may refer to the description of the distribution of global data in the method corresponding to FIG. 3 , which will not be described again.

[0156] S530: The base station determines that the terminal device calculates the second importance weight.

[0157] S540, the base station sends third indication information to the terminal device, where the third indication information instructs the terminal device to calculate the second importance weight.

[0158] Correspondingly, the terminal device receives the third indication information sent by the base station.

[0159] S550: The terminal device calculates a second importance weight according to a second parameter.

[0160] Optionally, the method may further include: the terminal device receives the distribution of global data sent by the base station, and calculates a second importance weight according to the second parameter and the distribution of the global data.

[0161] In some embodiments, the second importance weight satisfies the following conditions:

[0162]

[0163] Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, Γ N for The smallest non-negative value in .

[0164] S560, the terminal device trains the next local model according to the second importance weight.

[0165] The communication method provided in the embodiment of the present application can locally update the importance weight for terminal devices with surplus computing power or for which the federal update timeliness is not strong, thereby reducing the pressure of system scheduling and further improving the overall efficiency.

[0166] FIG6 shows another example of a communication method provided in an embodiment of the present application. As shown in FIG6 , the method includes:

[0167] S610: The system is initialized.

[0168] Specifically, the initialization includes the preset of system parameters, the initialization of the local model of the terminal device, the initialization of the importance weights of each category corresponding to the terminal device, and the configuration of training hyperparameters.

[0169] S620: The base station performs terminal-cloud interaction preset for the current global period.

[0170] Optionally, the central server may be a base station.

[0171] S630, each terminal device among the multiple terminal devices samples a preset number of samples of the local training data according to the importance weight for updating the local model.

[0172] For example, the process of training the local model based on the importance weight of the nth terminal device may be:

[0173] Step 6-1, the nth terminal device samples each type of data sample according to the importance weight to form a data batch for training the local model;

[0174] Step 6-2, the nth terminal device performs back propagation on the sampled data batches in turn and updates the local model;

[0175] In step 6-3, the nth terminal device checks whether the local training round is completed. If completed, the training ends. If not completed, the training goes to step 6-1 to continue the training process.

[0176] S640, after each of the multiple terminal devices completes a local training round, it uploads the local model to the base station.

[0177] S650, the base station aggregates the models according to the local models of each terminal device according to the preset federation update rule to obtain the global model of this time.

[0178] S660, the base station estimates the gradient Hiplitz coefficient of each category of data samples of each terminal device on the local model of the corresponding terminal device on a preset data set based on the local model of each terminal device and the global model.

[0179] The preset data set may be a pre-set data set that is independent of each terminal device, or may be a segment of data set intercepted from the data of each terminal device.

[0180] For example, the base station may estimate the gradient Hiplitz coefficient of each category of data samples of multiple categories of the n-th terminal device on the local model of the n-th terminal device.

[0181] S670: The base station calculates the importance weight according to the gradient Hiplitz coefficient.

[0182] In other words, the base station updates the importance weight according to the gradient Hiplitz coefficient.

[0183] For example, as shown in Figure 7, a schematic diagram of a process for a base station to calculate the importance weight of an n-th terminal device is shown. As shown in Figure 7, the process may be:

[0184] Step 710, the base station initializes the candidate parameter set to be empty and sets the category number to 0;

[0185] Step 720, the base station calculates the gradient Lipschitz coefficient deviation of the current category according to the formula If there is a deviation If it is less than 0, Add the candidate parameter set and increase the category label by one, otherwise just increase the category label by one;

[0186] Step 730, the base station determines whether to traverse all analogs, if yes, proceed to step 740, if no, jump to step 720;

[0187] Step 740: The base station selects the minimum non-negative value in the parameter candidate set as the optimal parameter Γ N , and then get the importance weight of each category of data samples in the nth terminal device

[0188] S680: The base station sends the current global model and the corresponding importance weight to each terminal device among the multiple terminal devices.

[0189] S690, the base station checks whether the global training round is completed. If completed, the model training is terminated. If not completed, the base station jumps to S620 to loop.

[0190] The method provided in the embodiment of the present application solves the problems of poor model accuracy and slow convergence caused by the non-independent and identically distributed local data of each terminal device by introducing importance weighted sampling of sample data of each category of terminal devices and the interaction mechanism between the base station and the terminal device. The importance weight derived based on relevant theories not only takes into account the uneven distribution of samples of different categories, but also takes into account the gradient Lipschitz coefficient that affects the convergence of the model, so that the pattern features of samples of different categories are learned by the local model while also ensuring that the deviation between local models is controllable. In addition, the update of the importance weight is deployed in the base station, which can make full use of the computing power advantage of the base station and liberate the computing power of the terminal device.

[0191] FIG8 shows another example of a communication method provided in an embodiment of the present application. As shown in FIG8 , the method includes:

[0192] S810: The system is initialized.

[0193] Specifically, the initialization includes the preset of system parameters, the initialization of the local model of the terminal device, the initialization of the importance weights of each category corresponding to the terminal device, and the configuration of training hyperparameters.

[0194] S820: The base station performs terminal-cloud interaction preset for the current global period.

[0195] S830, each terminal device among the multiple terminal devices samples a preset number of samples of the local training data according to the importance weight for updating the local model.

[0196] For example, the process of the nth terminal device training the local model based on the importance weight can refer to the description in step S630 and will not be repeated here.

[0197] S840, after each of the multiple terminal devices completes a local training round, it uploads the local model to the base station.

[0198] S850, the base station aggregates the models according to the local models of each terminal device according to the preset federation update rule to obtain the global model of this time.

[0199] S860, the base station sends instruction information to the terminal device, instructing the terminal device to calculate the importance weight.

[0200] It should be noted that this step is optional.

[0201] S870, the terminal device obtains the global model and auxiliary parameters of this time from the base station.

[0202] S880: The terminal device calculates the importance weight.

[0203] It should be noted that the calculation of the importance weight may refer to the description in S670, and only the base station needs to be replaced by the terminal device.

[0204] S890, the base station checks whether the global training round is completed. If completed, the model training ends. If not completed, jump to S820 to loop.

[0205] The communication method provided in the embodiment of the present application can locally update the importance weights of distributed users with surplus computing power or for whom the timeliness of federated updates is not strong, thereby reducing the pressure of system scheduling and further improving overall efficiency.

[0206] Figure 9 is an example of a schematic diagram of the effect of the communication method provided in an embodiment of the present application. As shown in Figure 9, it is a comparison curve of model accuracy with communication rounds. Under the set non-independent and identically distributed data distribution, after 25 rounds of federated updates, the classification accuracy of the global model on the standard test set is 64% on average, which is only about 6% lower than the model obtained by centralized training using all non-independent and identically distributed data.

[0207] FIG10 is an example of a schematic diagram of the effect of the communication method provided in an embodiment of the present application. As shown in FIG10, with respect to data sampling efficiency, the situation in which local data is partially involved in training is considered, that is, in each round of local training iteration, only a certain proportion of data is sampled for updating the local model. As the sampling ratio decreases, the global model classification accuracy of the federated learning algorithm designed in the embodiment of the present application decreases less, and the ability to cope with insufficient local device computing resources, computing time or insufficient samples is also significantly stronger than the random sampling method.

[0208] FIG11 is a schematic diagram of an example of a communication device provided in an embodiment of the present application. As shown in FIG11 , the communication device 1100 includes a transceiver unit 1110 and a processing unit 1120 .

[0209] In some embodiments, the communication device 1100 may be used to implement the functions of the terminal device in any of the above methods. For example, the communication device 1100 may correspond to the terminal device.

[0210] The communication device 1100 may be a terminal device and execute the steps executed by the terminal device in the above method embodiment. The transceiver unit 1110 may be used to support the communication device 1100 to communicate, such as executing the sending and / or receiving actions executed by the terminal device in the above method embodiment, and the processing unit 1120 may be used to support the communication device 1100 to execute the processing actions in the above method embodiment, such as executing the processing actions executed by the terminal device in the above method embodiment.

[0211] Optionally, the communication device may further include a storage unit 1130 (not shown in FIG. 11 ) for storing program codes and data of the communication device.

[0212] In some embodiments, the communication device 1100 may be used to implement the functions related to the base station in any of the above methods. For example, the communication device 1100 may correspond to a base station.

[0213] The communication device 1100 may be a base station and execute the steps performed by the base station in the above method embodiment. The transceiver unit 1110 may be used to support the communication device 1100 to communicate, for example, to perform the sending and / or receiving actions performed by the base station in the above method embodiment, and the processing unit 1120 may be used to support the communication device 1100 to perform the processing actions in the above method embodiment, for example, to perform the processing actions performed by the base station in the above method embodiment.

[0214] Optionally, the communication device may further include a storage unit 1130 (not shown in FIG. 11 ) for storing program codes and data of the communication device.

[0215] FIG12 is a schematic diagram of another example of a communication device provided in an embodiment of the present application. As shown in FIG12, the device 1200 includes: a transceiver 1210, a processor 1220, and a memory 1230. The memory 1230 is used to store instructions. The processor 1220 is coupled to the memory 1230 to execute the instructions stored in the memory to execute the method provided in the embodiment of the present application.

[0216] Specifically, the transceiver 1210 in the device 1200 may correspond to the transceiver unit 1110 in the device 1100 , and the processor 1220 in the communication device 1200 may correspond to the processing unit 1120 in the communication device 1100 .

[0217] It should be understood that the memory 1210 and the processor 1220 may be combined into one processing device, and the processor 1220 is used to execute the program code stored in the memory 1230 to implement the above functions. In specific implementation, the memory 1230 may also be integrated into the processor 1220, or independent of the processor 1220.

[0218] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0219] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0220] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0223] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0224] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

A model training method, characterized in that: include: The first device calculates the gradient Lipschitz coefficient of each category of data samples in a plurality of categories of data samples in the terminal device according to the distribution of local data of the terminal device; The first device calculates the importance weight of each category of data samples in the multiple categories of data samples based on the gradient Lipschitz coefficient of the data samples in each category of the multiple categories of data samples, and the importance weights of the data samples in the multiple categories are used by the terminal device to train the local model next time. The method according to claim 1, characterized in that The importance weights satisfy the following conditions: Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, The gradient Hiplitz deviation for each category is Said represents the global model, represents the local model, the N for The smallest non-negative value in . The method according to claim 1, characterized in that The first device is a base station. The method according to claim 1, characterized in that The first device is the terminal device. A communication method, characterized in that: include: The base station determines to calculate a first importance weight, where the first importance weight is an importance weight of each category of data samples in multiple categories of data samples corresponding to the terminal device; The base station sends first indication information to the terminal device, where the first indication information is used to instruct the terminal device to send a first parameter, and the first parameter is used by the base station to calculate the first importance weight; The base station receives the first parameter sent by the terminal device; Calculating, by the base station, the first importance weight according to the first parameter; The base station sends the first importance weight to the terminal device. The method according to claim 5, characterized in that The first parameter includes distribution of local data of the terminal device. The method according to claim 6, characterized in that The first parameter also includes a gradient corresponding to the local data of the terminal device. The method according to any one of claims 5 to 7, characterized in that The method also includes: the base station determines the distribution of global data, the distribution of the global data is the distribution of local data corresponding to multiple terminal devices associated with the base station when performing federated training; the base station calculates the first importance weight according to the first parameter, including: the base station calculates the first importance weight according to the first parameter and the distribution of the global data. The method according to any one of claims 5 to 8, characterized in that The first importance weight satisfies the following conditions: Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, the Γ N for The smallest non-negative value in . A communication method, characterized in that: include: The terminal device receives first indication information sent by the base station, where the first indication information is used to indicate that the first indication information is used to instruct the terminal device to send a first parameter, and the first parameter is used by the base station to calculate the first importance weight; The terminal device sends the first parameter to the base station according to the first indication information; The terminal device receives the first importance weight sent by the base station; The terminal device trains the next local model according to the first importance weight. The method according to claim 10, characterized in that The first parameter includes distribution of local data of the terminal device. The method according to claim 11, characterized in that The first parameter also includes a gradient corresponding to the local data of the terminal device. The method according to any one of claims 10 to 12, characterized in that The first importance weight satisfies the following conditions: Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, the Γ N for The smallest non-negative value in . A communication method, characterized in that: include: The base station sends second indication information to the terminal device, where the second indication information instructs the terminal device to send a second parameter, where the second parameter is used to calculate a second importance weight, where the second importance weight is an importance weight of each category of data samples in multiple categories corresponding to the terminal device; The base station receives the second parameter sent by the terminal device; The base station determines that the second importance weight is calculated by the terminal device; The base station sends third indication information to the terminal device, where the third indication information instructs the terminal device to calculate the second importance weight. The method according to claim 14, characterized in that The second parameter includes the distribution of local data of the terminal device. The method according to claim 15, characterized in that The second parameter also includes a gradient corresponding to the local data of the terminal device. The method according to any one of claims 14 to 16, characterized in that The method also includes: the base station determines the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station when performing federated training; and the base station sends the distribution of global data to the terminal device. The method according to any one of claims 14 to 17, characterized in that The second importance weight satisfies the following conditions: Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, the Γ N for The smallest non-negative value in . A communication method, characterized in that: include: The terminal device receives second indication information sent by the base station, where the second indication information instructs the terminal device to send a second parameter to the base station, where the second parameter is used to calculate a second importance weight, where the second importance weight is an importance weight of each category of data samples in multiple categories corresponding to the terminal device; The terminal device sends a second parameter to the base station according to the second indication information; the terminal device receives third indication information sent by the base station, and the third indication information indicates that the terminal device calculates the second importance weight; The terminal device calculates the second importance weight according to the second parameter; The terminal device trains the next local model according to the second importance weight. The method according to claim 19, characterized in that The second parameter includes the distribution of local data of the terminal device. The method according to claim 20, characterized in that The second parameter also includes a gradient corresponding to the local data of the terminal device. The method according to any one of claims 19 to 21, characterized in that The method also includes: the terminal device receives the distribution of global data sent by the base station, the distribution of the global data is the distribution of local data corresponding to multiple terminal devices associated with the base station when performing federated training; the terminal device calculates the second importance weight according to the second parameter, including: the terminal device calculates the second importance weight according to the second parameter and the distribution of the global data. The method according to any one of claims 19 to 22, characterized in that: The second importance weight satisfies the following conditions: Among them, the p N,n represents the distribution of global data, where the distribution of global data is the distribution of local data corresponding to multiple terminal devices associated with the base station during federated training. n is the distribution of local data corresponding to the terminal device, N represents the number of the plurality of terminal devices, n represents that the terminal device is the nth terminal device, and α N,n is the gradient Hiplitz deviation for each class, the Γ N for The smallest non-negative value in . A communication device, characterized in that: include: Memory, used to store program instructions and data; A processor, configured to be coupled to the memory and execute instructions in the memory to implement the method according to any one of claims 5 to 9, or to implement the method according to any one of claims 14 to 18. A communication device, characterized in that: include: Memory, used to store program instructions and data; A processor, configured to be coupled to the memory and execute instructions in the memory to implement the method according to any one of claims 10 to 13, or to implement the method according to any one of claims 19 to 23. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 5 to 13, or to execute the method according to any one of claims 14 to 23. A chip, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run a computer medium stored in the memory to execute the method according to any one of claims 5 to 13, or execute the method according to claim 14 or 23.

Images