Neural network training method, data processing method, device and storage medium

Abstract

The application relates to a neural network training method, a data processing method, a device and a storage medium. The neural network training method comprises the following steps: a server sends a corresponding first tensor core set to one or more terminal devices; the terminal device acquires the first tensor core set, determines a first neural network model according to the first tensor core set, determines a first target function according to local data and the first neural network model, determines a second target function according to the first target function, the first neural network model and the first tensor core set, obtains a trained first tensor core set according to the second target function, and sends the trained first tensor core set to the server; the server aggregates the trained first tensor core sets sent by the terminal devices, determines a tensor core set meeting a predetermined convergence condition, and determines a second neural network model. Therefore, the communication cost between the server and the terminal device can be saved, and a neural network model with better performance can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a neural network training method based on federated learning, a data processing method, an apparatus and a storage medium. BACKGROUND

[0002] In recent years, neural networks have been widely used in the field of computer vision (such as target detection, picture classification, semantic segmentation, etc.). High-quality data plays an irreplaceable important role in the training of neural networks, so different manufacturers pay more and more attention to data protection, which easily leads to the problem of data islands, that is, high-quality data does not circulate between different manufacturers. Based on this, federated learning (FL) has emerged. Federated learning, also known as federated machine learning, joint learning, alliance learning, etc., can effectively help multiple client devices to conduct data use and machine learning modeling under the requirements of user privacy protection, data security and government regulations. That is, federated learning can train neural networks using the data on each client under the premise of not sharing data between different clients, protecting user data privacy while improving the performance of neural networks.

[0003] There are currently various neural network training methods based on federated learning in the industry. For example, a federated learning method is called federated averaging (FedAvg). FedAvg generally includes a server and some clients in structure, and the technical process is that the server issues a neural network model to each client, each client trains the model respectively, the server receives the trained models on each client for aggregation, and the above process is iterated until the final model is obtained. The above method is suitable for the case where the dimensions of neural networks of each client are the same. If the dimensions of the trained neural networks on each client are different, the server can only perform a rough superposition and aggregation, and cannot better fuse the features of each neural network, so the performance of the obtained neural network is poor. Therefore, there is an urgent need for a neural network training method to make the performance of the obtained neural network model better. SUMMARY

[0004] Therefore, a neural network training method based on federated learning, a data processing method, an apparatus and a storage medium are proposed.

[0005] In a first aspect, embodiments of the present application provide a neural network training method based on federated learning. The method is used for a terminal device, and the method comprises: obtaining a first tensor core set from a server, the first tensor core set comprising decomposition results of tensors, the tensors comprising parameters of a first neural network model; determining the first neural network model according to the first tensor core set; determining a first target function corresponding to the first neural network model according to local data and the first neural network model; determining a second target function according to the first target function, the first neural network model and the first tensor core set; training the first tensor core set according to the second target function to obtain a trained first tensor core set; and sending the trained first tensor core set to the server, wherein the trained first tensor core set is used by the server to aggregate with trained first tensor core sets obtained from other terminal devices to obtain a tensor core set satisfying a predetermined convergence condition, and a second neural network model is determined according to the tensor core set satisfying the predetermined convergence condition.

[0006] According to embodiments of the present application, the data exchanged between the server and the terminal devices can be a tensor core set, the tensor core set can comprise decomposition results of tensors, and the tensors can comprise parameters of a neural network model, thereby reducing the communication volume in the exchange process. Each terminal device can train the tensor core set according to a target function determined according to local data, a neural network model and a tensor core set sent by the server, and each terminal device can thereby obtain a trained tensor core set and send the trained tensor core set to the server for aggregation of the tensor cores. Thus, even if the dimensions of the model parameters on each terminal device are different, the tensor cores obtained by decomposing the parameters corresponding to the tensors can still be fused with each other, and therefore the server can better fuse the features of models of different dimensions to determine a tensor core set satisfying a predetermined convergence condition, thereby determining a global neural network model on the server, and the performance of the model is better.

[0007] According to a first aspect, in a first possible implementation manner of the neural network training method based on federated learning, the method further comprises: training the first neural network model according to the first target function or the second target function to obtain a trained first neural network model.

[0008] Thus, the training of the first neural network model can be implemented, and a more personalized neural network model can be obtained on each terminal device.

[0009] In a second possible implementation of the neural network training method based on federated learning according to the first aspect or the first possible implementation of the first aspect, the server aggregates the trained first tensor core set obtained from other terminal devices to obtain a tensor core set satisfying a predetermined convergence condition, and further comprises: in a case where the server determines that the aggregated tensor core set does not satisfy the predetermined convergence condition, determining, by the server, a new first tensor core set according to the aggregated tensor core set; obtaining, from the server, the new first tensor core set, determining a new first neural network model according to the new first tensor core set or the trained first neural network model, and repeatedly performing the steps of determining the first target function corresponding to the first neural network model according to the local data and the first neural network model, and the subsequent steps until the predetermined convergence condition is satisfied, to obtain a tensor core set satisfying the predetermined convergence condition.

[0010] In this way, the neural network model with better performance can be obtained through the iterative process between the server and the terminal device.

[0011] In a third possible implementation of the neural network training method based on federated learning according to the first aspect or the first or second possible implementation of the first aspect, the aggregation comprises aggregating the trained first tensor core sets on the terminal devices according to a predetermined weight.

[0012] According to the embodiments of the present application, by aggregating the trained first tensor core sets on the terminal devices according to the predetermined weight, the server can adaptively adjust the number of aggregated tensor cores, so that the training process is more flexible.

[0013] In a fourth possible implementation of the neural network training method based on federated learning according to the first aspect or the first, second or third possible implementation of the first aspect, the first tensor core set corresponds to a preset neural network dimension on the terminal device.

[0014] In this way, even if the dimensions of the model parameters are different, the tensor cores obtained by decomposing the parameters can still be fused with each other, and the server can better fuse the features of models with different dimensions to determine the first tensor core set satisfying the predetermined convergence condition, so that the performance of the finally determined model is better.

[0015] In a fifth possible implementation of the neural network training method based on federated learning according to the first aspect or the first, second, third or fourth possible implementation of the first aspect, sending the trained first tensor core set to the server comprises: determining a second tensor core set from the trained first tensor core set according to an upload ratio; and sending the second tensor core set to the server.

[0016] According to the embodiment of the present application, by determining the second tensor core set according to the uploading proportion, the number of uploaded tensor cores can be adjusted as needed, saving the communication cost. And thus the personalized part of the tensor core of each terminal device can be retained without uploading, so that the neural network model of the terminal device locally can have better personalized ability.

[0017] In a sixth possible implementation manner of the neural network training method based on federated learning according to the fifth possible implementation manner of the first aspect, the method further comprises: obtaining the uploading proportion sent by the server, wherein the uploading proportion is related to the accuracy of the trained first tensor core set.

[0018] According to the embodiment of the present application, since the uploading proportion is related to the accuracy of the trained first tensor core set, the uploaded tensor core set is more concise, which can save the communication cost and make the server aggregate a better tensor core set for training, so as to obtain a neural network model with better performance.

[0019] In a seventh possible implementation manner of the neural network training method based on federated learning according to the first aspect or the first, second, third, fourth, fifth or sixth possible implementation manner of the first aspect, each decomposed tensor is any one of the following types of tensors: a tensor after CP decomposition, a tensor after Tucker decomposition, a tensor after tensor chain TT decomposition, and a tensor after tensor ring TR decomposition.

[0020] Thus, the structure of the tensor core can be flexibly selected as needed, so that the application range of the method of the present application is wider.

[0021] In a second aspect, embodiments of the present application provide a neural network training method based on federated learning. The method is used for a server, and the method comprises: sending, to one or more terminal devices, a first set of tensor cores corresponding to each terminal device, the first set of tensor cores comprising decomposition results of tensors, the tensors comprising parameters of a first neural network model; obtaining a trained first set of tensor cores sent by each terminal device, wherein the trained first set of tensor cores is obtained by training the first set of tensor cores according to a second objective function by the terminal device, the second objective function being determined according to a first objective function corresponding to the first neural network model, the first neural network model and the first set of tensor cores, the first objective function being determined according to local data of the terminal device and the first neural network model, and the first neural network model being determined according to the first set of tensor cores; aggregating each trained first set of tensor cores to obtain a set of tensor cores satisfying a predetermined convergence condition; and determining a second neural network model according to the set of tensor cores satisfying the predetermined convergence condition.

[0022] According to embodiments of the present application, the data exchanged between the server and each terminal device can be a set of tensor cores, the set of tensor cores can comprise decomposition results of tensors, and the tensors can comprise parameters of a neural network model, thereby reducing the communication volume in the exchange process. Each terminal device can train the set of tensor cores according to an objective function determined according to local data, a neural network model and a set of tensor cores sent by the server, and can thereby obtain a trained set of tensor cores and send the trained set of tensor cores to the server for aggregation of the tensor cores. The dimensions of the model parameters on each terminal device are different, but the tensor cores obtained by decomposing the parameters corresponding to the tensors can still be fused with each other, and therefore the server can better fuse the features of models of different dimensions to determine a set of tensor cores satisfying a predetermined convergence condition, thereby determining a global neural network model on the server, and the performance of the model is better.

[0023] According to the second aspect, in a first possible implementation manner of the neural network training method based on federated learning, aggregating each trained first set of tensor cores to obtain a set of tensor cores satisfying a predetermined convergence condition comprises: determining whether the aggregated set of tensor cores satisfies the predetermined convergence condition; in a case where the aggregated set of tensor cores does not satisfy the predetermined convergence condition, determining a new first set of tensor cores according to the aggregated set of tensor cores, and repeatedly performing the steps of sending, to one or more terminal devices, a first set of tensor cores corresponding to each terminal device and the subsequent steps until the aggregated set of tensor cores satisfies the predetermined convergence condition, to obtain a set of tensor cores satisfying the predetermined convergence condition.

[0024] Therefore, through the iterative process between the server and the terminal device, a neural network model with better performance can be obtained.

[0025] According to the second aspect or the first possible implementation manner of the second aspect, in a second possible implementation manner of the neural network training method based on the federated learning, the aggregation of the trained first tensor core set of each terminal device to obtain the tensor core set satisfying the predetermined convergence condition comprises: aggregating the trained first tensor core set of each terminal device according to a predetermined weight to obtain the tensor core set satisfying the predetermined convergence condition.

[0026] According to the embodiments of the present application, by aggregating the trained first tensor core set of each terminal device according to a predetermined weight, the server can adaptively adjust the number of aggregated tensor cores, and the training process is more flexible.

[0027] According to the second aspect or the first or second possible implementation manner of the second aspect, in a third possible implementation manner of the neural network training method based on the federated learning, the first tensor core set corresponds to a preset neural network dimension on each terminal device.

[0028] Therefore, even if the dimensions of the model parameters are different, the tensor cores obtained after the decomposition of the parameters can still be fused with each other, and the server can better fuse the features of models with different dimensions to determine the first tensor core set satisfying the predetermined convergence condition, so that the performance of the finally determined model is better.

[0029] According to the second aspect or the first, second or third possible implementation manner of the second aspect, in a fourth possible implementation manner of the neural network training method based on the federated learning, the trained first tensor core set sent by each terminal device is obtained, comprising: obtaining the second tensor core set sent by each terminal device, wherein the second tensor core set is determined by each terminal device from the trained first tensor core set according to an upload ratio.

[0030] According to the embodiments of the present application, by obtaining the second tensor core set determined from the trained first tensor core set according to the upload ratio, the number of uploaded tensor cores can be adjusted as needed, and the communication cost is saved. Therefore, each terminal device can retain the personalized part of the tensor core and does not upload it, so that the local neural network model of the terminal device can have better personalized ability.

[0031] In a fifth possible implementation form of the method of training a neural network based on federated learning according to the fourth possible implementation form of the second aspect, the method further includes: determining the upload ratio corresponding to each terminal device respectively according to the accuracy of the trained first set of tensor cores sent by each terminal device; and sending the upload ratio corresponding to each terminal device to each terminal device.

[0032] According to the embodiments of the present application, since the upload ratio is determined by the server according to the accuracy of the trained first set of tensor cores, the uploaded set of tensor cores is more concise, and the server can aggregate a more optimal set of tensor cores to perform training on the basis of saving communication costs, so that a neural network model with better performance is obtained.

[0033] In a sixth possible implementation form of the method of training a neural network based on federated learning according to the second aspect or the first, second, third, fourth or fifth possible implementation form of the second aspect, each tensor after decomposition is any one of the following types of tensors: a tensor after CP decomposition, a tensor after Tucker decomposition, a tensor after tensor chain TT decomposition, and a tensor after tensor ring TR decomposition.

[0034] In this way, the structure of the tensor core can be flexibly selected as needed, so that the method of the embodiments of the present application has a wider range of application.

[0035] In a third aspect, the embodiments of the present application provide a device for training a neural network based on federated learning. The device is used in a terminal device, and the device includes: a first obtaining module configured to obtain a first set of tensor cores from a server, the first set of tensor cores including decomposition results of tensors, and the tensors including parameters of a first neural network model; a first determining module configured to determine the first neural network model according to the first set of tensor cores; a second determining module configured to determine a first target function corresponding to the first neural network model according to local data and the first neural network model; a third determining module configured to determine a second target function according to the first target function, the first neural network model and the first set of tensor cores; a first training module configured to train the first set of tensor cores according to the second target function to obtain a trained first set of tensor cores; and a first sending module configured to send the trained first set of tensor cores to the server, wherein the trained first set of tensor cores is used by the server to aggregate with a trained first set of tensor cores obtained from other terminal devices to obtain a set of tensor cores satisfying a predetermined convergence condition, and a second neural network model is determined according to the set of tensor cores satisfying the predetermined convergence condition.

[0036] In a first possible implementation form of the neural network training apparatus based on federated learning according to the third aspect, the apparatus further includes a second training module configured to train the first neural network model according to the first target function or the second target function to obtain a trained first neural network model.

[0037] In a second possible implementation form of the neural network training apparatus based on federated learning according to the third aspect or the first possible implementation form of the third aspect, the server aggregates the trained first tensor core set obtained from other terminal devices to obtain a tensor core set satisfying a predetermined convergence condition, and the apparatus further includes: in a case where the server determines that the aggregated tensor core set does not satisfy the predetermined convergence condition, the server determines a new first tensor core set according to the aggregated tensor core set; the terminal device obtains the new first tensor core set from the server, determines a new first neural network model according to the new first tensor core set or the trained first neural network model, and repeatedly executes the steps of determining the first target function corresponding to the first neural network model according to the local data and the first neural network model, and the subsequent steps until the predetermined convergence condition is satisfied, to obtain the tensor core set satisfying the predetermined convergence condition.

[0038] In a third possible implementation form of the neural network training apparatus based on federated learning according to the third aspect or the first or second possible implementation form of the third aspect, the aggregation includes aggregating the trained first tensor core sets on the respective terminal devices according to a predetermined weight.

[0039] In a fourth possible implementation form of the neural network training apparatus based on federated learning according to the third aspect or the first, second or third possible implementation form of the third aspect, the first tensor core set corresponds to a preset neural network dimension on the terminal device.

[0040] In a fifth possible implementation form of the neural network training apparatus based on federated learning according to the third aspect or the first, second, third or fourth possible implementation form of the third aspect, the first sending module includes: determining a second tensor core set from the trained first tensor core set according to an upload proportion; and sending the second tensor core set to the server.

[0041] In a sixth possible implementation form of the neural network training apparatus based on federated learning according to the fifth possible implementation form of the third aspect, the apparatus further includes a third obtaining module configured to obtain the upload proportion sent by the server, where the upload proportion is related to an accuracy of the trained first tensor core set.

[0042] In a seventh possible implementation manner of the neural network training apparatus based on federated learning according to the third aspect or the first or the second or the third or the fourth or the fifth or the sixth possible implementation manner, the decomposed tensors are any one of the following types of tensors: CP decomposed tensors, Tucker decomposed tensors, tensor train TT decomposed tensors, and tensor ring TR decomposed tensors.

[0043] In a fourth aspect, the embodiments of the present application provide a neural network training apparatus based on federated learning. The apparatus is used for a server, and the apparatus comprises: a second sending module configured to send, to one or more terminal devices, a first tensor core set corresponding to each terminal device, the first tensor core set comprising a decomposition result of a tensor, and the tensor comprising parameters of a first neural network model; a second obtaining module configured to obtain a trained first tensor core set sent by each terminal device, wherein the trained first tensor core set is obtained by training the first tensor core set according to a second objective function by the terminal device, the second objective function is determined according to a first objective function corresponding to the first neural network model, the first neural network model and the first tensor core set, the first objective function is determined according to local data of the terminal device and the first neural network model, and the first neural network model is determined according to the first tensor core set; an aggregation module configured to aggregate each trained first tensor core set to determine a tensor core set satisfying a predetermined convergence condition; and a fourth determining module configured to determine a second neural network model according to the tensor core set satisfying the predetermined convergence condition.

[0044] In a first possible implementation manner of the neural network training apparatus based on federated learning according to the fourth aspect, the aggregation module comprises: a judging module configured to judge whether the aggregated tensor core set satisfies the predetermined convergence condition; and a determining module configured to, in a case where the aggregated tensor core set does not satisfy the predetermined convergence condition, determine a new first tensor core set according to the aggregated tensor core set, and repeatedly execute the sending, to one or more terminal devices, of a first tensor core set corresponding to each terminal device and the subsequent steps until the aggregated tensor core set satisfies the predetermined convergence condition, so as to obtain the tensor core set satisfying the predetermined convergence condition.

[0045] In a second possible implementation manner of the neural network training apparatus based on federated learning according to the fourth aspect or the first possible implementation manner of the fourth aspect, the aggregation module comprises: an aggregating module configured to aggregate the trained first tensor core set on each terminal device according to a predetermined weight to obtain the tensor core set satisfying the predetermined convergence condition.

[0046] In a third possible implementation manner of the neural network training apparatus based on federated learning according to the fourth aspect or the first or second possible implementation manner of the fourth aspect, the first tensor core set corresponds to a preset neural network dimension on each terminal device.

[0047] In a fourth possible implementation manner of the neural network training apparatus based on federated learning according to the fourth aspect or the first, second or third possible implementation manner of the fourth aspect, the second acquisition module comprises: acquiring a second tensor core set sent by each terminal device, the second tensor core set being determined by each terminal device in the trained first tensor core set according to an upload ratio.

[0048] In a fifth possible implementation manner of the neural network training apparatus based on federated learning according to the fourth possible implementation manner of the fourth aspect, the apparatus further comprises: a determination module configured to determine the upload ratio corresponding to each terminal device according to the accuracy of the trained first tensor core set sent by each terminal device; and a third sending module configured to send the upload ratio corresponding to each terminal device to each terminal device.

[0049] In a sixth possible implementation manner of the neural network training apparatus based on federated learning according to the fourth aspect or the first, second, third, fourth or fifth possible implementation manner of the fourth aspect, the decomposed tensors are any one of the following types of tensors: CP decomposed tensors, Tucker decomposed tensors, tensor chain TT decomposed tensors, and tensor ring TR decomposed tensors.

[0050] In a fifth aspect, an embodiment of the present application provides a data processing method, which comprises: acquiring input data; and inputting the input data into a second neural network model to obtain a processing result, wherein the second neural network model is obtained by the neural network training method based on federated learning of the first aspect or one or more of the possible implementation manners of the first aspect, or is obtained by the neural network training method based on federated learning of the second aspect or one or more of the possible implementation manners of the second aspect.

[0051] In a sixth aspect, an embodiment of the present application provides a neural network training apparatus based on federated learning, which comprises: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the neural network training method based on federated learning of the first aspect or one or more of the possible implementation manners of the first aspect, or the neural network training method based on federated learning of the second aspect or one or more of the possible implementation manners of the second aspect when executing the instructions.

[0052] In a seventh aspect, an embodiment of the present application provides a non-transitory computer readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the neural network training method based on federated learning of the first aspect or one or more of the possible implementation manners of the first aspect, or implement the neural network training method based on federated learning of the second aspect or one or more of the possible implementation manners of the second aspect.

[0053] In an eighth aspect, an embodiment of the present application provides a terminal device, which can execute the neural network training method based on federated learning of the first aspect or one or more of the possible implementation manners of the first aspect, or execute the neural network training method based on federated learning of the second aspect or one or more of the possible implementation manners of the second aspect.

[0054] In a ninth aspect, an embodiment of the present application provides a computer program product, including computer readable code or a non-transitory computer readable storage medium carrying computer readable code, when the computer readable code is run in an electronic device, a processor in the electronic device executes the neural network training method based on federated learning of the first aspect or one or more of the possible implementation manners of the first aspect, or executes the neural network training method based on federated learning of the second aspect or one or more of the possible implementation manners of the second aspect.

[0055] These and other aspects of the present application will become more fully understood from the following (a few) embodiment descriptions. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings, which are incorporated in and form a part of the specification, illustrate examples of the present application and, together with the description, serve to explain the principles of the present application.

[0057] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application.

[0058] Figure 2 A flowchart showing a neural network training method based on federated learning according to an embodiment of the present application.

[0059] Figure 3 A structural diagram of a neural network training apparatus based on federated learning according to an embodiment of the present application.

[0060] Figure 4 A structural diagram of a neural network training apparatus based on federated learning according to an embodiment of the present application.

[0061] Figure 5 A structural diagram of an electronic device 100 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0062] Various exemplary embodiments, features and aspects of the present application will be explained in greater detail below with reference to the accompanying drawings. The same reference numerals are used in different drawings to refer to functional or structural identical or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0063] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0064] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the application are omitted. It will be appreciated that those skilled in the art will be able to devise various modes of implementing the application without the benefits of these particulars. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the application.

[0065] With the wide application of neural networks, the data silo problem is generated. The federated learning is a machine learning framework proposed for the existence of the data silo problem, which can train the neural network with the data on each client under the premise of not sharing the data between different clients, protect the user data privacy while improving the performance of the neural network. The current federated learning method, such as federated averaging (FedAvg), is only applicable to the case where the dimensions of the neural networks on each client are the same. If the dimensions of the neural networks trained on each client are different, the neural networks can only be aggregated after being roughly stacked in the server, and the features of the neural networks on each client cannot be better fused, so the performance of the neural network obtained in the server is poor.

[0066] In view of this, the application provides a neural network training method based on federated learning. The neural network training method can be used in the process of data interaction between a server and terminal devices (i.e., clients) to achieve federated learning. In this process, the data exchanged between the server and the terminal devices can be a tensor core set, which can include decomposition results of tensors, and the tensors can include parameters of a neural network model, thereby reducing the communication volume. The terminal devices can determine the neural network model according to the tensor core set, train the tensor core set according to local data, the neural network model, and a target function corresponding to the neural network model, obtain a trained tensor core set, and send the trained tensor core set to the server for aggregation of the tensor cores. Thus, even if the dimensions of the model parameters on the terminal devices are different, the tensor cores obtained by decomposing the parameters can still be fused with each other, so that the server can better fuse the features of the models with different dimensions to determine a tensor core set that meets a predetermined convergence condition, thereby determining a global neural network model on the server. The model is the target result of federated learning and has better performance.

[0067] Figure 1 A schematic diagram of an application scenario according to an embodiment of the application is shown. As shown in FIG. 1, the application scenario includes a server and a plurality of terminal devices (i.e., clients). The server and the terminal devices can be connected to each other through a network. Figure 1As shown, the neural network training method based on federated learning in the embodiments of the present application can be used for the server 101 and at least one terminal device. For example, the at least one terminal device includes the terminal devices 111-114. The server 101 involved in the present application can be located in the cloud or locally, and can be a physical device or a virtual device such as a virtual machine, a container, etc., and has a wireless communication function. The terminal devices 111-114 of the present application can be touch screen or non-touch screen, or even without a screen. The touch screen can be controlled by clicking, sliding, etc. on the display screen through a finger, a stylus, etc. The non-touch screen device can be connected to a mouse, a keyboard, a touch panel, etc. input device, and the terminal device is controlled through the input device. The device without a screen, for example, can be a Bluetooth speaker without a screen. For example, the terminal device of the present application can be at least one of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or a smart city device. The embodiments of the present application do not specially limit the specific type of the terminal device. The server 101 and the terminal devices 111-114 can communicate through a wireless connection, for example, through 2G / 3G / 4G / 5G mobile communication technology, and Wi-Fi, Bluetooth, frequency modulation (FM), data radio, satellite communication, etc. wireless communication mode to realize interaction. The server 101 can be used to send the first tensor core set corresponding to each terminal device to the terminal devices 111-114 respectively, and obtain the trained first tensor core set sent by each terminal device, and aggregate each trained first tensor core set to determine the tensor core set meeting the predetermined convergence condition, and determine the second neural network model (i.e. the target result of federated learning) according to the tensor core set meeting the predetermined convergence condition.The terminal devices 111-114 can obtain the first tensor core set from the server 101, determine the first neural network model according to the first tensor core set, and determine the first target function corresponding to the first neural network model according to the local data and the first neural model. The terminal devices 111-114 can determine the second target function according to the first target function, the first neural network model, and the first tensor core set, train the first tensor core set according to the second target function, obtain the trained first tensor core set, and send the trained first tensor core set to the server 101. The first tensor core set exchanged between the server 101 and the terminal devices 111-114 can be referred to. Figure 1 The quadrilaterals shown in FIG. 2A (such as the quadrilaterals A (1) , A (2) , A (3) , A (4) may represent different types of tensor cores after decomposition, and the number, dimension, and size of each dimension (such as the letters a, b, c, and d shown in the figure, which can be referred to as the shape of the tensor core) of each type of tensor core can be different, where r can represent the rank of the corresponding tensor core. The first tensor core set on each terminal device can include the same or different types of tensor cores. For example, the terminal device 111 can include a tensor core corresponding to type A (1) , the terminal device 112 can include a tensor core corresponding to type A (1) , A (2) , the terminal device 113 can include a tensor core corresponding to type A (1) , A (2) , A (3) , and the terminal device 114 can include a tensor core corresponding to type A (1) , A (2) , A (3) , A (4) . The server 101 can aggregate the tensor core set of each type (including the union of the tensor cores of each type on each terminal device).

[0068] In a possible implementation, the second neural network model obtained by the neural network training method of the embodiments of the present application can be deployed in a server, and each first neural network model can also be trained on each terminal device to obtain each trained first neural network model for deployment on each terminal device, which can be used in text recognition, speech recognition, image recognition, and the like.

[0069] The following describes the neural network training method based on federated learning of the embodiments of the present application in detail based on Figure 1 . Referring to Figure 2 , a flowchart of the neural network training method based on federated learning according to an embodiment of the present application is shown. As Figure 2 indicated, the method includes the following steps.

[0070] In step S201, the server sends a first tensor core set corresponding to each terminal device to one or more terminal devices.

[0071] The first tensor core set includes decomposition results of tensors, and the tensors include parameters of the first neural network model, such as parameters of neurons in an intermediate layer.

[0072] The decomposition of the tensors can include any one of the following types: CP decomposition, Tucker decomposition, tensor train (TT) decomposition, and tensor ring (TR) decomposition. In different decomposition manners, the forms of the tensor cores obtained after the decomposition of the tensors can also be different. For example, different decomposition manners can obtain tensor cores of different orders, different sizes (which can refer to the shapes of the tensor cores), and different numbers. For example, CP decomposition can decompose an n-order tensor into multiplication of n n-1-dimensional tensor cores (the shapes of the n tensor cores can be different), TT decomposition can decompose an n-order tensor into multiplication of n-2 3-order tensor cores (the shapes of the n-2 tensor cores can be different) and 2 2-order tensor cores (the shapes of the 2 tensor cores can be different), and so on. Figure 1 A (1) A (2) A (3) A (4) may represent each type of tensor core obtained after decomposition in any of the above tensor decomposition manners. For example, in the CP decomposition manner, A (1) A (2) A (3) A (4) may correspond to tensor cores of different shapes (i.e., different sizes of dimensions) respectively. After decomposition, the ranks of the tensor cores can be the same, for example, all r.

[0073] It should be noted that the decomposition manner of the tensors can also be other decomposition manners other than the above types, which are not limited in the present application.

[0074] The first tensor core set can correspond to a preset neural network dimension on each terminal device. That is, the first tensor core set on each terminal device can include different types of tensor core sets, and the types of the tensor cores included in the first tensor core set sent by the server to each terminal device can be different. For example, in the case where the aggregated tensor core set obtained by the server 101 in the previous iteration includes tensor cores of types A (1) A (2) A (3) A (4) , if the type of the tensor core after decomposition under the preset neural network dimension on the terminal device 112 is A (1) A (2)If the types correspond, the server 101 can send the aggregated tensor core set corresponding to the type A (1) , A (2) to the terminal device 112. The process can be seen from the following formula (1):

[0075]

[0076] wherein {A (n)} can represent the tensor core set corresponding to the type, N can represent the total number of tensor core types in the first tensor core set corresponding to the terminal device, n can represent the serial number of the corresponding tensor core type, k can represent the serial number of the corresponding terminal device, and t can represent the number of iterations of the server, or in other words, the number of times the server sends the first tensor set to the terminal device. For example, t = 0 can represent that the corresponding first tensor core set is the initial first tensor core set.

[0077] In step S202, each terminal device obtains a first tensor core set from the server.

[0078] As described above, the first tensor core set obtained by each terminal device can be the same or different. For example, when obtaining the first tensor core set from the server for the first time (i.e., t = 0), the first tensor core set obtained by the terminal device 111 can be the first tensor core set obtained by the terminal device 112 can be the first tensor core set obtained by the terminal device 113 can be the first tensor core set obtained by the terminal device 114 can be respectively corresponding to the preset neural network dimensions on each terminal device.

[0079] In step S203, each terminal device determines a first neural network model according to the first tensor core set.

[0080] wherein each terminal device can store the architecture of the first neural network model itself, and since the tensor core in the first tensor core set includes the parameters of the first neural network model, the first neural network model can be determined based on the first tensor core set. The specific type of the first neural network model determined on each terminal device can be different, and the application does not limit this. The tasks of the first neural network model on each terminal device can also be different, for example, used for identifying different types of images, and the application does not limit this.

[0081] In step S204, each terminal device determines a first target function corresponding to the first neural network model according to local data and the first neural network model.

[0082] The local data can be data used by the terminal devices to train the first neural network model, and the local data on the terminal devices can be the same or different. The first target function can be used as a loss function when training the first neural network model, and the first target function can be determined according to a training task of the first neural network model. The application does not limit the manner of determining the first target function.

[0083] In step S205, each terminal device determines a second target function according to the first target function, the first neural network model, and the first tensor core set. The second target function can also be used as a loss function when training the first neural network model. In the second target function, factors of the first target function, the first neural network model, and the first tensor core set are introduced, so that the first tensor core set can also be trained when the second target function is used as a loss function for training.

[0084] An exemplary manner of determining the second target function can be seen from formula (2):

[0085]

[0086] wherein F k (w) can represent the second target function, w can represent the first tensor core set A (1) ⊙…⊙A (n-1) ⊙A (n+1) ⊙…⊙A (N) . f k (θ k ) can represent the first target function, θ k may represent the first neural network model.

[0087] In one possible implementation, the method further includes:

[0088] In step S206, the first tensor core set is trained according to the second target function to obtain a trained first tensor core set.

[0089] In one possible implementation, the method can further include:

[0090] In step S207, the first neural network model is trained according to the first target function or the second target function to obtain a trained first neural network model.

[0091] If the first neural network model is trained separately by using the first objective function in step S207, the first tensor core set can be trained according to the second objective function in step S206. The first neural network model and the first tensor core set can also be trained by using the second objective function at the same time, and then S207 can not be performed. In this process, the trained first tensor core set can be directly obtained without extracting the tensor from the first neural network model and then decomposing the tensor, so that the calculation resources are saved.

[0092] The manner of training the first neural network model by using the first objective function can be that the first neural network model is iteratively trained by using a loss optimization manner according to the first objective function until the first objective function value is less than a predetermined threshold, and the trained first neural network model is obtained. The loss optimization manner can be a gradient descent method, and the present application does not limit this. The manner of training the first neural network model and the first tensor core set by using the second objective function at the same time is described below.

[0093] In the case of training the parameters of the neural network model and the first tensor core set at the same time, the first tensor core set can be obtained in the tensor layer, and the first neural network model can include the tensor layer. In a possible implementation manner, the tensor layer can also be deployed in the terminal device independently of the first neural network model. The training manner can be that the first neural network model and the first tensor core set are iteratively trained by using a loss optimization manner according to the second objective function until the second objective function value is less than a predetermined threshold, and the trained first neural network model and the trained first tensor core set are obtained. The loss optimization manner can be a gradient descent method, and the present application does not limit this. The obtained trained first neural network model can be referred to formula (3) as follows:

[0094]

[0095] wherein θ k may represent the first neural network model, R d may represent a real number field, A (1) ⊙…⊙A (n-1) ⊙A (n+1) ⊙…⊙A (N) may represent the first tensor core set, and λ may represent a preset regularization parameter. It should be noted that in formula (2) and formula (3), only the distance calculated by using the 2-norm is taken as an example to introduce a manner of calculating the distance of θ k and w (that is, A (1) ⊙…⊙A (n-1) ⊙A (n+1) ⊙…⊙A (N) ) in the present embodiment, the distance of θ kThe distance mode with w can also be calculated by using other norms (such as 0 norm, 1 norm), or by using cosine (cos) distance, similarity function, and the like, and the application does not make any limitation in this regard. It should be noted that the application does not make any limitation on the training mode in steps S206 and S207, and the trained first neural network model and the trained first tensor core set can also be obtained by using other training modes.

[0096] Since the first tensor core set is trained in this process rather than directly obtained according to the parameters of the trained first neural network model, the parameters of the trained first neural network model and the trained first tensor core set can not be exactly the same, and each terminal device can obtain a more personalized model.

[0097] In step S208, each terminal device sends the trained first tensor core set to the server.

[0098] The terminal device can send all the tensor cores in the trained first tensor core set to the server, or send part of the tensor cores to the server (see the description in steps S2010-S2012), and the application does not make any limitation in this regard. For example, since each terminal device can perform different neural network tasks, each terminal device can also retain the personalized part (determined according to the differentiated part of each neural network task) of the tensor core and not send it to the server, so that the first neural network model finally trained by each terminal device can have higher personalization capability.

[0099] In step S209, the server obtains the trained first tensor core set sent by each terminal device.

[0100] The first tensor core set sent by each terminal device can include different types of tensor cores. The server obtains each first tensor core set (for example, including all the tensor cores in the first tensor core set) to update the first tensor core set stored in the server. The process can be referred to formula (4):

[0101]

[0102] Wherein, may represent the trained first tensor core set sent by the terminal device with the corresponding serial number k, and M can represent the number of iterations of the first tensor core set on the corresponding terminal device in step S206. N represents the number of tensor cores, may represent all the tensor cores in the trained first tensor core set on the corresponding terminal device.

[0103] In one possible implementation, the method further includes:

[0104] In step S2010, the server determines the upload ratio corresponding to each terminal device according to the accuracy of the trained first tensor core set sent by each terminal device.

[0105] The accuracy can represent the inference accuracy of the neural network model determined according to the trained first tensor core set, and the upload ratio can be used to indicate the upload number of each type of tensor core in the trained first tensor core set on the terminal device.

[0106] For example, when the accuracy is greater than 95%, the upload ratio corresponding to the corresponding terminal device is determined to be 1, when the accuracy is 90%-95%, the upload ratio of the corresponding terminal device is determined to be 0.5, and otherwise the upload ratio is 0. The upload ratio of each terminal device can also be determined in other manners, which is not limited in the present application.

[0107] In a possible implementation, the server can also determine the size p of the rank corresponding to each terminal device in the first tensor core set uploaded by each terminal device according to the accuracy of the trained first tensor core set sent by each terminal device, that is, in the case of a total rank number r, the tensor core corresponding to the first p ranks in the sorted order from small to large in each type of tensor core in the uploaded first tensor core set.

[0108] Therefore, the server can determine the number of tensor cores uploaded by each terminal device according to the training quality of each terminal device, so that the terminal device with better training quality can upload more tensor cores, and the performance of the finally obtained model is better.

[0109] In step S2011, the server sends the upload ratio corresponding to each terminal device to each terminal device.

[0110] The server can send the upload ratio corresponding to each terminal device to each terminal device once every iteration, or can send the upload ratio corresponding to each terminal device to each terminal device once every predetermined number of iteration training times, which is not limited in the present application.

[0111] In step S2012, each terminal device obtains the upload ratio sent by the server.

[0112] The terminal device can obtain the upload ratio once every server iteration, or can obtain the upload ratio once every predetermined number of server iterations, which is not limited in the present application.

[0113] In a possible implementation, the upload proportion and / or the size p of the rank can also be preset, for example, can be determined according to a predetermined training data amount, data quality (for example, for image data, the data quality can include the definition, resolution and the like of the image), for example, the greater the predetermined training data amount or predetermined data quality, the greater the corresponding upload proportion and / or the size p of the rank. Alternatively, the upload proportion and / or the size p of the rank can also be determined according to the sparsity or update change degree of the first tensor core set in the training process, which is not limited in the present application.

[0114] In a possible implementation, step S208 comprises: each terminal device determines a second tensor core set from the trained first tensor core set according to the upload proportion; and each terminal device sends the second tensor core set to the server.

[0115] In a possible implementation, in the iteration of the current time, each terminal device in step S208 can determine a second tensor core set from the trained first tensor core set according to the upload proportion obtained in steps S2010-S2012 in the last iteration.

[0116] In a possible implementation, the second tensor core set can include a predetermined number of tensor cores in the first tensor core set determined according to the upload proportion. For example, in the case where the first tensor core set includes tensor cores of types A (1) , A (2) , A (3) and the upload proportion is 0.5, half the number (which can be randomly determined) of tensor cores can be determined from the corresponding tensor cores of A (1) , A (2) , A (3) to form the second tensor core set.

[0117] In a possible implementation, in the case where the size p of the corresponding rank in the first tensor core set uploaded by each terminal device is determined, the second tensor core can include tensor cores of ranks 1-p in the first tensor core set. For example, in the case where the first tensor core set includes tensor cores of types A (1) , A (2) , A (3) and p is determined to be 4, tensor cores of ranks 1-4 can be determined from the corresponding tensor cores of A (1) , A (2) , A (3) to form the second tensor core set.

[0118] In a possible implementation, step S209 comprises: the server obtains the second tensor core set sent by each terminal device.

[0119] The process can refer to step S209 described above.

[0120] At step S2013, the server aggregates the trained first tensor core sets to determine a tensor core set satisfying a predetermined convergence condition.

[0121] In the process, the server can determine the tensor core set satisfying the predetermined convergence condition in an iterative manner. In one iteration, the server can aggregate the first tensor core sets corresponding to part of the terminal devices according to formula (5):

[0122]

[0123] wherein, β can represent a predetermined coefficient, and S may represent the tensor core set obtained by the server after aggregation in the t+1th iteration, which can include the parameters of the global neural network model generated on the server. β can represent a predetermined coefficient, and S t may represent the set of terminal devices corresponding to the first tensor core sets selected for aggregation at the tth iteration, which can include terminal devices pre-set. In one possible implementation, a predetermined weight can also be set for each terminal device, and the first tensor core sets on each terminal device can be aggregated by weighting. The manner in which the server aggregates the trained first tensor core sets can be average aggregation, or other manners, which are not limited in the present application. For example, the average value of the same type of tensor cores (e.g., each tensor core of type A (1) may be calculated to achieve average aggregation.

[0124] Referring to formula (5), when aggregating the trained first tensor core sets, the server can also retain the tensor core features in the last iteration, so that the global neural network model obtained has stronger self-adaptation capability.

[0125] In one possible implementation, at step S2013, the server aggregates the trained first tensor core sets on each terminal device according to a predetermined weight to determine a tensor core set satisfying a predetermined convergence condition.

[0126] In one possible implementation, at step S2013, the server aggregates the trained first tensor core sets on each terminal device according to a predetermined weight to determine a tensor core set satisfying a predetermined convergence condition. (1) may be calculated to achieve average aggregation.

[0127] The manner in which the tensor core set satisfying the predetermined convergence condition is determined according to the aggregated first tensor core set can be seen below.

[0128] In a possible implementation, the step S2013 includes: determining, by the server, whether the aggregated tensor core set satisfies the predetermined convergence condition; and in a case where the aggregated tensor core set does not satisfy the predetermined convergence condition, determining a new first tensor core set according to the aggregated tensor core set, and repeating the step of sending the first tensor core set corresponding to each terminal device to the one or more terminal devices and the subsequent steps, i.e., returning to step S201, until the aggregated tensor core set satisfies the predetermined convergence condition, to obtain the tensor core set satisfying the predetermined convergence condition.

[0129] The server can calculate a loss optimization function value of the aggregated tensor core set, and in a case where the loss optimization function value is less than a predetermined threshold (i.e., the predetermined convergence condition), determine the tensor core set at this time as the tensor core set satisfying the predetermined convergence condition. The application does not limit the specific form of the predetermined convergence condition.

[0130] In a case where the predetermined convergence condition is not satisfied, after each terminal device obtains a new first tensor core set from the server, a new first neural network model can be determined according to the new first tensor core set or the trained first neural network model, and the step of determining the first target function corresponding to the first neural network model according to the local data and the first neural network model and the subsequent steps are repeated, i.e., steps S204 and the subsequent steps are performed, until the server determines that the aggregated tensor core set satisfies the predetermined convergence condition, to obtain the tensor core set satisfying the predetermined convergence condition. That is, after the server issues a new first tensor core set, each terminal device can determine a first neural network model using the new first tensor core set, or can use the trained first neural network model obtained in the last iteration as the initial first neural network model in this iteration, and the application does not limit this.

[0131] The step S2014 includes: determining, by the server, the second neural network model according to the tensor core set satisfying the predetermined convergence condition.

[0132] The aggregated tensor core set satisfying the predetermined convergence condition can include parameters of the second neural network model, and the second neural network model can be determined according to the tensor core set satisfying the predetermined convergence condition.

[0133] According to the embodiments of this application, since the data exchanged between the server and each terminal device can be a set of tensor kernels, which can include the decomposition results of each tensor and the parameters of the neural network model, the amount of communication during the interaction is reduced. Each terminal device can train the tensor kernel set using the objective function determined based on local data, the neural network model, and the tensor kernel set issued by the server. Each terminal device can then obtain a trained tensor kernel set and send it to the server for aggregation. Thus, even if the dimensions of the model parameters on each terminal device are different, the tensor kernels obtained after the tensor decomposition of the corresponding parameters can still be fused together. Therefore, the server can better fuse the features of models with different dimensions to determine the set of tensor kernels that meet the predetermined convergence conditions, thereby determining the global neural network model on the server, and the performance of the obtained model is better.

[0134] Figure 3 This diagram illustrates a structural diagram of a neural network training apparatus based on federated learning according to an embodiment of this application. The apparatus is used in terminal devices, such as… Figure 3 As shown, the device includes:

[0135] The first acquisition module 301 is used to acquire a first tensor kernel set from the server. The first tensor kernel set includes the decomposition results of each tensor, and the tensors include the parameters of the first neural network model.

[0136] The first determining module 302 is used to determine the first neural network model based on the first tensor kernel set;

[0137] The second determining module 303 is used to determine the first objective function corresponding to the first neural network model based on local data and the first neural network model.

[0138] The third determining module 304 is used to determine the second objective function based on the first objective function, the first neural network model, and the first tensor kernel set;

[0139] The first training module 305 is used to train the first tensor kernel set according to the second objective function to obtain the trained first tensor kernel set;

[0140] The first sending module 306 is used to send the trained first tensor kernel set to the server, wherein the trained first tensor kernel set is used by the server to aggregate with the trained first tensor kernel set obtained from other terminal devices to obtain a tensor kernel set that meets a predetermined convergence condition, and a second neural network model is determined based on the tensor kernel set that meets the predetermined convergence condition.

[0141] According to the embodiment of the present application, the data exchanged between the server and each terminal device can be a tensor core set, the tensor core set can include decomposition results of tensors, and the tensors can include parameters of a neural network model, thereby reducing the communication amount in the exchange process. Each terminal device can train the tensor core set by using a target function determined according to local data, a neural network model, and the tensor core set issued by the server. Each terminal device can obtain a trained tensor core set and send the trained tensor core set to the server, and the server aggregates the tensor cores. Thus, even if the dimensions of the model parameters on each terminal device are different, the tensor cores obtained by decomposing the parameters corresponding to the tensors can be fused with each other, so that the server can better fuse the features of the models of different dimensions to determine a tensor core set that meets a predetermined convergence condition, thereby determining a global neural network model on the server, and the performance of the model is better.

[0142] In a possible implementation, the decomposed tensors are any one of the following types of tensors: a tensor decomposed by CP, a tensor decomposed by Tucker, a tensor decomposed by tensor chain TT, and a tensor decomposed by tensor ring TR.

[0143] Thus, the structure of the tensor core can be flexibly selected as needed, so that the device of the embodiment of the present application has a wider range of applications.

[0144] In a possible implementation, the device further includes a second training module configured to train the first neural network model according to the first target function or the second target function to obtain a trained first neural network model.

[0145] Thus, the training of the first neural network model can be implemented, and a more personalized neural network model can be obtained on each terminal device.

[0146] In a possible implementation, the first tensor core set corresponds to a preset neural network dimension on the terminal device.

[0147] Thus, even if the dimensions of the model parameters are different, the tensor cores obtained by decomposing the parameters corresponding to the tensors can be fused with each other, and the server can better fuse the features of the models of different dimensions to determine a first tensor core set that meets a predetermined convergence condition, so that the performance of the finally determined model is better.

[0148] In a possible implementation, the aggregation includes aggregating the trained first tensor core sets on each terminal device according to a predetermined weight.

[0149] According to an embodiment of the present application, by aggregating the first set of trained tensor cores according to the predetermined weight, the server can adaptively adjust the number of aggregated tensor cores, making the training process more flexible.

[0150] In a possible implementation, the first sending module 306 includes: determining a second set of tensor cores from the set of trained first tensor cores according to the upload proportion; and sending the second set of tensor cores to the server.

[0151] According to an embodiment of the present application, by determining the second set of tensor cores according to the upload proportion, the number of uploaded tensor cores can be adjusted as needed, saving communication costs. Moreover, the personalized part of the tensor cores can be retained by each terminal device without uploading, so that the local neural network model of the terminal device can have better personalized capabilities.

[0152] In a possible implementation, the apparatus further includes a third obtaining module configured to obtain an upload proportion sent by the server, wherein the upload proportion is related to the accuracy of the set of trained first tensor cores.

[0153] According to an embodiment of the present application, since the upload proportion is related to the accuracy of the set of trained first tensor cores, the uploaded set of tensor cores is more concise, and the server can obtain a better set of tensor cores for training on the basis of saving communication costs, thereby obtaining a neural network model with better performance.

[0154] In a possible implementation, the server aggregates the set of trained first tensor cores obtained from other terminal devices to obtain a set of tensor cores that meets a predetermined convergence condition, and further includes: in a case where the server determines that the aggregated set of tensor cores does not meet the predetermined convergence condition, determining, by the server, a new set of first tensor cores according to the aggregated set of tensor cores; obtaining, from the server, the new set of first tensor cores, determining a new first neural network model according to the new set of first tensor cores or the trained first neural network model, and repeatedly performing the steps of determining the first target function corresponding to the first neural network model according to the local data and the first neural network model and the subsequent steps until the predetermined convergence condition is met, to obtain a set of tensor cores that meets the predetermined convergence condition.

[0155] Thus, through the iterative process between the server and the terminal device, a neural network model with better performance can be obtained.

[0156] Figure 4 A structural diagram of a neural network training apparatus based on federated learning according to an embodiment of the present application is shown. The apparatus is used for a server, as shown in the figure, which includes: Figure 4 ​

[0157] The second sending module 401 is configured to send, to one or more terminal devices, a first tensor core set corresponding to each terminal device, where the first tensor core set comprises decomposition results of tensors, and the tensors comprise parameters of a first neural network model.

[0158] The second obtaining module 402 is configured to obtain a trained first tensor core set sent by each terminal device, where the trained first tensor core set is obtained by training the first tensor core set according to a second objective function, the second objective function is determined according to a first objective function corresponding to the first neural network model, the first neural network model and the first tensor core set, the first objective function is determined according to local data of the terminal device and the first neural network model, and the first neural network model is determined according to the first tensor core set.

[0159] The aggregation module 403 is configured to aggregate the trained first tensor core sets to determine a tensor core set meeting a predetermined convergence condition.

[0160] The fourth determining module 404 is configured to determine a second neural network model according to the tensor core set meeting the predetermined convergence condition.

[0161] According to the embodiments of the present application, the data exchanged between the server and the terminal devices can be tensor core sets, the tensor core sets can comprise decomposition results of tensors, and the tensors can comprise parameters of neural network models, thereby reducing the communication amount in the exchange process. The terminal devices can train the tensor core sets according to the objective function determined according to the local data, the neural network model and the tensor core set sent by the server, and can obtain the trained tensor core sets and send the trained tensor core sets to the server for aggregation of the tensor cores. Thus, the model parameters on the terminal devices have different dimensions, but the tensor cores obtained by decomposing the parameters corresponding to the tensors can still be fused with each other, so that the server can better fuse the features of the models with different dimensions to determine the tensor core set meeting the predetermined convergence condition, thereby determining the global neural network model on the server, and the performance of the model is better.

[0162] In a possible implementation, the decomposed tensors are any one of the following types of tensors: tensors decomposed by CP, tensors decomposed by Tucker, tensors decomposed by tensor chain TT, and tensors decomposed by tensor ring TR.

[0163] Thus, the structure of the tensor cores can be flexibly selected as needed, so that the device of the embodiments of the present application has a wider application range.

[0164] In a possible implementation, the first tensor core set corresponds to a preset neural network dimension on each terminal device.

[0165] Therefore, even if the dimensions of the model parameters are different, the tensor cores obtained after tensor decomposition of the parameters can be fused with each other, the server can better fuse the features of models of different dimensions to determine a first tensor core set that satisfies a predetermined convergence condition, and the finally determined model has better performance.

[0166] In a possible implementation, the aggregation module 403 comprises: aggregating the first tensor core sets trained on each terminal device according to a predetermined weight to obtain a tensor core set that satisfies a predetermined convergence condition.

[0167] According to an embodiment of the present application, by aggregating the first tensor core sets trained on each terminal device according to a predetermined weight, the server can adaptively adjust the number of aggregated tensor cores, and the training process is more flexible.

[0168] In a possible implementation, the second acquisition module 402 comprises: acquiring a second tensor core set sent by each terminal device, wherein the second tensor core set is determined by each terminal device from the trained first tensor core set according to an upload ratio.

[0169] According to an embodiment of the present application, by acquiring the second tensor core set and determining the second tensor core set from the trained first tensor core set according to an upload ratio, the number of uploaded tensor cores can be adjusted as needed, and the communication cost is saved. Therefore, each terminal device can retain personalized tensor cores and does not upload them, so that the local neural network model of the terminal device has better personalized ability.

[0170] In a possible implementation, the apparatus further comprises: a determination module configured to determine the upload ratio corresponding to each terminal device according to the accuracy of the trained first tensor core set sent by each terminal device; and a third sending module configured to send the upload ratio corresponding to each terminal device to each terminal device.

[0171] According to an embodiment of the present application, since the upload ratio is determined by the server according to the accuracy of the trained first tensor core set, the uploaded tensor core set is more concise, the server can aggregate a better tensor core set to perform training on the basis of saving communication cost, and thus a neural network model with better performance is obtained.

[0172] In a possible implementation, the aggregation module 403 includes: determining whether the aggregated tensor core set meets the predetermined convergence condition; in a case where the aggregated tensor core set does not meet the predetermined convergence condition, determining a new first tensor core set according to the aggregated tensor core set, and repeatedly performing the step of sending the first tensor core set corresponding to each terminal device to the one or more terminal devices and the subsequent steps until the aggregated tensor core set meets the predetermined convergence condition, to obtain a tensor core set meeting the predetermined convergence condition.

[0173] In this way, a neural network model with better performance can be obtained through an iterative process between the server and the terminal device.

[0174] Embodiments of the present application provide a data processing method, which includes: obtaining input data; inputting the input data into a second neural network model to obtain a processing result, wherein the second neural network model is obtained by the above method.

[0175] Embodiments of the present application provide a neural network training device based on federated learning, which includes: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

[0176] Embodiments of the present application provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above method.

[0177] Embodiments of the present application provide a computer program product, including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0178] Figure 5 A structural diagram of an electronic device 100 according to an embodiment of the present application is shown. As shown in the figure, the electronic device 100 can be the server 101 described above, or the terminal device 111-114 described above, and performs the functions of the server or the terminal device in the neural network training method based on federated learning shown in any one of the above methods. The electronic device 100 includes at least one processor 1801, at least one memory 1802, and at least one communication interface 1803. In addition, the electronic device can also include general components such as antennas, which are not described in detail here. Figure 5 Figures 1-2 The functions of the server or the terminal device in the neural network training method based on federated learning shown in any one of the above methods are performed.

[0179] The various constituent components of the electronic device 100 will be described in detail below. Figure 5

[0180] ​​The processor 1801 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling program execution of the above solutions. The processor 1801 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0181] The communication interface 1803 is configured to communicate with other electronic devices or communication networks, such as Ethernet, a radio access network (RAN), a core network, a wireless local area network (WLAN), etc. For example, referring to Figure 1 , the electronic device 100 is a server 101, and the communication interface 1803 can be configured to communicate with the terminal devices 111-114. For another example, the electronic device 100 is a terminal device 111-114, and the communication interface 1803 can be configured to communicate with the server 101.

[0182] The memory 1802 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

[0183] The memory 1802 is configured to store application program codes for implementing the above solutions, and the processor 1801 is configured to control the execution of the application program codes.

[0184] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0185] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital video disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium can be any tangible storage medium which can retain and store instructions for use by an instruction execution device.

[0186] Computer readable program instructions or code can be downloaded to a starting device, or to another computer or device from an external computer or storage device through a network, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0187] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0188] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0189] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable program instructions can also be downloaded to a computer, other programmable data processing apparatus, or other device from a computer readable storage medium or to an external computer or external storage device via a data signal that can be transmitted for example via a wired medium or a wireless medium such as the Internet or Wireless Application Protocol (WAP) signaling.

[0190] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0191] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts.

[0192] It is also important to note that each of the boxes in the block diagrams and / or flowchart illustrations, and combinations of boxes in the block diagrams and / or flowchart illustrations, can be implemented by hardware, for example, circuitry or an ASIC (Application Specific Integrated Circuit), or can be implemented by a combination of hardware and software, such as firmware or the like.

[0193] Although the present application has been described in connection with various embodiments thereof, it will be understood that other modifications can be made by those skilled in the art to the application described herein without departing from the scope of the application as defined by the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The reference signs in the claims should not be construed as limiting the scope of the application.

[0194] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. Use of the terms "preferably," "preferred," "desired," etc., are intended to present a choice between one embodiment and another, and are not intended to limit the context in which the terms are used. The choice of terms is intended to best explain the principle, practical application or improvement over the technology in the field that the various embodiments are directed to, or to enable others skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A neural network training method based on federated learning, characterized in that, The method is used in a terminal device, and the method includes: Obtain a first set of tensor kernels from the server. The first set of tensor kernels includes the decomposition results of each tensor, and the tensors include the parameters of the first neural network model. The first neural network model is determined based on the first set of tensor kernels; Based on local data and the first neural network model, determine the first objective function corresponding to the first neural network model; The second objective function is determined based on the first objective function, the first neural network model, and the first tensor kernel set; Based on the second objective function, the first tensor kernel set is trained to obtain a trained first tensor kernel set. The trained first tensor kernel set is sent to the server, wherein the trained first tensor kernel set is used by the server to aggregate with the trained first tensor kernel set obtained from other terminal devices to obtain a tensor kernel set that satisfies a predetermined convergence condition, and the second neural network model is determined based on the tensor kernel set that satisfies the predetermined convergence condition.

2. The method according to claim 1, characterized in that, The method further includes: The first neural network model is trained according to the first objective function or the second objective function to obtain a trained first neural network model.

3. The method according to claim 1 or 2, characterized in that, The server aggregates the pre-trained first tensor kernel set obtained from other terminal devices to obtain a tensor kernel set that satisfies a predetermined convergence condition, and further includes: If the server determines that the aggregated tensor kernel set does not meet the predetermined convergence condition, the server determines a new first tensor kernel set based on the aggregated tensor kernel set. The new first tensor kernel set is obtained from the server. A new first neural network model is determined based on the new first tensor kernel set or the trained first neural network model. The steps of determining the first objective function corresponding to the first neural network model based on local data and the first neural network model and subsequent steps are repeated until the predetermined convergence condition is met, and a tensor kernel set that meets the predetermined convergence condition is obtained.

4. The method according to claim 1, characterized in that, The aggregation includes aggregating the first tensor kernel set trained on each terminal device according to predetermined weights.

5. The method according to claim 1, characterized in that, The first set of tensor kernels corresponds to the preset neural network dimension on the terminal device.

6. The method according to claim 1, characterized in that, Sending the trained first set of kernels to the server includes: Based on the upload ratio, determine the second set of kernels from the first set of kernels that has been trained; The second tensor kernel set is sent to the server.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the upload ratio sent by the server, wherein the upload ratio is related to the accuracy of the trained first tensor kernel set.

8. The method according to claim 1, characterized in that, The decomposed tensors are any of the following types of tensors: tensors decomposed by CP, tensors decomposed by Tucker, tensors decomposed by tensor chain TT, and tensors decomposed by tensor ring TR.

9. A neural network training method based on federated learning, characterized in that, The method is used on a server, and the method includes: Send a first tensor kernel set corresponding to each terminal device to one or more terminal devices. The first tensor kernel set includes the decomposition results of each tensor, and the tensor includes the parameters of a first neural network model. Obtain the first tensor set of training sent by each terminal device, wherein the first tensor set of training is obtained by the terminal device training the first tensor set according to the second objective function, the second objective function is determined according to the first objective function corresponding to the first neural network model, the first neural network model and the first tensor set, the first objective function is determined according to the local data of the terminal device and the first neural network model, and the first neural network model is determined according to the first tensor set; Aggregate the sets of trained first tensor kernels to obtain a set of tensor kernels that satisfy the predetermined convergence conditions; The second neural network model is determined based on the set of tensor kernels that satisfy the predetermined convergence conditions.

10. The method according to claim 9, characterized in that, The aggregation of the trained first tensor kernel sets to obtain a tensor kernel set that satisfies the predetermined convergence condition includes: Determine whether the aggregated tensor kernel set satisfies the predetermined convergence condition; If the aggregated tensor kernel set does not meet the predetermined convergence condition, a new first tensor kernel set is determined based on the aggregated tensor kernel set, and the steps of sending the first tensor kernel set corresponding to each terminal device to one or more terminal devices and subsequent steps are repeated until the aggregated tensor kernel set meets the predetermined convergence condition, thus obtaining a tensor kernel set that meets the predetermined convergence condition.

11. The method according to claim 9 or 10, characterized in that, The aggregation of the trained first tensor kernel sets to obtain a tensor kernel set that satisfies the predetermined convergence condition includes: The first set of tensor kernels trained on each terminal device is aggregated according to predetermined weights to obtain a set of tensor kernels that meet predetermined convergence conditions.

12. The method according to claim 9, characterized in that, The first set of tensor kernels corresponds to the preset neural network dimensions on each terminal device.

13. The method according to claim 9, characterized in that, The step of obtaining the first set of trained kernels sent by each terminal device includes: Obtain the second set of volume kernels sent by each terminal device. The second set of volume kernels is determined by each terminal device from the first set of volume kernels that has been trained, based on the upload ratio.

14. The method according to claim 13, characterized in that, The method further includes: Based on the accuracy of the first trained kernel set sent by each terminal device, the upload ratio corresponding to each terminal device is determined. Send the upload ratio corresponding to each terminal device to each terminal device.

15. The method according to claim 9, characterized in that, The decomposed tensors are any of the following types of tensors: tensors decomposed by CP, tensors decomposed by Tucker, tensors decomposed by tensor chain TT, and tensors decomposed by tensor ring TR.

16. A neural network training device based on federated learning, characterized in that, The device is used in a terminal device, and the device includes: The first acquisition module is used to acquire a first tensor kernel set from the server. The first tensor kernel set includes the decomposition results of each tensor, and the tensors include the parameters of the first neural network model. The first determining module is used to determine the first neural network model based on the first tensor kernel set; The second determining module is used to determine the first objective function corresponding to the first neural network model based on local data and the first neural network model. The third determining module is used to determine the second objective function based on the first objective function, the first neural network model, and the first tensor kernel set; The first training module is used to train the first tensor kernel set according to the second objective function to obtain the trained first tensor kernel set. The first sending module is used to send the trained first tensor kernel set to the server, wherein the trained first tensor kernel set is used by the server to aggregate with trained first tensor kernel sets obtained from other terminal devices to obtain a tensor kernel set that satisfies a predetermined convergence condition, and a second neural network model is determined based on the tensor kernel set that satisfies the predetermined convergence condition.

17. A neural network training device based on federated learning, characterized in that, The device is used for a server, and the device includes: The second sending module is used to send a first tensor kernel set corresponding to each terminal device to one or more terminal devices. The first tensor kernel set includes the decomposition results of each tensor, and the tensor includes the parameters of the first neural network model. The second acquisition module is used to acquire the trained first tensor kernel set sent by each terminal device, wherein the trained first tensor kernel set is obtained by the terminal device training the first tensor kernel set according to the second objective function, the second objective function is determined according to the first objective function corresponding to the first neural network model, the first neural network model and the first tensor kernel set, the first objective function is determined according to the local data of the terminal device and the first neural network model, and the first neural network model is determined according to the first tensor kernel set; The aggregation module is used to aggregate the sets of trained first tensor kernels to determine the set of tensor kernels that meet the predetermined convergence conditions. The fourth determining module is used to determine the second neural network model based on the set of tensor kernels that satisfy the predetermined convergence conditions.

18. A data processing method, characterized in that, The method includes: Get the input data; The input data is input into a second neural network model to obtain a processing result, wherein the second neural network model is obtained by the method described in any one of claims 1-8, or by the method described in any one of claims 9-15.

19. A neural network training device based on federated learning, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-8 or the method of any one of claims 9-15 when executing the instructions.

20. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8, or the method described in any one of claims 9-15.

21. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device performs the method of any one of claims 1-8, or performs the method of any one of claims 9-15.

Images