Neural network training method and related equipment

Abstract

The embodiment of the invention discloses a neural network training method, and the method comprises the steps: a plurality of first devices are respectively provided with a first module of a federated neural network, the first module comprises a feature extraction module, and a second device is provided with a plurality of second modules in the federated neural network, and an early leaving module connected with each second module. Therefore, in the federated learning process, the second equipment can comprise a plurality of early-quit nodes, each early-quit node corresponds to one second module, and the corresponding second module is connected with the early-quit module. Thus, after federal learning, when the target network is deployed in the first device, the structure of the target network can have various forms based on the plurality of early exit nodes. That is, target networks of different structures can be flexibly deployed in different first devices, for example, flexible scheduling can be performed according to resource conditions of different first devices, so that each first device can realize efficient data processing through a target network of a proper scale.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 25, 2023, with Chinese application number 202311398570.6 and invention name “A model training method, device and system”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a neural network training method and related equipment. Background Art

[0003] In recent years, large-scale neural networks have been widely used in many fields. In order to provide sufficient computing power, federated learning can also be used to combine multiple electronic devices to achieve large-scale neural network training and deployment.

[0004] However, in application scenarios such as the Internet of Things and mobile Internet, electronic devices such as smartphones, smart bracelets and microcontrollers have limited resources in terms of storage, computing power and power supply. During the federated learning process, the training and reasoning operations that can be carried out are relatively limited.

[0005] One way to solve the above problem is to use model early exit technology. Model early exit refers to an inference acceleration technology that ends inference in advance according to the difficulty of input data in a multi-stage inference model, thereby reducing redundant calculations. A neural network can be considered as a multi-stage model containing multiple layers connected in series. A neural network that uses model early exit technology can be called an early-exit neural network.

[0006] At present, in the federated learning system combined with the early-retirement neural network, the neural network is usually pre-divided into two sub-networks, and one sub-network is deployed in a terminal device and the other sub-network is deployed in a cloud server. However, this deployment method usually forms a two-level early-retirement neural network. In actual application, it is difficult to flexibly schedule resources and it is difficult to give full play to the advantages of the model early-retirement technology, resulting in low data processing efficiency. Summary of the invention

[0007] The embodiment of the present application provides a neural network training method to solve the problem that the method of deploying early-retirement neural networks in the current neural network training system is difficult to flexibly schedule resources, resulting in low data processing efficiency. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0008] In a first aspect, the present application provides a neural network training method, which is applied to multiple first devices and second devices, wherein each of the multiple first devices is deployed with a first module of a federated neural network, the first module includes a feature extraction module, and the second device is deployed with multiple second modules in the federated neural network, and an early exit module is connected to each second module.

[0009] In the method, the first module, the multiple second modules and the early-exit modules connected to each second module can be trained according to the first data of the multiple first devices to obtain the trained first module, the trained multiple second modules and the trained early-exit modules; the multiple first devices receive indication information from the second device, the indication information is used to instruct the multiple first devices to deploy target networks respectively, the target network deployed by device A among the multiple first devices includes module A among the trained multiple second modules and the trained early-exit modules connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the trained multiple second modules and the trained early-exit modules connected to module B; the multiple first devices deploy the target network according to the indication message.

[0010] In the first aspect, during the federated learning process, the second device may include a plurality of early-leaving nodes, each early-leaving node corresponds to a second module and the corresponding early-leaving module is connected to the second module.

[0011] In this way, after federated learning, when the target network is deployed in the first device, based on multiple early-leaving nodes, the structure of the target network can have multiple forms. In other words, target networks with different structures can be flexibly deployed in different first devices. Specifically, the target network deployed by device A among multiple first devices includes module A among multiple trained second modules and the trained early-leaving module connected to module A, and the target network deployed by device B among multiple first devices includes module B among multiple trained second modules and the trained early-leaving module connected to module B.

[0012] It can be seen that the deployment method of the target network in the first device is relatively flexible and can be deployed according to the needs of the current scenario. For example, it can be flexibly scheduled according to the resource conditions of different first devices, so that each first device can achieve efficient data processing through a target network of appropriate size.

[0013] In a possible implementation of the first aspect, the training process includes one or more stages; according to the first data of multiple first devices, the first module, multiple second modules and the early exit modules connected to each second module are trained, including: in the i-th stage, the first device A processes the first data of the i-th stage through the first module of the i-th stage to obtain the first feature tensor of the i-th stage, where i is a positive integer; the first device A sends the first feature tensor of the i-th stage to the second device; the second device trains the multiple second modules of the i-th stage according to the received first feature tensor of the i-th stage The first device processes the first feature tensor of the i-th stage through the multiple second modules after the training of the i-th stage to obtain the first probability distribution of the i-th stage; the second device sends the first probability distribution of the i-th stage to device A; device A performs multiple iterative training on the first module of the i-th stage according to the first data of the i-th stage and the received first probability distribution of the i-th stage to obtain the first module after the training of the i-th stage.

[0014] In this possible implementation, device A can train the first module of the i-th stage according to the first probability distribution, so as to further improve the performance of the first module of device A through the knowledge learned by multiple second modules in the second device.

[0015] In a possible implementation of the first aspect, device A performs multiple iterative training on the first module of the i-th stage according to the first data of the i-th stage and the received first probability distribution of the i-th stage to obtain the first module of the i-th stage after training, including: device A performs knowledge distillation on the first module of the i-th stage according to the first probability distribution of the i-th stage and the first data of the i-th stage to obtain the first module of the i-th stage after training; wherein the knowledge distillation is implemented according to a loss function, the loss function includes a first loss term, and the first loss term is used to evaluate the similarity between the second probability distribution obtained by the first module according to the first data of the i-th stage in the knowledge distillation process and the first probability distribution of the i-th stage.

[0016] In this possible implementation, the multiple second modules and the early exit modules connected to each second module are located in the second device, and the second device can be a device such as a cloud platform, so it can have a large amount of computing resources, and the scale of the multiple second modules and the early exit modules connected to each second module is usually larger than the first module. In addition, there can be multiple first devices, so the multiple second modules in the second device and the early exit modules connected to each second module can learn more knowledge than the first module in a single first device based on the first feature tensors uploaded by multiple first devices.

[0017] Based on this, in this possible implementation method, the multiple second modules after training in the i-th stage can be used as teacher models, and the first module of the i-th stage can be used as a student model. Through knowledge distillation, based on the first probability distribution, the knowledge in the multiple second modules after training in the i-th stage can be transferred to the first module of the i-th stage to obtain the first module after training in the i-th stage.

[0018] In a possible implementation of the first aspect, the method also includes: the second device obtains device performance information of device A and / or communication performance information between device A and the second device; the second device determines the target network of device A based on the device performance information of device A and / or communication performance information between device A and the second device.

[0019] In this possible implementation, a target network that meets the resource conditions of device A can be determined based on the device performance information, so that in subsequent reasoning processes, the computing resources and storage resources of device A can meet the needs of subsequent reasoning processes, thereby improving the efficiency of the overall reasoning process.

[0020] The communication performance information may reflect the communication rate between device A and device 2. Exemplarily, the communication performance information may include the communication bandwidth between device A and device 2. Determining the target network based on the communication performance information may ensure that after the target network is deployed, during the inference process, the amount of data that may be transmitted between device A and device 2 will not result in a long transmission time, thereby improving the efficiency of the overall inference process.

[0021] In a possible implementation of the first aspect, after device A deploys the target network according to the indication message, the method also includes: device A fixes the parameters of the second module in the target network, and trains the early exit module in the target network according to the first data in device A to update the target network.

[0022] In this possible implementation, after implementing federated learning and deploying the early-leaving neural network, the target network in device A can also be fine-tuned. When fine-tuning, the trained first module and the trained second module in device A can be fixed, and the weight of the early-leaving module in the target network can be fine-tuned according to the local first data in device A, so as to improve the performance of the early-leaving module in the target network according to the local data in device A, so that the early-leaving module in the target network can better predict the local samples of device A.

[0023] In a possible implementation of the first aspect, the method also includes: device A obtains data to be processed; device A obtains a first prediction result of at least one early exit module in the target network regarding the data to be processed; device A obtains a target prediction result of the data to be processed based on an early exit condition and the first prediction result, and the early exit condition includes a confidence threshold; wherein the confidence threshold is determined based on one or more of the following information: the prediction accuracy of at least one early exit module in device A, the prediction time consumption of at least one early exit module in device A, and the communication performance between device A and a second device.

[0024] In this possible implementation, device A can sequentially obtain the first prediction results output by the early exit module along the depth direction, and determine whether the first prediction result satisfies the early exit condition each time a first prediction result is obtained. Any first prediction result may include a probability distribution. If the maximum confidence in a first prediction result is greater than the confidence threshold, it means that the credibility of the first prediction result is high, and the first prediction result satisfies the early exit condition. The first prediction result that satisfies the early exit condition can be used as the target prediction result, thereby realizing the early exit of the model; otherwise, the target prediction result can be obtained through the second module in the second device, and the target prediction result is fed back to device A by the second device.

[0025] Exemplarily, the confidence threshold can be determined by the user according to the current scenario requirements and a curve describing the corresponding relationship between the prediction accuracy and prediction time of at least one early-retirement module under different preset confidences. In addition, in some examples, the confidence threshold can also be determined based on the communication performance between device A and the second device. For example, if the communication performance is poor, it can be considered to complete the prediction on device A as much as possible, and reduce the possibility of data transmission between device A and the second device, and then a smaller confidence threshold can be determined.

[0026] In a possible implementation of the first aspect, device A obtains a target prediction result of the data to be processed based on the early exit condition and the first prediction result, including: if any first prediction result satisfies the early exit condition, device A uses the first prediction result that satisfies the early exit condition as the target prediction result.

[0027] In this possible implementation, if a first prediction result is greater than the confidence threshold, it means that the first prediction result is highly credible and meets the early exit condition. The first prediction result that meets the early exit condition can be used as the target prediction result, thereby achieving early exit of the model.

[0028] In a possible implementation of the first aspect, device A obtains a target prediction result of the data to be processed based on an early exit condition and a first prediction result, including: if there is no first prediction result that satisfies the early exit condition, device A sends a second feature tensor to a second device, wherein the second feature tensor is a feature tensor output by a target module in device A based on the data to be processed, and the target module is a trained first module in device A or a trained second module in device A, and the target module is determined based on one or more of the following information: the dimension of the output data of the trained first module in device A, the dimension of the output data of the trained second module in device A, and the communication performance between device A and the second device; the second device obtains the target prediction result based on the received second feature tensor through a second module located after the target module among multiple trained second modules; the second device sends the target prediction result to device A; and device A receives the target prediction result from the second device.

[0029] In this possible implementation, when early retirement is not possible, the output of device A can be flexibly determined from the output of the first module of device A after training and the output of the second module in the target network so that the corresponding reasoning performance is optimal (for example, the end-to-end delay of the reasoning process is optimal).

[0030] A second aspect of the present application provides a neural network training method, which is applied to a second device, wherein the second device is deployed with multiple second modules in a federated neural network, and each of the second modules is respectively connected to an early-leaving module, and each of the first devices is respectively deployed with a first module of the federated neural network, and the method comprises: the second device receives respective first feature tensors from the multiple first devices; the second device trains the multiple second modules and the early-leaving modules respectively connected to each of the second modules according to the first feature tensors to obtain multiple trained second modules and the trained early-leaving modules, wherein the trained multiple second modules include module A and module B, and the trained early-leaving modules include the trained early-leaving modules connected to module A and the trained early-leaving modules connected to module B; the second device sends indication information to the multiple first devices, wherein the indication information is used to instruct the multiple first devices to respectively deploy a target network, wherein the target network deployed by device A among the multiple first devices includes module A among the multiple trained second modules and the trained early-leaving modules connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple trained second modules and the trained early-leaving modules connected to module B.

[0031] In a possible implementation of the second aspect, the second device trains multiple second modules and the early-leaving modules connected to each second module according to the first feature tensor, including: in the i-th stage, the second device receives the first feature tensor of the i-th stage from the device A, where i is a positive integer; the second device trains the multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the received first feature tensor of the i-th stage, and obtains the multiple second modules after training of the i-th stage and the early-leaving module after training of the i-th stage; the second device processes the first feature tensor of the i-th stage through the multiple second modules after training of the i-th stage, and obtains the first probability distribution of the i-th stage; the second device sends the first probability distribution of the i-th stage to the device A.

[0032] In a possible implementation of the second aspect, the i-th stage includes one or more iterative processes; in the j-th iterative process of the i-th stage, the second device trains multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the received first feature tensor of the i-th stage, including: the second device obtains the output probability distribution of each early-leaving module of the j-th iterative process according to the first feature tensor of the j-th iterative process, where j is a positive integer; the second device updates the second modules of the j-th iterative process and the early-leaving module of the j-th iterative process according to each output probability distribution, each corresponding weight of each output probability distribution and a label corresponding to the first feature tensor of the j-th iterative process, to obtain the trained second modules of the j-th iterative process and the trained early-leaving module of the j-th iterative process.

[0033] In a possible implementation of the second aspect, the method also includes: the second device obtains device performance information of device A and / or communication performance information between device A and the second device; the second device determines the target network of device A based on the device performance information of device A and / or communication performance information between device A and the second device.

[0034] In a possible implementation of the second aspect, the method also includes: the second device receives a second feature tensor from device A, the second feature tensor is a feature tensor output by a target module in device A according to data to be processed, and the target module is a trained first module in device A or a trained second module in device A; the second device obtains a target prediction result according to the received second feature tensor through a second module located after the target module among multiple trained second modules; and the second device sends the target prediction result to device A.

[0035] In a third aspect, the present application provides a neural network training system, which includes a plurality of first devices and a second device, and the system has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an interface module (such as a first interface module and a second interface module) and a processing module (such as a first processing module and a second processing module).

[0036] In a fourth aspect, the present application provides a neural network training system, which includes multiple first devices and second devices, wherein the multiple first devices and second devices include a processor and a memory, wherein the memory stores computer-executable instructions that can be run on the processor, and when the computer-executable instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0037] The fifth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0038] In a sixth aspect, the present application provides a computer program product storing one or more computer-executable instructions. The computer program product includes computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0039] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting a neural network training system to implement the functions involved in the first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory for storing necessary program instructions and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0040] In an eighth aspect, the present application provides a neural network training device, which has the function of implementing the method of the second aspect or any possible implementation of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a second interface module and a second processing module.

[0041] In a ninth aspect of the present application, a second device is provided, which includes a processor and a memory, wherein the memory stores computer execution instructions that can be run on the processor, and when the computer execution instructions are executed by the processor, the processor executes a method as described in the second aspect above or any possible implementation method of the second aspect.

[0042] The tenth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the second aspect or any possible implementation of the second aspect.

[0043] In the eleventh aspect of the present application, there is provided a computer program product storing one or more computer-executable instructions. The computer program product includes computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as in the second aspect or any possible implementation of the second aspect.

[0044] In a twelfth aspect of the present application, a chip system is provided, which includes a processor for supporting a second device to implement the functions involved in the above-mentioned second aspect or any possible implementation of the second aspect. In a possible design, the chip system may also include a memory, which is used to store necessary program instructions and data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0045] Among them, the technical effects brought about by the second to twelfth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is an exemplary schematic diagram of a data center provided in an embodiment of the present application;

[0047] Figure 2 is an exemplary schematic diagram of the system architecture provided by an embodiment of the present application;

[0048] Figure 3 is an exemplary schematic diagram of the processing stage provided in the embodiment of the present application;

[0049] Figure 4 is a schematic diagram of an embodiment of a neural network training method provided in an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of an embodiment of a neural network training method provided in an embodiment of the present application;

[0051] Figure 6 is an exemplary schematic diagram of the system architecture provided in the embodiment of the present application;

[0052] Figure 7 is an exemplary schematic diagram of a curve describing the corresponding relationship between the prediction accuracy and the prediction time consumption of at least one early-retirement module under different preset confidences provided in an embodiment of the present application;

[0053] Figure 8 is an exemplary schematic diagram of the system architecture provided by an embodiment of the present application;

[0054] Fig. 9 is a schematic diagram of an embodiment of a neural network training system provided in an embodiment of the present application;

[0055] Fig.10 is a schematic diagram of an embodiment of a neural network training device provided in an embodiment of the present application;

[0056] Fig.11 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0057] Fig.12 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0058] Fig.13 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0059] Fig.14 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0060] Fig.15 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0061] Fig.16 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0064] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0065] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0066] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0067] (1) Neural Network

[0068] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0069]

[0070] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0071] (2) Loss Function

[0072] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0073] (3) Back propagation algorithm

[0074] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0075] (4) Federated Learning

[0076] Federated learning is an artificial intelligence technology. Federated learning aims to enable multiple participants to jointly train neural networks with their data without leaking plaintext data, making data available but invisible. During the training and reasoning of neural networks, each participant in the federated learning system follows the privacy principle that local data should not be out of the local area. Federated learning is an artificial intelligence technology. The goal of federated learning is to carry out efficient machine learning among multiple participants while ensuring information security during big data exchange, protecting terminal data and personal data privacy, and ensuring legality and compliance.

[0077] Depending on the different data distribution among the participants, federated learning is usually divided into horizontal federated learning and vertical federated learning.

[0078] Among them, horizontal federated learning is aimed at the scenario where the training data of different users is stored in different participants; therefore, the purpose of horizontal federated learning is to train the overall neural network by using the combined training data of each participant on the basis of adhering to the privacy principle of local data not leaving the local area.

[0079] Vertical federated learning uses the combination of different features of the same users among all participants to train the overall neural network. Different participants in vertical federated learning have data in the same sample space and different feature spaces. Through the user data of shared users, secure joint modeling is performed, which has a wide range of application scenarios in finance, advertising and other fields.

[0080] (5) Model early exit

[0081] Model early exit refers to the inference acceleration technology that adds early exit modules such as classifiers at different depth positions of the neural network and terminates the inference in advance according to the difficulty of the input data, thereby reducing redundant calculations. A neural network can be considered as a multi-stage model containing multiple layers in series, and a neural network that uses model early exit technology can be called an early-exit neural network.

[0082] It can be seen that the early-exit modules such as classifiers added in different depth positions of the neural network can be used to dynamically adjust the reasoning route. In the actual reasoning process, not all samples need to be predicted using a deepened neural network. For example, a shallow network can better handle overall, coarse-grained features, and a deep network can better handle local, fine-grained features. If the depth of the neural network does not match the characteristics of the sample, it will have a negative impact. In the reasoning process using the early-exit neural network, it can be decided whether to exit the reasoning process early based on the content characteristics of each input data, thereby significantly improving the reasoning efficiency and improving the reasoning accuracy.

[0083] (6) Knowledge distillation (KD)

[0084] Knowledge distillation is a training method based on the "teacher-student network concept" that distills the knowledge contained in the trained teacher model into the student model. Knowledge distillation can be considered a model compression method. In knowledge distillation, a large-scale model can be regarded as a teacher model, and the knowledge extracted from the teacher model can be transferred to a simple student model, that is, to a small-scale neural network. After the migration is completed, the small-scale student model can be deployed on devices with limited storage and computing power.

[0085] The embodiment of the present application provides a neural network training method, which can flexibly deploy early-retirement neural networks in a neural network training system, thereby facilitating flexible resource scheduling and data processing according to actual scenarios and improving data processing efficiency.

[0086] In the embodiment of the present application, this can be achieved through a neural network training system, which can be considered as a federated learning system.

[0087] The neural network training system may include multiple first devices and second devices.

[0088] Each first device may be a computing device cluster including one or more computing devices. The types and quantities of the computing devices included in different first devices may be the same or different.

[0089] The second device may also be a computing device cluster including one or more computing devices.

[0090] The type of any computing device is not limited herein. Exemplarily, any computing device may be a terminal device, or a server, a container, or a virtual machine.

[0091] In the neural network training system, the types of different devices may be different.

[0092] For example, in one example, any first device can be considered as a client device, and the second device can be used to implement the cloud platform. In this way, in the neural network training system, one or more client devices can combine the resources of the cloud platform to implement operations such as federated learning training and model deployment.

[0093] In this exemplary scenario, the cloud platform can provide computing, network and storage capabilities based on services of hardware resources and software resources. Exemplarily, the cloud platform can include one or more data centers to provide cloud resources through one or more data centers.

[0094] Combine the following Figure 1 An architectural diagram is shown to introduce a data center.

[0095] Figure 1 In the data center, the cloud management platform communicates with one or more servers (such as Figure 1The server 1 and server 2 in the server exchange information. The server includes a hardware layer and a software layer. The hardware layer includes the hardware configured by the server, wherein the PCI device may be, for example, a network card, a graphics processing unit (GPU), an offload card, etc., which can be inserted into the peripheral component interconnect standard (PCI) or peripheral component interconnect express (PCIe) slot of the server. The software layer includes an operating system installed and running on the server (relative to the operating system of the virtual machine, it can be called the host operating system). The host operating system is provided with a virtual machine manager (also called a hypervisor). The role of the virtual machine manager is to realize the computing virtualization, network virtualization and storage virtualization of the virtual machine, and is responsible for managing the virtual machine. Among them, a virtual machine (virtual machine) refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. Figure 1 In the system architecture shown, a data center is equipped with multiple servers, which can be used to run virtual machines. The specifications of each virtual machine can be the same or different. Among them, the virtual machine can also be called a cloud server (elastic compute service, ECS), elastic instance, etc. Different cloud service providers may have different names.

[0096] In one example, the cloud platform may be a public cloud platform. In this case, cloud service providers such as individuals or software developers with cloud resource development capabilities may provide cloud services to users. Users obtain cloud services through the Internet but do not own cloud computing resources.

[0097] Specifically, in Figure 1 In the example shown, the cloud management platform can provide an access interface (such as an interface or application programming interface (API)). Users of the cloud platform and cloud service providers can operate the client remote access access interface to register a cloud account and password on the cloud management platform, and log in to the cloud management platform after the cloud management platform successfully authenticates the cloud account and password, thereby creating, managing, logging in and operating virtual machines in the cloud data center.

[0098] Of course, the cloud platform may also be other types of cloud platforms, for example, a private cloud platform or a hybrid cloud platform, etc., which is not limited in the embodiments of the present application.

[0099] In the embodiment of the present application, the first device and the second device can be connected to each other in communication. The communication connection method between each first device and the second device is not limited here, and the communication connection methods used when different first devices and second devices are connected can be the same or different.

[0100] In the embodiment of the present application, the neural network trained by federated learning is called a federated neural network. The type and structure of the federated neural network are not limited here. Exemplarily, the federated neural network can be a combination of one or more of a convolutional neural network (CNN), a recurrent neural network (RNN), a residual network (ResNet), a support vector machine, a decision tree, a random forest, a logistic regression, etc. After the training is completed, functions such as image processing, text processing, or speech recognition can be implemented through the federated neural network.

[0101] In the embodiment of the present application, an early withdrawal network that can implement the model early withdrawal technology can be constructed based on the federated neural network.

[0102] Specifically, early exit modules can be set at different depth positions. Figure 2 In the exemplary system architecture shown, the federated neural network may include but is not limited to a first module and a plurality of second modules (e.g. Figure 2 The second module 1, second module 2, ... second module n shown in the figure, the multiple first devices may include the first device 1 to the first device p, the first device 1 can obtain the local first data 1, the first device p can obtain the local first data p, the first module can be located in each first device, the first module includes a feature extraction module, and in some examples, during federated learning, the first module can also be connected to a first classifier. The multiple second modules are located in the second device, each second module can be connected in series, and each second module is connected to an early exit module (for example Figure 2 The early-leaving modules 1 to n are shown in the figure, thereby obtaining an early-leaving neural network capable of realizing early-leaving of the model. Among them, the possible early-leaving nodes are the early-leaving modules.

[0103] The specific structure of each second module is not limited here. The structure and number of layers included in different second modules can be the same or different.

[0104] In some examples, the last layer of any second module can be an activation layer. In addition, in some examples, the network parameter quantities of each second module along the depth direction can be roughly the same, thereby facilitating providing a variety of resource cost options in early model withdrawal during subsequent data processing.

[0105] The specific structure of the early exit module is not limited here.

[0106] Exemplarily, any early-leaving module may include a classifier such as a fully connected layer, and may also include a pooling layer, wherein the classifier is used to output a prediction result, and the pooling layer may map the intermediate layer features output by the corresponding second module to a fixed dimension to ensure that the computational overhead of the classifiers in each early-leaving module is the same.

[0107] In addition, in order to facilitate training, a first classifier can be set in the first device, and the first classifier is connected after the first module.

[0108] In this way, each first device may be deployed with a first module. In addition, in the federated learning process, in some examples, such as Figure 2 As shown, the first module can also be connected to the first classifier, and the second device can include multiple second modules and early exit modules connected to each second module.

[0109] Based on the first module in each first device and the multiple second modules in the second device, and the early exit modules connected to each second module, a neural network training method of an embodiment of the present application can realize federated learning and network deployment after federated learning. In addition, in some examples, fine-tuning can be performed after network deployment, and after fine-tuning, the corresponding prediction task is performed.

[0110] It can be seen that Figure 3 As shown, the embodiment of the present application may include one or more of the following processing stages:

[0111] Federated learning, network deployment, network fine-tuning, and performing inference tasks.

[0112] Below, each processing stage is introduced exemplarily.

[0113] 1. Federated learning.

[0114] In some embodiments, the federated learning process may include: Figure 4 Step 401 is shown.

[0115] Step 401, based on the first data of the plurality of first devices, train the first module, the plurality of second modules and the early exit module connected to each second module to obtain the trained first module, the trained plurality of second modules and the trained early exit module.

[0116] In an embodiment of the present application, during the training process, the first feature tensor output by the first module of each first device can be transmitted to the second device, so that the second device trains the second module and the early exit module of the second device according to the first feature tensor output by the first module during the joint learning process. In addition, in some examples, during the training process, the second device can feedback information to multiple first devices, so that multiple first devices can be trained according to the information fed back by the second device, so that multiple first devices can learn the knowledge fed back by the second device.

[0117] The specific steps of the training process are not limited here. For example, the first module and the second sub-network may be updated in each iteration of the training process, or the first module may be iterated multiple times before the second sub-network is iterated multiple times.

[0118] An exemplary implementation of the training process is introduced below by taking the federated learning between device A and the second device among multiple first devices as an example.

[0119] In some embodiments, the training process includes one or more stages, and the i-th stage is taken as an example for introduction below.

[0120] i is a positive integer, that is, the i-th stage can be any stage in the training process, and other stages in the training process may include more steps than the i-th stage, or may include fewer steps than the i-th stage, or may include steps different from the steps of the i-th stage, which is not limited in the embodiments of the present application.

[0121] like Figure 5 As shown, in the i-th stage, the method includes steps 501-508.

[0122] Step 501: Device A processes the first data of the i-th stage through the first module of the i-th stage to obtain the first feature tensor of the i-th stage.

[0123] In the embodiment of the present application, the first data of the i-th stage may be part of or all of the first data of device A.

[0124] In some examples, before the first stage, the first module can be pre-trained, and the pre-trained first module is used as the first module of the first stage, and the first modules of other stages except the first stage are the first modules trained in the previous stage of the other stages.

[0125] In this example, during the pre-training process of the first module, the first module can be iterated once or multiple times according to the first data to obtain the pre-trained first module as the first module of the first stage.

[0126] In addition, in some examples, device A sends the first feature tensor of the i-th stage and the label corresponding to the first data of the i-th stage to the second device.

[0127] Alternatively, in some examples, if the first data involved in each stage is the same, device A may send the label corresponding to the first data to device II only in the first stage, and there is no need to upload the label corresponding to the first data in other stages except the first stage.

[0128] The specific content of the label is determined according to the specific application scenario. For example, in a classification scenario, the label can be a category; in a regression scenario, the label can be a regression result.

[0129] Step 502: Device A sends the first feature tensor of the i-th stage to device II.

[0130] Step 503: The second device receives the first feature tensor of the i-th stage from the first device.

[0131] Step 504: The second device trains multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the first feature tensor of the i-th stage to obtain multiple second modules of the i-th stage and the early-leaving module of the i-th stage after training.

[0132] The first feature tensor of the i-th stage can be used as input data of a network in the second device in the i-th stage, which includes multiple second modules and early-retirement modules connected to each second module.

[0133] During the training process of the second sub-network in the i-th stage, the multiple second modules of the i-th stage and the early-leaving module of the i-th stage can be iterated once or multiple times according to the first feature tensor of the i-th stage to obtain the multiple second modules after training of the i-th stage and the early-leaving module after training of the i-th stage.

[0134] Exemplarily, the i-th stage includes one or more iterations;

[0135] In the j-th iteration of the i-th stage, step 504 includes:

[0136] The second device obtains the output probability distribution of each early-leaving module in the j-th iteration process according to the first feature tensor of the j-th iteration process, where j is a positive integer;

[0137] The second device updates the second module of the j-th iterative process and the early-leaving module of the j-th iterative process according to each output probability distribution, the weight corresponding to each output probability distribution and the label corresponding to the first feature tensor of the j-th iterative process, and obtains the trained second module of the j-th iterative process and the trained early-leaving module of the j-th iterative process.

[0138] In some examples, the second device obtains the output probability distribution of each early-leaving module of the j-th iteration process based on the first feature tensor of the j-th iteration process, and can also obtain the output probability distribution of the federated neural network.

[0139] Then, the total loss value can be calculated according to the output probability distributions and the labels corresponding to the first feature tensor of the j-th iteration process, as well as the loss function; then, according to the total loss value, the gradients of the second module of the j-th iteration process and the early exit module of the j-th iteration process are obtained through back propagation to update the second module of the j-th iteration process and the early exit module of the j-th iteration process, and obtain the trained second module of the j-th iteration process and the trained early exit module of the j-th iteration process.

[0140] The specific form of the loss function is not limited here. Exemplarily, the loss function may be a cross entropy function.

[0141] In addition, there are multiple ways to calculate the total loss value.

[0142] In one example, a weight may be set for each early-leaving module. In addition, when each output probability distribution includes an output probability distribution of a federated neural network, a weight may be set for the last layer of the federated neural network. For example, the weight of each early-leaving module and the last layer of the federated neural network may be the same, or may be changed according to the depth of each early-leaving module and the last layer of the federated neural network.

[0143] Then, the output probability distribution of each early-leaving module in the j-th iteration process and the output probability distribution of the last layer of the federated neural network in the j-th iteration process, the loss value between each output probability distribution and the label is calculated, and a weighted calculation is performed according to the loss value corresponding to each output probability distribution and the weight corresponding to each output probability distribution to obtain the total loss value.

[0144] In some examples, when there are multiple first devices, different numbers of first feature tensors sent to the second device by different first devices may cause different first devices to have different degrees of influence on the second device during training.

[0145] For example, in one example, in the i-th stage, each first device may send its own first feature tensor to the second device respectively, and the number of first feature tensors sent by each first device may be the same or different.

[0146] During the j-th iteration of the i-th stage, multiple second modules of the i-th stage and the early exit module of the i-th stage can be trained according to the first feature tensor corresponding to each device A during the j-th iteration.

[0147] Among them, according to the first feature tensor corresponding to each first device in the j-th iteration process, the gradient of the second module of the j-th iteration process and the early-leaving module of the j-th iteration process corresponding to each first device can be obtained by referring to the above example, which obtains the gradient of the second module of the j-th iteration process and the early-leaving module of the j-th iteration process according to the labels corresponding to the first feature tensor of the j-th iteration process and the loss function.

[0148] In this example, the more first feature tensors the second device transmits to the second device during the j-th iteration, the more user data the device A has, and the importance of the device A is usually higher. Therefore, the impact of the device A on the total loss value is correspondingly higher.

[0149] Step 505: The second device processes the first feature tensor of the i-th stage through multiple second modules trained in the i-th stage to obtain the first probability distribution of the i-th stage.

[0150] After obtaining the multiple second modules after training of the i-th stage, the first feature tensor of the i-th stage can be processed by the multiple second modules after training of the i-th stage to obtain the first probability distribution of the i-th stage. The first probability distribution can be the probability distribution output by the network deployed on the second device in the federated neural network after the i-th stage training by processing the first feature tensor of the i-th stage.

[0151] Step 506: The second device sends the first probability distribution of the i-th stage to the first device.

[0152] Step 507: Device A receives the first probability distribution of the i-th stage from the second device.

[0153] In step 508, device A performs multiple iterative training on the first module of the i-th stage according to the first data of the i-th stage and the received first probability distribution of the i-th stage to obtain the trained first module of the i-th stage.

[0154] In the present application examples, reference Figure 3In the example shown, device A can perform knowledge distillation on the first module of the i-th stage according to the first probability distribution, so as to further improve the performance of the first module of device A through the knowledge learned by multiple second modules in the second device.

[0155] Specifically, in some embodiments, the above step 508 includes:

[0156] Device A performs knowledge distillation on the first module of the i-th stage according to the first probability distribution of the i-th stage and the first data of the i-th stage to obtain the first module after training of the i-th stage;

[0157] Among them, knowledge distillation is implemented according to a loss function, and the loss function includes a first loss term, which is used to evaluate the similarity between the second probability distribution obtained by the first module in the knowledge distillation process according to the first data of the i-th stage and the first probability distribution of the i-th stage.

[0158] Knowledge distillation is a training method based on the "teacher-student network idea", which distills the knowledge contained in the teacher model into the student model.

[0159] In an embodiment of the present application, multiple second modules and the early exit module connected to each second module are located in the second device, and the second device can be a device such as a cloud platform, so it can have a large amount of computing resources, and the scale of the multiple second modules and the early exit module connected to each second module is usually larger than the first module. In addition, there can be multiple first devices, so the multiple second modules in the second device and the early exit module connected to each second module can learn more knowledge than the first module in a single first device based on the first feature tensors uploaded by multiple first devices.

[0160] Based on this, in an embodiment of the present application, the multiple second modules after training in the i-th stage can be used as teacher models, and the first module of the i-th stage can be used as a student model. Through knowledge distillation, based on the first probability distribution, the knowledge in the multiple second modules after training in the i-th stage can be transferred to the first module of the i-th stage to obtain the first module after training in the i-th stage.

[0161] The knowledge distillation can be implemented by a corresponding loss function. The loss function corresponding to the knowledge distillation can be different from the loss function used when training the multiple second modules of the i-th stage and the early withdrawal module of the i-th stage.

[0162] Specifically, the loss function may include a first loss term. Furthermore, in some examples, the loss function may also include a second loss term.

[0163] Among them, the first loss term is used to evaluate the similarity between the second probability distribution obtained by the first module in the knowledge distillation process according to the first data of the i-th stage and the first probability distribution of the i-th stage.

[0164] Exemplarily, the first loss term may be a KL divergence loss between the second probability distribution and the first probability distribution.

[0165] The second loss term is used to evaluate the difference between the second probability distribution obtained by the first module of the i-th stage based on the first data of the i-th stage and the corresponding label during the knowledge distillation process.

[0166] Exemplarily, the second loss term may be a cross entropy loss between the second probability distribution and the corresponding label.

[0167] The knowledge distillation of the i-th stage may include one or more iterative processes, that is, based on the first probability distribution and the loss function, the knowledge in the multiple second modules after training of the i-th stage can be transferred to the first module of the i-th stage through one or more iterative processes to obtain the first module after training of the i-th stage.

[0168] Referring to the steps of the above-mentioned i-th stage, one or more stages of training are performed until the number of trainings reaches a threshold number, or until the first module, multiple second modules, and the early-leaving modules connected to each second module converge to the desired state, then the above-mentioned federated learning is completed, so that device A obtains the trained first module, and the second device obtains the trained multiple second modules and the trained early-leaving module.

[0169] 2. Network deployment.

[0170] In some embodiments, Figure 4 As shown, after obtaining the trained first module, the trained multiple second modules and the trained early-leaving module, the neural network training method can execute steps 402-405 to achieve network deployment in device A.

[0171] Specifically, in step 402, the second device generates indication information.

[0172] The indication information is used to instruct multiple first devices to respectively deploy a target network, wherein the target network deployed by device A among the multiple first devices includes module A among the multiple second modules after training and a trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple second modules after training and a trained early-leaving module connected to module B.

[0173] After obtaining the trained first module, the trained second module and the trained early-leaving module, the second device can determine a segmentation method for the early-leaving neural network obtained after joint learning, and determine the target network to be deployed by each first device based on the segmentation method, thereby generating indication information to instruct each first device to perform network deployment according to the indication information.

[0174] There are many specific ways to determine the target network. For example, taking device A as an example, the second device may determine the network data volume suitable for the situation of device A based on one or more of information such as the device performance of device A and / or the communication performance between device A and the second device, thereby determining the modules included in the target network.

[0175] The following uses the device performance and / or communication performance of device A as an example to illustrate the specific method of determining the target network.

[0176] 1) Determine the target network of device A based on the device performance information of device A.

[0177] In this example, the second device can determine the target network according to the device performance information of the first device, the trained second module and the trained early exit module.

[0178] The device performance information of device A can indicate one or more performances of device A, such as computing performance (e.g., the number of processors, processing latency), storage performance (the size of storage devices such as memory and hard disk in device A), etc., thereby reflecting the resource status of device A, such as computing resources and / or storage resources.

[0179] The manner in which the second device obtains the device performance information of the first device is not limited here.

[0180] In one example, the second device may obtain device performance information of device A from other devices (eg, an electronic device that manages device A).

[0181] In another example, device A may send device performance information of device A to device II.

[0182] Device A can collect device performance information of device A during the federated learning process in the above step 401, such as device performance information such as the computing delay of device A and the average free memory size, and send the device performance information of device A to the second device after the federated learning is completed or during the federated learning process.

[0183] Exemplarily, the device performance information can be used to evaluate the total amount of data that device A can carry and efficiently operate. The second device can determine the target network that meets the resource situation of device A based on the data volume of the parameters of the trained multiple second modules and the trained early-exit module, and generate indication information based on the target network. Alternatively, the device performance information can be used to evaluate the amount of data that device A can carry and efficiently operate in addition to the first module. The second device can determine the target network that meets the resource situation of device A based on the trained multiple second modules and the trained early-exit module, and generate indication information about device A based on the target network.

[0184] It can be seen that based on the device performance information, a target network that meets the resource conditions of device A can be determined, so that in the subsequent reasoning process, the computing resources and storage resources of device A can meet the needs of the subsequent reasoning process, thereby improving the efficiency of the overall reasoning process.

[0185] 2) Determine the target network of device A based on the communication performance information.

[0186] In this example, the second device can determine the target network of device A based on the communication performance information between device A and the second device and the trained multiple second modules and the trained early exit module.

[0187] In one example, the second device may collect the communication performance information based on the communication status between the second device and the first device.

[0188] In another example, the communication performance information between the device A and the second device may be sent by the device A to the second device.

[0189] The communication performance information may reflect the communication rate between device A and device II.

[0190] Exemplarily, the communication performance information may include the communication bandwidth between device A and device II.

[0191] When the communication performance information indicates that the communication rate between device A and the second device is slow, the second device tends to make the amount of data output by the target network deployed by device A smaller, so as to reduce the amount of data to be transmitted between device A and the second device in subsequent application processes.

[0192] When the communication performance information indicates that the communication rate between device A and the second device is faster, the second device tends to make the output data volume of the target network deployed by device A larger. In the subsequent application process, even if the amount of data to be transmitted between device A and the second device is larger, it will not result in a longer transmission time.

[0193] It can be seen that determining the target network based on the communication performance information can ensure that after the target network is deployed, during the reasoning process, the amount of data that may be transmitted between device A and device II will not result in a long transmission time, thereby improving the efficiency of the overall reasoning process.

[0194] The target network may be in various situations, which are not limited here.

[0195] In one example, the instruction information of the second device indicates that a first device only deploys the trained first module, and does not need to deploy the trained second module and the trained early exit module. In this example, it can be considered that the target network of the first device only includes the trained first module.

[0196] In another example, the target network of the first device may include one or more early-leaving branches, that is, including at least one trained second module and an early-leaving module connected to the at least one second module. In addition, the first device also has a trained first module deployed therein.

[0197] When a target network of a first device includes at least one trained second module and an early-leaving module connected to the at least one second module, the indication information may include at least one trained second module in the target network and an early-leaving module connected to the at least one second module.

[0198] When the target network of a first device includes only the trained first module but does not include the second module and the early-retirement module, the indication information may include only instructions to indicate that device A only needs to deploy the trained first module.

[0199] It can be understood that, among the multiple first devices, the target networks of two first devices may be the same or different.

[0200] Take device A and device B among the plurality of first devices as an example. The target network of device A is different from the target network of device B.

[0201] Specifically, the target network deployed by device A among the plurality of first devices includes module A among the plurality of trained second modules and the trained early-leaving module connected to module A, and the target network deployed by device B among the plurality of first devices includes module B among the plurality of trained second modules and the trained early-leaving module connected to module B. Module A is different from module B.

[0202] Among them, device A may include more or less second modules and corresponding early exit modules than device B.

[0203] For example, if device A includes more second modules than device B, the target network deployed by device A includes module A among the trained multiple second modules and the trained early-leaving module connected to module A, and module B among the trained multiple second modules and the trained early-leaving module connected to module B. The target network deployed by device B does not include module A among the trained multiple second modules and the trained early-leaving module connected to module A.

[0204] Step 403: The second device sends indication information to multiple first devices.

[0205] Among them, the second device can send the indication information of the corresponding first device to each first device respectively, that is, the second device can send the indication information of device A to device A, and send the indication information of device B to device B, and the indication information of device A is different from the indication information of device B.

[0206] Step 404: multiple first devices receive indication information from the second device.

[0207] Step 405: Multiple first devices deploy a target network according to the instruction message.

[0208] In some examples, during the federated learning process, the first module may be subsequently connected to the first classifier to train the first module through the first classifier. However, during deployment, the target network may not include the first classifier to which the trained first module is connected.

[0209] For device A, device A can receive module A among multiple trained second modules and the trained early-retirement module connected to module A from the second device and deploy them.

[0210] As for the device B, the device B may receive the trained module B among the multiple second modules and the trained early-retirement module connected to the module B from the second device.

[0211] It can be seen that in the embodiment of the present application, in the federated learning process, the second device may include multiple early-leaving nodes, each early-leaving node corresponds to a second module and the corresponding second module is connected to the early-leaving module.

[0212] In this way, after federated learning, when the target network is deployed in the first device, based on multiple early-leaving nodes, the structure of the target network can have multiple forms. In other words, target networks with different structures can be flexibly deployed in different first devices. Specifically, the target network deployed by device A among multiple first devices includes module A among multiple trained second modules and the trained early-leaving module connected to module A, and the target network deployed by device B among multiple first devices includes module B among multiple trained second modules and the trained early-leaving module connected to module B.

[0213] It can be seen that the deployment method of the target network in the first device is relatively flexible and can be deployed according to the needs of the current scenario. For example, it can be flexibly scheduled according to the current resource conditions, so that each first device can achieve efficient data processing through a target network of appropriate size.

[0214] It can be understood that the structures of the target networks deployed in different A devices may be the same or different.

[0215] For example, Figure 6 The figure is a schematic diagram of an exemplary structure of the system architecture after network deployment.

[0216] The device performance of the first device 1, the first device 2, the first device 3 and the first device n is enhanced in sequence. At this time, the target networks corresponding to the first device 1, the first device 2, the first device 3 and the first device n are different. For example, the first device 2 can be considered as device A and the first device 3 can be considered as device B.

[0217] Among them, the first device 1 only includes the first module, and the backend network corresponding to the target network of the first device 1 includes all the second modules in the second device. The first device 2 includes the first module and also includes a first-level early exit branch, the early exit branch includes the second module 1 and the early exit module 1; the backend network corresponding to the target network of the first device 2 includes the second module 2 in the second device and the subsequent second module. The first device 3 includes two-level early exit branches; the backend network corresponding to the target network of the first device 3 includes the second module 3 in the second device and the subsequent second module. The first device n includes n-1-level early exit branches; the backend network corresponding to the target network of the first device n includes the second module n in the second device and the subsequent second module.

[0218] 3. Network fine-tuning.

[0219] In some embodiments, after implementing federated learning and deploying the early-falloff neural network, the target network in the first device may also be fine-tuned.

[0220] Taking device A as an example, specifically, in some embodiments, after device A deploys the target network according to the instruction message, the method further includes:

[0221] Device A fixes the parameters of the second module in the target network, and trains the early-retirement module in the target network according to the first data in device A to update the target network.

[0222] During fine-tuning, the trained first module and the trained second module in device A can be fixed, and the weight of the early-leaving module in the target network can be fine-tuned based on the local first data in device A, so as to improve the performance of the early-leaving module in the target network based on the local data in device A, so that the early-leaving module in the target network can better predict the local samples of device A.

[0223] After updating the target network through fine-tuning, device A can perform inference tasks based on the target network.

[0224] 4. Perform reasoning tasks.

[0225] Among them, when performing the reasoning task, it can be determined whether to exit early according to the trained early exit module deployed in the first device. For the convenience of description, taking device A as an example, in the subsequent embodiments, the target network deployed in device A and the back-end network corresponding to the target network in the second device are referred to as early exit neural networks. The back-end network corresponding to the target network of device A can include a second module in the federated neural network, which is located after the last second module of the target network.

[0226] The following is an introduction to the inference scenario of device A.

[0227] In some embodiments, the method further comprises:

[0228] Device A obtains the data to be processed;

[0229] Device A obtains a first prediction result of at least one early-leaving module on to-be-processed data in the target network;

[0230] Device A obtains the target prediction result of the data to be processed according to the early exit condition and the first prediction result.

[0231] Early exit conditions include confidence thresholds.

[0232] The confidence threshold is determined based on one or more of the following information:

[0233] The prediction accuracy of at least one early-leaving module in device A, the prediction time consumption of at least one early-leaving module in device A, and the communication performance between device A and a second device.

[0234] Next, the specific configuration method of the early exit condition is introduced.

[0235] The early exit condition may include a confidence threshold, which may be determined based on a pre-set confidence test result and user configuration.

[0236] For example, the corresponding relationship between the prediction accuracy and the prediction time consumption of at least one early-leaving module under different preset confidence levels may be pre-tested.

[0237] For example, the preset confidence can be determined to be 0.8, and then the maximum confidence in the output probability distribution of each early-exit branch in the target network can be obtained on the local data set of device A (for example, including 1,000 samples). If the maximum confidence corresponding to a certain early-exit branch is greater than the preset confidence, it can be determined that early exit is achieved from the early-exit branch, and the corresponding prediction result can be obtained. By traversing each sample in the local data set, the prediction time (which can also be considered as end-to-end delay) of each sample in the reasoning process can be obtained when the preset confidence is 0.8, and the average of the prediction time corresponding to each sample is used as the prediction time of the early-exit neural network. In addition, based on the prediction result corresponding to each sample and the corresponding label, the prediction accuracy of the early-exit neural network can be obtained when the preset confidence is 0.8.

[0238] Then, other values ​​of the preset confidence can be determined, and referring to the test process when the preset confidence is 0.8, the prediction accuracy and prediction time of the early-retirement neural network under other values ​​of the preset confidence can be obtained.

[0239] In this way, a curve describing the corresponding relationship between the prediction accuracy and the prediction time consumption of at least one early-leaving module under different preset confidence levels can be obtained.

[0240] For example, Figure 7 The figure shows an exemplary schematic diagram of the curve.

[0241] Figure 7 In the example shown, the horizontal axis can reflect the prediction time of the early-departure neural network, and the vertical axis is the prediction accuracy of the early-departure neural network. Each point in the curve corresponds to a preset confidence, which can be an attribute of the point but is not shown in Figure 7 in the curve.

[0242] Through testing, we know that Figure 7 In the curve, when the preset confidence is low, the prediction accuracy of the early-retirement neural network is low, but early retirement can usually be achieved in the shallower early-retirement module, the possibility of achieving early retirement is higher, and the prediction time is shorter; when the preset confidence is high, the prediction accuracy of the early-retirement neural network is high, and early retirement usually needs to be achieved in the deeper early-retirement module, the possibility of achieving early retirement is lower, and the prediction time is longer.

[0243] Of course, in some other examples, the confidence threshold can be determined by other forms of information. For example, a relationship curve between different preset confidences and the prediction accuracy of the early exit module and a relationship curve between different preset confidences and the prediction time consumption can be constructed respectively, so as to determine the confidence threshold in the current scenario based on the two curves.

[0244] After obtaining the corresponding relationship between the prediction accuracy and the prediction time consumption of at least one early exit module under different preset confidences, the user can determine the confidence threshold in the early exit condition from multiple preset confidences based on scenario requirements.

[0245] For example, a user can refer to the correspondence between the prediction accuracy and prediction time of at least one early exit module under different preset confidences described in the example shown in 7, and determine the confidence threshold from multiple preset confidences according to the requirements for prediction accuracy and / or prediction time in actual scenarios.

[0246] In addition, in some examples, the confidence threshold may be determined based on the communication performance between device A and the second device. For example, if the communication performance is poor, it may be considered to complete the prediction on device A as much as possible, and reduce the possibility of data transmission between device A and the second device, and a smaller confidence threshold may be determined.

[0247] In the embodiment of the present application, according to the early exit condition, determining whether the first prediction result meets the early exit condition, thereby determining the target prediction result may include the following two situations:

[0248] 1) Implement early exit of the model.

[0249] The early exit condition may include a confidence threshold. Device A may sequentially obtain the first prediction results output by the early exit module along the depth direction, and determine that the first prediction result satisfies the early exit condition each time a first prediction result is obtained.

[0250] Any first prediction result may include a probability distribution. If the maximum confidence in a first prediction result is greater than the confidence threshold, it means that the first prediction result is highly credible and meets the early exit condition. The first prediction result that meets the early exit condition may be used as the target prediction result, thereby achieving early exit of the model.

[0251] For example, Figure 8 In the example shown, device A is the first device n, and the early exit module n-1 in the first device n includes a classifier, which is used to output a first prediction result of the early exit module n-1.

[0252] Then, when the maximum confidence in the first prediction result of the early-leaving module n-1 is greater than the confidence threshold, the first prediction result of the early-leaving module n-1 is used as the target prediction result.

[0253] 2) Early exit of the model cannot be achieved.

[0254] If each first prediction result is not greater than the confidence threshold, it means that the model early exit cannot be achieved through the target network in device A. At this time, the target prediction result can be obtained through the target network in device A and the backend network corresponding to the target network in the second device.

[0255] In one example, the output of the last layer second module of the target network of device A can be transmitted to the second device, and then the target prediction result can be obtained based on the target network of device A in the backend network of the second device.

[0256] For example, Figure 8 In the example shown, device A is the first device 2 , and the early exit module 1 in the first device 2 includes a classifier, which is used to output a first prediction result of the early exit module 1 .

[0257] Then, when the maximum confidence in the first prediction result of the early exit module 1 is not greater than the confidence threshold, the first prediction result of the early exit module 1 is uploaded to the second device, so as to obtain the target prediction result through the trained second module 2 and subsequent modules in the second device, and then return it to the first device 2.

[0258] In another example, device A obtains a target prediction result of the data to be processed according to the early exit condition and the first prediction result, including:

[0259] If there is no first prediction result that satisfies the early exit condition, the first device sends the second feature tensor to the second device;

[0260] The second device obtains a target prediction result according to the received second feature tensor from a second module located after the target module among the plurality of second modules after training;

[0261] The second device sends the target prediction result to the first device;

[0262] Device A receives the target prediction result from device II.

[0263] Among them, the second feature tensor is the feature tensor output by the target module in device A according to the data to be processed, the target module is the trained first module in device A or the trained second module in device A, and the target module is determined based on one or more of the following information: the dimension of the output data of the trained first module in device A, the dimension of the output data of the trained second module in device A, and the communication performance between device A and the second device.

[0264] It can be seen that in this example, when early retirement is not possible, the output of device A can be flexibly determined from the output of the first module of device A after training and the output of the second module in the target network, so that the corresponding reasoning performance is optimal (for example, the end-to-end delay of the reasoning process is optimal).

[0265] Among them, the target module can be determined from the trained first module of device A and the second module in the target network, so that the dimension of the second feature tensor output by the target module conforms to the communication performance status between device A and the second device.

[0266] Specifically, if the communication performance between device A and the second device is good, the dimension of the second feature tensor output by the determined target module can be made larger; if the communication performance between device A and the second device is poor, the dimension of the second feature tensor output by the determined target module can be made smaller.

[0267] In this way, the second feature tensor output by the target module does not take a long time to be transmitted between device A and device II, thereby ensuring that the end-to-end inference delay is short.

[0268] The target module may be the first module, a relatively shallow module in the target network, or a relatively deep module, and there is no limitation here.

[0269] In one example, when the second device is used to implement a cloud platform, considering that the resources of the cloud platform are usually sufficient, when the dimension of the second feature tensor matches the communication performance between device A and the second device, the target module can be made a shallower module to make full use of the resources of the cloud platform and complete the reasoning process of the back-end network of the target module at a high speed to obtain the target prediction result, thereby improving the end-to-end reasoning efficiency.

[0270] For example, Figure 8In the example shown, device A is the first device 3. In the first prediction result output by the classifier in the early-leaving module 2 in the first device 3, the maximum confidence is not greater than the confidence threshold, and the communication performance between the first device 3 and the second device is poor, and the feature tensor output by the second module 1 is smaller than the feature tensor output by the second module 2. Then, the feature tensor output by the second module 1 in the first device 3 can be uploaded to the second device with less time, so that the target prediction result can be obtained through the second module 2 and subsequent modules in the trained second subnetwork in the second device, and then returned to the first device 3.

[0271] In addition, in some embodiments, when a first device only includes a trained first module, a trained second module and a trained early-retirement module, a target prediction result can be obtained according to the output of the trained first module.

[0272] In one example, a first device may be deployed with a trained first module and a trained first classifier.

[0273] If the maximum confidence of the output of the trained first classifier is greater than the confidence threshold, the output of the trained first classifier can be used as the target prediction result.

[0274] In other examples, since it is usually difficult for the trained first module to obtain a high-precision prediction result, the target prediction result can be obtained through the trained first module in the first device and the back-end network corresponding to the first module in the second device.

[0275] Specifically, the method further includes:

[0276] The first device obtains data to be processed;

[0277] The first device obtains a third feature tensor of the trained first module about the data to be processed;

[0278] The first device sends a third feature tensor to the second device;

[0279] The second device receives a third feature tensor from the first device;

[0280] The second device obtains a target prediction result according to the third feature tensor through the trained second module;

[0281] The second device sends the target prediction result to the first device;

[0282] The first device receives a target prediction result from the second device.

[0283] For example, Figure 8In the example shown, after the first device 1 obtains the third feature tensor of the trained first module about the data to be processed, it uploads the third feature tensor to the second device to obtain the target prediction result through the trained multiple second modules in the second device, and returns it to the first device 1.

[0284] The neural network training method provided in the embodiment of the present application is introduced above from multiple aspects. The neural network training system provided in the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0285] like Fig. 9 As shown, an embodiment of the present application provides a neural network training system 90, which includes multiple first devices 901 and second devices 902. The multiple first devices 901 are each deployed with a first module of a federated neural network, the first module includes a feature extraction module, the second device 902 is deployed with multiple second modules in the federated neural network, and each second module is respectively connected to an early exit module, the multiple first devices 901 include a first processing module 9011 and a first interface module 9012, and the second device 902 includes a second processing module 9021 and a second interface module 9022;

[0286] The plurality of first devices 901 and the second device 902 are used to: train the first module, the plurality of second modules and the early-leaving modules connected to each second module according to the first data of the plurality of first devices, and obtain the trained first module, the trained plurality of second modules and the trained early-leaving module;

[0287] The first interface module 9012 is used to: receive instruction information from the second device, the instruction information is used to instruct the multiple first devices to respectively deploy a target network, the target network deployed by device A among the multiple first devices includes module A among the multiple trained second modules and a trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple trained second modules and a trained early-leaving module connected to module B;

[0288] The first processing module 9011 is used to deploy a target network according to the instruction message.

[0289] Optionally, the training process includes one or more stages;

[0290] The first processing module 9011 is used to: in the i-th stage, process the first data of the i-th stage through the first module of the i-th stage to obtain the first feature tensor of the i-th stage, where i is a positive integer;

[0291] The first interface module 9012 is used to: send the first feature tensor of the i-th stage to the second device;

[0292] The second processing module 9021 is used for:

[0293] According to the received first feature tensor of the i-th stage, multiple second modules of the i-th stage and the early-leaving module of the i-th stage are trained to obtain multiple second modules of the i-th stage after training and the early-leaving module of the i-th stage after training;

[0294] Processing the first feature tensor of the i-th stage through the plurality of second modules after the training of the i-th stage to obtain the first probability distribution of the i-th stage;

[0295] The second interface module 9022 is used to: send the first probability distribution of the i-th stage to device A;

[0296] The first processing module 9011 is used to perform multiple iterative training on the first module of the i-th stage according to the first data of the i-th stage and the received first probability distribution of the i-th stage, so as to obtain the trained first module of the i-th stage.

[0297] Optionally, the first processing module 9011 is used to: perform knowledge distillation on the first module of the i-th stage according to the first probability distribution of the i-th stage and the first data of the i-th stage to obtain the first module after training of the i-th stage;

[0298] Among them, knowledge distillation is implemented according to a loss function, and the loss function includes a first loss term, which is used to evaluate the similarity between the second probability distribution obtained by the first module in the knowledge distillation process according to the first data of the i-th stage and the first probability distribution of the i-th stage.

[0299] Optionally, the second interface module 9022 is used to: obtain device performance information of device A, and / or communication performance information between device A and a second device;

[0300] The second processing module 9021 is used to determine the target network of device A according to the device performance information of device A and / or the communication performance information between device A and the second device.

[0301] Optionally, the first processing module 9011 is used to: fix the parameters of the second module in the target network, and train the early exit module in the target network according to the first data in device A to update the target network.

[0302] Optionally, the first interface module 9012 is used to: obtain data to be processed;

[0303] The first processing module 9011 is used for:

[0304] Obtaining a first prediction result of at least one early-retirement module in the target network regarding the data to be processed;

[0305] Obtaining a target prediction result of the data to be processed according to an early exit condition and the first prediction result, wherein the early exit condition includes a confidence threshold;

[0306] The confidence threshold is determined based on one or more of the following information:

[0307] The prediction accuracy of at least one early-leaving module in device A, the prediction time consumption of at least one early-leaving module in device A, and the communication performance between device A and a second device.

[0308] Optionally, the first processing module 9011 is used for: if any first prediction result satisfies an early exit condition, then device A uses the first prediction result satisfying the early exit condition as the target prediction result.

[0309] Optionally, the first interface module 9012 is used to: if there is no first prediction result that satisfies the early exit condition, send a second feature tensor to the second device, wherein the second feature tensor is a feature tensor output by a target module in device A according to the data to be processed, the target module is a trained first module in device A or a trained second module in device A, and the target module is determined according to one or more of the following information: the dimension of the output data of the trained first module in device A, the dimension of the output data of the trained second module in device A, and the communication performance between device A and the second device;

[0310] The second processing module 9021 is used to: obtain a target prediction result according to the received second feature tensor through a second module located after the target module among the plurality of trained second modules;

[0311] The second interface module 9022 is used to: send the target prediction result to device A;

[0312] The first interface module 9012 is used to receive the target prediction result from the second device.

[0313] like Fig.10 As shown, the embodiment of the present application provides a neural network training device 100, which is applied to a second device, the second device is deployed with multiple second modules in a federated neural network, and each second module is respectively connected to an early exit module, and multiple first devices are respectively deployed with a first module of the federated neural network, and the device 100 includes a second processing module 1001 and a second interface module 1002;

[0314] The second interface module 1002 is used to: receive respective first feature tensors from a plurality of first devices;

[0315] The second processing module 1001 is used to: train the multiple second modules and the early-leaving modules connected to each of the second modules according to the first feature tensor, to obtain the multiple second modules after training and the trained early-leaving modules, wherein the multiple second modules after training include module A and module B, and the trained early-leaving modules include the trained early-leaving modules connected to module A and the trained early-leaving modules connected to module B;

[0316] The second interface module 1002 is used to: send indication information to multiple first devices, the indication information is used to instruct the multiple first devices to respectively deploy a target network, the target network deployed by device A among the multiple first devices includes module A among the multiple trained second modules and a trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple trained second modules and a trained early-leaving module connected to module B.

[0317] Optionally, in the i-th phase:

[0318] The second interface module 1002 is used to: receive the first feature tensor of the i-th stage from device A, where i is a positive integer;

[0319] The second processing module 1001 is used to: train multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the received first feature tensor of the i-th stage, and obtain multiple second modules of the i-th stage after training and the early-leaving module of the i-th stage after training;

[0320] The second processing module 1001 is used to: process the first feature tensor of the i-th stage through the multiple second modules after the training of the i-th stage to obtain the first probability distribution of the i-th stage;

[0321] The second interface module 1002 is used to send the first probability distribution of the i-th stage to device A.

[0322] Optionally, the i-th stage includes one or more iterations;

[0323] The second processing module 1001 is used for:

[0324] According to the first feature tensor of the j-th iteration process, the output probability distribution of each early-leaving module of the j-th iteration process is obtained, where j is a positive integer;

[0325] According to each output probability distribution, the weight corresponding to each output probability distribution and the label corresponding to the first feature tensor of the j-th iteration process, the second module of the j-th iteration process and the early exit module of the j-th iteration process are updated to obtain the trained second module of the j-th iteration process and the trained early exit module of the j-th iteration process.

[0326] Optionally, the second interface module 1002 is used to: obtain device performance information of device A, and / or communication performance information between device A and a second device;

[0327] The second processing module 1001 is used to determine a target network of device A according to device performance information of device A and / or communication performance information between device A and a second device.

[0328] Optionally, the second interface module 1002 is used to: receive a second feature tensor from device A, where the second feature tensor is a feature tensor output by a target module in device A according to the data to be processed, and the target module is a trained first module in device A or a trained second module in device A;

[0329] The second processing module 1001 is used to: obtain a target prediction result according to the received second feature tensor through a second module located after the target module among the plurality of trained second modules;

[0330] The second interface module 1002 is used to send the target prediction result to device A.

[0331] In an embodiment of the present application, a module is used as an example of a software functional unit, and a neural network training system and / or a neural network training device may include code running on a computing instance. Among them, the computing instance may be at least one of a physical host (computing device), a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computing device may be one or more. For example, a neural network training system and / or a neural network training device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same available zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0332] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0333] As an example of a hardware functional unit, the neural network training system and / or the neural network training device may include at least one computing device, such as a server, etc. Alternatively, the neural network training system and / or the neural network training device may also be a device implemented by an application specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex PLD (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0334] The multiple computing devices included in the neural network training system and / or the neural network training apparatus can be distributed in the same region or in different regions. The multiple computing devices included in the neural network training system and / or the neural network training apparatus can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the neural network training system and / or the neural network training apparatus can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0335] It should be noted that, in other embodiments, the interface module (such as the first interface module and / or the second interface module) can be used to execute any step in the neural network training method, and the processing module (such as the first processing module and / or the second processing module) can be used to execute any step in the neural network training method. The steps that the processing module and the interface module are responsible for implementing can be specified as needed. The full functions of the neural network training system and / or the neural network training device are realized by respectively implementing different steps in the neural network training method through the processing module and the interface module.

[0336] In the embodiment of the present application, any first device among the multiple first devices may be a computing device cluster, which includes one or more computing devices.

[0337] The following is an exemplary introduction to the hardware structure of the computing device involved when the first device is a computing device cluster and the hardware structure of the computing device cluster.

[0338] The present application embodiment also provides a computing device 110. Fig.11As shown, the computing device 110 includes: a bus 112, a processor 114, a memory 116, and a communication interface 118. The processor 114, the memory 116, and the communication interface 118 communicate with each other through the bus 112. The computing device 110 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 110.

[0339] The bus 112 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 The bus 114 may include a path for transmitting information between various components of the computing device 110 (eg, the memory 116, the processor 114, and the communication interface 118).

[0340] The processor 114 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0341] The memory 116 may include a volatile memory, such as a random access memory (RAM). The processor 114 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0342] The memory 116 stores executable program codes, and the processor 114 executes the executable program codes to respectively implement the functions of the aforementioned first interface module and the first processing module, thereby implementing the neural network training method applied to the first device (e.g., device A) in the above-mentioned embodiment. That is, the memory 116 stores instructions for executing the neural network training method applied to the first device in the above-mentioned embodiment.

[0343] The communication interface 118 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 110 and other devices or a communication network.

[0344] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0345] like Fig.12 As shown, the computing device cluster includes at least one computing device 110. The memory 116 in one or more computing devices 110 in the computing device cluster may store the same instructions for executing the neural network training method applied to the first device in the above embodiment.

[0346] In some possible implementations, the memory 116 of one or more computing devices 110 in the computing device cluster may also respectively store some instructions for executing the neural network training method applied to the first device in the above embodiment. In other words, the combination of one or more computing devices 110 may jointly execute instructions for executing the neural network training method applied to the first device in the above embodiment.

[0347] It should be noted that the memory 116 in different computing devices 110 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the neural network training method applied to the first device in the above embodiment. That is, the instructions stored in the memory 116 in different computing devices 110 can implement the functions of one or more modules in the first interface module and the first processing module.

[0348] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.13 A possible implementation is shown. Fig.13 As shown, two computing devices 110A and 110B are connected via a network. Specifically, the network is connected via the communication interface in each computing device. In this type of possible implementation, the memory 116 in the computing device 110A may store instructions for executing the functions of the first processing module. At the same time, the memory 116 in the computing device 110B may store instructions for executing the functions of the first interface module. Alternatively, the memory 116 in the computing device 110A may store instructions for executing part of the functions of the first processing module. At the same time, the memory 116 in the computing device 110B may store instructions for executing another part of the functions of the first processing module.

[0349] It should be understood that Fig.13 The functionality of the computing device 110A shown in FIG. 1 may also be implemented by multiple computing devices 110 . Similarly, the functionality of the computing device 110B may also be implemented by multiple computing devices 110 .

[0350] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.12 and Fig.13 Connection mode of computing device cluster. The difference is that the memory 116 in one or more computing devices 110 in the computing device cluster may store the same instructions for executing the neural network training method applied to the first device in the above embodiment.

[0351] In some possible implementations, the memory 116 of one or more computing devices 110 in the computing device cluster may also respectively store some instructions for executing the neural network training method applied to the first device in the above embodiment. In other words, the combination of one or more computing devices 110 may jointly execute instructions for executing the neural network training method applied to the first device in the above embodiment.

[0352] It should be noted that the memory 116 in different computing devices 110 in the computing device cluster may store different instructions for executing part of the functions of the neural network training method applied to the first device in the above embodiment. That is, the instructions stored in the memory 116 in different computing devices 110 may implement the functions of one or more modules in the first interface module and the first processing module.

[0353] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the neural network training method applied to the first device in the above embodiment.

[0354] The present application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the neural network training method applied to the first device in the above embodiment.

[0355] The embodiment of the present application also provides a chip system, which includes a processor, and the processor is used to implement the steps performed by the above-mentioned computing device cluster. In a possible design, the chip system may also include a memory, and the memory is used to store necessary program instructions and data. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0356] In the embodiment of the present application, the second device may be a computing device cluster, which includes one or more computing devices.

[0357] The following is an exemplary introduction to the hardware structure of the computing device involved when the second device is a computing device cluster and the hardware structure of the computing device cluster.

[0358] The present application embodiment also provides a computing device 140. Fig.14 As shown, the computing device 140 includes: a bus 142, a processor 144, a memory 146, and a communication interface 148. The processor 144, the memory 146, and the communication interface 148 communicate with each other through the bus 142. The computing device 140 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 140.

[0359] The bus 142 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 The bus 144 may include a path for transmitting information between various components of the computing device 140 (eg, the memory 146, the processor 144, and the communication interface 148).

[0360] The processor 144 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0361] The memory 146 may include a volatile memory, such as a random access memory (RAM). The processor 144 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0362] The memory 146 stores executable program codes, and the processor 144 executes the executable program codes to respectively implement the functions of the aforementioned second interface module and the second processing module, thereby implementing the neural network training method applied to the second device in the above-mentioned embodiment, that is, the memory 146 stores instructions for executing the neural network training method applied to the second device in the above-mentioned embodiment.

[0363] The communication interface 148 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 140 and other devices or a communication network.

[0364] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0365] like Fig.15 As shown, the computing device cluster includes at least one computing device 140. The memory 146 in one or more computing devices 140 in the computing device cluster may store the same instructions for executing the neural network training method applied to the second device in the above embodiment.

[0366] In some possible implementations, the memory 146 of one or more computing devices 140 in the computing device cluster may also respectively store some instructions for executing the neural network training method applied to the second device in the above embodiment. In other words, the combination of one or more computing devices 140 can jointly execute the instructions for executing the neural network training method applied to the second device in the above embodiment.

[0367] It should be noted that the memory 146 in different computing devices 140 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the neural network training method applied to the second device in the above embodiment. That is, the instructions stored in the memory 146 in different computing devices 140 can implement the functions of one or more modules in the second interface module and the second processing module.

[0368] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.16 A possible implementation is shown. Fig.16 As shown, two computing devices 140A and 140B are connected via a network. Specifically, the network is connected via the communication interface in each computing device. In this type of possible implementation, the memory 146 in the computing device 140A may store instructions for executing the functions of the second interface module. At the same time, the memory 146 in the computing device 140B may store instructions for executing the functions of the second processing module. Alternatively, the memory 146 in the computing device 140A may store instructions for executing part of the functions of the second processing module. At the same time, the memory 146 in the computing device 140B may store instructions for executing another part of the functions of the second processing module.

[0369] It should be understood that Fig.16 The functionality of computing device 140A shown in FIG. 1 may also be performed by multiple computing devices 140. Similarly, the functionality of computing device 140B may also be performed by multiple computing devices 140.

[0370] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.15 and Fig.16 Connection mode of computing device cluster. The difference is that the memory 146 in one or more computing devices 140 in the computing device cluster may store the same instructions for executing the neural network training method applied to the second device in the above embodiment.

[0371] In some possible implementations, the memory 146 of one or more computing devices 140 in the computing device cluster may also respectively store some instructions for executing the neural network training method applied to the second device in the above embodiment. In other words, the combination of one or more computing devices 140 can jointly execute the instructions for executing the neural network training method applied to the second device in the above embodiment.

[0372] It should be noted that the memory 146 in different computing devices 140 in the computing device cluster may store different instructions for executing part of the functions of the neural network training method applied to the second device in the above embodiment. That is, the instructions stored in the memory 146 in different computing devices 140 may implement the functions of one or more modules in the second interface module and the second processing module.

[0373] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the neural network training method applied to the second device in the above embodiment.

[0374] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the neural network training method applied to the second device in the above embodiment.

[0375] The embodiment of the present application also provides a chip system, which includes a processor, and the processor is used to implement the steps performed by the above-mentioned computing device cluster. In a possible design, the chip system may also include a memory, and the memory is used to store necessary program instructions and data. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0376] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0377] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0378] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0379] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0380] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A neural network training method, characterized in that: Applied to a plurality of first devices and a second device, each of the plurality of first devices is deployed with a first module of a federated neural network, the first module includes a feature extraction module, the second device is deployed with a plurality of second modules in the federated neural network, and an early exit module connected to each of the second modules, the method includes: According to the first data of the multiple first devices, the first module, the multiple second modules, and the early exit module connected to each of the second modules are trained to obtain the trained first module, the trained multiple second modules, and the trained early exit module; The multiple first devices receive indication information from the second device, the indication information is used to instruct the multiple first devices to respectively deploy a target network, the target network deployed by device A among the multiple first devices includes module A among the multiple trained second modules and the trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple trained second modules and the trained early-leaving module connected to module B; The multiple first devices deploy the target network according to the indication message.

2. The method according to claim 1, characterized in that The training process consists of one or more phases; The step of training the first module, the plurality of second modules, and the early exit module connected to each of the second modules according to the first data of the plurality of first devices comprises: In the i-th stage, the device A processes the first data of the i-th stage through the first module of the i-th stage to obtain the first feature tensor of the i-th stage, where i is a positive integer; The device A sends the first feature tensor of the i-th stage to the second device; The second device trains the multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the received first feature tensor of the i-th stage, and obtains the multiple second modules of the i-th stage and the early-leaving module of the i-th stage after training; The second device processes the first feature tensor of the i-th stage through the multiple second modules trained in the i-th stage to obtain the first probability distribution of the i-th stage; The second device sends the first probability distribution of the i-th stage to the first device; The device A performs multiple iterative training on the first module of the i-th stage according to the first data of the i-th stage and the received first probability distribution of the i-th stage to obtain the trained first module of the i-th stage.

3. The method according to claim 2, characterized in that The device A performs multiple iterative training on the first module of the i-th stage according to the first data of the i-th stage and the received first probability distribution of the i-th stage to obtain the trained first module of the i-th stage, including: The device A performs knowledge distillation on the first module of the i-th stage according to the first probability distribution of the i-th stage and the first data of the i-th stage to obtain the first module after training of the i-th stage; In which, the knowledge distillation is implemented according to a loss function, and the loss function includes a first loss term, and the first loss term is used to evaluate the similarity between the second probability distribution obtained by the first module in the knowledge distillation process based on the first data of the i-th stage and the first probability distribution of the i-th stage.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The second device acquires device performance information of the first device and / or communication performance information between the first device and the second device; The second device determines the target network of the device A according to the device performance information of the device A and / or the communication performance information between the device A and the second device.

5. The method according to any one of claims 1 to 4, characterized in that: After the device A deploys the target network according to the instruction message, the method further includes: The device A fixes the parameters of the second module in the target network, and trains the early-retirement module in the target network according to the first data in the device A to update the target network.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The device A obtains data to be processed; The device A obtains a first prediction result of at least one of the early exit modules on the data to be processed in the target network; The device A obtains the target prediction result of the to-be-processed data according to the early exit condition and the first prediction result, wherein the early exit condition includes a confidence threshold; The confidence threshold is determined based on one or more of the following information: The prediction accuracy of at least one of the early exit modules in the device A, the prediction time consumption of at least one of the early exit modules in the device A, and the communication performance between the device A and the second device.

7. The method according to claim 6, characterized in that The device A obtains the target prediction result of the data to be processed according to the early exit condition and the first prediction result, including: If any of the first prediction results meets the early exit condition, the device A uses the first prediction result that meets the early exit condition as the target prediction result.

8. The method according to claim 6, characterized in that The device A obtains the target prediction result of the data to be processed according to the early exit condition and the first prediction result, including: If there is no first prediction result that satisfies the early exit condition, the device A sends a second feature tensor to the second device, wherein the second feature tensor is a feature tensor output by a target module in the device A according to the data to be processed, the target module is a trained first module in the device A or a trained second module in the device A, and the target module is determined according to one or more of the following information: the dimension of the output data of the trained first module in the device A, the dimension of the output data of the trained second module in the device A, and the communication performance between the device A and the second device; The second device obtains the target prediction result according to the received second feature tensor through a second module located after the target module among the plurality of trained second modules; The second device sends the target prediction result to the first device; The device A receives the target prediction result from the second device.

9. A neural network training method, characterized in that: Applied to a second device, the second device is deployed with a plurality of second modules in a federated neural network, and an early exit module to which each of the second modules is respectively connected, and a plurality of first devices are respectively deployed with a first module of the federated neural network, the method comprising: The second device receives respective first feature tensors from the plurality of first devices; The second device trains the multiple second modules and the early-leaving modules connected to each of the second modules according to the first feature tensor to obtain the multiple second modules after training and the trained early-leaving modules, wherein the multiple second modules after training include module A and module B, and the trained early-leaving modules include the trained early-leaving modules connected to the module A and the trained early-leaving modules connected to the module B; The second device sends indication information to the multiple first devices, and the indication information is used to instruct the multiple first devices to respectively deploy a target network, the target network deployed by device A among the multiple first devices includes module A among the multiple second modules after training and the trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple second modules after training and the trained early-leaving module connected to module B.

10. The method according to claim 9, characterized in that The second device trains the plurality of second modules and the early-leaving modules connected to each of the second modules according to the first feature tensor, including: In the i-th stage, the second device receives the first feature tensor of the i-th stage from the first device, where i is a positive integer; The second device trains the multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the received first feature tensor of the i-th stage, and obtains the multiple second modules of the i-th stage and the early-leaving module of the i-th stage after training; The second device processes the first feature tensor of the i-th stage through the multiple second modules trained in the i-th stage to obtain the first probability distribution of the i-th stage; The second device sends the first probability distribution of the i-th stage to the first device.

11. The method according to claim 10, characterized in that The i-th stage includes one or more iterations; In the j-th iteration process of the i-th stage, the second device trains the multiple second modules of the i-th stage and the early-leaving module of the i-th stage according to the received first feature tensor of the i-th stage, including: The second device obtains the output probability distribution of each early-leaving module of the j-th iterative process according to the first feature tensor of the j-th iterative process, where j is a positive integer; The second device updates the second module of the j-th iterative process and the early-leaving module of the j-th iterative process according to each output probability distribution, the weights corresponding to each output probability distribution and the label corresponding to the first feature tensor of the j-th iterative process, and obtains the trained second module of the j-th iterative process and the trained early-leaving module of the j-th iterative process.

12. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: The second device acquires device performance information of the first device and / or communication performance information between the first device and the second device; The second device determines the target network of the device A according to the device performance information of the device A and / or the communication performance information between the device A and the second device.

13. The method according to any one of claims 9 to 12, characterized in that: The method further comprises: The second device receives a second feature tensor from the device A, where the second feature tensor is a feature tensor output by a target module in the device A according to data to be processed, and the target module is a trained first module in the device A or a trained second module in the device A; The second device obtains the target prediction result according to the received second feature tensor through a second module located after the target module among the plurality of trained second modules; The second device sends the target prediction result to the first device.

14. A neural network training system, characterized in that: The neural network training system includes a plurality of first devices and a second device, each of the plurality of first devices is deployed with a first module of a federated neural network, the first module includes a feature extraction module, the second device is deployed with a plurality of second modules in the federated neural network, and an early withdrawal module connected to each of the second modules, the plurality of first devices include a first processing module and a first interface module, and the second device includes a second processing module and a second interface module; The multiple first devices and the second device are used to: train the first module, the multiple second modules and the early exit module connected to each of the second modules according to the first data of the multiple first devices, to obtain the trained first module, the trained multiple second modules and the trained early exit module; The first interface module is used to: receive indication information from the second device, the indication information is used to instruct the multiple first devices to respectively deploy a target network, the target network deployed by device A among the multiple first devices includes module A among the multiple trained second modules and the trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple trained second modules and the trained early-leaving module connected to module B; The first processing module is used to deploy the target network according to the indication message.

15. The system according to claim 14, characterized in that The second interface module is used to: obtain device performance information of the device A, and / or communication performance information between the device A and the second device; The second processing module is used to determine the target network of the device A according to the device performance information of the device A and / or the communication performance information between the device A and the second device.

16. The system according to claim 14 or 15, characterized in that The first interface module is used to: obtain data to be processed; The first processing module is used for: Obtaining a first prediction result of at least one of the early-leaving modules in the target network regarding the data to be processed; Obtaining a target prediction result of the data to be processed according to an early exit condition and the first prediction result, wherein the early exit condition includes a confidence threshold; The confidence threshold is determined based on one or more of the following information: The prediction accuracy of at least one of the early exit modules in the device A, the prediction time consumption of at least one of the early exit modules in the device A, and the communication performance between the device A and the second device.

17. The system according to claim 16, characterized in that The first interface module is used for: if there is no first prediction result satisfying the early exit condition, sending a second feature tensor to the second device, wherein the second feature tensor is a feature tensor output by a target module in the device A according to the data to be processed, the target module is a trained first module in the device A or a trained second module in the device A, and the target module is determined according to one or more of the following information: the dimension of the output data of the trained first module in the device A, the dimension of the output data of the trained second module in the device A, and the communication performance between the device A and the second device; The second processing module is used to: obtain the target prediction result according to the received second feature tensor through a second module located after the target module among the plurality of trained second modules; The second interface module is used to: send the target prediction result to the device A; The first interface module is used to receive the target prediction result from the second device.

18. A neural network training device, characterized in that: Applied to a second device, the second device is deployed with a plurality of second modules in a federated neural network, and an early exit module to which each of the second modules is respectively connected, a plurality of first devices are respectively deployed with a first module of the federated neural network, and the apparatus includes a second processing module and a second interface module; The second interface module is used to: receive respective first feature tensors from the plurality of first devices; The second processing module is used for: training the plurality of second modules and the early-leaving modules connected to each of the second modules according to the first feature tensor to obtain the plurality of trained second modules and the trained early-leaving modules, wherein the plurality of trained second modules include module A and module B, and the trained early-leaving modules include the trained early-leaving modules connected to the module A and the trained early-leaving modules connected to the module B; The second interface module is used to: send indication information to the multiple first devices, and the indication information is used to instruct the multiple first devices to respectively deploy a target network, the target network deployed by device A among the multiple first devices includes module A among the multiple second modules after training and the trained early-leaving module connected to module A, and the target network deployed by device B among the multiple first devices includes module B among the multiple second modules after training and the trained early-leaving module connected to module B.

19. The device according to claim 18, characterized in that The second interface module is used to: obtain device performance information of the device A, and / or communication performance information between the device A and the second device; The second processing module is used to determine the target network of the device A according to the device performance information of the device A and / or the communication performance information between the device A and the second device.

20. The device according to claim 18 or 19, characterized in that The second interface module is used to: receive a second feature tensor from the device A, where the second feature tensor is a feature tensor output by a target module in the device A according to the data to be processed, and the target module is a trained first module in the device A or a trained second module in the device A; The second processing module is used to: obtain the target prediction result according to the received second feature tensor through a second module located after the target module among the plurality of trained second modules; The second interface module is used to send the target prediction result to the device A.

21. A neural network training system, characterized in that: The neural network training system includes a plurality of first devices and a second device, wherein the plurality of first devices and the second device include a processor, a memory, and instructions stored in the memory and executable by the processor, and the processor executes the instructions to implement the steps of the method described in any one of claims 1 to 8.

22. A second device, characterized in that: The second device includes a processor, a memory, and instructions stored in the memory and executable by the processor, and the processor executes the instructions to implement the steps of the method described in any one of claims 9 to 13.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images