Efficient communication and computation for split training of machine learning models

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for an improved machine learning training process. Input data for a first portion of the neural network is accessed, and output data is generated based on processing the input data using the first portion of the neural network. At the first point in time, the output data is sent to the second computing system for the second computing system to update one or more parameters of the second portion of the neural network based on the output data. At a second point in time after the first point in time, it is determined that one or more communication criteria are not satisfied. In response to the determination, reduced communication training of the neural network is performed, the performing including reducing the amount of data transmitted for one or more rounds of training.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 463,108, filed September 7, 2023, which has been assigned to the assignee of this application and is expressly incorporated herein by reference in its entirety, as fully set forth below and for all applicable purposes. Background Technology

[0003] This disclosure relates to various aspects of machine learning.

[0004] A wide variety of machine learning model architectures have proliferated and are used to provide solutions to a wide range of prediction and inference problems. Many machine learning model architectures, especially those for complex tasks (e.g., deep learning models such as neural networks), have a large number of parameters. Furthermore, as task complexity increases (and as desired accuracy increases), the number of parameters used has increased rapidly (e.g., easily reaching billions in some modern models). This large model size makes training (and in some cases, inference) on many devices (especially resource-constrained systems such as wearables and Internet of Things (IoT) devices) impractical (and in some cases impossible).

[0005] Various methods have been proposed to mitigate this computational cost. Some of these solutions provide techniques for jointly training machine learning models on two or more devices. As an example, one approach involves splitting the training, where each participating device or system trains a portion of a larger model. This reduces the computational cost on each system but results in significant communication overhead between the systems (e.g., for sending features and / or gradients between systems for each round of training). Summary of the Invention

[0006] Some aspects provide a method comprising: accessing a first element of input data for a first part of a neural network; generating a first element of output data by a first computing system processing the first element of the input data using the first part of the neural network; sending the first element of the output data from the first computing system to a second computing system at a first time point for the second computing system to update one or more parameters of a second part of the neural network based on the first element of the output data; determining by the first computing system at a second time point after the first time point that one or more communication criteria are not satisfied; and, in response to the determination, performing reduced communication training of the neural network, the execution comprising reducing the amount of data sent by the first computing system for one or more rounds of training.

[0007] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0008] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0009] The accompanying drawings depict certain features of one or more aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0010] FIG. 1 Example environments for reducing communication splitting learning are illustrated according to various aspects of this disclosure.

[0011] FIG. 2A An example workflow for learning reduced communication splits in complete (e.g., uplink and downlink) data states, according to various aspects of this disclosure, is described.

[0012] FIG. 2B An example workflow for learning reduced communication splitting in reduced downlink data states, based on various aspects of this disclosure, is described.

[0013] FIG. 2C An example workflow for learning reduced communication splitting in reduced uplink states, based on various aspects of this disclosure, is described.

[0014] FIG. 2D An example workflow for learning reduced communication splits in a bidirectional reduced state, according to various aspects of this disclosure, is described.

[0015] FIG. 3A An example workflow for learning reduced communication splits using dynamic uplink compression, according to various aspects of this disclosure, is described.

[0016] FIG. 3B An example workflow for learning reduced communication splits using dynamic downlink compression, based on various aspects of this disclosure, is described.

[0017] FIG. 4This is a flowchart depicting an example method for reducing communication splitting learning according to various aspects of this disclosure.

[0018] FIG. 5 This is a flowchart depicting example methods for asynchronous split learning performed by a client system, according to various aspects of this disclosure.

[0019] FIG. 6 This is a flowchart depicting example methods for asynchronous split learning performed by a server system, according to various aspects of this disclosure.

[0020] FIG. 7 This is a flowchart depicting an example method for reduced communication training of machine learning models according to various aspects of this disclosure.

[0021] FIG. 8 Example processing systems configured to perform various aspects of this disclosure are described.

[0022] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0023] This disclosure provides apparatus, methods, processing systems, and computer-readable media for reduced communication training and reduced computation training of machine learning models.

[0024] In some conventional methods of split training, the increased communication overhead introduced by splitting the model into parts (and allowing each system to train the corresponding part) can reduce or even eliminate the benefits of split training. Aspects of this disclosure provide techniques for dynamically modifying the split training process to reduce communication and / or computational overhead. This results in an improved training process because computational and communication costs can be reduced (or at least deferred to a later time when such costs have a smaller impact on other operations of the participating devices).

[0025] In some aspects, reduced communication training may include the use of various techniques, including asynchronous training and dynamic compression. As discussed in more detail below, asynchronous communication or training can be used to reduce the number of updates exchanged between participating systems. This significantly reduces communication overhead and computational load and cost (e.g., by reducing the amount of data expected to be processed by each side at a given time). Additionally, as discussed in more detail below, dynamic compression can be used to dynamically select the amount of compression to utilize when exchanging data between participating systems. This (e.g., compared to simply compressing all exchanged data) can significantly reduce communication overhead while also reducing errors introduced by compression. In various aspects of this disclosure, reduced communication training can be performed using asynchronous training, dynamic compression, or a combination of both, as discussed in more detail below.

[0026] In some respects, dynamic signaling can be used to exchange information between participating devices to enable reduced communication protocols (e.g., to indicate whether uplink and / or downlink data should be reduced or suspended, to indicate which compression scheme(s) should be used, etc.). This enables enhanced training of machine learning models that allows additional devices (which might otherwise not be able to participate in the training) to participate, and reduces the computational and communication costs of such participation.

[0027] Example environment for reduced-communication split learning

[0028] FIG. 1 An example environment 100 for reducing communication split learning according to various aspects of this disclosure is illustrated.

[0029] In the illustrated example, client system 105 is communicatively coupled to server system 110 via link 120 (which may include network 115). Link 120 generally represents any suitable communication system and technology. Link 120 may include a combination of wired and / or wireless links. In some respects, network 115 corresponds to the Internet.

[0030] Although depicted as discrete systems for clarity of concept, in various respects, client system 105 and / or server system 110 may generally be implemented as independent computing systems and / or as components of one or more broader systems. For example, client system 105 may correspond to a machine learning component on a wearable device, while server system 110 may be implemented as part of a cloud-based deployment. In some respects, client system 105 is a relatively more resource-constrained computing system compared to server system 110. Although a single client system 105 and a single server system 110 are depicted for clarity of concept, in various respects, multiple client systems 105 and / or multiple server systems 110 may participate in the training process.

[0031] For example, client system 105 may correspond to wearable devices or IoT devices (e.g., with relatively low clock cycles, small amounts of memory, limited storage capacity, limited or no parallel processing capabilities, limited power or battery capacity, etc.), while server system 110 may correspond to a relatively more powerful system (e.g., a server cluster with high clock cycles, large amounts of memory and storage, high parallel capacity, etc.). In some aspects, client system 105 and / or server system 110 may be constrained based on various hard and / or soft constraints. For example, hard constraints may refer to hardware limitations (e.g., memory size), while soft constraints may refer to dynamic or time constraints (e.g., sometimes resources may be allocated to higher priority tasks, leaving a relatively small portion of those resources for training). In some aspects, as discussed in more detail below, these and other factors may be evaluated to determine which(s) of the reduced communication methods should be utilized at any given time to facilitate efficient training operations.

[0032] In the illustrated example, client system 105 and server system 110 are jointly training a machine learning model, wherein client system 105 trains a first part 125A of the machine learning model, and server system 110 trains a second part 125B of the machine learning model (collectively referred to as part 125). In some respects, part 125 may each be referred to as the machine learning model. For example, if the architecture of the model is a neural network, then parts 125A and 125B may be individually referred to as a neural network, a subnetwork or sub-network, or a part of a neural network.

[0033] As illustrated, the first portion 125A comprises one or more layers 130A to 130N, while the second portion 125B comprises one or more layers 135A to 135N. In some aspects, layers 130A to 130N may be referred to as initial layers to indicate that layers 130A to 130N comprise the beginning or first layer of the model but not the final layer. Similarly, layers 135A to 135N may be referred to as final layers to indicate that layers 135A to 135N comprise the last layer of the model but not the first layer. In various aspects, portions 125A and 125B may each comprise any number of layers (including a single layer), and portions 125A and 125B together form the entire model. For example, the model may be defined as a sequence of layers, and these layers may be depicted as two portions 125 at partition points (also referred to in some aspects as split points), each portion containing one or more layers. Various techniques may be used to select partition points to balance one or more objectives (e.g., minimizing latency and / or power consumption).

[0034] In some aspects, the client system 105 defines the model architecture (e.g., determines the values ​​of hyperparameters such as learning rate, number of layers, etc.) and selects partition points (e.g., determines the partition points for the first...). The layer is retained in part 125A, and the last one will be... The layer is devolved to part 125B). In other aspects, server system 110 defines the architecture and partition points. In some aspects, client system 105 and server system 110 may jointly or collaboratively define the architecture and partition points.

[0035] In each respect, during training, client system 105 can process input data using a portion 125A of the model (e.g., using initial layers 130A to 130N) to generate a set of features (also referred to as intermediate features or feature tensors) output from intermediate layers of the model (e.g., layer 130N). These features are then sent to server system 110, which performs a forward pass by processing the features using portion 125B (e.g., starting from layer 135A) to generate an output from layer 135N. This output can then be used to train the model (e.g., by generating a loss and backpropagating through portion 125B). In each respect, the gradients at layer 135A (also referred to as gradient tensors) are then sent to client system 105, and client system 105 can update portion 125A by continuing backpropagation using these gradients.

[0036] In this manner, for each training round, features are sent from client system 105 to server system 110, and gradients are sent from server system 110 to client system 105. This communication can introduce significant overhead, especially since many rounds of such training are typically preferred. In the illustrated example, therefore, client system 105 and server system 110 can use dynamically reduced communication training techniques to significantly reduce this overhead. For example, as discussed in more detail below, client system 105 and server system 110 can selectively exchange data. Some non-limiting examples of selectively exchanging data may include: for one or more iterations, abandoning the sending of features from client system 105 to server system 110; for one or more iterations, abandoning the sending of gradients from server system 110 to client system 105; for one or more iterations, avoiding the exchange of both features and gradients, etc. As another example, as discussed in more detail below, client system 105 and server system 110 may dynamically select compression schemes or parameters for each iteration in order to reduce communication and / or computation costs while minimizing (or at least reducing) errors introduced by (lossy) compression techniques.

[0037] In some aspects, participating systems may use one or more finite state machines (FSMs) to determine which state to use. For example, suppose each communication pattern is represented by a state or node in the FSM (e.g., a first state for full data operation, a second state for reduced downlink operation, a third state for reduced uplink operation, and a fourth state for bidirectional reduced operation). Transition criteria can then be defined between each state, allowing participating systems to move from one state to another (or remain in the current state) based on whether the corresponding transition criterion is met. In some aspects, depending on the specific implementation, training may operate within a subset of possible operating states (e.g., switching between a full data state and a reduced downlink state, but without using a reduced uplink state or a bidirectional reduced data state).

[0038] For example, suppose the training loss at server system 110 is used as a metric to trigger an operational state transition (e.g., the difference in loss between two adjacent training epochs or iterations can be compared to determine whether a transition occurs, where a smaller change in loss indicates slower training progress). Further suppose the participating systems expect to continue updating the second part 125B of the model frequently or normally (e.g., the final part), while updating the first part 125A intermittently or less frequently when training progress is slow. In some aspects, if the change in loss satisfies defined criteria (e.g., meets or exceeds a threshold), client system 105 and server system 110 may determine to use the full data state, where features and gradients are exchanged and both parts 125A and 125B are updated. In some aspects, if the change in loss does not satisfy defined criteria (e.g., is less than a threshold), client system 105 and server system 110 may determine to use the reduced downlink data state, where features are sent to server system 110 and part 125B is updated, but gradients are not sent to client system 105 and part 125A is frozen.

[0039] As another example, suppose the channel condition between client system 105 and server system 110 is used as a metric to trigger operational state transitions (e.g., measured by data rate). Further suppose the participating systems expect to continue training both parts 125A and 125B when the data rate is sufficiently high and avoid exchanging data when the data rate is low. In some aspects, if the data rate meets defined criteria (e.g., meets or exceeds a threshold), client system 105 and server system 110 may determine to use the full data state, where features and gradients are exchanged and both parts 125A and 125B are updated. In some aspects, if the data rate does not meet defined criteria (e.g., is less than a threshold), client system 105 and server system 110 may determine to use a different state, such as a bidirectional reduced data state, where features are not sent to server system 110 and gradients are not sent to client system 105. In some aspects, server system 110 may pause updates to part 125B. In other aspects, server system 110 may continue training part 125B based on one or more previously received features (discussed in more detail below).

[0040] As another example, suppose that the characteristics of the training data available on client system 105 (e.g., the signal-to-noise ratio (SNR) of the available data in the current iteration, the number of available samples, and / or the distribution of the available samples) are used as metrics to trigger operational state transitions. Further suppose that the participating systems expect to continue training both parts 125A and 125B when the training data characteristics meet the criteria, and avoid exchanging data when the characteristics do not meet the criteria. In some aspects, if the characteristics meet the defined criteria (e.g., meet or exceed a threshold), client system 105 and server system 110 may determine to use the full data state, where features and gradients are exchanged and both parts 125A and 125B are updated. In some aspects, if the characteristics do not meet the defined criteria (e.g., are less than a threshold), client system 105 and server system 110 may determine to use a different state, such as a bidirectionally reduced data state, where features are not sent to server system 110 and gradients are not sent to client system 105. In some aspects, server system 110 may pause updates to part 125B. In other respects, server system 110 may continue training part 125B based on one or more previously received features (discussed in more detail below).

[0041] In some respects, as discussed above, client system 105 and server system 110 can similarly use FSM and transformation criteria to determine which compression(s) to use in any given iteration.

[0042] In these ways, using various aspects of this disclosure, the reduced communication training techniques described herein can significantly reduce the costs incurred in training models, thereby allowing increased participation from client systems 105 and server systems 110, enabling the training of larger and more complex models, and reducing the bandwidth consumed by such training.

[0043] Example workflow for reduced-communication split learning

[0044] FIG. 2A to FIG. 2D An example workflow for learning to reduce communication splitting, based on various aspects of this disclosure, is described. Specifically, FIG. 2A to FIG. 2D Each of these describes a different reduced communication state, which can be used to reduce the communication and computational overhead for split training of machine learning models. In some respects, the participating system can move dynamically between these states based on a variety of criteria (e.g., in...). FIG. 2A to FIG. 2D (Switching between workflows described herein). In some respects, any participating system can initiate a transition to a different operational state. In other respects, participating systems negotiate or agree on the operational state at the beginning of each iteration, as discussed in more detail below.

[0045] The criteria or factors used to determine whether to switch to a new state may vary depending on the specific implementation. As an example, and not a limitation, participating systems may consider or evaluate the state of one or more communication links between participating computing systems (e.g., current data rate, connection stability or uptime, transmission retry rate, etc.), resource availability on either or both participating systems (e.g., whether either system is currently handling other higher-priority operations, leaving little or no resources available for training), indications of the model's training progress (e.g., comparing the current loss value with a previous loss value to determine if training is in progress, where the system can determine to move to a different state if the change in the loss value is less than or greater than one or more thresholds), characteristics of the training data used to train the model (e.g., whether there is enough data for another round of training, whether the signal-to-noise ratio (SNR) of the current batch of training data meets specified criteria, whether the distribution of the current batch meets specified criteria, etc.), etc. See below for reference. FIG. 4 Some example evaluations of the criteria used for dynamically switching between training modes or states are discussed in more detail.

[0046] FIG. 2A An example workflow 200A for reduced communication split learning in a complete data state is described according to various aspects of this disclosure.

[0047] In the illustrated example, the "complete data" state can refer to an operational or training state in which client system 105 and server system 110 exchange complete training data (e.g., for each sample of the training data, client system 105 sends features to server system 110, and server system 110 sends gradients to client system 105). Specifically, as illustrated, client system 105 uses a portion 125A of the model (e.g., using initial layers 130A to 130N) to process input data 205 to generate a feature tensor 210. As discussed above, this feature tensor 210 generally corresponds to intermediate features output by the network's inner or hidden layers (e.g., by layer 130N).

[0048] As illustrated, during the “complete data” operation, client system 105 sends feature tensor 210 to server system 110. Server system 110 uses portions 125B of the model (e.g., final layers 135A to 135N) to process the received feature tensor 210 to generate output data 215. The specific content and format of the output data 215 may vary depending on the specific implementation and task. For example, in a classification task, output data 215 may include predicted classifications of input data 205 (e.g., probabilities or confidence levels for one or more classifications). For a regression task, output data 215 may include predicted values ​​of input data 205 (e.g., continuous numerical values).

[0049] In the illustrated example, as depicted in operation 220, the server system 110 may then evaluate the output data 215 to generate a loss 225. For example, if a benchmark truth value is available for the input data 205 (e.g., if the server system 110 already knows the benchmark truth value, such as if the client system 105 indicates a benchmark truth value for the input data 205), the server system 110 may use various loss formulas (e.g., cross-entropy) to compare the output data 215 with that benchmark truth value to generate the loss 225.

[0050] In some respects, to protect privacy, server system 110 may optionally send output data 215 to client system 105 (or to another system with a reference truth label). In such respects, client system 105 (or the other system) may compare output data 215 with the reference truth to generate loss 225, and thereafter client system 105 (or the other system) may send loss 225 back to server system 110.

[0051] As illustrated, server system 110 then refines part 125B using loss 225. That is, server system 110 may update one or more parameters of part 125B (e.g., weights of layers 135A to 135N) by means of backpropagation. In some aspects, updating part 125B involves generating gradients at each layer 135 of part 125B and using these gradients to update the parameters of layer 135 (and generating gradients for each previous layer), a process that moves from the final layer of the model (e.g., layer 135N) through the first layer (e.g., layer 135A) of part 125B.

[0052] As illustrated, in the complete data workflow 200A, the server system 110 then sends the gradient tensor 230 from layer 135A to the client system 105. The client system 105 uses this received gradient tensor 230 to update portion 125A (e.g., updating one or more parameters of layers 130A to 130N). For example, as discussed above with reference to updating portion 125B, the client system 105 can backpropagate the gradient tensor 230 through portion 125A (e.g., starting from layer 130N and moving through layer 130A). In this way, the client system 105 trains portion 125A based on input data 205 and labels, while the server system 110 trains portion 125B based on input data 205 and labels.

[0053] In some respects, this process can be repeated any number of times. For example, client system 105 can generate any number of feature tensors 210 based on any number of samples of input data 205, and then send each feature tensor 210 (or an aggregated representation of the feature tensors) to server system 110. In general, the depicted system can be trained using stochastic gradient descent (e.g., generating a separate loss 225 for each corresponding input data 205) or batch gradient descent (e.g., generating an aggregated loss based on a batch of input samples).

[0054] In some respects, client system 105 and server system 110 may use the complete data workflow 200A for one or more iterations or rounds. For example, client system 105 and / or server system 110 may periodically re-evaluate communication criteria (e.g., at the end of each iteration and / or round, at each...). After a training sample, iteration, and / or round, etc., it is determined whether to remain in the complete data state (e.g., continue using workflow 200A to exchange feature tensor 210 and gradient tensor 230) or transition to another communication state (e.g., see below for reference). FIG. 2B , FIG. 2C and FIG. 2D (One of the states discussed in workflows 200B, 200C, and 200D).

[0055] In some respects, at defined intervals (e.g., at the start of each iteration or round), client system 105 and server system 110 may negotiate or indicate which state to use. For example, each system may evaluate various communication criteria to determine which state to use. Client system 105 and server system 110 may then exchange management or control frames to determine which state should be used (e.g., by selecting the state with the least data exchange, as discussed in more detail below). For example, if one system determines to use the full data state while another system determines to use the reduced data state, these systems may agree to use the reduced data state, as discussed in more detail below. As another example, if one system selects one type of reduced data state while another system selects a different type of reduced data state, these systems may agree to use any suitable selection criteria to use one of the selected states (or a third, not yet selected state).

[0056] FIG. 2B An example workflow 200B is described for learning reduced communication splitting in reduced downlink data states according to various aspects of this disclosure.

[0057] In the illustrated example, the "reduced downlink" state can refer to an operational or training state in which client system 105 sends feature data to server system 110, but server system 110 does not send complete gradient data to client system 105. Specifically, as illustrated, client system 105 uses a portion 125A of the model (e.g., using initial layers 130A to 130N) to process input data 205 to generate feature tensor 210. As discussed above, this feature tensor 210 generally corresponds to intermediate features output by the network's inner or hidden layers (e.g., by layer 130N).

[0058] As illustrated, during the “reduced downlink” operation, client system 105 sends feature tensor 210 to server system 110. Server system 110 uses portions 125B of the model (e.g., final layers 135A to 135N) to process the received feature tensor 210 to generate output data 215. As discussed above, the specific content and format of output data 215 may vary depending on the specific implementation and task. For example, in a classification task, output data 215 may include predicted classifications of input data 205 (e.g., probabilities or confidence levels for one or more classifications). For a regression task, output data 215 may include predicted values ​​of input data 205 (e.g., continuous numerical values).

[0059] As depicted in operation 220, server system 110 may then evaluate output data 215 to generate loss 225, as discussed above. For example, if a benchmark truth value is available for input data 205 (e.g., if server system 110 already knows the benchmark truth value, such as if client system 105 indicates a benchmark truth value for input data 205), server system 110 may use various loss formulas (e.g., cross-entropy) to compare output data 215 with that benchmark truth value to generate loss 225.

[0060] In some respects, to protect privacy, server system 110 may optionally send output data 215 to client system 105 (or to another system with a reference truth label). In such respects, client system 105 (or the other system) may compare output data 215 with the reference truth to generate loss 225, and thereafter client system 105 (or the other system) may send loss 225 back to server system 110.

[0061] As illustrated, server system 110 then refines part 125B using loss 225. That is, server system 110 may update one or more parameters of part 125B (e.g., weights of layers 135A to 135N) by means of backpropagation. In some aspects, updating part 125B involves generating gradients at each layer 135 of part 125B and using these gradients to update the parameters of layer 135 (and generating gradients for each previous layer), a process that moves from the final layer of the model (e.g., layer 135N) through the first layer (e.g., layer 135A) of part 125B.

[0062] In the illustrated reduced downlink workflow 200B, server system 110 avoids sending gradient tensors to client system 105. That is, because client system 105 and / or server system 110 have determined to operate in a reduced downlink communication state, server system 110 can avoid sending gradients to client system 105 even if client system 105 continues to send feature tensors 210. In this way, although portion 125B continues to be updated, portion 125A is effectively frozen during workflow 200B. That is, the parameters of portion 125A remain unchanged during the depicted workflow 200B, and server system 110 continues to update the parameters of portion 125B.

[0063] Although the illustrated example depicts server system 110 avoiding sending any gradients to client system 105, in some respects, server system 110 may alternatively send a compressed or reduced set of gradients to client system 105 during the reduced downlink workflow 200B. For example, as discussed in more detail below, server system 110 may compress the gradient tensor using one or more compression operations before sending the gradient tensor to client system 105. This allows client system 105 to continue updating portion 125A while significantly reducing the communication overhead of the training process (although possibly with less accurate gradients due to the loss in the compression). As another example, server system 110 may send a subset of gradients (e.g., for each received gradient) Each feature tensor 210 returns a gradient tensor, thereby enabling at least some updates of part 125A with reduced communication overhead.

[0064] In all respects, the reduced downlink data state generally corresponds to the following time period: the client system 105 sends the complete data (e.g., the uncompressed feature tensor 210 or the feature tensor 210 compressed using a lossless algorithm) to the server system 110, while the server system 110 avoids sending gradients to the client system 105, or sends a reduced version (e.g., every other gradient tensor, or sends a gradient tensor compressed using one or more lossy compression algorithms).

[0065] In some respects, this process can be repeated any number of times. For example, client system 105 can generate any number of feature tensors 210 based on any number of samples of input data 205, and then send each feature tensor 210 (or an aggregated representation of the feature tensors) to server system 110. In general, the depicted system can be trained using stochastic gradient descent (e.g., generating a separate loss 225 for each corresponding input data 205) or batch gradient descent (e.g., generating an aggregated loss based on a batch of input samples).

[0066] In some respects, client system 105 and server system 110 may use a reduced downlink workflow 200B for one or more iterations or rounds, as discussed above. For example, client system 105 and / or server system 110 may periodically re-evaluate communication criteria (e.g., at the end of each iteration and / or round, at each...). After a training sample, iteration, or round, etc., it is determined whether to remain in the reduced downlink state (e.g., continue using workflow 200B to exchange reduced data) or transition to another communication state (e.g., as mentioned above). FIG. 2A The complete data status discussed in Workflow 200A or refer to the following text. FIG. 2C and FIG. 2D(One of the states discussed in workflows 200C and 200D).

[0067] For example, as discussed above, each system can evaluate various communication criteria to determine which state to use. Client system 105 and server system 110 can then exchange management or control frames to determine which state should be used (e.g., by selecting the state with the least data exchange, as discussed in more detail below). For example, if one system determines to remain in a reduced downlink state while another system determines to transition to a full data state, these systems can agree to use the reduced downlink state. As another example, if one system selects a type of reduced data state while another system selects a different type of reduced data state, these systems can agree to use any suitable selection criteria to use one of the selected states (or a third, unselected state).

[0068] FIG. 2C An example workflow 200C is described for learning reduced communication splitting in reduced uplink states according to various aspects of this disclosure.

[0069] In the illustrated example, the "reduced uplink" state can refer to an operational or training state in which client system 105 avoids sending feature data to server system 110, but server system 110 sends gradient data to client system 105. Specifically, as illustrated, client system 105 may or may not use a portion 125A of the model (e.g., using initial layers 130A to 130N) to process the input data to generate a feature tensor. However, server system 110 continues to send gradient tensor 230 to client system 105.

[0070] As illustrated, during the “reduced uplink” operation, server system 110 may use portions 125B of the model (e.g., final layers 135A through 135N) to process one or more previously received feature tensors to generate output data 215. As discussed above, the specific content and format of the output data 215 may vary depending on the specific implementation and task. For example, in a classification task, output data 215 may include predicted classifications of input data 205 (e.g., probabilities or confidence levels for one or more classifications). For a regression task, output data 215 may include predicted values ​​of input data 205 (e.g., continuous numerical values).

[0071] In other words, even though server system 110 does not receive new feature tensors during the depicted workflow 200C, server system 110 can reprocess feature tensors previously received from client system 105 (e.g., during a previous full data state or a reduced downlink state), which server system 110 may have already stored for later use. This allows server system 110 to reuse previous data to continue training part 125B even if no additional feature tensors are received during workflow 200C.

[0072] As depicted in operation 220, server system 110 (or another system) may then evaluate output data 215 to generate loss 225, as discussed above. As illustrated, server system 110 then uses loss 225 to refine part 125B. That is, server system 110 may update one or more parameters of part 125B (e.g., weights of layers 135A to 135N) by means of backpropagation. In some aspects, updating part 125B involves generating gradients at each layer 135 of part 125B and using these gradients to update the parameters of layer 135 (and generating gradients for each previous layer), a process that moves from the final layer of the model (e.g., layer 135N) through the first layer of part 125B (e.g., layer 135A).

[0073] In the illustrated reduced uplink workflow 200C, the server system 110 then sends the gradient tensor 230 to the client system 105. The client system 105 can use this received gradient tensor 230 to update part 125A (e.g., update one or more parameters of layers 130A to 130N). For example, as discussed above, the client system 105 can backpropagate the gradient tensor 230 through part 125A (e.g., starting from layer 130N and moving through layers 130A). In this way, the client system 105 trains part 125A based on the previously generated feature tensor.

[0074] In some respects, by avoiding the generation of new feature tensors and / or sending new feature tensors to server system 110, client system 105 can significantly reduce the workload of the client system (e.g., because there is no need to generate new feature tensors) and the communication overhead of the training process (e.g., because there is no need to send new feature tensors to server system 110).

[0075] Although the illustrated example depicts client system 105 avoiding sending any features to server system 110, in some respects, client system 105 may alternatively send a compressed or reduced set of features to server system 110 during the reduced uplink workflow 200C. For example, as discussed in more detail below, client system 105 may compress the feature tensors using one or more compression operations before sending them to server system 110. This allows server system 110 to continue updating part 125B with new features while significantly reducing the communication overhead of the training process (although possibly with less accurate features due to the loss in the compression). As another example, client system 105 may send a subset of features (e.g., send fewer feature tensors) to achieve at least some new training with reduced communication overhead.

[0076] In all respects, the reduced uplink data state generally corresponds to the following time period: server system 110 sends the complete data (e.g., uncompressed gradient tensor 230 or gradient tensor 230 compressed using a lossless algorithm) to client system 105, while client system 105 avoids sending features to server system 110, or sends a reduced version (e.g., feature tensor compressed using one or more lossy compression algorithms).

[0077] In some respects, the process can be repeated any number of times. For example, server system 110 can generate any number of gradient tensors 230 based on any number of previously received and stored feature tensors, thereby sending each gradient tensor 230 (or an aggregated representation of the gradient tensors) to client system 105. In general, the depicted system can train the model using stochastic gradient descent (e.g., generating a separate loss 225 for each corresponding feature tensor) or batch gradient descent (e.g., generating an aggregated loss based on a batch of input samples).

[0078] In some respects, client system 105 and server system 110 may use a reduced uplink workflow 200C for one or more iterations or rounds, as discussed above. For example, client system 105 and / or server system 110 may periodically re-evaluate communication criteria (e.g., at the end of each iteration and / or round, at each...). After a training sample, iteration, or round, etc., it is determined whether to remain in the reduced uplink state (e.g., continue using workflow 200C to exchange reduced data) or transition to another communication state (e.g., as mentioned above). FIG. 2A The complete data status discussed in Workflow 200A, as mentioned above (reference). FIG. 2B The reduced downlink state or reference discussed in Workflow 200B FIG. 2D The communication status discussed in Workflow 200D).

[0079] FIG. 2D An example workflow 200D for learning reduced communication splits in a bidirectional reduced state is described according to various aspects of this disclosure.

[0080] In the illustrated example, a "bidirectional reduction" state can refer to an operational or training state in which client system 105 avoids sending feature data to server system 110 and server system 110 similarly avoids sending gradient data to client system 105. Specifically, as illustrated, client system 105 may or may not use a portion 125A of the model (e.g., using initial layers 130A to 130N) to process the input data to generate feature tensors.

[0081] As illustrated, during the “bidirectional reduction” operation, server system 110 may use portions 125B of the model (e.g., final layers 135A to 135N) to process one or more previously received feature tensors to generate output data 215. As discussed above, the specific content and format of the output data 215 may vary depending on the specific implementation and task. For example, in a classification task, output data 215 may include the predicted classification of input data 205. For a regression task, output data 215 may include the predicted values ​​(e.g., continuous numerical values) of input data 205.

[0082] In other words, even though server system 110 does not receive new feature tensors during the depicted workflow 200D, server system 110 can reprocess feature tensors previously received from client system 105 (e.g., during a previous full data state or reduced downlink state), which server system 110 may have already stored for later use. This allows server system 110 to reuse previous data to continue training part 125B even if no additional feature tensors are received during workflow 200D.

[0083] As depicted in operation 220, server system 110 (or another system) may then evaluate output data 215 to generate loss 225, as discussed above. As illustrated, server system 110 then uses loss 225 to refine part 125B. That is, server system 110 may update one or more parameters of part 125B (e.g., weights of layers 135A to 135N) by means of backpropagation. In some aspects, updating part 125B involves generating gradients at each layer 135 of part 125B and using these gradients to update the parameters of layer 135 (and generating gradients for each previous layer), a process that moves from the final layer of the model (e.g., layer 135N) through the first layer of part 125B (e.g., layer 135A).

[0084] In the illustrated bidirectional reduced data workflow 200D, server system 110 avoids sending gradient tensors to client system 105. That is, because client system 105 and / or server system 110 have determined to be operating in a reduced communication state (e.g., a bidirectional reduced state), server system 110 avoids sending gradients to client system 105. In this way, although portion 125B can continue to be updated based on previously received features, portion 125A is effectively frozen during workflow 200D. That is, the parameters of portion 125A remain unchanged during the depicted workflow 200D, and server system 110 continues to update the parameters of portion 125B.

[0085] In some respects, by avoiding the generation of new feature tensors and / or sending new feature tensors to server system 110, client system 105 can significantly reduce the workload of the client system (e.g., because there is no need to generate new feature tensors) and the communication overhead of the training process (e.g., because there is no need to send new feature tensors to server system 110). Similarly, by avoiding sending new gradient tensors to client system 105, server system 110 can significantly reduce the communication overhead of the training process.

[0086] Although the illustrated example depicts client system 105 and server system 110 avoiding sending any features or gradients, in some respects, client system 105 and / or server system 110 may alternatively send compressed or reduced datasets to another system during workflow 200D. For example, as discussed in more detail below, client system 105 and / or server system 110 may compress the feature tensor and / or gradient tensor respectively using one or more compression operations before sending the tensors. This allows the system to continue updating the model with new features and gradients while significantly reducing the communication overhead of the training process (albeit with less accurate results due to the loss in the compression).

[0087] In all respects, the bidirectional reduction state generally corresponds to the following period: neither server system 110 nor client system 105 sends complete data (e.g., uncompressed data or data compressed using lossless algorithms) to the other system. Instead, both client system 105 and server system 110 avoid sending data to the other system or both send reduced versions (e.g., tensors compressed using one or more lossy compression algorithms).

[0088] In some respects, the process can be repeated any number of times. For example, server system 110 can generate any number of losses 225 based on any number of previously received and stored feature tensors, thereby updating part 125B any number of times.

[0089] In some respects, client system 105 and server system 110 may use a bidirectional reduced workflow 200D for one or more iterations or rounds, as discussed above. For example, client system 105 and / or server system 110 may periodically re-evaluate communication criteria (e.g., at the end of each iteration and / or round, at each...). After a training sample, iteration, or round, etc., it is determined whether to remain in the bidirectional reduction state (e.g., continue using workflow 200D) or transition to another communication state (e.g., as mentioned above). FIG. 2A The complete data status discussed in Workflow 200A, as mentioned above (reference). FIG. 2B The reduced downlink state discussed in Workflow 200B or the above reference FIG. 2C The workflow 200C discusses reduced uplink states.

[0090] In some respects, the above references FIG. 2A The complete data state described and discussed may be referred to as the first state, the first communication state, the first training state, the first computation state, etc. Each other training state (such as those mentioned above) FIG. 2B The reduced downlink states discussed above are referenced. FIG. 2C The reduced uplink states discussed and / or the above reference FIG. 2D The bidirectional reduction state discussed can be similarly referred to as the second state, the second communication state, the second training state, the second computation state, etc. Similarly, each other state can be referred to as the third state, the fourth state, etc.

[0091] Example workflow for reduced-communication split learning using dynamic compression

[0092] FIG. 3A An example workflow 300A for learning reduced communication splits using dynamic uplink compression, according to various aspects of this disclosure, is described.

[0093] In some respects, Workflow 300A can be used during any communication or training state when uplink data is reduced or eliminated. For example, Workflow 300A can be used in the above reference. FIG. 2C Workflow 200C discusses reduced uplink states and / or the above references FIG. 2D During the bidirectional reduced data state discussed in workflow 200D, compressed or reduced uplink data is sent (from client system 105 to server system 110). That is, although the above reference FIG. 2C and FIG. 2DThe reduced uplink state and reduced bidirectional state discussed do not depict the client system 105 sending feature tensors, but in some respects, workflow 300A may additionally or alternatively be used to send a relatively reduced amount of data (e.g., compressed feature tensors compared to full precision or uncompressed feature tensors in full data state) from client system 105 to server system 110, thereby significantly reducing uplink data throughput while the training process continues.

[0094] In the illustrated example, as discussed above, the client system 105 can use a portion 125A of the model (e.g., using initial layers 130A to 130N) to process the input data to generate a feature tensor (in... FIG. 2A In the description but FIG. 2C or FIG. 2D (Not depicted in the text). As discussed above, this feature tensor generally corresponds to intermediate features output by the network's internal or hidden layers (e.g., by layer 130N).

[0095] As illustrated, during workflow 300A, client system 105 may select one or more compression operations to use before sending feature data to server system 110. Specifically, as illustrated by switch 305, client system 105 may select compression operations 310A to 310N (collectively referred to as compression operations 310). In some respects, compression operations 310 each have different compression ratios and / or different levels of loss. For example, compression operation 310A may compress the data with a relatively small compression ratio (e.g., where the compressed data is only slightly smaller than the original data) but with a relatively small amount of loss (e.g., a relatively small amount of errors or noise introduced). Compression operation 310B may have a higher compression ratio than compression operation 310A while introducing a relatively large amount of decompression or reconstruction errors or noise (resulting in reduced compressed data). Furthermore, compression operation 310N may have an even higher compression ratio while introducing even larger amounts of decompression or reconstruction errors or noise (e.g., resulting in significantly reduced compressed data size). Although three compression operations 310 are depicted, client system 105 may use any number of compression operations.

[0096] Generally, the specific configuration of compression operation 310 can vary depending on the specific implementation. For example, in some aspects, one or more of compression operations 310 may correspond to an autoencoder. As used herein, an autoencoder is a type of unsupervised artificial neural network that has been trained (along with the autodecoder) to generate an efficient data representation of the input data. The encoder part generally learns to map the input vector (e.g., a feature tensor) to a latent vector space (typically of reduced size compared to the input). As discussed in more detail below, the decoder part (represented by decompression operations 315A to 315N) generally learns to map this latent vector to a reconstructed version of the input (e.g., a reconstruction mapped to the original feature tensor). Sending the latent tensor instead of the full feature tensor generated by the client system 105 eliminates a significant amount of data traffic during the split learning process.

[0097] In some respects, the client system 105 selects which compression operation 310 to use for a given state or feature tensor. That is, based on communication criteria (e.g., based on congestion or channel state in the communication network connecting the systems), the client system 105 may select one of compression operations 310A to 310N to balance compression ratio and reconstruction error. For example, when channel congestion is high, the client system 105 may select a compression operation 310 with a high compression ratio (even with high reconstruction error). When channel congestion is relatively low, the client system 105 may select a compression operation 310 with a relatively low compression ratio (resulting in relatively low reconstruction error).

[0098] As illustrated, the client system 105 then sends these compressed features to the server system 110. In the illustrated example, the server system 110 then uses a corresponding decompression operation 315 to decompress the received data to reconstruct the original feature tensor, and the server system 110 uses a portion 125B of the model (e.g., the final layers 135A to 135N) to process the decompressed data to generate output data. As discussed above, the server system 110 can then evaluate the output data to generate a loss, and use this loss (e.g., by using backpropagation) to update one or more parameters of portion 125B (e.g., the weights of layers 135A to 135N).

[0099] In some respects, as discussed above, client system 105 and server system 110 may negotiate or indicate which compression operation will be used. For example, at defined intervals (e.g., at the beginning of each iteration or round), one or each system may evaluate various communication criteria to determine which pair of compression operation 310 and decompression operation 315 should be used. Client system 105 and server system 110 may then exchange management or control frames to determine which compression alternative should be used (e.g., by selecting the operation with the least data exchange). For example, if one system determines to use compression operation 310A and decompression operation 315A (e.g., with the least introduced error) while another system determines to use compression operation 310N and decompression operation 315N (e.g., with a higher compression ratio but also with higher errors), then these systems may agree to use compression operation 310N. As another example, if one system selects compression operation 310A and decompression operation 315A while another system determines not to use compression, then these systems may agree to use compression operation 310A and decompression operation 315A.

[0100] FIG. 3B An example workflow 300B for learning reduced communication splitting using dynamic downlink compression is described according to various aspects of this disclosure.

[0101] In some respects, Workflow 300B can be used during any communication or training state when downlink data is reduced or eliminated. For example, Workflow 300B can be used in the reference above. FIG. 2B The reduced downlink state discussed in Workflow 200B and / or the above reference FIG. 2D During the bidirectional reduced data state discussed in workflow 200D, compressed or reduced downlink data is sent (from server system 110 to client system 105). That is, although the above reference FIG. 2B and FIG. 2D The reduced downlink state and reduced bidirectional state discussed do not depict the server system 110 sending gradient tensors, but in some respects, workflow 300B may additionally or alternatively be used to send a relatively reduced amount of data (e.g., compressed gradient tensors compared to full accuracy or uncompressed gradient tensors in full data state) from server system 110 to client system 105, thereby significantly reducing downlink data throughput while the training process continues.

[0102] In the illustrated example, as discussed above, server system 110 can use a portion 125B of the model (e.g., using final layers 135A to 135N) to process (newly received or previously received and stored) feature data to generate the model's output. As discussed above, this output generally includes predictions for the corresponding inputs. As discussed above, server system 110 can then use this output to generate a loss, which can be used to update the parameters of portion 125B (e.g., using backpropagation). As discussed above, this update process produces gradient tensors from layer 135N (in... FIG. 2A In the description but FIG. 2B or FIG. 2D (Not depicted in the text), this gradient tensor can be provided to the client system 105.

[0103] As illustrated, during workflow 300B, server system 110 may select one or more compression operations to use before sending gradient data to client system 105. Specifically, as illustrated by switch 320, server system 110 may select compression operations 310A to 310N (collectively referred to as compression operations 310). In some respects, as discussed above, compression operations 310 each have different compression ratios and different reconstruction errors. Although three compression operations 310 are depicted, server system 110 may use any number of compression operations. In general, the specific configuration of compression operations 310 may vary depending on the specific implementation. For example, as discussed above, one or more of compression operations 310 may correspond to an autoencoder.

[0104] In some respects, server system 110 selects which compression operation 310 to use for a given state or gradient tensor. That is, based on communication criteria (e.g., based on congestion or channel state in the communication network connecting the systems), server system 110 may select one of compression operations 310A to 310N to balance compression ratio and reconstruction error. For example, when channel congestion is high, server system 110 may select compression operation 310 with a high compression ratio (even with high reconstruction error). When channel congestion is relatively low, server system 110 may select compression operation 310 with a relatively low compression ratio (resulting in relatively low reconstruction error).

[0105] As illustrated, server system 110 then sends these compressed gradients to client system 105. In the illustrated example, client system 105 then uses the corresponding decompression operation 315 to decompress the received data to reconstruct the original feature tensor as discussed above. Client system 105 can then process the decompressed data to update the parameters of parts 125A of the model (e.g., initial layers 130A to 130N), such as by using backpropagation.

[0106] In some respects, as discussed above, client system 105 and server system 110 may negotiate or indicate which compression operation will be used for the gradient. For example, at defined intervals (e.g., at the beginning of each iteration or round), each system may evaluate various communication criteria to determine which pair of compression operation 310 and decompression operation 315 should be used. Client system 105 and server system 110 may then exchange management or control frames to determine which compression alternative should be used (e.g., by selecting the operation with the least data exchange). For example, if one system determines to use compression operation 310A and decompression operation 315A (e.g., with the least introduced error) while another system determines to use compression operation 310N and decompression operation 315N (e.g., with a higher compression ratio but also with higher error), then these systems may agree to use compression operation 310N. As another example, if one system selects compression operation 310A and decompression operation 315A while another system determines not to use compression, then these systems may agree to use compression operation 310A and decompression operation 315A.

[0107] Example method for reduced-communication split learning

[0108] FIG. 4 This is a flowchart depicting an example method 400 for reducing communication splitting during learning according to various aspects of this disclosure. In some aspects, method 400 is performed by a client system (such as...) participating in the splitting training of a machine learning model. FIG. 1 The client system 105) executes the method. In some respects, method 400 is performed by a server system (such as...) that participates in the split training of the machine learning model. FIG. 1 The method 400 can be executed by any computing system involved in the split training of the machine learning model (server system 110).

[0109] At box 405, the computational system determines the communication state to be used for training in the current round or iteration. For example, in some aspects, the computational system may evaluate various communication criteria to determine whether to exchange complete data during an iteration (e.g., using complete data operations or workflows, such as those mentioned above). FIG. 2A (as described) or by exchanging reduced amounts of data (e.g., using the reference above). FIG. 2B , FIG. 2C , FIG. 2D , FIG. 3A and FIG. 3B (One of the workflows discussed).

[0110] Generally, the specific criteria used to determine the state of communication can vary depending on the particular implementation. As an example, and not a limitation, a computing system may evaluate information such as the channel state (e.g., data rate, channel stability, congestion, etc.) of a communication link between participating systems. For example, if the channel state fails to meet the criteria (e.g., congestion is above a threshold, or data rate and / or stability is below a threshold), the computing system may determine to use one or more reduced communication operations.

[0111] As another example, a computing system can assess the availability of its resources (e.g., by determining the current workload of the computing system and which resources(s) are available for training, which may include computational or processing resources for processing training data and / or communication resources for receiving and / or sending data). For example, if available resources fail to meet criteria (e.g., available resources are less than a threshold), the computing system can determine to use reduced communication and / or reduced computational operations.

[0112] As another example, a computational system can evaluate the characteristics of training data (e.g., based on the quantity, quality, and / or distribution of data available for the current iteration). For instance, if there is an insufficient number of data samples or the training data fails to meet quality criteria, the computational system can determine to use a reduced communication state.

[0113] As another example, a computational system can evaluate one or more indicators of the model's training progress (e.g., by comparing the loss of one iteration with the loss of previous iterations). For instance, if the change in loss is less than a threshold, the computational system can determine to use a reduced communication process.

[0114] In some aspects, at box 405, the computing system evaluates various criteria to select a communication state. The computing system may indicate the selected state to another participating system(s) (e.g., via one or more coordination or management messages). In some aspects, the other system(s) may respond based on the selected action (e.g., if the computing system is leading or designated as the decision-making state). In some aspects, the other system(s) may similarly evaluate criteria and indicate the preferred state for these systems. These systems may then negotiate or agree on the state (e.g., selecting a recommended state with the lowest communication and / or processing overhead). In some aspects, the computing system may determine the state by waiting for one or more other systems to indicate to the computing system which state to use.

[0115] In some respects, participating systems do not communicate their preferred state, but rather share relevant information for determining or selecting a state (e.g., indicating the available resources for these participating systems), thereby allowing a designated leader system (which may or may not be a participating system in training) to select a state.

[0116] As discussed above, determining the state generally includes determining whether to use full, reduced (e.g., compressed) uplink transmission or no uplink transmission, and / or to use full, reduced (e.g., compressed) downlink transmission or no downlink transmission. In some aspects, if reduced uplink and / or downlink transmission is selected, the computing system may further determine which compression operation(s) to use (e.g., which autoencoder). For example, the computing system may select a first autoencoder / autodecoder pair for uplink transmission and a second (different) autoencoder / autodecoder pair (or no compression at all) for downlink transmission.

[0117] In some respects, as discussed above, participating systems can use one or more FSMs to determine which state to use for training in subsequent iterations.

[0118] At box 410, the computing system determines whether the determined state (or the information used to determine the state) satisfies one or more communication criteria. For example, as discussed above, the computing system may determine whether the channel is sufficiently clear, whether sufficient computing resources are available, whether sufficient training data is available, and whether training should continue.

[0119] If so, method 400 continues to box 415, where the computational system exchanges complete data during model training (e.g., operating in the first state). That is, the computational system performs complete communication training (e.g., using...). FIG. 2A Workflow 200A). As discussed above, features and gradients are exchanged between participating systems in a complete data state without using lossy compression or reduction techniques.

[0120] Once the current round or iteration of training is complete, method 400 returns to box 405 to select a communication state / action for the subsequent rounds or iterations of training.

[0121] Returning to box 410, if the computing system determines that the criteria are not met (e.g., available bandwidth is below a threshold, computing resources are below a threshold, etc.), method 400 continues to box 420. At box 420, the computing system exchanges reduced data during model training (e.g., operating in a second state). That is, the computing system performs reduced-communication training of the model. For example, as discussed above, the computing system can use… FIG. 2B , FIG. 2C and FIG. 2D One or more of the respective workflows 200B, 200C, or 200D, and / or the references above may be used. FIG. 3A and FIG. 3B The respective workflows 300A and 300B discuss one or more compression operations.

[0122] In the illustrated example, once the training for the current round or iteration is complete, method 400 returns to box 405 to select a communication state / action for the subsequent training iteration.

[0123] Generally, the repeatable method can be performed 400 times, such as until a training termination criterion is met. Once training is complete, the model can be deployed to one or more systems for inference. For example, in some aspects, another participating system can send its trained portions to a computational system, which can then aggregate these portions to produce a fully trained machine learning model, which can then be used for inference. In other aspects, the computational system similarly sends its trained portions to another participating system, allowing that other system to instantiate the complete model.

[0124] In some respects, computational systems may also use a split arrangement during inference. For example, a client system may continue to use a first part of the model to process the input data and generate features, which are then sent to a server system. The server system can then use the final part of the model to process these features to generate a prediction for the input data. Depending on the specific implementation, this prediction can then be sent to the client system for use, used by the server system, and / or provided to another system for use.

[0125] Example method for performing asynchronous split learning by a client system

[0126] FIG. 5 This is a flowchart depicting an example method 500 for asynchronous split learning performed by a client system according to various aspects of this disclosure. In some aspects, method 500 is performed by a client system (such as...) participating in the split training of a machine learning model. FIG. 1 The client system 105) executes the method. Generally speaking, method 500 can be executed by any computing system involved in the split training of the machine learning model.

[0127] At box 505, the client system determines which communication state to use for training in the current iteration. For example, as discussed above, the client system can evaluate various criteria such as communication channel metrics, data characteristics, training progress, and computational resource availability to determine the state (either individually or in combination with another participating system).

[0128] At box 510, the client system determines whether a complete state (or first state) has been selected (e.g., whether to send complete or lossless features and receive complete or lossless gradients). If so, method 500 continues to box 515, where the client system uses the first part of the machine learning model (e.g., ... FIG. 2A Part 125A) processes input data (e.g., FIG. 2AThe input data 205) is used to generate features (e.g., FIG. 2A The feature tensor set (210). At box 520, the client system sends the generated features to the server system (e.g., FIG. 2A Server system 110).

[0129] At box 525, the client system receives gradients from the server system (e.g., FIG. 2A The gradient tensor is 230). For example, as discussed above, the server system can use the second part of the model (e.g., FIG. 2A Part 125B processes the features (received from the client system) to generate an output, which can then be compared with a ground truth (which may be provided by the client system) to generate a loss. This loss can then be used to update the second part (e.g., via backpropagation) to generate gradients, which are then forwarded to the client system.

[0130] At box 530, the client system updates one or more model parameters based on the received gradients (e.g., FIG. 2A (Parameters of part 125A). In this way, both parts of the model are updated. Then, method 500 returns to box 505 to determine the new state for the next iteration of training.

[0131] While the illustrated example depicts updating model parameters independently based on each input sample (e.g., using stochastic gradient descent), in some aspects, the system may alternatively update model parameters based on batch inputs in each iteration or round (e.g., using batch gradient descent). Furthermore, while the illustrated example depicts returning to box 505 after each model update, in some aspects, the system may remain in the current state for multiple such iterations before re-evaluating criteria to select the next state. Additionally, in some aspects, method 500 does not perform a defined number of iterations or rounds, but instead returns to box 505 in response to the occurrence of various criteria, such as identifying a sudden drop in the data rate of the communication channel.

[0132] Returning to box 510, if the client system determines that the full data state has not been selected or should not be used, method 500 continues to box 535. At box 535, the client system determines whether a reduced downlink state (or a second state) has been selected (e.g., whether full or lossless features are being sent without receiving full or lossless gradients). If so, method 500 continues to box 540, where the client system uses the first part of the machine learning model (e.g., ... FIG. 2B Part 125A) processes input data (e.g., FIG. 2B The input data 205) is used to generate features (e.g., FIG. 2BThe feature tensor set (210). At box 545, the client system sends the generated features to the server system (e.g., FIG. 2B Server system 110).

[0133] Then, method 500 returns to box 505 to determine a new state for training in the next iteration. Although the illustrated example depicts returning to box 505 after sending features, in some respects, the system can remain in the current state for multiple iterations before re-evaluating criteria to select the next state. Additionally, in some respects, instead of performing a defined number of iterations or rounds, method 500 returns to box 505 in response to the occurrence of various criteria, such as identifying a sudden drop in the data rate of the communication channel.

[0134] Returning to box 535, if the client system determines that a reduced downlink data state has not been selected or should not be used, method 500 continues to box 550. At box 550, the client system determines whether a reduced uplink state (or a third state) has been selected (e.g., whether to avoid sending full or lossless features while continuing to receive full or lossless gradients). If so, method 500 continues to box 555, where the client system receives gradients from the server system (e.g., ...). FIG. 2C The gradient tensor is 230). For example, as discussed above, the server system can use the second part of the model (e.g., FIG. 2C Part 125B processes features previously received from the client system (e.g., features received in the immediate preceding training round or during the last round of feature reception) to generate an output, which can then be compared with a ground truth (which may be provided by the client system) to generate a loss. This loss can then be used to update the second part (e.g., via backpropagation) to generate gradients, which are then forwarded to the client system.

[0135] At box 560, the client system updates one or more model parameters based on the received gradients (e.g., FIG. 2C (Parameters of part 125A). In this way, both parts of the model are updated. Then, method 500 returns to box 505 to determine the new state for the next iteration of training.

[0136] While the illustrated example depicts updating model parameters independently based on each input sample (e.g., using stochastic gradient descent), in some aspects, the system may alternatively update model parameters based on batch inputs in each iteration or round (e.g., using batch gradient descent). Furthermore, while the illustrated example depicts returning to box 505 after each model update, in some aspects, the system may remain in the current state for multiple such iterations before re-evaluating criteria to select the next state. Additionally, in some aspects, method 500 does not perform a defined number of iterations or rounds, but instead returns to box 505 in response to the occurrence of various criteria, such as identifying a sudden drop in the data rate of the communication channel.

[0137] Returning to box 550, if the client system determines that a reduced uplink data state has not been selected or should not be used, method 500 returns to box 505. In some respects, as discussed above, the server system may continue training for one or more iterations or rounds during the offline state (or fourth state) (e.g., using previously provided features). However, the client system may wait until a subsequent iteration or round in which a state transition occurs before rejoining training.

[0138] Although the illustrated examples depict reduced communication operations involving avoiding the exchange of data in one or both directions (e.g., from the client system to the server system and / or from the server system to the client system), in some aspects the system may additionally or alternatively use dynamic compression, as discussed above. For example, at boxes 520 and / or 545, firstly, the client system may use one or more compression techniques selected based on communication criteria (e.g., using... FIG. 3A The compression operation 310 is used to compress the features and send the compressed features (instead of the complete uncompressed features).

[0139] Similarly, at boxes 530 and / or 560, firstly, the client system can use one or more decompression techniques selected based on the compression operation used by the server system (e.g., using...). FIG. 3B The decompression operation 315) is used to decompress the received gradients and the decompressed gradients are used to update the model.

[0140] Example method for performing asynchronous split learning by a server system

[0141] FIG. 6 This is a flowchart depicting an example method 600 for asynchronous split learning performed by a server system according to various aspects of this disclosure. In some aspects, method 600 is performed by a server system (such as...) participating in the split training of a machine learning model. FIG. 1 The method 600 can be executed by any computing system involved in the split training of the machine learning model (server system 110).

[0142] At box 605, the server system determines which communication state to use for the training of the current iteration. For example, as discussed above, the server system may evaluate various criteria such as communication channel metrics, data characteristics, training progress, and computational resource availability to determine the state (either individually or in combination with another participating system).

[0143] At box 610, the server system determines whether a complete state (or first state) has been selected (e.g., whether to receive complete or lossless features and send complete or lossless gradients). If so, method 600 continues to box 615, where the client system receives information from the client system (e.g., ...). FIG. 2A The client system 105) receives features (e.g., FIG. 2A The set of feature tensors (210).

[0144] At box 620, the server system updates one or more model parameters based on the received features. For example, as discussed above, the server system can use a second part of the model (e.g., FIG. 2A Part 125B processes the features (received from the client system) to generate an output, which can then be compared with a baseline ground truth (which can be provided by the client system) to generate a loss. This loss can then be used to update a second part of the model (e.g., via backpropagation).

[0145] At box 625, the server system generates gradients based on features (e.g., FIG. 2A The gradient tensor 230). For example, as discussed above, backpropagating the loss through the second part of the model can produce a set of gradients at the first layer of the final part (e.g., layer 135A). These gradients can then be sent to the client system for use in the first part of updating the model. In this way, both parts of the model are updated. Method 600 then returns to box 605 to determine the new state for training the next iteration.

[0146] While the illustrated example depicts updating model parameters independently based on each input sample (e.g., using stochastic gradient descent), in some aspects, the system may alternatively update model parameters based on batch inputs in each iteration or round (e.g., using batch gradient descent). Furthermore, while the illustrated example depicts returning to box 605 after each model update, in some aspects, the system may remain in the current state for multiple such iterations before re-evaluating criteria to select the next state. Additionally, in some aspects, method 600 does not perform a defined number of iterations or rounds, but instead returns to box 605 in response to the occurrence of various criteria, such as identifying a sudden drop in the data rate of the communication channel.

[0147] Returning to box 610, if the server system determines that the full data state has not been selected or should not be used, method 600 continues to box 635. At box 635, the client system determines whether a reduced downlink state (or a second state) has been selected (e.g., whether full or lossless features are received without sending full or lossless gradients). If so, method 600 continues to box 640, where the server system receives features from the client system (e.g., ...). FIG. 2B The feature tensor set (210). At box 645, the server system updates the second part of the model based on the received features, as discussed above.

[0148] Then, method 600 returns to box 605 to determine a new state for training in the next iteration. Although the illustrated example depicts returning to box 605 after sending features, in some respects, the system may remain in the current state for multiple iterations before re-evaluating criteria to select the next state. Additionally, in some respects, instead of performing a defined number of iterations or rounds, method 600 returns to box 605 in response to the occurrence of various criteria, such as identifying a sudden drop in the data rate of the communication channel.

[0149] Returning to box 635, if the server system determines that a reduced downlink data state has not been selected or should not be used, method 600 continues to box 650. At box 650, the server system determines whether a reduced uplink state (or third state) has been selected (e.g., whether a full or lossless gradient is being sent without receiving full or lossless features).

[0150] If so, method 600 continues to box 655, where the server system updates the model parameters based on the previous feature set (e.g., FIG. 2C (Parameters of part 125B). For example, as discussed above, the server system can use the second part of the model (e.g., FIG. 2C Part 125B processes features previously received from the client system (e.g., features received in the immediate preceding training round or during the last round of feature reception) to generate an output, which can then be compared with a ground truth (which may be provided by the client system) to generate a loss. This loss can then be used to update the second part (e.g., via backpropagation) to generate gradients.

[0151] At box 660, the server system generates gradients based on previous features (e.g., FIG. 2C The gradient tensor is 230). For example, as discussed above, backpropagating the loss through the second part of the model can be done in the first layer of the final part (e.g., FIG. 2C A gradient set is generated at layer 135A.

[0152] At box 665, these gradients can then be sent to the client system for use in the first part of updating the model. In this way, both parts of the model are updated. Then, method 600 returns to box 605 to determine the new state for the next iteration of training.

[0153] While the illustrated example depicts updating model parameters independently based on each input sample (e.g., using stochastic gradient descent), in some aspects, the system may alternatively update model parameters based on batch inputs in each iteration or round (e.g., using batch gradient descent). Furthermore, while the illustrated example depicts returning to box 605 after each model update, in some aspects, the system may remain in the current state for multiple such iterations before re-evaluating criteria to select the next state. Additionally, in some aspects, method 600 does not perform a defined number of iterations or rounds, but instead returns to box 605 in response to the occurrence of various criteria, such as identifying a sudden drop in the data rate of the communication channel.

[0154] Returning to box 650, if the server system determines that a reduced uplink data state has not been selected or should not be used, method 600 returns to box 605. In some aspects, as discussed above, the server system may continue training for one or more iterations or rounds during the offline state (or fourth state) (e.g., using previously provided features) (e.g., executing boxes 655 and 660). However, the client system may wait until a subsequent iteration or round in which a state transition occurs before rejoining training.

[0155] Although the illustrated examples depict reduced communication operations involving avoiding the exchange of data in one or both directions (e.g., from the client system to the server system and / or from the server system to the client system), in some aspects the system may additionally or alternatively use dynamic compression, as discussed above. For example, at boxes 630 and / or 665, firstly, the server system may use one or more compression techniques selected based on communication criteria (e.g., using... FIG. 3B The compression operation 310 is used to compress the gradient and send the compressed gradient (instead of the full uncompressed gradient).

[0156] Similarly, at boxes 615 and / or 640, firstly, the server system can use one or more decompression techniques selected based on the compression operation used by the client system (e.g., using...). FIG. 3A The decompression operation 315) is used to decompress the received features and use the decompressed features to update the model.

[0157] Example method for reduced-communication training of a machine learning model

[0158] FIG. 7This is a flowchart depicting an example method 700 for reduced communication training of a machine learning model according to various aspects of this disclosure. In some aspects, method 700 involves client systems (such as...) participating in the split training of the machine learning model. FIG. 1 to FIG. 3B The client system 105) executes the method. In some respects, method 700 is performed by a server system (such as...) that participates in the split training of the machine learning model. FIG. 1 to FIG. 3B The server system 110) executes the method. Generally speaking, method 700 can be executed by the first computing system (also referred to in some respects as the processing system) that participates in the split training of the machine learning model.

[0159] At box 705, access the first element of the input data used for the first part of the neural network.

[0160] At box 710, the first element of the output data is generated based on the first element of the input data processed by the first part of the neural network.

[0161] At box 715, at a first time point, a first element of the output data is sent to a second computing system for the second computing system to update one or more parameters of a second part of the neural network based on the first element of the output data. For example, in the case where the first computing system is a client system, the second computing system can be a server system. As another example, in the case where the first computing system is a server system, the second computing system can be a client system.

[0162] At box 720, at a second time point after the first time point, it is determined that one or more communication criteria are not met.

[0163] At box 725, in response to this determination, reduced communication training of the neural network is performed, which includes reducing the amount of data sent for one or more rounds of training.

[0164] In some aspects, one or more of the communication criteria include at least one of the following: the status of one or more communication links between the first computing system and the second computing system, resource availability on at least one of the first computing system or the second computing system, one or more characteristics of the training data used to train the neural network, or one or more indications of the training progress of the neural network.

[0165] In some respects, wherein: the first part of the neural network includes one or more initial layers of the neural network, the first element of the output data includes feature tensors output by intermediate layers of the neural network, and the second part of the neural network includes one or more final layers of the neural network.

[0166] In some respects, method 700 also includes updating one or more parameters of a first part of the neural network by the first computing system based on a set of gradients received from the second computing system at the first computing system.

[0167] In some respects, the first part of the neural network includes one or more final layers of the neural network, the first element of the output data includes a gradient tensor, and the second part of the neural network includes one or more initial layers of the neural network.

[0168] In some aspects, method 700 also includes generating a second element of output data by a first computing system based on a first element of the input data processed by a first part of the neural network, and updating one or more parameters of the first part of the neural network based on the second element of the output data.

[0169] In some respects, the reduced communication training of the neural network includes avoiding sending at least a second element of the output data from the first computing system to the second computing system.

[0170] In some aspects, the reduced communication training of the neural network also includes receiving a second element of the input data from a second computing system, and updating one or more parameters of the first part of the neural network based on the second element of the input data.

[0171] In some aspects, updating one or more parameters of the first part of the neural network includes: generating decompressed data based on decompressing a second element of the input data from the second computing system, and updating one or more parameters of the first part of the neural network based on the decompressed data.

[0172] In some aspects, performing reduced communication training of the neural network includes sending at least a second element of the output data from a first computing system to a second computing system, and the first computing system does not receive a second element of the input data from the second computing system while performing reduced communication training of the neural network.

[0173] In some aspects, sending a second element of the output data from a first computing system to a second computing system includes: at the first computing system, generating compressed output data based on compressing the second element of the output data using one or more compression operations, and sending the compressed output data from the first computing system to the second computing system.

[0174] In some aspects, the reduced communication training of the neural network includes generating compressed output data at a first computing system based on at least a second element of the output data compressed using one or more compression operations, and sending the compressed output data from the first computing system to a second computing system.

[0175] In some aspects, generating compressed output data includes: selecting a first compression operation from one or more compression operations based on communication criteria, and using the first compression operation to compress at least a second element of the output data to generate compressed output data.

[0176] In some aspects, the reduced communication training of the neural network also includes generating a second element of the input data based on decompressing data received from a second computing system, and updating one or more parameters of the first part of the neural network based on the second element of the input data.

[0177] Example processing system for efficient machine learning

[0178] In some respects, refer to FIG. 1 to FIG. 7 The described workflows, techniques, and methods can be implemented on one or more devices or systems. FIG. 8 Various aspects configured to perform this disclosure are described (including, for example, regarding...). FIG. 1 to FIG. 7 The described techniques and methods are used to illustrate an example processing system 800 (also referred to in some respects as a computing system). In some respects, the processing system 800 may correspond to a client system, such as... FIG. 1 to FIG. 3B The client system 105. In some respects, the processing system 800 may correspond to a server system, such as... FIG. 1 to FIG. 3B Server system 110. Generally speaking, processing system 800 can correspond to any computing system involved in the split training of machine learning models, as discussed above. Although depicted as a single system for clarity of concept, in some respects, as discussed above, the operations described below with respect to processing system 800 can be distributed across any number of devices or systems.

[0179] The processing system 800 includes a central processing unit (CPU) 802, which in some examples may be a multi-core CPU. Instructions executed at the CPU 802 may be loaded, for example, from program memory associated with the CPU 802 or from a memory partition (e.g., a partition of memory 824).

[0180] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, a multimedia component 810 (e.g., a multimedia processing unit), and a wireless connectivity component 812.

[0181] NPUs (such as the NPU 808) are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0182] NPUs such as the NPU 808 are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, an NPU may be part of a dedicated neural network accelerator.

[0183] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0184] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0185] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take new data segments as input and rapidly process those segments through a trained model to generate model outputs (e.g., inference).

[0186] In some implementations, the NPU 808 is part of one or more of the CPU 802, GPU 804, and / or DSP 806.

[0187] In some examples, the wireless connectivity component 812 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and / or other wireless data transmission standards. The wireless connectivity component 812 is further coupled to one or more antennas 814.

[0188] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or a navigation processor 820, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0189] The processing system 800 may also include one or more input and / or output devices 822, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0190] In some examples, one or more processors in the processing system 800 may be based on the ARM or RISC-V instruction set.

[0191] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0192] Specifically, in this example, memory 824 includes a state component 824A, a compression component 824B, a communication component 824C, and a training component 824D. Memory 824 also includes model parameters 824E for one or more models or portions thereof (e.g., parameters of layer 130 in portion 125A and / or parameters of layer 135 in portion 125B). Although not included in the illustrated example, in some respects memory 824 may also include other data, such as training data (e.g., ...). FIG. 2A Input data 205). Although for the sake of conceptual clarity, FIG. 8 The components are depicted as discrete components, but in various respects, the illustrated components (and other components not depicted) may be implemented together or separately.

[0193] The processing system 800 also includes a state circuit 826, a compression circuit 827, a communication circuit 828, and a training circuit 829. The depicted circuits and other circuits not depicted can be configured to perform various aspects of the techniques described herein.

[0194] For example, state component 824A and / or state circuit 826 can be used to determine or select a communication or training state, as discussed above. For example, state component 824A and / or state circuit 826 can evaluate criteria such as channel state, data characteristics, training progress, and / or resource availability to determine which state should be used for a given training epoch.

[0195] Compression component 824B and / or compression circuit 827 can be used to compress and / or decompress data (e.g., such as FIG. 2A to FIG. 2D Features of the feature tensor 210 and / or features such as FIG. 2A to FIG. 2D The gradient of the gradient tensor 230), as discussed above. For example, compression component 824B and / or compression circuit 827 may use one or more compression operations (such as... FIG. 3A and FIG. 3B Compression operation 310) is used to dynamically and / or selectively compress data before sending it to another participating system(s), and / or use one or more decompression operations (such as compression operation 310) to compress data dynamically and / or selectively before sending it to another participating system(s). FIG. 3A and FIG. 3B The decompression operation 315 is used to decompress any compressed data received from other participating systems.

[0196] The communication component 824C and / or the communication circuit 828 can be used to communicate relevant data during training, as discussed above. For example, the communication component 824C and / or the communication circuit 828 can be used to exchange management data (e.g., select communication state and / or compression operation) with another participating system(s), and / or exchange the training data itself (e.g., features and / or gradients).

[0197] The training component 824D and / or training circuit 829 can be used to process data using a machine learning model (or a portion thereof) that is being trained, and / or to update the model (or a portion thereof), as discussed above. For example, the training component 824D and / or training circuit 829 can be used by employing a first part of the model (e.g., FIG. 2A and FIG. 2B Part 125A) processes input data (e.g., FIG. 2A and FIG. 2B The input data 205) is used to generate features (e.g., FIG. 2A and FIG. 2B The feature tensor 210). As another example, the training component 824D and / or the training circuit 829 can be trained by using the second part of the model (e.g., the feature tensor 210). As another example, the training component 824D and / or the training circuit 829 can be trained by using the second part of the model (e.g., FIG. 2A to FIG. 2D Part 125B) processes feature data (e.g., FIG. 2A and FIG. 2B The feature tensor 210) is used to generate output data (e.g., FIG. 2A to FIG. 2D The output data 215). As another example, the training component 824D and / or the training circuit 829 may be based on gradients generated during training (e.g., FIG. 2A and FIG. 2C The gradient tensor (230) is used to update the parameters of the model (e.g., the parameters of part 125A and / or 125B), as discussed above.

[0198] Although for the sake of clarity FIG. 8 While the circuits are depicted as separate components and circuits, the state circuit 826, compression circuit 827, communication circuit 828, and training circuit 829 may be implemented jointly or separately in other processing devices of the processing system 800, such as in the CPU 802, GPU 804, DSP 806, NPU 808, etc.

[0199] Generally speaking, the processing system 800 and / or its components can be configured to perform the methods described herein.

[0200] It is worth noting that, in other aspects, such as when the processing system 800 is a server computer, components of the processing system 800 may be omitted. For example, in other aspects, the multimedia component 810, the wireless connectivity component 812, the sensor processing unit 816, the ISP 818, and / or the navigation processor 820 may be omitted. Furthermore, various aspects of the processing system 800 may be distributed among multiple devices.

[0201] Example clauses

[0202] Specific implementation examples are described in the following numbered clauses: Clause 1: A method comprising: accessing a first element of input data for a first part of a neural network; generating a first element of output data by a first computing system based on the first element of the input data processed using the first part of the neural network; sending the first element of the output data from the first computing system to a second computing system at a first time point for the second computing system to update one or more parameters of a second part of the neural network based on the first element of the output data; determining by the first computing system at a second time point after the first time point that one or more communication criteria are not satisfied; and in response to the determination, performing reduced communication training of the neural network, the execution comprising reducing the amount of data sent by the first computing system for one or more rounds of training.

[0203] Clause 2: According to the method of Clause 1, the one or more communication criteria include at least one of the following: the status of one or more communication links between the first computing system and the second computing system, resource availability on at least one of the first computing system or the second computing system, one or more characteristics of training data for training the neural network, or one or more indications of the training progress of the neural network.

[0204] Clause 3: The method according to any one of Clauses 1 to 2, wherein: the first portion of the neural network includes one or more initial layers of the neural network, the first element of the output data includes a feature tensor output by an intermediate layer of the neural network, and the second portion of the neural network includes one or more final layers of the neural network.

[0205] Clause 4: The method according to Clause 3 further includes updating one or more parameters of the first portion of the neural network by the first computing system based on a set of gradients received at the first computing system from the second computing system.

[0206] Clause 5: The method according to any one of Clauses 1 to 4, wherein: the first portion of the neural network includes one or more final layers of the neural network, the first element of the output data includes a gradient tensor, and the second portion of the neural network includes one or more initial layers of the neural network.

[0207] Clause 6: The method according to Clause 5 further comprises: generating a second element of output data by the first computing system based on a first element of the input data processed using the first part of the neural network; and updating one or more parameters of the first part of the neural network based on the second element of the output data.

[0208] Clause 7: The method according to any one of Clauses 1 to 6, wherein performing the reduced communication training of the neural network includes avoiding sending at least a second element of the output data from the first computing system to the second computing system.

[0209] Clause 8: The method according to Clause 7, wherein performing the reduced communication training of the neural network further comprises: receiving a second element of input data from the second computing system; and updating one or more parameters of the first portion of the neural network based on the second element of the input data.

[0210] Clause 9: The method according to Clause 8, wherein updating one or more parameters of the first portion of the neural network comprises: generating decompressed data based on decompressing the second element of input data from the second computing system; and updating one or more parameters of the first portion of the neural network based on the decompressed data.

[0211] Clause 10: The method according to any one of Clauses 1 to 9, wherein: performing the reduced communication training of the neural network includes sending at least a second element of the output data from the first computing system to the second computing system, and the first computing system does not receive a second element of the input data from the second computing system when performing the reduced communication training of the neural network.

[0212] Clause 11: The method according to Clause 10, wherein sending the second element of the output data from the first computing system to the second computing system comprises: at the first computing system, generating compressed output data based on compressing the second element of the output data using one or more compression operations; and sending the compressed output data from the first computing system to the second computing system.

[0213] Clause 12: The method according to any one of Clauses 1 to 11, wherein performing the reduced communication training of the neural network comprises: at the first computing system, generating compressed output data based on at least a second element of the output data compressed using one or more compression operations; and transmitting the compressed output data from the first computing system to the second computing system.

[0214] Clause 13: The method according to Clause 12, wherein generating the compressed output data comprises: selecting a first compression operation among the one or more compression operations based on the communication criteria; and using the first compression operation to compress the at least second element of the output data to generate the compressed output data.

[0215] Clause 14: The method according to any one of Clauses 1 to 13, wherein performing the reduced communication training of the neural network further comprises: generating a second element of input data based on decompressing data received from the second computing system; and updating one or more parameters of the first portion of the neural network based on the second element of the input data.

[0216] Clause 15: A method comprising: accessing a first element of input data for a first part of a neural network; generating a first element of output data based on the first element of the input data processed using the first part of the neural network; determining at a first time point that one or more communication criteria are not satisfied; and in response to the determination: generating compressed output data based on the first element of the output data compressed using one or more compression operations; and transmitting the compressed output data from a first computing system to a second computing system for the second computing system to update one or more parameters of a second part of the neural network based on the compressed output data.

[0217] Clause 16: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of Clauses 1 to 15.

[0218] Clause 17: A processing system comprising components for performing the method according to any one of Clauses 1 to 15.

[0219] Clause 18: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method according to any one of Clauses 1 to 15.

[0220] Clause 19: A non-transitory computer-readable medium encoding logic, which, when executed by a processing system, causes the processing system to perform the method according to any one of Clauses 1 to 15.

[0221] Clause 20: An apparatus comprising logic circuitry configured to perform the method according to any one of Clauses 1 to 15.

[0222] Clause 21: A computer program product embodied on a computer-readable storage medium, the computer-readable storage medium including code for performing the method according to any one of Clauses 1 to 15.

[0223] Additional notes

[0224] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of the disclosure herein may be embodied by one or more elements of the claims.

[0225] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0226] As used in this article, the phrase “at least one of” in a list of items refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0227] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0228] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0229] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC § 112(f) unless the element is expressly referred to by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known later, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A first processing system, the first processing system comprising: One or more memories, the one or more memories including processor-executable instructions; and One or more processors, the one or more processors being configured to execute processor-executable instructions and enable the first processing system: Access the first element of the input data used for the first part of the neural network; The first processing system generates a first element of output data, wherein, in order to generate the first element of the output data, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to process the first element of the input data using the first portion of the neural network; At a first time point, the first element of the output data is sent from the first processing system to the second processing system so that the second processing system can update one or more parameters of the second part of the neural network based on the first element of the output data; The first processing system determines at a second time point after the first time point that one or more communication criteria are not met. as well as In response to the determination, reduced communication training of the neural network is performed, wherein, in order to perform the reduced communication training of the neural network, the one or more processors are configured to execute processor-executable instructions to reduce the amount of data sent by the first processing system for one or more rounds of training.

2. The first processing system according to claim 1, wherein the one or more communication criteria include at least one of the following: The status of one or more communication links between the first processing system and the second processing system. Resource availability on at least one of the first processing system or the second processing system One or more features of the training data used to train the neural network, or One or more indications of the training progress of the neural network.

3. The first processing system according to claim 1, wherein: The first portion of the neural network includes one or more initial layers of the neural network. The first element of the output data includes a feature tensor output by the intermediate layer of the neural network, and The second part of the neural network includes one or more final layers of the neural network.

4. The first processing system of claim 3, wherein the one or more processors are configured to further execute processor-executable instructions to cause the first processing system to update one or more parameters of the first portion of the neural network based on a set of gradients received at the first processing system from the second processing system.

5. The first processing system according to claim 1, wherein: The first part of the neural network includes one or more final layers of the neural network. The first element of the output data includes the gradient tensor, and The second part of the neural network includes one or more initial layers of the neural network.

6. The first processing system of claim 5, wherein the one or more processors are configured to further execute processor-executable instructions to cause the first processing system to: The second element of the output data is generated based on the first element of the input data processed using the first part of the neural network; and One or more parameters of the first part of the neural network are updated based on the second element of the output data.

7. The first processing system of claim 1, wherein, in order to perform the reduced communication training of the neural network, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to avoid sending at least a second element of the output data from the first processing system to the second processing system.

8. The first processing system of claim 7, wherein, in order to perform the reduced communication training of the neural network, the one or more processors are configured to execute processor-executable instructions to further cause the first processing system to: The second element of the input data received from the second processing system; and The first part of the neural network is updated based on the second element of the input data.

9. The first processing system of claim 8, wherein, in order to update one or more parameters of the first portion of the neural network, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to: Decompressed data is generated by decompressing the second element of the input data from the second processing system; and One or more parameters of the first part of the neural network are updated based on the decompressed data.

10. The first processing system according to claim 1, wherein: To perform the reduced communication training of the neural network, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to send at least a second element of the output data from the first processing system to the second processing system, and The first processing system does not receive a second element of the input data from the second processing system when performing the reduced communication training of the neural network.

11. The first processing system of claim 10, wherein, in order to send the second element of the output data from the first processing system to the second processing system, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to: Generate compressed output data, wherein, in order to generate the compressed output data, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to compress the second element of the output data using one or more compression operations; and The compressed output data is sent from the first processing system to the second processing system.

12. The first processing system of claim 1, wherein, in order to perform the reduced communication training of the neural network, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to: Generate compressed output data, wherein, in order to generate the compressed output data, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to compress at least a second element of the output data using one or more compression operations; and The compressed output data is sent from the first processing system to the second processing system.

13. The first processing system of claim 12, wherein, in order to generate the compressed output data, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to: The first compression operation among the one or more compression operations is selected based on the communication criteria; and The at least second element of the compressed output data, wherein, in order to compress the at least second element of the output data, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to generate the compressed output data using the first compression operation.

14. The first processing system of claim 1, wherein, in order to further perform the reduced communication training of the neural network, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to: Generating a second element of the input data, wherein, in order to generate the second element of the input data, the one or more processors are configured to execute processor-executable instructions to cause the first processing system to decompress data received from the second processing system; and The first part of the neural network is updated based on the second element of the input data.

15. A processor-implemented method, the processor-implemented method comprising: Access the first element of the input data used for the first part of the neural network; The first computing system generates the first element of the output data based on the first element of the input data processed using the first part of the neural network; At a first time point, the first element of the output data is sent from the first computing system to the second computing system so that the second computing system can update one or more parameters of the second part of the neural network based on the first element of the output data; The first computing system determines at a second time point after the first time point that one or more communication criteria are not met; as well as In response to the determination, reduced communication training of the neural network is performed, the performance including reducing the amount of data sent by the first computing system for one or more rounds of training.

16. The processor-implemented method of claim 15, wherein the one or more communication criteria include at least one of the following: The status of one or more communication links between the first computing system and the second computing system. Resource availability on at least one of the first computing system or the second computing system. One or more features of the training data used to train the neural network, or One or more indications of the training progress of the neural network.

17. The processor-implemented method according to claim 15, wherein: The first portion of the neural network includes one or more initial layers of the neural network. The first element of the output data includes a feature tensor output by the intermediate layer of the neural network, and The second part of the neural network includes one or more final layers of the neural network.

18. The processor-implemented method of claim 17, further comprising updating one or more parameters of the first portion of the neural network by the first computing system based on a set of gradients received at the first computing system from the second computing system.

19. The processor-implemented method according to claim 15, wherein: The first part of the neural network includes one or more final layers of the neural network. The first element of the output data includes the gradient tensor, and The second part of the neural network includes one or more initial layers of the neural network.

20. The processor-implemented method according to claim 19, further comprising: The first computing system generates the second element of the output data based on the first element of the input data processed using the first part of the neural network; as well as One or more parameters of the first part of the neural network are updated based on the second element of the output data.

21. The processor-implemented method of claim 15, wherein performing the reduced communication training of the neural network includes avoiding sending at least a second element of the output data from the first computing system to the second computing system.

22. The processor-implemented method of claim 21, wherein performing the reduced communication training of the neural network further comprises: Receive the second element of the input data from the second computing system; as well as The first part of the neural network is updated based on the second element of the input data.

23. The processor-implemented method of claim 22, wherein updating one or more parameters of the first portion of the neural network comprises: Decompressed data is generated by decompressing the second element of the input data from the second computing system; as well as One or more parameters of the first part of the neural network are updated based on the decompressed data.

24. The processor-implemented method according to claim 15, wherein: Performing the reduced communication training of the neural network includes sending at least a second element of the output data from the first computing system to the second computing system, and The first computing system does not receive a second element of the input data from the second computing system when performing the reduced communication training of the neural network.

25. The processor-implemented method of claim 24, wherein sending the second element of the output data from the first computing system to the second computing system comprises: At the first computing system, compressed output data is generated based on the second element of the output data compressed using one or more compression operations; as well as The compressed output data is sent from the first computing system to the second computing system.

26. The processor-implemented method of claim 15, wherein performing the reduced communication training of the neural network comprises: At the first computing system, compressed output data is generated based on at least a second element of the output data compressed using one or more compression operations; as well as The compressed output data is sent from the first computing system to the second computing system.

27. The processor-implemented method of claim 26, wherein generating the compressed output data comprises: The first compression operation among the one or more compression operations is selected based on the communication criteria. as well as The first compression operation is used to compress the at least second element of the output data to generate the compressed output data.

28. The processor-implemented method of claim 15, wherein performing the reduced communication training of the neural network further comprises: The second element of the input data is generated by decompressing the data received from the second computing system. as well as The first part of the neural network is updated based on the second element of the input data.

29. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media comprising computer-executable instructions, which, when executed by one or more processors of one or more processing systems, cause the one or more processing systems to: Access elements of the input data used for the first part of the neural network; Elements of output data generated by a first processing system, wherein, in order to generate the elements of the output data, the one or more processors are configured to execute computer-executable instructions to cause the first processing system to process the elements of the input data using the first portion of the neural network; At a first time point, the elements of the output data are sent from the first processing system to the second processing system for the second processing system to update one or more parameters of the second part of the neural network based on the elements of the output data; The first processing system determines at a second time point after the first time point that one or more communication criteria are not met. as well as In response to the determination, reduced communication training of the neural network is performed, wherein, in order to perform the reduced communication training of the neural network, the one or more processors are configured to execute computer-executable instructions to reduce the amount of data sent by the first processing system for one or more rounds of training.

30. A first processing system, the first processing system comprising: A component for accessing elements of the input data used in the first part of a neural network; A component for generating output data by the first processing system based on the elements of the input data processed using the first part of the neural network; A component for sending the element of the output data from the first processing system to the second processing system at a first time point so that the second processing system can update one or more parameters of the second part of the neural network based on the element of the output data; Components for determining by the first processing system at a second time point after the first time point that one or more communication criteria are not met; and Components for performing reduced communication training of the neural network in response to the determination, the execution including reducing the amount of data sent by the first processing system for one or more rounds of training.

Images