Performing model-parallel training with local learning in processor-based devices

Abstract

Performing model-parallel training with local learning in processor-based devices is disclosed herein. In some aspects, a distributed training system comprises a plurality of processor-based nodes, wherein each node is configured to receive input data, and process the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM). Each node calculates a local loss function, and calculates local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. Each node is also configured to forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. Each next subsequent node comprises a fusion block that is configured to fuse local node data of a previous node with local node data of the next subsequent node.

Description

PERFORMING MODEL-PARALLEL TRAINING WITH LOCAL LEARNING IN PROCESSOR-BASED DEVICESPRIORITY APPLICATION

[0001] The present application claims priority to Greek Patent Application Serial No. 20240100065, filed February 1, 2024 and entitled “PERFORMING MODELPARALLEL TRAINING WITH LOCAL LEARNING IN PROCESSOR DEVICES,” which is incorporated herein by reference in its entirety.BACKGROUNDI. Field of the Disclosure

[0002] The technology of the disclosure relates generally to machine learning, and, in particular, to mechanisms for training machine learning models.IL Background

[0003] Machine learning is a subfield of artificial intelligence (Al) that uses algorithms trained on data sets to create models that enable computers to perform tasks such as image categorization or data analysis. One type of machine learning model is a large language model (LLM), which can be used to predict and generate language sequences by determining the probability of words or sequences of words that follow a given token or sequence of tokens. LLMs are useful for automating tasks including text generation, language translation, and query responses. However, training an LLM may be challenging when using a single, centralized setting such as a single processor-based device, due to the large number of parameters (e.g., on the order of millions or billions) used by many LLMs. In such a centralized setting, it may be impossible to fit the LLM within the memory of the processor-based device, which may require the processor-based device to perform prohibitively slow memory swapping using non-volatile storage.

[0004] To address this challenge, techniques such as distributed training of LLMs using multiple interconnected processor-based nodes have been developed. Conventional distributed training approaches can be generally divided into model-parallel approaches and data-parallel approaches. With model-parallel approaches, each node of a plurality of nodes receives a subset of layers of the LLM to process. The first node computes itsown forward activations and sends them to the next subsequent node, and so on in like fashion. The last node then computes a loss function and gradients, and sends that data back through the nodes in reverse order. Model-parallel approaches, though, may result in sub-optimal hardware utilization, as each node is idle after it generates and sends its forward activations until it receives gradients from the next subsequent node.

[0005] In contrast, with data-parallel approaches, each node of the plurality of nodes receives a full replica of the LLM, but works on a smaller set of training data. After processing the training data during a training interval, the updates generated by each node must be synchronized with the other nodes. This information exchange between nodes during synchronization may require a high-capacity communications link between the nodes.

[0006] Accordingly, it is desirable to implement an approach for training an LLM that is more efficient in terms of hardware utilization and has lower communication bandwidth requirements, while providing performance comparable or superior to existing approaches.SUMMARY OF THE DISCLOSURE

[0007] Aspects disclosed in the detailed description include performing modelparallel training with local learning in processor-based devices. Related apparatus and methods are also disclosed. In this regard, in some exemplary aspects disclosed herein, a distributed training system includes a plurality of processor-based nodes that are configured to employ a local learning model-parallel (LLMP) training approach. Each node is assigned a plurality of modules that comprise a distinct subset of layers of a large language model (LLM). Upon receiving input data, each node processes the input data using its plurality of modules, and calculates a local loss function. Each node then calculates local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. Each node forwards local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. The next subsequent node fuses the local node data of a previous node with local node data of the next subsequent node (e.g., by fusing an intermediate representation from the previous node with a local input representation). In some aspects, fusing the local node data of the previous node with thelocal node data of the next subsequent node uses a linear combiner. By dividing the LLM among multiple nodes, the LLMP training approach assigns fewer LLM parameters to each node, which results in lower processing, memory, and storage requirements per node in comparison to conventional data-parallel approaches. Additionally, because gradients are not communicated between nodes and intermediate representations and / or logits are only communicated in a forward direction, each node is able to achieve higher hardware utilization when compared to conventional model-parallel approaches.

[0008] In another aspect, a distributed training system is disclosed. The distributed training system comprises a plurality of processor-based nodes, wherein each node is configured to receive input data, and process the input data using a plurality of modules that comprise a distinct subset of layers of an LLM. Each node is further configured to calculate a local loss function, and calculate local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. Each node is also configured to forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. Each next subsequent node comprises a fusion block that is configured to fuse local node data of a previous node with local node data of the next subsequent node.

[0009] In another aspect, a distributed training system is disclosed. The distributed training system comprises a plurality of processor-based nodes, wherein each node comprises means for receiving input data. Each node further comprises means for processing the input data using a plurality of modules that comprise a distinct subset of layers of an LLM. Each node also comprises means for calculating a local loss function. Each node additionally comprises means for calculating local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. Each node further comprises means for forwarding local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. Each next subsequent node comprises means for fusing local node data of a previous node with local node data of the next subsequent node.

[0010] In another aspect, a method for performing model-parallel training with local learning in processor-based devices is disclosed. The method comprises, for each nodeof a plurality of processor-based nodes of a distributed training system, receiving, by the node, input data. The method further comprises processing, by the node, the input data using a plurality of modules that comprise a distinct subset of layers of the LLM. The method also comprises calculating, by the node, a local loss function. The method additionally comprises calculating, by the node, local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. The method further comprises forwarding, by the node, local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. The method also comprises fusing, by the next subsequent node, local node data of a previous node with local node data of the next subsequent node.

[0011] In another aspect, a non-transitory computer-readable medium is disclosed. The non-transitory computer-readable medium stores computer-executable instructions that, when executed, cause a processor of each node of a plurality of processor-based nodes of a distributed training system to receive input data. The computer-executable instructions further cause the processor to process the input data using a plurality of modules that comprise a distinct subset of layers of an LLM. The computer-executable instructions also cause the processor to calculate a local loss function. The computerexecutable instructions additionally cause the processor to calculate local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. The computer-executable instructions further cause the processor to forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. The computer-executable instructions further cause a processor of each next subsequent node to fuse local node data of a previous node with local node data of the next subsequent node.BRIEF DESCRIPTION OF THE FIGURES

[0012] Figures 1A and IB are block diagrams illustrating conventional centralized and data-parallel approaches, respectively, to large language model (LLM) training;

[0013] Figure 2 is a block diagram illustrating an exemplary distributed training system that is configured to perform model-parallel training with local learning, according to some aspects;

[0014] Figure 3 provides a flowchart illustrating exemplary operations of processorbased nodes of the distributed training system of Figure 2 for performing model -parallel training with local learning, according to some aspects; and

[0015] Figure 4 is a block diagram of an exemplary processor-based device that can include the processor-based nodes of Figure 2.DETAILED DESCRIPTION

[0016] With reference now to the drawing figures, several exemplary aspects of the present disclosure are described. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. The terms “first,” “second,” and the like used herein are intended to distinguish between similarly named elements, and do not indicate an ordinal relationship between such elements unless otherwise indicated.

[0017] Aspects disclosed in the detailed description include performing modelparallel training with local learning in processor-based devices. Related apparatus and methods are also disclosed. In this regard, in some exemplary aspects disclosed herein, a distributed training system includes a plurality of processor-based nodes that are configured to employ a local learning model-parallel (LLMP) training approach. Each node is assigned a plurality of modules that comprise a distinct subset of layers of a large language model (LLM). Upon receiving input data, each node processes the input data using its plurality of modules, and calculates a local loss function. Each node then calculates local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes. Each node forwards local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit. The next subsequent node fuses the local node data of a previous node with local node data of the next subsequent node (e.g., by fusing an intermediate representation from the previous node with a local input representation). In some aspects, fusing the local node data of the previous node with thelocal node data of the next subsequent node uses a linear combiner. By dividing the LLM among multiple nodes, the LLMP training approach assigns fewer LLM parameters to each node, which results in lower processing, memory, and storage requirements per node in comparison to conventional data-parallel approaches. Additionally, because gradients are not communicated between nodes and intermediate representations and / or logits are only communicated in a forward direction, each node is able to achieve higher hardware utilization when compared to conventional model-parallel approaches.

[0018] Before describing LLMP training in greater detail, conventional centralized and data-parallel approaches to LLM training are first discussed. In this regard, Figure 1A illustrates a centralized training approach using a single processor-based node (captioned as “NODE” in Figure 1A) 100. The node 100 is configured to process input data 102 using modules 104(0)-104(N) that each comprise one or more layers (not shown) of an LLM (not shown), and that together represent the entire LLM. As the input data 102 is processed through the modules 104(0)-104(N), forward activations between the modules 104(0)-104(N), represented by rightward arrows, are performed. The node 100 then calculates a loss function 106, and gradients, represented by leftward arrows, are sent back through the modules 104(0)-104(N) in reverse order and used to update the LLM. As noted above, though, the large number of parameters used by the LLM may be impossible to fit within the memory of the node 100, which may require the node 100 to perform prohibitively slow memory swapping using non-volatile storage (not shown).

[0019] Figure IB illustrates a data-parallel training approach that addresses some of the shortcomings of the centralized approach of Figure 1A. In Figure IB, two (2) processor-based nodes (captioned as “NODE” in Figure IB) 108(0) and 108(1) are each configured to use the modules 104(0)-104(N) representing the entire LLM to process smaller subsets of input data 110(0) and 110(1). The nodes 108(0) and 108(1) process the respective input data 110(0) and 110(1) through the modules 104(0)-104(N) (as represented by rightward arrows), calculate corresponding loss functions 112(0) and 112(1), and send gradients back through the modules 104(0)-104(N) in reverse order (represented by leftward arrows) to update each local copy of the LLM. The nodes 108(0) and 108(1) synchronize their respective updates, as represented by vertical arrows. However, this synchronization may require a high-capacity communications link between the nodes 108(0) and 108(1).

[0020] In this regard, Figure 2 illustrates an exemplary distributed training system 200 that is configured to perform LLMP training. The distributed training system 200 comprises a plurality of processor-based nodes (captioned as “NODE” in Figure 2) 202(0)-202(D), two (2) of which are shown in the example of Figure 2 (i.e., D=2). Each of the nodes 202(0)-202(D) comprises a processor-based device such as those discussed in greater detail below with respect to Figure 4. The nodes 202(0)-202(D) are each configured to process a plurality of modules 204(0)-204(N) that comprise a distinct subset of layers of an LLM (not shown), and that correspond to the modules 104(0)-104(N) of Figures 1 A-1B. In this example, the node 202(0) processes the modules 204(0)-204(M), while the node 202(D) processes the modules 204(M+l)-204(N). Both of the nodes 202(0) and 202(D) process the same input data 206. It is to be understood that, while only two (2) nodes are shown in the example of Figure 2, some aspects may comprise more than two (2) nodes.

[0021] In exemplary operation, the node 202(0) of the distributed training system 200 receives the input data 206 and processes the input data 206 using the modules 204(0)- 204(M), with forward activations between the modules 204(0)-204(M) represented by rightward arrows. The node 202(0) then calculates a local loss function 208, and also calculates local gradients, such as a local gradient 210, based on the local loss function 208. The node 202(0) does not communicate the local gradients to other nodes of the plurality of processor-based nodes 202(0)-202(D), but instead updates its own modules 204(0)-204(M) without having to idle to wait for gradients from other nodes. The node 202(0) also forwards local node data 212 (e.g., an intermediate representation and / or a logit) to the next subsequent node 202(D), which continues processing using the modules 204(M+l)-204(N) and a loss function 214. The next subsequent node 202(D) uses a fusion block 216 and / or a fusion block 218 to fuse the local node data 212 of the previous node 202(0) with local node data 220 of the next subsequent node 202(D). The local node data 220 of the node 202(D) may comprise, e.g., a local input representation of the input data 206, as a non-limiting example. In some aspects, the fusion block 216 and / or the fusion block 218 may comprise a linear combiner. It is to be understood that nodes other than the node 202(0) may operate both as the “previous node” and as the “next subsequent node” depending on context, and thus may perform the operations attributed herein to the node 202(0) and the node 202(D).

[0022] To illustrate exemplary operations performed by the processor-based nodes 202(0)-202(D) of Figure 2 for performing model-parallel training with local learning according to some aspects, Figure 3 provides a flowchart showing exemplary operations 300. For the sake of clarity, elements of Figure 2 are referenced in describing Figure 3. It is to be understood that some aspects may provide that some operations illustrated in Figure 3 may be performed in an order other than that illustrated herein, and / or may be omitted.

[0023] The exemplary operations 300 begin in Figure 3 with each node of a plurality of processor-based nodes of a distributed training system (e.g., the node 202(0) of the plurality of processor-based nodes 202(0)-202(D) of the distributed training system 200 of Figure 2) performing a series of operations (block 302). The node 202(0) receives input data (such as the input data 206 of Figure 1) (block 304). The node 202(0) then processes the input data 206 using a plurality of modules (e.g., the modules 204(0)- 204(M) of Figure 2) that comprise a distinct subset of layers of an LLM (block 306). The node 202(0) calculates a local loss function (e.g., the local loss function 208 of Figure 2) (block 308). The node 202(0) calculates local gradients (such as the local gradient 210) based on the local loss function 208, without communicating the local gradients 210 to other nodes of the plurality of processor-based nodes 202(0)-202(D) (block 310). The node 202(0) forwards local node data (e.g., local node data 212 of Figure 2) to a next subsequent node (such as the node 202(D) of Figure 2), wherein the local node data 212 comprises one or more of an intermediate representation and a logit (block 312). The next subsequent node 202(D) fuses the local node data 212 of a previous node (i.e., the node 202(0) of Figure 2) with local node data (e.g., the local node data 220 of Figure 2) of the next subsequent node 202(D) (block 314). In some aspects, the operations of block 314 for fusing the local node data 212 of the previous node 202(0) with the local node data 220 of the next subsequent node 202(D) use a linear combiner (block 316).

[0024] The processor-based nodes according to aspects disclosed herein and discussed with reference to Figures 2 and 3 may be provided in or integrated into any processor-based device. Examples, without limitation, include a set top box, an entertainment unit, a navigation device, a communications device, a fixed location data unit, a mobile location data unit, a global positioning system (GPS) device, a mobile phone, a cellular phone, a smart phone, a session initiation protocol (SIP) phone, a tablet,a phablet, a server, a computer, a portable computer, a mobile computing device, laptop computer, a wearable computing device (e.g., a smart watch, a health or fitness tracker, eyewear, etc.), a desktop computer, a personal digital assistant (PDA), a monitor, a computer monitor, a television, a tuner, a radio, a satellite radio, a music player, a digital music player, a portable music player, a digital video player, a video player, a digital video disc (DVD) player, a portable digital video player, an automobile, a vehicle component, an avionics system, a drone, and a multicopter.

[0025] In this regard, Figure 4 illustrates an example of a processor-based device 400, which corresponds in functionality to the processor-based nodes 202(0)-202(D) of Figure 2. The processor-based device 400 includes a processor device 402 which comprises one or more CPUs 404 coupled to a cache memory 406. The CPU(s) 404 is also coupled to a system bus 408 and can intercouple devices included in the processor-based device 400. As is well known, the CPU(s) 404 communicates with these other devices by exchanging address, control, and data information over the system bus 408. For example, the CPU(s) 404 can communicate bus transaction requests to a memory controller 410. Although not illustrated in Figure 4, multiple system buses 408 could be provided, wherein each system bus 408 constitutes a different fabric.

[0026] Other devices may be connected to the system bus 408. As illustrated in Figure 4, these devices can include a memory system 412, one or more input devices 414, one or more output devices 416, one or more network interface devices 418, and one or more display controllers 420, as examples. The input device(s) 414 can include any type of input device, including, but not limited to, input keys, switches, voice processors, etc. The output device(s) 416 can include any type of output device, including, but not limited to, audio, video, other visual indicators, etc. The network interface device(s) 418 can be any devices configured to allow exchange of data to and from a network 422. The network 422 can be any type of network, including, but not limited to, a wired or wireless network, a private or public network, a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a BLUETOOTH™ network, and the Internet. The network interface device(s) 418 can be configured to support any type of communications protocol desired. The memory system 412 can include the memory controller 410 coupled to one or more memory arrays 424.

[0027] The CPU(s) 404 may also be configured to access the display controller(s) 420 over the system bus 408 to control information sent to one or more displays 426. The display controller(s) 420 sends information to the display(s) 426 to be displayed via one or more video processors 428, which process the information to be displayed into a format suitable for the display(s) 426. The display(s) 426 can include any type of display, including, but not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, etc.

[0028] Those of skill in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithms described in connection with the aspects disclosed herein may be implemented as electronic hardware, instructions stored in memory or in another computer readable medium and executed by a processor device. The master devices and slave devices described herein may be employed in any circuit, hardware component, integrated circuit (IC), or IC chip, as examples. Memory disclosed herein may be any type and size of memory and may be configured to store any type of information desired. To clearly illustrate this interchangeability, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. How such functionality is implemented depends upon the particular application, design choices, and / or design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0029] The various illustrative logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a processor device, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor device may be a microprocessor, but in the alternative, the processor device may be any conventional processor device, controller, microcontroller, or state machine. A processor device may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).

[0030] The aspects disclosed herein may be embodied in hardware and in instructions that are stored in hardware, and may reside, for example, in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, a hard disk, a removable disk, a CD-ROM, or any other form of computer readable medium known in the art. An exemplary storage medium is coupled to the processor device such that the processor device can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor device. The processor device and the storage medium may reside in an ASIC. The ASIC may reside in a remote station. In the alternative, the processor device and the storage medium may reside as discrete components in a remote station, base station, or server.

[0031] It is also noted that the operational steps described in any of the exemplary aspects herein are described to provide examples and discussion. The operations described may be performed in numerous different sequences other than the illustrated sequences. Furthermore, operations described in a single operational step may actually be performed in a number of different steps. Additionally, one or more operational steps discussed in the exemplary aspects may be combined. It is to be understood that the operational steps illustrated in the flowchart diagrams may be subject to numerous different modifications as will be readily apparent to one of skill in the art. Those of skill in the art will also understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0032] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0033] Implementation examples are described in the following numbered clauses:1. A distributed training system comprising a plurality of processor-based nodes, wherein: each node is configured to: receive input data; process the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM); calculate a local loss function; calculate local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; and each next subsequent node comprises a fusion block that is configured to fuse local node data of a previous node with local node data of the next subsequent node.2. The distributed training system of clause 1, wherein the fusion block is configured to fuse the local node data of the previous node with local node data of the next subsequent node using a linear combiner.3. The distributed training system of any one of clauses 1-2, integrated into a device selected from the group consisting of a set top box; an entertainment unit; a navigation device; a communications device; a fixed location data unit; a mobile location data unit; a global positioning system (GPS) device; a mobile phone; a cellular phone; a smart phone; a session initiation protocol (SIP) phone; a tablet; a phablet; a server; a computer; a portable computer; a mobile computing device; a wearable computing device; a desktop computer; a personal digital assistant (PDA); a monitor; a computer monitor; a television; a tuner; a radio; a satellite radio; a music player; a digital music player; a portable music player; a digital video player; a video player; a digital video disc (DVD) player; a portable digital video player; an automobile; a vehicle component; avionics systems; a drone; and a multicopter.4. A distributed training system comprising a plurality of processor-based nodes, wherein: each node comprises: means for receiving input data; means for processing the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM); means for calculating a local loss function; means for calculating local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and means for forwarding local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; and each next subsequent node comprises means for fusing local node data of a previous node with local node data of the next subsequent node.5. A method for performing model-parallel training with local learning in processorbased devices, comprising: for each node of a plurality of processor-based nodes of a distributed training system: receiving, by the node, input data; processing, by the node, the input data using a plurality of modules that comprise a distinct subset of layers of the LLM; calculating, by the node, a local loss function; calculating, by the node, local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and forwarding, by the node, local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; andfusing, by the next subsequent node, local node data of a previous node with local node data of the next subsequent node.6. The method of clause 5, comprising fusing the local node data of the previous node with the local node data of the next subsequent node using a linear combiner.7. A non-transitory computer-readable medium, having stored thereon computerexecutable instructions that, when executed, cause a processor of each node of a plurality of processor-based nodes of a distributed training system to: receive input data; process the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM); calculate a local loss function; calculate local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; wherein the computer-executable instructions further cause a processor of each next subsequent node to fuse local node data of a previous node with local node data of the next subsequent node.8. The non-transitory computer-readable medium of clause 7, wherein the computerexecutable instructions further cause the processor of each next subsequent node to fuse local node data of the previous node with local node data of the next subsequent node using a linear combiner.

Claims

What is claimed is:

1. A distributed training system comprising a plurality of processor-based nodes, wherein: each node is configured to: receive input data; process the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM); calculate a local loss function; calculate local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; and each next subsequent node comprises a fusion block that is configured to fuse local node data of a previous node with local node data of the next subsequent node.

2. The distributed training system of claim 1, wherein the fusion block is configured to fuse the local node data of the previous node with local node data of the next subsequent node using a linear combiner.

3. The distributed training system of claim 1, integrated into a device selected from the group consisting of: a set top box; an entertainment unit; a navigation device; a communications device; a fixed location data unit; a mobile location data unit; a global positioning system (GPS) device; a mobile phone; a cellular phone; a smart phone; a session initiation protocol (SIP) phone; a tablet; a phablet; a server; a computer; a portable computer; a mobile computing device; a wearable computing device; a desktop computer; a personal digital assistant (PDA); a monitor; a computer monitor; a television; a tuner; a radio; a satellite radio; a music player; a digital music player; a portable music player; a digital video player; a video player; a digital video disc (DVD) player; a portable digitalvideo player; an automobile; a vehicle component; avionics systems; a drone; and a multicopter.

4. A distributed training system comprising a plurality of processor-based nodes, wherein: each node comprises: means for receiving input data; means for processing the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM); means for calculating a local loss function; means for calculating local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and means for forwarding local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; and each next subsequent node comprises means for fusing local node data of a previous node with local node data of the next subsequent node.

5. A method for performing model-parallel training with local learning in processorbased devices, comprising: for each node of a plurality of processor-based nodes of a distributed training system: receiving, by the node, input data; processing, by the node, the input data using a plurality of modules that comprise a distinct subset of layers of the LLM; calculating, by the node, a local loss function; calculating, by the node, local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; andforwarding, by the node, local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; and fusing, by the next subsequent node, local node data of a previous node with local node data of the next subsequent node.

6. The method of claim 5, comprising fusing the local node data of the previous node with the local node data of the next subsequent node using a linear combiner.

7. A non-transitory computer-readable medium, having stored thereon computerexecutable instructions that, when executed, cause a processor of each node of a plurality of processor-based nodes of a distributed training system to: receive input data; process the input data using a plurality of modules that comprise a distinct subset of layers of a large language model (LLM); calculate a local loss function; calculate local gradients based on the local loss function, without communicating the local gradients to other nodes of the plurality of processor-based nodes; and forward local node data to a next subsequent node, wherein the local node data comprises one or more of an intermediate representation and a logit; wherein the computer-executable instructions further cause a processor of each next subsequent node to fuse local node data of a previous node with local node data of the next subsequent node.

8. The non-transitory computer-readable medium of claim 7, wherein the computerexecutable instructions further cause the processor of each next subsequent node to fuse local node data of the previous node with local node data of the next subsequent node using a linear combiner.

Images