Apparatus and method for supporting graph neural network learning

Abstract

A graph neural network learning support device and method having a multi-queue and layer-dependent sampling structure for efficient learning in a multi-graphics processing unit environment are disclosed. A graph neural network learning support device according to one embodiment may include an interface unit for data input and output; a mini-batch generation unit that generates a mini-batch for parallel processing based on data received through the interface unit; and two or more neural network learning units that update a graph neural network by calculating a gradient based on the mini-batch.

Description

Technology Field

[0001] This invention relates to a graph neural network learning support device and method having a multi-queue and layer-dependent sampling structure for efficient learning in a multi-graphics processing unit environment. Background Technology

[0003] Various real-world data, such as social networks, gene expression networks, and knowledge graphs, take the form of graphs. Accordingly, Graph Neural Networks (GNNs) are growing into an important field in machine learning.

[0004] However, resource-efficient GNN training for massive graphs is a major challenge. In applications, graphs are typically very large, containing hundreds of millions to billions of nodes and tens of billions to hundreds of billions of edges. Furthermore, nodes and edges are coupled with feature vectors to be fed as input to the GNN. Consequently, training massive graphs on a single graphics processing unit (GPU) or a single machine is very difficult due to limited resources. Prior art literature

[0006] Republic of Korea Published Patent No. 10-2023-0170397 The problem to be solved

[0007] The purpose is to provide a graph neural network learning support device and method having a multi-queue and layer-dependent sampling structure for efficient learning in a multi-graphics processing unit environment. means of solving the problem

[0009] According to one aspect, a graph neural network learning support device may include an interface unit for data input / output; a mini-batch generation unit that generates a mini-batch for parallel processing based on data received through the interface unit; and two or more neural network learning units that update a graph neural network by calculating a gradient based on the mini-batch.

[0010] The minibatch generation unit may include a central processing unit (CPU) that samples data received through the interface unit layer by layer to generate two or more minibatches for parallel processing, and two or more parallel-connected first minibatch buffers that each store the generated two or more minibatches.

[0011] The neural network learning unit may include a second mini-batch buffer that receives a mini-batch from any one of two or more first mini-batch buffers connected in parallel, a Graphics Processing Unit (GPU) that computes a gradient for training a graph neural network based on the mini-batch stored in the second mini-batch buffer, and a gradient buffer that stores the computed gradient.

[0012] The number of neural network learning units can be equal to the number of the first mini-batch buffer.

[0013] Two or more neural network learning units can be connected in parallel for parallel computation.

[0014] The neural network learning unit checks the gradient buffers of other neural network learning units connected in parallel, and if gradients for the same layer are stored in the gradient buffers of two or more neural network learning units, the graph neural network can be updated by synchronizing the gradients.

[0015] Each graphics processing unit included in the neural network learning unit can asynchronously calculate the gradient and store it in the gradient buffer, independently of the operation of the graphics processing units of other neural network learning units.

[0016] According to one aspect, a method performed in a computing device having one or more processors and a memory for storing one or more programs executed by one or more processors may include the step of generating a mini-batch for parallel processing based on data received through an interface; and the step of updating a graph neural network by calculating a gradient based on the mini-batch.

[0017] The step of generating a minibatch may utilize a minibatch generator comprising a central processing unit (CPU) that samples data received through an interface layer by layer to generate two or more minibatches for parallel processing, and two or more parallel-connected first minibatch buffers that each store the generated two or more minibatches.

[0018] The step of updating a graph neural network may utilize two or more neural network learners comprising: a second mini-batch buffer that receives a mini-batch from any one of two or more first mini-batch buffers connected in parallel; a Graphics Processing Unit (GPU) that computes a gradient to train the graph neural network based on the mini-batch stored in the second mini-batch buffer; and a gradient buffer that stores the computed gradient.

[0019] The step of updating the graph neural network checks the gradient buffers of other neural network learning units connected in parallel, and if the gradients for the same layer are stored in the gradient buffers of two or more neural network learning units, the graph neural network can be updated by synchronizing the gradients.

[0020] Each graphics processing unit included in a neural network learner can asynchronously calculate gradients and store them in a gradient buffer, independently of the computational operations of the graphics processing units of other neural network learners.

[0021] According to one aspect, a computer program stored on a non-transitory computer-readable storage medium comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device may be configured to perform the steps of: generating a mini-batch for parallel processing based on data received through an interface; and updating a graph neural network by computing a gradient based on the mini-batch. Effects of the invention

[0023] The computational efficiency of parallel-connected GPUs can be increased by using mini-batch buffers and gradient buffers. Brief explanation of the drawing

[0025] Figure 1 is a configuration diagram of a graph neural network learning support device in one embodiment. FIG. 2 is an exemplary diagram illustrating the configuration of a graph neural network learning support device according to one embodiment. FIG. 3 is a flowchart illustrating a graph neural network learning support method according to one embodiment. FIG. 4 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. Specific details for implementing the invention

[0026] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0027] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0028] Additionally, terms such as "first," "second," etc., may be used to describe various components, but the components should not be limited by these terms. The terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0029] Figure 1 is a configuration diagram of a graph neural network learning support device in one embodiment.

[0030] Referring to FIG. 1, a graph neural network learning support device (100) may include an interface unit (110) for data input / output, a mini-batch generation unit (120) for generating a mini-batch for parallel processing based on data received through the interface unit (110), and two or more neural network learning units (130) for updating a graph neural network by calculating a gradient based on the mini-batch.

[0031] For example, a graph neural network learning support device (100) may have MQ-LAS (Multi-Queued and Layer-dependent Sampling based), which is a learning pipeline structure for a graph neural network (GNN). The graph neural network learning support device (100) can increase GPU utilization by interleaving data access, transmission, and computation and utilizing layer-dependent sampling. Accordingly, the graph neural network learning support device (100) can perform data computations more accurately and quickly compared to existing systems.

[0032] For example, a graph neural network learning support device (100) can perform asynchronous learning to reduce the learning time. The graph neural network learning support device (100) may be composed of a pipeline for asynchronous data movement to maintain mini-batches on both the CPU and the GPU, and a pipeline for maintaining gradients only on the GPU to maintain consistency of the graph neural network model. Here, the mini-batches are buffered in a data buffer and hidden at the time of mini-batch creation, but can be used in the learning pipeline. Additionally, the graph neural network model can be updated by performing asynchronous gradient accumulation on multiple GPUs. In particular, the graph neural network learning support device (100) can accumulate gradients in a gradient buffer and update the graph neural network model as soon as the gradient accumulation for the mini-batches at a specific point in time is finished on each GPU. That is, gradient accumulation on the GPUs is performed asynchronously on each GPU, but synchronization can be achieved by performing the update of the graph neural network model at a specific point in time on each GPU or at the time when the gradient operation of the mini-batches for the feature layer is completed and buffered in the buffer.

[0033] FIG. 2 is an exemplary diagram illustrating the configuration of a graph neural network learning support device according to one embodiment.

[0034] According to one embodiment, the mini-batch generation unit (120) may include a central processing unit (CPU) (121) that samples data received through the interface unit (110) layer by layer to generate two or more mini-batches for parallel processing, and two or more parallel-connected first mini-batch buffers (123) that each store the generated two or more mini-batches.

[0035] For example, given a graph G = (V, E), V = (v1, v2, ...v N ) is the set of nodes and E = (e ij |vi , v j ∈ V) is node v i and v j It can be a set of edges connecting.

[0036] The mini-batch generation unit (120) can continuously and simultaneously generate mini-batches and insert them into the first mini-batch buffer (123). The mini-batches generated in this way can be transmitted to the neural network learning unit (130) through the first mini-batch buffer (123). When the mini-batches arrive at the neural network learning unit (130), the mini-batches can be extracted for training a graph neural network model. For example, the neural network learning unit (130) can optimize GPU utilization by continuously using a queue of new mini-batches that are continuously flowing in. Accordingly, continuous gradient operations to the next mini-batch can be performed without considering intermediate gradient synchronization with the GPUs of other neural network learning units. Subsequently, when all gradients are ready, accumulation occurs, and model updates can be performed on all GPUs.

[0037] According to one example, the mini-batch generator (120) receives a dataset and can generate mini-batches and store them in the first mini-batch buffer (123). Nodes sampled from a specific layer to the next layer can be represented as a union of neighbors as follows.

[0038] [Mathematical Formula 1]

[0039]

[0041] Here, is a node sampled from layer l-1, and node v i It is the set of neighbor nodes of. S l is a set of nodes sampled from the l-th layer, and is v i It is the set of neighbors of. During the sampling process, probability assignment is V, the nodes designated by l-1It is limited to. In particular, important probabilities can be defined as follows.

[0042] [Mathematical Formula 2]

[0043]

[0045] Here, the set of nodes sampled from the l-th and l-1-th layers can be determined. And for each node v i probability can be assigned. That diagonal matrix It can be defined as follows.

[0046] [Mathematical Formula 3]

[0047]

[0049] Here, s l-1 is the number of nodes sampled from layer l-1.

[0050] For example, the minibatch generator (120) uses a basic sampling method because there is no information about intermediate embeddings or activation matrices while defining samples in the l-th layer, which is a row selection matrix R l and may depend on the normalized Laplacian matrix L. Here, the row selection matrix R l It can be defined as follows.

[0051] [Mathematical Formula 4]

[0052]

[0054] Here, and is a node set S l and S l-1 It means that. The Laplacian matrix L can be defined as follows.

[0055] [Mathematical Formula 5]

[0056]

[0058] At this time, In this, A is an adjacency matrix, and Is It is a diagonal matrix satisfying . The sampled mini-batch consists of the Laplacian adjacency matrix and the accompanying embeddings (or feature values ​​from Layer 0) and It can be represented as. is a normalized Laplacian matrix to prevent gradient vanishing or explosion.

[0059] According to one embodiment, the neural network learning unit (130) may include a second mini-batch buffer (131) that receives a mini-batch from any one of two or more first mini-batch buffers (123) connected in parallel, a Graphics Processing Unit (GPU) (133) that calculates a gradient for training a graph neural network based on the mini-batch stored in the second mini-batch buffer, and a gradient buffer (135) that stores the calculated gradient.

[0060] For example, the number of neural network learning units (130) may be equal to the number of first mini-batch buffers (123). For example, as shown in FIG. 2, each first mini-batch buffer (123) may be connected to one neural network learning unit (130-1, ..., 130-n). At this time, the neural network learning units (130-1, ..., 130-n) may be connected in parallel with each other for parallel operation.

[0061] According to one example, the generated mini-batches are transferred to a second mini-batch buffer (131) of the neural network learning unit (130). This optimizes the data transfer mechanism to minimize waiting time and allows the GPU to access the mini-batches quickly and efficiently.

[0062] According to one example, the GPU (133) of the neural network learning unit (130) can generate a gradient by computing the input mini-batch. After generating the mini-batch and adding it to a queue, the mini-batch can be used to train the graph neural network as follows.

[0063] [Mathematical Formula 6]

[0064]

[0065]

[0067] Here, H l (v) is the feature set of neighbors of node v sampled in the l-th layer. H l+1 (v) is the embedding value at the (l+1)th layer of v. This is formed by accumulating neighbor information for the GraphSAGE model, which uses a neighbor aggregation method, and then concatenating the embedding values ​​of node v in layer l. In the GCN model, the above Z l+1 The neighbor accumulation of (v) can be replaced as follows.

[0068] [Mathematical Formula 7]

[0069]

[0071] According to one example, multiple neural network learning units (130-1, ..., 130-n) can store gradients in a gradient buffer (135) instead of immediately synchronizing gradients with other neural network learning units. This allows the graph neural network training process to smoothly continue to the next batch of data, thereby increasing the utilization of the GPU. In other words, it is possible to ensure that computational efficiency is not lost during the training process by moving on to the computation of a mini-batch corresponding to the next layer without waiting while the GPU of another neural network learning unit is performing computations.

[0072] According to one example, the GPU in the neural network training unit performs forward propagation, and at this time, the loss value s and the corresponding gradient It can be calculated as follows and input into the gradient buffer (135).

[0073] [Mathematical Formula 8]

[0074]

[0075]

[0077] Here, y represents the actual label value and pr represents the predicted value.

[0078] According to one embodiment, the neural network learning unit (130) checks the gradient buffer of another neural network learning unit connected in parallel, and if the gradient for the same layer is stored in the gradient buffer of all two or more neural network learning units, the graph neural network can be updated by synchronizing the gradients. At this time, each graphics processing unit included in the neural network learning unit can calculate the gradient asynchronously and store it in the gradient buffer, regardless of the operation of the graphics processing unit of the other neural network learning unit.

[0079] Referring to FIG. 3, each GPU of the neural network learning unit (130) can compute a gradient and store it in a gradient buffer (135). At this time, since the computation speed or time may differ in each GPU, the gradients stored in the gradient buffer may not be synchronized. For example, the GPU may sequentially input the gradient computed by the mini-batch of a specific layer into the gradient buffer. For example, the gradient computed by the mini-batch of a specific layer may be input into the gradient buffer (135-1). At this time, if the computation of the neural network learning unit (130-2) is delayed, the gradient computed by the mini-batch of a specific layer may not be input into the gradient buffer (135-1) of the neural network learning unit (130-2). In such a case, other neural network learning units may wait without training the graph neural network model. On the other hand, another neural network learning unit can calculate the gradient by calculating the next step's mini-batch regardless of the computational delay of the neural network learning unit (130-2).

[0080] According to one example, a neural network training unit can perform gradient synchronization and accumulation between GPUs. For instance, a neural network training unit can synchronize gradients obtained from the GPUs of other neural network training units. Synchronized gradient updates can be performed synchronously after gradient operations have finished on all GPUs. In other words, gradient operations are performed asynchronously and stored in a gradient buffer, but gradient updates can be performed synchronously.

[0081] For example, to synchronize gradients, the neural network learning unit can accumulate all gradients as follows.

[0082] [Mathematical Formula 9]

[0083]

[0085] Here, is the accumulated gradient from Layer 1, and is a predetermined mini-batch of GPU d i It is the gradient at.

[0086] According to one example, the neural network learning unit can update the graph neural network model on each GPU based on accumulated gradients. For example, the neural network learning unit can update the graph neural network model as shown in the mathematical equation below.

[0087] [Mathematical Formula 10]

[0088]

[0090] Here, W l is the model's weight in layer l. is the learning rate.

[0091] FIG. 3 is a flowchart illustrating a graph neural network learning support method according to one embodiment.

[0092] According to one embodiment, a graph neural network learning support device may be a computing device having one or more processors and a memory that stores one or more programs executed by one or more processors.

[0093] According to one embodiment, a graph neural network learning support device can generate a mini-batch for parallel processing based on data received through an interface (310), and can update the graph neural network by calculating a gradient based on the mini-batch (320).

[0094] According to one example, a graph neural network learning support device may utilize a minibatch generator comprising a central processing unit (CPU) that samples data received through an interface layer by layer to generate two or more minibatches for parallel processing in the step of generating minibatches, and two or more parallel-connected first minibatch buffers that each store the generated two or more minibatches. Additionally, in the step of updating a graph neural network, the graph neural network learning support device may utilize two or more neural network learners comprising a second minibatch buffer that receives a minibatch from any one of two or more parallel-connected first minibatch buffers, a Graphics Processing Unit (GPU) that computes a gradient to train the graph neural network based on the minibatch stored in the second minibatch buffer, and a gradient buffer that stores the computed gradient.

[0095] According to one embodiment, a graph neural network learning support device checks the gradient buffers of other neural network learning units connected in parallel during the step of updating a graph neural network, and if gradients for the same layer are stored in the gradient buffers of two or more neural network learning units, the graph neural network can be updated by synchronizing the gradients. At this time, each graphics processing unit included in the neural network learner can asynchronously calculate the gradients and store them in the gradient buffers, regardless of the computational operation of the graphics processing units of other neural network learners.

[0096] Among the embodiments of FIG. 3, embodiments that overlap with the descriptions made with reference to FIG. 1 and FIG. 2 have been omitted.

[0097] FIG. 4 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0098] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a video representation learning device.

[0099] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). One or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0100] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0101] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0102] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0103] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0105] 100: Graph Neural Network Learning Support Device 110: Interface section 120: Minibatch creation section 121: Central Processing Unit 123: 1st minibatch buffer 130: Neural network learning unit 131: 2nd minibatch buffer 133: Graphics Processing Unit 135: Gradient buffer

Claims

Claim 1 An interface unit for data input / output; a mini-batch generation unit that generates a mini-batch for parallel processing based on data received through the interface unit; and two or more neural network learning units that update a graph neural network by calculating a gradient based on the mini-batch, wherein the mini-batch generation unit generates two or more mini-batches including a set of sampling nodes per layer by determining a set of sampling nodes of the next layer based on a set of neighbor nodes of nodes sampled in the previous layer for a plurality of nodes included in the data received through the interface unit; a central processing unit (CPU) that generates two or more mini-batches including a set of sampling nodes per layer; and two or more parallel-connected first mini-batch buffers that each store the generated two or more mini-batches, and the neural network learning unit includes a second mini-batch buffer that receives a mini-batch from any one of the two or more parallel-connected first mini-batch buffers; and a Graphics Processing Unit (GPU) that calculates a gradient for training a graph neural network based on the mini-batch stored in the second mini-batch buffer. A graph neural network learning support device comprising a gradient buffer for storing the calculated gradient, wherein the neural network learning unit checks the gradient buffer of another neural network learning unit connected in parallel, and updates the graph neural network by synchronizing the gradient only when the gradients for the same layer are all stored in the gradient buffers of two or more neural network learning units, and each graphics processing unit included in the neural network learning unit asynchronously calculates the gradient and stores it in the gradient buffer regardless of the operation of the graphics processing unit of the other neural network learning unit. Claim 2 delete Claim 3 delete Claim 4 A graph neural network learning support device according to claim 1, wherein the number of neural network learning units is equal to the number of first mini-batch buffers. Claim 5 A graph neural network learning support device according to claim 1, wherein the two or more neural network learning units are connected in parallel with each other for parallel computation. Claim 6 delete Claim 7 delete Claim 8 A method performed in a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising the step of generating a minibatch for parallel processing based on data received through an interface; The method includes the step of updating a graph neural network by calculating a gradient based on the mini-batch, wherein the step of generating the mini-batch utilizes a mini-batch generator comprising a central processing unit (CPU) that generates two or more mini-batches containing a set of sampling nodes per layer by determining a set of sampling nodes of the next layer based on a set of neighbor nodes of nodes sampled in the previous layer for a plurality of nodes included in data received through an interface, and two or more parallel-connected first mini-batch buffers that each store the generated two or more mini-batches; the step of updating the graph neural network utilizes two or more neural network learners comprising a second mini-batch buffer that receives a mini-batch from any one of the two or more parallel-connected first mini-batch buffers, a Graphics Processing Unit (GPU) that calculates a gradient for training the graph neural network based on the mini-batch stored in the second mini-batch buffer, and a gradient buffer that stores the calculated gradient; and the step of updating the graph neural network utilizes a gradient buffer of another parallel-connected neural network learning unit A method for supporting graph neural network learning, wherein, upon verification, the gradients for the same layer are all stored in the gradient buffers of two or more neural network learning units, the gradients are synchronized to update the graph neural network, and each graphics processing unit included in the neural network learning unit calculates the gradients asynchronously and stores them in the gradient buffers, regardless of the operation of the graphics processing unit of the other neural network learning unit. Claim 9 delete Claim 10 delete Claim 11 A graph neural network learning support method according to claim 8, wherein the number of neural network learners is equal to the number of the first mini-batch buffer. Claim 12 A graph neural network learning support method according to claim 8, wherein two or more neural network learners are connected in parallel for parallel computation. Claim 13 delete Claim 14 delete Claim 15 A computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises one or more instructions, and when the instructions are executed by a computing device having one or more processors, the computing device generates a minibatch for parallel processing based on data received through an interface;and perform the step of updating a graph neural network by calculating a gradient based on the mini-batch, wherein the step of generating the mini-batch utilizes a mini-batch generator comprising a central processing unit (CPU) that generates two or more mini-batches containing a set of sampling nodes per layer by determining a set of sampling nodes of the next layer based on a set of neighbor nodes of nodes sampled in the previous layer for a plurality of nodes included in data received through an interface, and two or more parallel-connected first mini-batch buffers that each store the generated two or more mini-batches, and the step of updating the graph neural network utilizes two or more neural network learners comprising a second mini-batch buffer that receives a mini-batch from any one of the two or more parallel-connected first mini-batch buffers, a Graphics Processing Unit (GPU) that calculates a gradient for training a graph neural network based on the mini-batch stored in the second mini-batch buffer, and a gradient buffer that stores the calculated gradient, and the step of updating the graph neural network utilizes a gradient buffer of another parallel-connected neural network learner, and two A computer program that updates a graph neural network by synchronizing gradients only when all gradients for the same layer are stored in the gradient buffers of the entire neural network learning unit, and wherein each graphics processing unit included in the neural network learning unit asynchronously calculates gradients and stores them in gradient buffers independently of the computational operations of the graphics processing units of other neural network learning units.

Images