A device for Computing Gradient-based Metric to Enable Adaptive Batching in Hybrid Parallelism and method for controlling the same
The gradient-based index calculation device addresses the challenge of calculating gradient similarity in hybrid parallelization by performing partial calculations and collecting scalar values, enhancing efficiency and reducing overhead in neural network training.
Patent Information
- Application Number
- KR1020250114452
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-08-18
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2045-08-18
AI Technical Summary
Conventional gradient similarity measurement methods are limited to data parallelization and cannot be applied in hybrid parallelization environments where various parallelization methods are mixed, leading to challenges in calculating gradient similarity in neural network training.
A gradient-based index calculation device utilizing an adaptive batch technique that performs partial calculations on GPUs, including inner product and squares, and collects scalar values to calculate cosine similarity, reducing memory overhead and preserving parallel processing structure.
Enables efficient calculation of gradient-based indices in hybrid parallelization environments, improving user convenience and reducing memory overhead and communication costs.
Smart Images

Figure 112025093984396-PAT00020_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a gradient-based index calculation device. More specifically, it relates to a gradient-based index calculation device for utilizing an adaptive batch technique in a hybrid parallelization environment and a method for controlling the same. Background Technology
[0002] With the current advancement of artificial intelligence technology, parallel processing technology for training Deep Neural Networks (DNNs) is receiving significant attention.
[0003] Multiple GPUs must be used for training large-scale Deep Neural Networks (DNNs). Data parallelism is commonly used. Data parallelism replicates the same model on each GPU, divides the data for training, and then aggregates (or sums) the resulting gradients across the GPUs. Gradient similarity metrics are calculated periodically to ensure training stability.
[0004] In conventional technology, a gradient-based metric called Gradient Similarity is utilized to measure it at regular intervals, and the batch size is increased or decreased based on a predetermined threshold value. This measurement of Gradient Similarity is performed through the following two steps, based on the premise of data parallelization.
[0005] First, the entire GPUs used for training are divided into two groups, and the gradients of each group are aggregated through an All-reduce operation within each group.
[0006] Second, the learning stability is evaluated by calculating the cosine similarity between the aggregated gradients from the two groups.
[0007] However, conventional gradient similarity measurement methods are limited to data parallelization, and in hybrid parallelization where various parallelization methods are mixed, the gradients of the entire neural network model do not exist on each GPU, so there was a problem in that gradient similarity could not be calculated in the same way. Prior art literature
[0008] Registered Patent Publication No. 10-2257530 B1 (Publication Date: May 24, 2021) The problem to be solved
[0009] The embodiments disclosed in this disclosure aim to provide a new gradient-based index calculation device applicable even in a hybrid parallelization environment.
[0010] The embodiment disclosed in this disclosure aims to provide a gradient-based indicator calculation device that applies Compute-and-Gather technology, a new method that changes the order of calculations to efficiently calculate gradient-based indicators even in a hybrid parallelization environment.
[0011] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem
[0012] A gradient-based index calculation device for utilizing an adaptive batch technique in a hybrid parallelization environment according to the present disclosure includes: a memory storing at least one process for performing an adaptive batch technique; and a processor performing an operation according to said process, wherein the processor controls the performance of a specific inner product operation on the gradient of a model layer allocated in a plurality of GPUs, sums the results of said operation by a specific operation within the group to generate a scalar value C (numerator) summed between a plurality of groups including a first group and a second group, calculates the squared value of said gradient, combines the results of said calculation into a single value through a specific operation to calculate a sum of squares for normalization for each group using said calculation results to generate scalar values A and B in each group, collects said scalar values A, B and C on a representative GPU of any one of said plurality of groups, and calculates cosine similarity.
[0013] In addition, the processor calculates the cosine similarity using the following mathematical formula.
[0014]
[0015] In addition, the processor calculates the cosine similarity using the following mathematical formula.
[0016]
[0017] Here, G1 and G2 represent aggregated gradients within two groups, and the above G1 and G2 are implemented by the following mathematical formula.
[0018]
[0019] The above n represents the total number of stages of pipeline parallelization.
[0020] In addition, the processor calculates the cosine similarity in a way that reduces memory overhead by transmitting the calculated scalar values A, B, and C instead of transmitting the entire gradient from each GPU.
[0021] In addition, the processor calculates the cosine similarity in a manner that preserves the gradient indicator definition while maintaining a parallel processing structure.
[0022] In addition, the processor performs partial calculations including gradient inner product and squares on the plurality of GPUs, and collects the scalar values resulting from the calculations to calculate the cosine similarity.
[0023] In addition, the processor divides the layers of the DNN model into n stages and distributes the n stages across multiple GPUs.
[0024] In addition, the processor performs training by utilizing an adaptive batching technique, which involves training with data of a predetermined batch size during the initial stages of training and then gradually expanding the batch size.
[0025] In addition, the aforementioned specific operation refers to an all-reduce sum operation in which values calculated by multiple GPUs are combined into a single sum, and the sum result is shared among multiple GPUs.
[0026] A gradient-based index calculation method for utilizing an adaptive placement technique in a hybrid parallelization environment, performed by a processor of a device according to the present disclosure, comprises: a step of controlling the execution of a specific inner product operation on the gradients of model layers allocated in a plurality of GPUs; a step of generating a scalar value C (numerator) summed between a plurality of groups including a first group and a second group by summing the results of the operation with a specific operation within the group; a step of calculating the squared value of the gradient; a step of generating scalar values A and B in each group by combining them into a single value through a specific operation to calculate a sum of squares for normalization for each group using the results of the calculation; and a step of collecting the scalar values A, B, and C on a representative GPU of any one of the plurality of groups and calculating cosine similarity.
[0027] In addition, it further includes the step of calculating the above cosine similarity using the following mathematical formula.
[0028]
[0029] In addition, it includes a step of calculating cosine similarity using the following mathematical formula.
[0030]
[0031] Here, G1 and G2 represent the aggregated gradients within the two groups, and
[0032] The above G1 and G2 are implemented by the following mathematical formulas.
[0033]
[0034] The above n represents the total number of stages of pipeline parallelization.
[0035] In addition, it further includes the step of calculating the cosine similarity in a way that reduces memory overhead by transmitting the calculated scalar values A, B, and C without transmitting the entire gradient from each of the above GPUs.
[0036] In addition, it further includes a step of calculating the cosine similarity in a manner that preserves the gradient indicator definition while maintaining a parallel processing structure.
[0037] Additionally, the method further includes the step of performing partial calculations including gradient inner product and squares on the plurality of GPUs; and the step of collecting scalar values that are the results of the calculations to calculate the cosine similarity.
[0038] Additionally, the method further includes the step of dividing the layers of the DNN model into n stages; and the step of distributing and arranging the n stages across a plurality of GPUs.
[0039] In addition, it includes a step in which training is conducted using an adaptive batching technique, which involves training with data of a fixed batch size during the initial stages and then gradually expanding the batch size.
[0040] In addition, the aforementioned specific operation refers to an all-reduce sum operation in which values calculated by multiple GPUs are combined into a single sum, and the sum result is shared among multiple GPUs.
[0041] In addition to this, a computer program stored on a computer-readable recording medium may be further provided to carry out a method for implementing the present disclosure.
[0042] In addition, a computer-readable recording medium for recording a computer program for executing a method for implementing the present disclosure may be further provided. Effects of the invention
[0043] According to the present invention, gradient-based indices can be calculated even in a hybrid parallelization environment where various parallel processing techniques are mixed, thereby overcoming the limitations of existing data parallel-based adaptive batching techniques and improving user convenience.
[0044] According to the present invention, the calculation of gradient-based indices is possible even in a hybrid parallelization environment where various parallel processing techniques are mixed, and thus the application of adaptive batching techniques is possible even in such an environment, thereby improving user convenience.
[0045] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing
[0046] FIG. 1 is a configuration diagram of a gradient-based index calculation device according to the present disclosure. FIG. 2 is a flowchart illustrating a gradient-based index calculation method according to the present disclosure. FIG. 3 is a diagram illustrating an embodiment showing gradient similarity for data parallelism (DP) in the prior art according to the present disclosure. FIG. 4 is a diagram illustrating an example of a gradient-based index calculation method of the present invention according to the present disclosure. Specific details for implementing the invention
[0047] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.
[0048] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.
[0049] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0050] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.
[0051] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.
[0052] Singular expressions include plural expressions unless there is an obvious exception in the context.
[0053] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.
[0054] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.
[0055] In this specification, the present invention may be implemented not only as a server system but also as various devices capable of performing computational processing and providing results to a user. For example, the present invention may include a computer, a server device, and a portable terminal, or may take the form of any one of them.
[0056] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.
[0057] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.
[0058] The above portable terminal may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).
[0059] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.
[0060] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using a number of learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.
[0061] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a Deep Neural Network (DNN), such as a Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the examples mentioned above.
[0062] The processor can create a neural network, train (or learn) the neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain the neural network.
[0063] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), ESN (Echo It will be understood by a person skilled in the art that any neural network may be included, but is not limited to, State Network, Deep Residual Network, Differential Neural Computer, Neural Turning Machine, Capsule Network, Kohonen Network, and Attention Network.
[0064] According to an exemplary embodiment of the present disclosure, the processor comprises a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4 for Natural Language Processing, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization for ResNet Data Intelligence, Various artificial intelligence structures and algorithms, such as recommendation and data creation, may be used, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.
[0065] FIG. 1 is a configuration diagram of a gradient-based index calculation device according to the present disclosure.
[0066] Referring to FIG. 1, the gradient-based index calculation device (100) includes an input module (110), a sensor module (120), a processor (130), a display module (140), a memory (150), a communication module (160), and a camera module (170).
[0067] The input module (110) collects gradient information of the model layer.
[0068] The input module (110) automatically collects gradient information of the model layer assigned from multiple GPUs.
[0069] The sensor module (120) senses gradient information.
[0070] The processor (130) performs a gradient-based indicator calculation method according to the process.
[0071] The processor (130) controls the performance of a specific inner product operation on the gradients of the model layers assigned to multiple GPUs, sums the results of the operation by a specific operation within the group to generate a scalar value C (numerator) summed between multiple groups including a first group and a second group, calculates the squared value of the gradient, and uses the results of the calculation to generate scalar values A and B in each group by combining them into one value through a specific operation to calculate a sum of squares for normalization for each group, collects the scalar values A, B, and C in a representative GPU of one of the multiple groups, and calculates the cosine similarity.
[0072] Here, a specific operation refers to an all-reduce sum operation in which values calculated by multiple GPUs are combined into a single sum, and the sum result is shared among multiple GPUs.
[0073] A detailed explanation of this is shown in Fig. 4.
[0074] The processor (130) calculates the cosine similarity using the following mathematical formula.
[0075]
[0076] A detailed explanation of this is shown in Fig. 4.
[0077] The processor (130) calculates the cosine similarity using the following mathematical formula.
[0078]
[0079] Here, G1 and G2 represent aggregated gradients within two groups, and the above G1 and G2 are implemented by the following mathematical formula.
[0080]
[0081] The above n represents the total number of stages of pipeline parallelization.
[0082] A detailed explanation of this is shown in Fig. 4.
[0083] The processor (130) calculates the cosine similarity in a way that reduces memory overhead by transmitting the calculated scalar values A, B, and C without transmitting the entire gradient from each GPU.
[0084] The processor (130) calculates the cosine similarity in a way that preserves the gradient indicator definition while maintaining a parallel processing structure.
[0085] The processor (130) performs partial calculations including gradient inner product and squares on the plurality of GPUs, and calculates the cosine similarity by collecting the scalar values that are the results of the calculations.
[0086] The processor (130) divides the layers of the DNN model into n stages and distributes the n stages among multiple GPUs.
[0087] A detailed explanation of this is shown in Fig. 4.
[0088] The processor (130) performs training using an adaptive batching technique in which it performs training with data of a predetermined batch size at the beginning of the training and then gradually expands the batch size.
[0089] A detailed explanation of this is shown in Fig. 4.
[0090] The display module (140) displays a graphic image according to a control command from the processor (130).
[0091] The memory (150) stores at least one process for performing operations and stores user input and data.
[0092] The communication module (160) transmits and receives data with an external device (200).
[0093] The communication module (160) can receive data from an external device (200).
[0094] Here, the external device (200) includes an external device such as a smartphone, PC, laptop, tablet PC, etc. The external device (200) includes an external mobile device (200) that includes a memory (260) for storing data.
[0095] The camera module (170) captures an image of the front.
[0096] The camera module (170) photographs a subject in front according to a control command from the processor (130).
[0097] However, the components illustrated in FIG. 1 are not essential for implementing the present invention according to the present disclosure, so the present invention described herein may have more or fewer components than the components listed above.
[0098] The communication module (160) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast receiving module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.
[0099] The input module (110) is for inputting video information (or signal), audio information (or signal), data, or information input from a user, and may include at least one of at least one camera, at least one microphone, and a user input unit. Voice data or image data collected by the input module (110) may be analyzed and processed into a user control command.
[0100] The display module (140) displays (outputs) information processed in the present invention. For example, the present invention may display execution screen information of a running application (e.g., an application), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.
[0101] The memory (150) can store data supporting various functions of the present invention and a program for the operation of the control unit, and can store input / output data (e.g., music files, still images, video, etc.), and can store a plurality of application programs (or applications) running in the present invention, data for the operation of the device, and commands. At least some of these application programs can be downloaded from an external server via wireless communication.
[0102] Such memory (150) may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory (150) may be a database connected via wired or wireless connection, although separate from the present invention, and may be implemented as a database system.
[0103] The processor (130) may be implemented with at least one core, a memory that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of the components within the present invention, and at least one processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.
[0104] In addition, the processor (130) can control one or a combination of the components described above in order to implement various embodiments according to the present disclosure described in FIGS. 2 to 4 below.
[0105] At least one component may be added or removed in response to the performance of the components illustrated in FIG. 1. Additionally, it will be readily understood by those skilled in the art that the relative positions of the components may be changed in response to the performance or structure of the system.
[0106] Meanwhile, each component illustrated in Fig. 1 refers to a software and / or hardware component such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).
[0107] Referring to FIG. 2, the processor (130) controls the performance of a specific inner product operation on the gradients of the model layers assigned to a plurality of GPUs (S210).
[0108] The processor (130) sums the above operation results by a specific operation within the group to generate a scalar value C (numerator) summed between a plurality of groups including a first group and a second group (S220).
[0109] The processor (130) calculates the squared value of the gradient (S230).
[0110] The processor (130) uses the above calculation results to calculate the sum of squares for normalization for each group by combining them into a single value through a specific operation to generate scalar values A and B for each group (S240).
[0111] The processor (130) collects the scalar values A, B, and C on a representative GPU of any one of the plurality of groups and calculates the cosine similarity (S250).
[0112] FIG. 3 is a diagram illustrating an embodiment showing gradient similarity for data parallelism (DP) in the prior art according to the present disclosure.
[0113] Referring to Fig. 3 (310), the process of calculating gradient similarity in data parallelism (DP) is explained.
[0114] Multiple GPUs (workers) replicate the same model and train by dividing different mini-batch data.
[0115] To calculate gradient similarity, follow the process below.
[0116] First, divide the GPUs into two groups, and aggregate the gradients within each group using an All-reduce operation.
[0117] Second, calculate the cosine similarity between the aggregated gradient vectors of the two groups.
[0118] The similarity indicator obtained in this way is used to determine learning stability or adaptive batching.
[0119] Next, we will explain the reasons why problems arise in hybrid parallelism.
[0120] In the case of hybrid parallelism techniques (e.g., using both pipeline parallelism and data parallelism), each GPU does not hold the entire model's gradients. That is, some GPUs handle only specific layers of the model, or the entire model gradient is not processed at once by being divided into pipeline stages.
[0121] Therefore, under the existing method—that is, the premise that all GPUs share the same total gradient—a problem arises in that gradient similarity metrics cannot be defined or calculated.
[0122] FIG. 4 is a diagram illustrating an example of a gradient-based index calculation method of the present invention according to the present disclosure.
[0123] Referring to Fig. 4 (410), a gradient-based indicator calculation method will be explained.
[0124] The present invention is performed by a processor (130).
[0125] The present invention performs Compute-and-Gather technology, a new method that changes the order of calculations, in order to efficiently calculate gradient-based indicators even in a hybrid parallelization environment.
[0126] The present invention first performs partial calculations such as gradient dot product and squares on each GPU, collects the scalar values resulting from the calculations, and calculates the final cosine similarity value.
[0127] As illustrated in Fig. 4 (410), since it consists of pipeline parallelization with 4 stages and 2 data parallelization groups, n=4 in the above formula.
[0128] As illustrated in FIG. 4 (410), the process of the present invention is configured as follows.
[0129] The processor (130) controls the performance of a specific inner product operation on the gradients of the model layers assigned to multiple GPUs (S10).
[0130] In multiple GPUs, each GPU multiplies its own share of layer gradients and sums them (dot product). This allows directional information to be summarized without collecting all gradient vectors.
[0131] The processor (130) sums the above operation results by a specific operation within the group to generate a scalar value C (numerator) summed between a plurality of groups including a first group and a second group (S20).
[0132] The GPUs sum the inner product values calculated individually across the entire group and generate a single scalar value (C) that can be compared between the two groups.
[0133] The processor (130) calculates the squared value of the gradient (S30).
[0134] Square each item to find the magnitude (norm) of the slope.
[0135] The processor (130) uses the above calculation results to calculate the sum of squares for normalization for each group by combining them into a single value through a specific operation to generate scalar values A and B for each group (S40).
[0136] Each group combines the squared value of its slope and the squared result to generate scalar values A and B.
[0137] The processor (130) collects the scalar values A, B, and C on a representative GPU of any one of the plurality of groups and calculates the cosine similarity (S50).
[0138] Here, since only summarized scalar values (inner product, sum of squares) are exchanged without transmitting multidimensional gradient tensors, there is an advantage in minimizing communication volume and memory usage.
[0139] Here, the specific operation refers to an all-reduce sum operation in which values calculated by multiple GPUs are combined into a single sum, and the sum result is shared among the multiple GPUs.
[0140] In the case of conventional technology, the gradient tensors (very large data) of all GPUs had to be combined to calculate cosine similarity. However, this had the disadvantage of increasing communication load and memory usage.
[0141] According to the present invention, since the same cosine similarity result can be obtained by exchanging only summarized scalar values (A, B, C), there is an advantage of reducing communication load and memory usage.
[0142] The processor (130) calculates the cosine similarity using the following mathematical formula.
[0143]
[0144] Specifically, the processor (130) calculates the cosine similarity using the following mathematical formula.
[0145]
[0146] Here, G1 and G2 represent aggregated gradients within two groups, and the above G1 and G2 are implemented by the following mathematical formula.
[0147]
[0148] Here, n represents the total number of stages of pipeline parallelization.
[0149] To calculate the final terms of mathematical formulas 1 through 3, you must collect the gradients in all stages and then calculate the denominator and numerator terms.
[0150] However, in the present invention, through the Compute-and-Gather technique, there is a function to perform local operations of gradients existing in each GPU and a function to execute the calculation results through an All-reduce operation within the group, so only the scalar values calculated locally are used in the All-reduce operation.
[0151] According to the present invention, the effect of alleviating memory usage and communication volume bottlenecks is exhibited, and the function of calculating gradient-based indicators can be performed even in a hybrid parallelization environment.
[0152] The processor (130) calculates the cosine similarity in a way that reduces memory overhead by transmitting the calculated scalar values A, B, and C without transmitting the entire gradient from each GPU.
[0153] The processor (130) calculates the cosine similarity in a way that preserves the gradient indicator definition while maintaining a parallel processing structure.
[0154] When n = 4, pipeline parallelization includes 4 stages and consists of 2 data parallelization groups.
[0155] The processor (130) performs partial calculations including gradient inner product and squares on the plurality of GPUs, and calculates the cosine similarity by collecting the scalar values that are the results of the calculations.
[0156] An example of dividing the stages is described.
[0157] The processor (130) divides the layers of the DNN model into n stages and distributes the n stages among multiple GPUs.
[0158] An example of an adaptive deployment technique is described.
[0159] The processor (130) performs training using an adaptive batching technique in which it performs training with data of a predetermined batch size at the beginning of the training and then gradually expands the batch size.
[0160] Explain distributed learning.
[0161] Distributed learning refers to a technology that utilizes multiple GPUs to increase the data throughput of Deep Neural Network (DNN) models and shorten training speeds.
[0162] This explains hybrid parallelism.
[0163] Data parallelism is a method in which the same DNN model is replicated on each GPU, and each GPU calculates the gradient for a partitioned dataset, synchronizing the results across all replicas through an All-reduce operation. However, as the size of DNN models is gradually increasing and GPU memory capacity is limited, the entire model cannot be loaded into the memory of a single GPU, so there are limitations to processing using only data parallelism.
[0164] To support large-scale DNN models, parallelization techniques such as pipeline parallelism and tensor parallelism can be used. In pipeline parallelism, the layers of a DNN model are divided into multiple groups, or stages, and distributed across multiple GPUs. The input mini-batch is divided into multiple micro-batches, and each GPU processes the micro-batches in a pipelined manner for its assigned stage.
[0165] The various parallel processing techniques mentioned above can be used individually, but by combining them, large-scale DNN models can be processed more efficiently. In this specification, a technique that integrates these various parallel processing strategies is referred to as Hybrid Parallelism.
[0166] This explains Adaptive Batching.
[0167] Adaptive Batching refers to a technique that dynamically adjusts the batch size according to the situation during the deep learning training process.
[0168] Since the synchronization phase of parallelization techniques for distributed learning can cause a decrease in training speed, a large batch size is intended to allow a large amount of training data to be computed on the GPU before the synchronization phase. However, training with large batch data all at once negatively affects convergence, leading to a decrease in the accuracy of the DNN model.
[0169] To address these issues, an adaptive batching technique is utilized, which starts with small batch sizes of data during the initial stages of training and gradually increases the batch size. As training progresses, the criteria for increasing the batch size are based on a gradient-based metric derived from the model's gradient information.
[0170] According to the present invention, memory overhead is reduced and communication costs are minimized by transmitting only the calculated scalar value instead of transmitting the entire gradient (multidimensional tensor) from each GPU. In addition, the definition of the gradient index can be preserved while maintaining a parallel processing structure.
[0171] The aforementioned program may include code encoded in a computer language such as C, C++, JAVA, or machine language, which can be read by the computer's processor (CPU) through the computer's device interface, in order for the computer to read the program and execute the methods implemented in the program. Such code may include functional code related to functions that define the necessary functions for executing the methods, and may include control code related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. Additionally, such code may further include memory reference code regarding where (address) additional information or media necessary for the computer's processor to execute the functions should be referenced in the computer's internal or external memory. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to transmit or receive during communication.
[0172] The above-mentioned storage medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the above-mentioned storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the above-mentioned program may be stored on various recording media on various servers that the computer can access, or on various recording media on the user's computer. Additionally, the above-mentioned medium may be distributed across networked computer systems, and computer-readable code may be stored in a distributed manner.
[0173] The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, a hard disk, a removable disk, a CD-ROM, or any form of computer-readable recording medium well known in the art to which the present disclosure belongs.
[0174] Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive.
[0175] The various embodiments of the present disclosure are not intended to list all possible combinations but to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.
Claims
Claim 1 A gradient-based index calculation device for utilizing an adaptive batch technique in a hybrid parallelization environment, comprising: a memory for storing at least one process for performing an adaptive batch technique; and a processor for performing an operation according to said process, wherein the processor controls the performance of a specific inner product operation on the gradients of model layers allocated in a plurality of GPUs, sums the results of said specific inner product operation by a specific operation within the group to generate a scalar value C summed between a plurality of groups including a first group and a second group, calculates the squared value of said gradient, combines the results of the calculation of said squared value into a single value through a specific operation to calculate a sum of squares for normalization for each group to generate scalar values A and B in each group, collects said scalar values A, B, and C on a representative GPU of any one of said plurality of groups, and calculates cosine similarity. Claim 2 In claim 1, the processor calculates the cosine similarity using the following mathematical formula, Gradient-based indicator calculation device. Claim 3 In claim 2, the processor calculates the cosine similarity using the following mathematical formula, Here, G1 and G2 represent aggregated gradients within two groups, and the above G1 and G2 are implemented by the following mathematical formula, and A gradient-based index calculation device, wherein n represents the total number of stages of pipeline parallelization. Claim 4 In claim 3, the processor is a gradient-based index calculation device that calculates the cosine similarity in a manner that reduces memory overhead by transmitting the calculated scalar values A, B, and C without transmitting the entire gradient from each GPU. Claim 5 In claim 4, the processor is a gradient-based index calculation device that calculates the cosine similarity in a manner that preserves the gradient index definition while maintaining a parallel processing structure. Claim 6 A gradient-based index calculation device according to claim 1, wherein the processor performs partial calculations including gradient inner products and squares in the plurality of GPUs, and collects scalar values resulting from the partial calculations to calculate the cosine similarity. Claim 7 In claim 1, the processor divides the layers of a DNN model into n stages and distributes the n stages across a plurality of GPUs, a gradient-based index calculation device. Claim 8 A gradient-based index calculation device according to claim 1, wherein the processor performs learning by utilizing an adaptive batching technique in which learning is performed with data of a predetermined batch size at the beginning of learning, and then the batch size is gradually expanded. Claim 9 A gradient-based index calculation device according to claim 1, wherein the specific operation refers to an all-reduce sum operation in which values calculated by a plurality of GPUs are collected and summed into one, and the sum result is shared by the plurality of GPUs. Claim 10 A gradient-based metric calculation method for utilizing an adaptive placement technique in a hybrid parallelization environment, performed by a processor of a device, comprising: a step of controlling the execution of a specific inner product operation on the gradients of model layers allocated on a plurality of GPUs; a step of generating a scalar value C summed between a plurality of groups including a first group and a second group by summing the results of the specific inner product operation with a specific operation within the group; a step of calculating the squared value of the gradient; a step of generating scalar values A and B in each group by combining them into a single value through a specific operation to calculate a sum of squares for normalization for each group using the results of the calculation of the squared value; and a step of collecting the scalar values A, B, and C on a representative GPU of any one of the plurality of groups and calculating cosine similarity. Claim 11 In claim 10, further comprising the step of calculating the cosine similarity using the following mathematical formula, Gradient-based indicator calculation method. Claim 12 In claim 11, further comprising the step of calculating the cosine similarity using the following mathematical formula, Here, G1 and G2 represent aggregated gradients within two groups, and the above G1 and G2 are implemented by the following mathematical formula, and A gradient-based metric calculation method in which n represents the total number of stages of pipeline parallelization. Claim 13 A gradient-based index calculation method according to claim 12, further comprising the step of calculating the cosine similarity in a manner that reduces memory overhead by transmitting the calculated scalar values A, B, and C without transmitting the entire gradient from each GPU. Claim 14 A gradient-based indicator calculation method according to claim 13, further comprising the step of calculating the cosine similarity in a manner that preserves the gradient indicator definition while maintaining a parallel processing structure. Claim 15 A gradient-based index calculation method according to claim 10, further comprising: a step of performing partial calculations including gradient inner products and squares on the plurality of GPUs; and a step of collecting scalar values resulting from the partial calculations to calculate the cosine similarity. Claim 16 A gradient-based index calculation method according to claim 10, further comprising the step of dividing the layers of a DNN model into n stages; and the step of distributing and arranging the n stages across a plurality of GPUs. Claim 17 A gradient-based indicator calculation method according to claim 10, further comprising a step of conducting training using an adaptive batching technique in which training is performed with data of a predetermined batch size during the initial stage of training, and then the batch size is gradually expanded. Claim 18 A gradient-based index calculation method according to claim 10, wherein the specific operation refers to an all-reduce sum operation in which values calculated by a plurality of GPUs are combined and summed into one, and the sum result is shared by the plurality of GPUs.
Citation Information
Patent Citations
Domain generalization method and system through domain gradient averaging in deep learning
KR1020250169832A
Method and apparatus for predicting object properties using similarity based information retrieval and modeling
US20150317344A1
