Method and apparatus for training transformer model based on multi-device

Abstract

In a multi-device-based transformer model training method according to the present invention, the transformer model is configured such that an initial first transformer block and a plurality of second transformer blocks are sequentially connected, and the first transformer block and the second transformer blocks include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), and the training method comprises the step of dividing the MHA and the MLP of the first transformer block and the plurality of second transformer blocks and distributing them to first and second devices; The method includes the step of training the transformer model in an intra-layer parallax manner through parallel computation of the first and second devices, wherein the first transformer block is configured such that the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device are summed and provided to the divided MLP of the first device and the divided MLP of the second device, and the second transformer block is configured such that the input of the second transformer block and the output of the divided MHA of the first device of the first transformer block are summed and provided to the divided MLP of the first device, and the input of the second transformer block and the output of the divided MHA of the second device of the first transformer block are summed and provided to the divided MLP of the second device.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for training a large-scale Transformer model using an intra-layer parallax method with multi-devices. Background Technology

[0002] Recently, Transformer models are evolving into large-scale Transformer models that expand the model size to improve performance. Large-scale Transformer models fundamentally have a structure consisting of sequentially connecting Transformer blocks composed of multi-head attention and multi-layer perceptrons.

[0003] Large-scale transformer models typically have billions of parameters that exceed the memory and computational limits of a single device, making multi-device based training across multiple devices essential.

[0004] Model parallelism is required to split deep learning models and train or infer using multiple devices. Model parallelism is a method of splitting a model and mapping it to each device, and is divided into inter-layer parallelism and intra-layer parallelism.

[0005] Inter-layer parallax, also known as pipeline parallax, is a method of separating a model layer by layer rather than separating the layers. While this can reduce the size of the model mapped to each device and increase computational throughput, it cannot reduce latency for a single request.

[0006] On the other hand, intra-layer parallax is a method of separating each layer. While this can increase throughput and reduce latency, it has the disadvantage that communication between devices is required after computation at each device due to the separation of layers. For example, to compute a multi-layer perceptron using the results of multi-head attention within a single transformer block, communication is required between devices to aggregate and synchronize the multi-head attention results.

[0007] Accordingly, as the number of devices using intra-layer parallax or the size of the data increases, a problem may arise in which the proportion of communication overhead in the training and inference time of deep learning models increases.

[0008] Figure 1 shows an example of the structure of a conventional large-scale transformer model.

[0009] A large-scale transformer model is configured such that multiple transformer blocks (100) are connected sequentially. Each transformer block (100) includes a layer normalization layer (120), a Multi-Head Attention unit (hereinafter MHA) (130) for dependency modeling, a residual connection (140) that sums the input (110) and the output of the MHA (130), a layer normalization layer (150), a Multi-Layer Perceptron unit (hereinafter MLP) (160) that performs a position-wise transformation using the dependency computed by the MHA (130), and a residual connection (170) that sums the result of the residual connection (140) and the output of the MLP (160).

[0010] The MHA (130) is configured such that the query, key, and value layers, the Scaled Dot-Product Attention layer, the concat layer, and the linear layer are connected sequentially. The MHA (130) transforms the input (110) into multiple heads through the query (Q), key (K), and value (V) layers (①), and then each head extracts dependencies between the input information individually through the Scaled Dot-Product Attention layer. The outputs of all heads are connected by the concat layer and merged by the linear layer (②) to form the final output of the MHA (130).

[0011] The output of the MHA (130), including dependencies between input information, is summed with the input (110) and passed through the layer normalization layer (150) to the MLP (160).

[0012] The MLP (160) is configured such that a linear layer, a non-linear layer of GeLU (or ReLU), and a linear layer are connected in sequence. The MLP (160) transforms the output of the MHA (130) to a higher dimension through the linear layer (③), applies a non-linear function of GeLU (or ReLU), and then transforms it back to the original dimension through the linear layer (④).

[0013] Figure 2 shows an example of a structure in which the Transformer blocks are divided into each device via intra-layer parallax for training a large-scale Transformer model.

[0014] In intra-layer parallax, the parameters of the MHA (130) and MLP (160) constituting the Transformer block (100) are divided in a row or column direction and distributed to the first device (GPU1) and the second device (GPU2). Thus, the MHA (130) is divided into MHA (131) and MHA (132), and the MLP (160) is divided into MLP (161) and MLP (162). For convenience, the part corresponding to the layer normalization layer (120, 150) in FIG. 2 has been omitted. Although a GPU is given as an example of a device in FIG. 2, the device may be various devices capable of performing computations for model training, such as CPUs, APs, and APUs, in addition to GPUs.

[0015] Referring to FIG. 2, the input (X) (110), the output (Z1) (131a) of the divided MHA (131) of GPU1, and the output (Z2) (132a) of the divided MHA (132) of GPU2 are summed and provided to the divided MLP (161) of GPU1 and the divided MLP (162) of GPU2.

[0016] The input X (110) is first broadcast to the devices (GPU1, GPU2) and then provided to each MHA (131, 132). The parameters of the query (Q), key (K), and value (V) layers are divided in the column direction of the matrix (①). Since each head calculates the attention output independently, no connection is needed between the MHAs (131, 132), and thus communication between the MHAs (131, 132) is prevented. The second linear layer (②) is divided in the row direction, which eliminates the need for data transmission and thus avoids communication between the MHAs (131, 132).

[0017] However, in order to pass the output of MHA (131, 132) to the next MLP (161, 162), it must be reconstructed into a complete matrix, so the output of MHA (131, 132) must be summed. This requires all-reduce communication between GPUs (similar to broadcasting after aggregation between consecutive transformer blocks) to sum the output of MHA (131, 132) and sum it with the input (X) (110) (see residual linkage (140) in FIG. 1) and then pass it to MLP (161, 162). All-reduce communication refers to communication to transmit the results computed on each GPU to other GPUs to synchronize the results and prepare for the next operation.

[0018] The two linear layers of the MLP (160) are also divided in the same way as the MHA (130), with the first linear layer in the column direction (③) and the second linear layer in the row direction (④). This division also requires additional data communication between GPUs (aggregated and then broadcast) to deliver the complete output to the next transformer block. The problem to be solved

[0019] The problem that the present invention aims to solve is to provide a multi-device-based transformer model training method and apparatus capable of reducing communication overhead by eliminating inter-device communication required to aggregate multi-head attention results between devices within a transformer block and transmit them to a multi-layer perceptron.

[0020] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0021] In a multi-device-based transformer model training method according to the present invention for solving the above technical problem, the transformer model is configured such that an initial first transformer block and a plurality of second transformer blocks are sequentially connected, and the first transformer block and the second transformer blocks include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), and the training method comprises the step of dividing the MHA and the MLP of the first transformer block and the plurality of second transformer blocks and distributing them to first and second devices; The method includes the step of training the transformer model in an intra-layer parallax manner through parallel computation of the first and second devices, wherein the first transformer block is configured such that the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device are summed and provided to the divided MLP of the first device and the divided MLP of the second device, and the second transformer block is configured such that the input of the second transformer block and the output of the divided MHA of the first device of the first transformer block are summed and provided to the divided MLP of the first device, and the input of the second transformer block and the output of the divided MHA of the second device of the first transformer block are summed and provided to the divided MLP of the second device.

[0022] A computer program according to the present invention for solving the above technical problem is stored on a computer-readable storage medium and executes the above-described multi-device-based transformer model training method on a computer.

[0023] A multi-device-based transformer model training device according to the present invention for solving the above technical problem comprises: a memory storing one or more programs for training a transformer model using a multi-device in an intra-layer parallax manner; and includes one or more processors that perform operations to train the transformer model in an intra-layer parallax manner using multi-devices according to one or more programs stored in the memory, wherein the transformer model is configured such that an initial first transformer block and a plurality of second transformer blocks are sequentially connected, and the first transformer block and the second transformer blocks include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), and the processor divides the MHA and the MLP of the first transformer block and the plurality of second transformer blocks and distributes them to first and second devices, and trains the transformer model in an intra-layer parallax manner through parallel operations of the first and second devices, and the first transformer block includes the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device It is configured to be summed and provided to the divided MLP of the first device and the divided MLP of the second device, and the second transformer block is configured such that the input of the second transformer block and the output of the divided MHA of the first device of the first transformer block are summed and provided to the divided MLP of the first device, and the input of the second transformer block and the output of the divided MHA of the second device of the first transformer block are summed and provided to the divided MLP of the second device.

[0024] A multi-device-based transformer model training device according to the present invention for solving the above technical problem comprises: a first device; The transformer model includes a first transformer block and a plurality of second transformer blocks connected sequentially, and the first transformer block and the second transformer block include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), and the MHA and MLP of the first transformer block and the plurality of second transformer blocks are divided and distributed to the first and second devices, and the transformer model is trained in an intra-layer parallax manner through parallel computation of the first and second devices, and the first transformer block is configured such that the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device are summed and provided to the divided MLP of the first device and the divided MLP of the second device, and the second transformer block is configured such that the second transformer block's The input and the output of the divided MHA of the first device of the first transformer block are summed and provided to the divided MLP of the first device, and the input of the second transformer block and the output of the divided MHA of the second device of the first transformer block are summed and provided to the divided MLP of the second device.

[0025] The second transformer block may be configured such that the output of the divided MHA of the first device of the second transformer block and the output of the divided MLP of the first device are summed, and the output of the divided MHA of the second device of the second transformer block and the output of the divided MLP of the second device are summed.

[0026] The above transformer model may be configured such that the sum of the output of the divided MHA of the first device and the output of the divided MLP of the first device, the sum of the output of the divided MHA of the second device and the output of the divided MLP of the second device, and the input of the second transformer block are summed and provided as the input of the next second transformer block.

[0027] The above MHA can be configured such that the query, key, value layer, Scaled Dot-Product Attention layer, concat layer, and linear layer are sequentially connected.

[0028] The above MLP can be configured such that a linear layer, a non-linear layer, and a linear layer are sequentially connected. Effects of the invention

[0029] According to the present invention described above, by utilizing the multi-head attention result of the first transformer block, communication overhead can be reduced by eliminating inter-device communication required to aggregate the multi-head attention results between devices within the transformer block and transmit them to a multi-layer perceptron.

[0030] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0031] Figure 1 shows an example of the structure of a conventional large-scale transformer model. Figure 2 shows an example of a structure in which the Transformer blocks are divided into each device via intra-layer parallax for training a large-scale Transformer model. Figure 3 shows the CKA similarity scores for the MHA output, MLP input (residual + MHA), and MLP output in adjacent transformer blocks. Figure 4 shows two scenarios: removing all MHAs (ALL MHA) and removing all MHA-MLP connections (All Connect). Figure 5 shows the normalized gradient magnitude of the MHA output in the transformer block of GPT-2 for various datasets. Figure 6 shows the results of MHA removal measured by perplexity. FIG. 7 shows the structure of a large-scale transformer model according to one embodiment of the present invention. FIG. 8 shows an example of a structure in which a second transformer block is divided into each device by intra-layer parallax for training a large-scale transformer model according to one embodiment of the present invention. FIG. 9 shows a flowchart of a multi-device-based transformer model training method according to one embodiment of the present invention. FIG. 10 shows a block diagram of a multi-device-based transformer model training device according to one embodiment of the present invention. Figure 11 shows the result of reducing the time required to train a transformer model according to an embodiment of the present invention. Figure 12 shows that a transformer model according to an embodiment of the present invention possesses the same learning ability as a conventional structure. Specific details for implementing the invention

[0032] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. In the following description and the attached drawings, substantially identical components are denoted by the same reference numerals to avoid redundant description. Furthermore, in describing the present invention, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the invention, such detailed description will be omitted.

[0033] An important principle of general transformer models is that the MLP must always receive the output of the most recent MHA within each transformer block. However, as previously explained, this results in all-reduce communication. Here, the inventor questions whether the MLP really needs the output of the most recent MHA, considering that the residual path of a transformer block already accumulates all attention outputs from the previous transformer block. To answer this, the inventor performed several analyses on GPT-2 using four language tasks: WikiText-2 [Merity et al., 2016], PTB [Marcus et al., 1993], BookCorpus [Zhu et al., 2015], and CC-News [Hamborg et al., 2017].

[0034] To investigate whether direct MHA-MLP coupling is strictly necessary, the inventors first measured how a single MHA output affects the MLP input within the transformer block. In particular, they compared the CKA similarity between the MHA output and the MLP input [Kornblith et al., 2019] to determine how significantly the most recent MHA output affects the representation after being added along with residual coupling that accumulates the previous MHA output. They measured how much the representation in the model changed using CKA similarity analysis from many previous studies.

[0035] Figure 3 shows the CKA similarity scores for the MHA output, MLP input (residual + MHA), and MLP output in adjacent transformer blocks. The MHA output, which varies significantly depending on the transformer block, is added to the MLP input, but the MLP input does not vary significantly depending on the transformer block. This suggests that the impact of a change in a single MHA output on the MLP input is limited because the residual connection already accumulates the attention signal from the previous block. Since the MLP still generates different outputs, new information can be generated even if the inputs are similar. This implies that the MLP may not always require the most recent MHA output (i.e., the MHA output within the same transformer block).

[0036] The inventors hypothesized that if the MHA-MLP connections themselves consistently generate similar representations, they can be selectively skipped or reconfigured to improve efficiency. To test this hypothesis, they compared removing the entire MHA with removing only the connections to the MLP.

[0037] Figure 4 illustrates two scenarios: removing all MHAs (ALL MHA) and removing all MHA-MLP connections (All Connect). As expected, removing all MHAs severely degrades model quality because attention cannot be used at all. In contrast, removing all connections recovers most of the performance lost compared to removing all MHAs. Although removing connections is less detrimental than removing the entire layer, the performance loss is still significant, indicating the need for alternative signals for the MLP.

[0038] Accordingly, the inventors focused on previous attention signals that could effectively replace or supplement the most recent attention. To investigate alternative signals for the MLP, the gradient magnitude of the MHA output in all transformer blocks was measured to identify important attention outputs that significantly influence the final prediction. Gradient magnitude is one of the widely used methods to measure which input features a model is focusing on as an importance score [Simonyan et al., 2013].

[0039] Figure 5 shows the normalized gradient magnitudes of the MHA outputs in the Transformer block of GPT-2 for various datasets. Figure 5 shows that the first MHA output consistently exhibits the highest gradient magnitudes, which demonstrates that fluctuations in the earliest attention results have a disproportionately large impact on the final prediction.

[0040] This was further verified by measuring perplexity after omitting the MHA in individual transformer blocks. Figure 6 shows the results of MHA removal measured by perplexity. As can be seen in Figure 6, removing the first attention results in a much greater increase in perplexity than removing subsequent layers, demonstrating the important role of the first attention in language modeling. These results suggest that utilizing the first attention more effectively can compensate for the information loss that occurs when skipping subsequent MHA-MLP connections.

[0041] The above analysis suggests that (1) the MHA output can be omitted but an alternative signal for the MLP is required, and (2) the first MHA output can be the key to resolving the performance gap. Based on these results, an embodiment of the present invention proposes a new transformer model designed to simplify the MHA-MLP connection using the first MHA.

[0042] The transformer model according to an embodiment of the present invention maintains the overall structure of the existing transformer model, but redirects the MHA output of the first transformer block to the MLP input instead of the attention within each transformer block.

[0043] FIG. 7 shows the structure of a large-scale transformer model according to one embodiment of the present invention.

[0044] The transformer model is configured such that an initial first transformer block (100) and a plurality of second transformer blocks (200) are connected sequentially. The first transformer block (100) is identical to the transformer block (180) of the existing transformer model of FIG. 1, but the second transformer block (200) has a different structure from the first transformer block (100). In FIG. 7, for the purpose of distinguishing it from the first transformer block (100), the second transformer block (200) is named SkipFormer Block for convenience.

[0045] Each second transformer block (200) includes a layer normalization layer (220), a Multi-Head Attention unit (hereinafter MHA) (230), a residual connection (240), a layer normalization layer (250), a Multi-Layer Perceptron unit (hereinafter MLP) (260), and a residual connection (270).

[0046] In the first transformer block (100), as shown in FIG. 1, the output of the MHA (130) within the block is provided as an input to the MLP (160), but in the second transformer block (200), the output of the MHA (230) within the same block is not provided as an input to the MLP (260). Instead, what is provided as an input to the MLP (260) is the output (Bottom-most Attention) (130a) of the MHA (130) of the first transformer block (100). That is, the output (Bottom-most Attention) (130a) of the MHA (130) of the first transformer block (100) is provided to each second transformer block (200) and is repeatedly utilized as an input to the MLP (260) in each second transformer block (200).

[0047] Accordingly, the residual connection (240) sums the input (210) and the output (Bottom-most Attention) (130a) of the MHA (130) of the first transformer block (100), and the summation result is provided to the MLP (260) through the layer normalization layer (250).

[0048] Additionally, the residual connection (270) generates the output (280) of the block by summing the input (210), the output of the MHA (230) of the block, and the output of the MLP (260), and provides it as the input to the next second transformer block (200). That is, in the first transformer block (100), the output of the MHA (130) within the same block is provided as the input to the MLP (160), but in the second transformer block (200), the output of the MHA (230) within the same block is not provided to the MLP (260), but instead is summed with the input (210) and the output of the MLP (260) and provided as the output to the second transformer block (200).

[0049] FIG. 8 shows an example of a structure in which a second transformer block (200) is divided into each device by intra-layer parallax for training a large-scale transformer model according to one embodiment of the present invention. The structure in which a first transformer block (100) is divided into each device by intra-layer parallax is the same as FIG. 2.

[0050] The parameters of the MHA (230) and MLP (260) constituting the second transformer block (200) are divided in a row or column direction and distributed to the first device (GPU1) and the second device (GPU2). Thus, the MHA (230) is divided into MHA (231) and MHA (232), and the MLP (260) is divided into MLP (261) and MLP (262). For convenience, the part corresponding to the layer normalization layer (220, 250) in FIG. 8 has been omitted. Although a GPU is given as an example of a device in FIG. 8, the device may be various devices capable of performing computations for model training, such as CPUs, APs, and APUs, in addition to GPUs.

[0051] Referring to FIG. 8, the input (X) (210) and the output (Bottom-most MHA output) (131a) (see FIG. 2) of the divided MHA (131) of GPU1 of the first transformer block (100) are summed and provided to the divided MLP (261) of GPU1. Meanwhile, the input (X) (210) and the output (Bottom-most MHA output) (132a) (see FIG. 2) of the divided MHA (132) of GPU2 of the first transformer block (100) are summed and provided to the divided MLP (262) of GPU2.

[0052] In this way, since the divided MLP (261) of GPU1 utilizes the output (131a) of the divided MHA (131) of GPU1 in the first transformer block (100) and the divided MLP (262) of GPU2 utilizes the output (132a) of the divided MHA (132) of GPU2 in the first transformer block (100), all-reduce communication between GPUs to sum the outputs of the divided MHAs (131, 132) and transmit them to the divided MLPs (161, 162) as shown in FIG. 2 is not required. That is, all-reduce communication between GPUs to sum the outputs of the divided MHAs (231, 232) and transmit them to the divided MLPs (261, 262) is not required. This can significantly improve training time performance by reducing communication overhead.

[0053] Additionally, since the split MLP (261) of GPU1 utilizes the output (131a) of the split MHA (131) of GPU1 of the first transformer block (100), the split MHA (231) and the split MLP (261) can be executed in parallel. Likewise, the split MHA (232) and the split MLP (262) can be executed in parallel. Thus, the training speed can be significantly increased.

[0054] The parameters of the query (Q), key (K), and value (V) layers constituting the MHA (230) are divided in the column direction of the matrix (①), and the first linear layer constituting the MLP (260) is also divided in the column direction (③). Additionally, the second linear layer constituting the MHA (230) is divided in the row direction of the matrix (②), and the second linear layer constituting the MLP (260) is also divided in the row direction (④). Therefore, the operation results of the divided MHA (231, 232) and the divided MLP (261, 262) have the same form.

[0055] In the second transformer block (200), the output (231a) of the divided MHA (231) of GPU1 and the output (261a) of the divided MLP (261) of GPU1 are summed, and the output (232a) of the divided MHA (232) of GPU2 and the output (262a) of the divided MLP (262) of GPU2 are summed. This summing does not cause communication between GPUs within the second transformer block (200). The sum of the output (231a) of the divided MHA (231) of GPU1 and the output (261a) of the divided MLP (261) of GPU1, the sum of the output (232a) of the divided MHA (232) of GPU2 and the output (262a) of the divided MLP (262) of GPU2, and the input (X) (210) of the second transformer block (200) are summed (280) and provided as the input to the next second transformer block.

[0056] FIG. 9 shows a flowchart of a multi-device-based transformer model training method according to one embodiment of the present invention.

[0057] In step 910, the MHA (130, 230) and MLP (160, 260) of the first transformer block (100) and a plurality of second transformer blocks (200) of the transformer model of FIG. 7 are divided as shown in FIG. 2 and FIG. 8 and distributed to the first and second devices.

[0058] In step 920, the Transformer model is trained in an intra-layer parallax manner through parallel computation of the first and second devices.

[0059] FIG. 10 shows a block diagram of a multi-device-based transformer model training device according to one embodiment of the present invention.

[0060] A multi-device-based transformer model training device (1000) trains a large-scale transformer model using a multi-device method in an intra-layer parallax manner. To this end, the multi-device-based transformer model training device (1000) may include one or more processors (1010), a computer-readable storage medium (1030), and a communication bus (1050).

[0061] The processor (1010) can control the operation of the multi-device-based transformer model training device (1000). For example, the processor (1010) can execute one or more programs (1031) stored in a computer-readable storage medium (1030). One or more programs (1031) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the multi-device-based transformer model training device (1000) to perform an operation to train a large-scale transformer model using multiple devices when executed on the processor (1010).

[0062] A computer-readable storage medium (1030) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information for training a large-scale Transformer model using multiple devices. A program (1031) stored in the computer-readable storage medium (1030) includes a set of instructions executable by a processor (1010). In one embodiment, the computer-readable storage medium (1030) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that are accessed by an image integration device (1000) and capable of storing desired information, or a suitable combination thereof.

[0063] The communication bus (1050) interconnects various other components of the multi-device-based transformer model training device (1000), including the processor (1010) and the computer-readable storage medium (1030).

[0064] The multi-device-based transformer model training device (1000) may also include one or more input / output interfaces (1070) and one or more communication interfaces (1090) that provide interfaces for one or more input / output devices. The input / output interfaces (1070) and the communication interfaces (1090) are connected to a communication bus (1050). An input / output device (not shown) may be connected to other components of the multi-device-based transformer model training device (1000) through the input / output interface (1070).

[0065] The processor (1010) divides the MHA (130, 230) and MLP (160, 260) of the first transformer block (100) and a plurality of second transformer blocks (200) of the transformer model of FIG. 7 as shown in FIG. 2 and FIG. 8 and distributes them to the first and second devices.

[0066] The processor (1010) trains the transformer model in an intra-layer parallax manner through parallel operation of the first and second devices.

[0067] Figure 11 shows the results of reducing the time required to train a transformer model according to an embodiment of the present invention. When comparing the training times of a transformer model with the same amount of parameters on four devices, the embodiment of the present invention reduces communication overhead compared to the existing structure, thereby reducing the training time by 36.2%.

[0068] FIG. 12 shows that the transformer model according to an embodiment of the present invention possesses the same learning ability as the existing structure. Compared to when the existing transformer model is trained with photographic data, the transformer model according to an embodiment of the present invention demonstrates that it maintains the same learning graph and that its learning ability does not deteriorate.

[0069] A multi-device-based transformer model training device according to embodiments of the present invention may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the interactive recommendation provider may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0070] In addition, the multi-device-based transformer model training device according to embodiments of the present invention may be mounted on a computing device or server equipped with hardware elements in the form of software, hardware, or a combination thereof. A computing device or server may refer to various devices that include, in whole or in part, a communication device such as a communication modem for communicating with various devices or wired / wireless communication networks, a memory for storing data for executing a program, and a microprocessor for executing a program to perform calculations and commands.

[0071] Operations according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium refers to any medium that participates in providing instructions to a processor for execution. A computer-readable medium may include program instructions, data files, data structures, or a combination thereof. Examples include magnetic media, optical recording media, memory, etc. Computer programs may be distributed over networked computer systems, and computer-readable code may be stored and executed in a distributed manner. Functional programs, code, and code segments for implementing the present embodiment will be readily deducible by programmers skilled in the art to which the present embodiment belongs.

[0072] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications, changes, and substitutions within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments and accompanying drawings. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention.

Claims

Claim 1 In a multi-device-based transformer model training method, the transformer model is configured such that an initial first transformer block and a plurality of second transformer blocks are sequentially connected, and the first transformer block and the second transformer blocks include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), and the training method comprises the step of dividing the MHA and the MLP of the first transformer block and the plurality of second transformer blocks and distributing them to first and second devices; A multi-device based transformer model training method comprising the step of training the transformer model in an intra-layer parallax manner through parallel computation of the first and second devices, wherein the first transformer block is configured such that the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device are summed and provided to the divided MLP of the first device and the divided MLP of the second device, and the second transformer block is configured such that the input of the second transformer block and the output of the divided MHA of the first device of the first transformer block are summed and provided to the divided MLP of the first device, and the input of the second transformer block and the output of the divided MHA of the second device of the first transformer block are summed and provided to the divided MLP of the second device. Claim 2 A multi-device based transformer model training method according to claim 1, wherein the second transformer block is configured such that the output of the divided MHA of the first device of the second transformer block and the output of the divided MLP of the first device are summed, and the output of the divided MHA of the second device of the second transformer block and the output of the divided MLP of the second device are summed. Claim 3 A multi-device based transformer model training method according to claim 2, wherein the transformer model is configured such that the sum of the output of the divided MHA of the first device and the output of the divided MLP of the first device, the sum of the output of the divided MHA of the second device and the output of the divided MLP of the second device, and the sum of the input of the second transformer block are summed and provided as the input of the next second transformer block. Claim 4 A multi-device based transformer model training method according to claim 1, wherein the MHA is configured such that a query, key, value layer, Scaled Dot-Product Attention layer, concat layer, and linear layer are sequentially connected. Claim 5 A multi-device based transformer model training method according to claim 1, wherein the MLP is configured such that a linear layer, a non-linear layer, and a linear layer are sequentially connected. Claim 6 A computer program stored on a computer-readable storage medium for executing a multi-device-based transformer model training method described in any one of claims 1 to 5 on a computer. Claim 7 A multi-device-based transformer model training device comprising: a memory storing one or more programs for training a transformer model in an intra-layer parallax manner using multi-devices; The system includes one or more processors that perform operations to train the transformer model in an intra-layer parallax manner using multi-devices according to one or more programs stored in the memory, wherein the transformer model is configured such that an initial first transformer block and a plurality of second transformer blocks are sequentially connected, and the first transformer block and the second transformer blocks include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), and the processor divides the MHA and the MLP of the first transformer block and the plurality of second transformer blocks and distributes them to first and second devices, and trains the transformer model in an intra-layer parallax manner through parallel operations of the first and second devices, and the first transformer block includes the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device. A multi-device based transformer model training device configured to be summed and provided to the divided MLP of the first device and the divided MLP of the second device, wherein the second transformer block is configured such that the input of the second transformer block and the output of the divided MHA of the first device of the first transformer block are summed and provided to the divided MLP of the first device, and the input of the second transformer block and the output of the divided MHA of the second device of the first transformer block are summed and provided to the divided MLP of the second device. Claim 8 A multi-device based transformer model training device according to claim 7, wherein the second transformer block is configured such that the output of the divided MHA of the first device of the second transformer block and the output of the divided MLP of the first device are summed, and the output of the divided MHA of the second device of the second transformer block and the output of the divided MLP of the second device are summed. Claim 9 A multi-device based transformer model training device according to claim 8, wherein the transformer model is configured such that the sum of the output of the divided MHA of the first device and the output of the divided MLP of the first device, the sum of the output of the divided MHA of the second device and the output of the divided MLP of the second device, and the input of the second transformer block are summed and provided as the input of the next second transformer block. Claim 10 In claim 7, the MHA is a multi-device based transformer model training device configured such that a query, key, value layer, Scaled Dot-Product Attention layer, concat layer, and linear layer are sequentially connected. Claim 11 In claim 7, the MLP is a multi-device based transformer model training device configured such that a linear layer, a non-linear layer, and a linear layer are sequentially connected. Claim 12 In a multi-device-based transformer model training device, a first device; The transformer model includes a first transformer block and a plurality of second transformer blocks, wherein the first transformer block and the second transformer block include a multi-head attention unit (hereinafter MHA) and a multi-layer perceptron unit (hereinafter MLP), wherein the MHA and the MLP of the first transformer block and the plurality of second transformer blocks are divided and distributed to the first and second devices, and the transformer model is trained through parallel operation of the first and second devices, wherein the first transformer block is configured such that the input of the first transformer block, the output of the divided MHA of the first device, and the output of the divided MHA of the second device are summed and provided to the divided MLP of the first device and the divided MLP of the second device, and the second transformer block is configured such that the input of the second transformer block and the first of the first transformer block A multi-device based transformer model training device configured such that the outputs of the divided MHAs of the device are summed and provided to the divided MLPs of the first device, and the inputs of the second transformer block and the outputs of the divided MHAs of the second device of the first transformer block are summed and provided to the divided MLPs of the second device. Claim 13 A multi-device based transformer model training device according to claim 12, wherein the second transformer block is configured such that the output of the divided MHA of the first device of the second transformer block and the output of the divided MLP of the first device are summed, and the output of the divided MHA of the second device of the second transformer block and the output of the divided MLP of the second device are summed. Claim 14 A multi-device based transformer model training device according to claim 13, wherein the transformer model is configured such that the sum of the output of the divided MHA of the first device and the output of the divided MLP of the first device, the sum of the output of the divided MHA of the second device and the output of the divided MLP of the second device, and the input of the second transformer block are summed and provided as the input of the next second transformer block. Claim 15 In claim 12, the above MHA is a multi-device based transformer model training device configured such that a query, key, value layer, Scaled Dot-Product Attention layer, concat layer, and linear layer are sequentially connected. Claim 16 In claim 12, the MLP is a multi-device based transformer model training device configured such that a linear layer, a non-linear layer, and a linear layer are sequentially connected.

Images