A multi-node cluster-based inference method and system using GPU-isolated allocation for pre-training and fine-tuning layers of a multi-deep learning model.

Abstract

The present invention relates to a multi-node cluster-based inference method by GPU-separated allocation of pre-training and fine-tuning layers of a multi-deep learning model. According to an embodiment of the present invention, the method includes the steps of: an endpoint unit of an inference server receiving input values ​​from a client; the endpoint unit of the inference server distributing the received input values ​​and transmitting the first input values ​​to a first computing node containing a container loaded with the neural network bundle of the first step; the first container of the first computing node using the received first input values ​​as input and performing calculations by the neural network layer of a GPU to generate a first output value; and the first container selecting a container loaded with the neural network bundle of the next step and transmitting the first output value to a container that executes the next step or to a computing node containing a container that executes the next step.

Description

Technical Field

[0001] The present invention relates to a multi-node cluster-based inference method and system for GPU separate allocation of a pre-training layer and a fine-tuning layer of a multi deep learning model. In particular, the pre-training layer and the fine-tuning layer of a model to which fine-tuning is applied to a GPU by a GPU partitioning virtualization technique can be separately stored, and the present invention relates to a multi-node cluster-based inference method and system for GPU separate allocation of a pre-training layer and a fine-tuning layer of a multi deep learning model that can perform calculations by multiple GPUs based on an input value received from a client.

Background Art

[0002] Many companies have developed and released ultra-large language models based on the Transformer architecture, such as ChatGPT (registered trademark), which began to be introduced to the general public at the end of 2022. These models commonly consist of numerous neural network layers composed of a very large number of parameters. While there are differences depending on the characteristics of the model itself, the quantization method used in the training and optimization processes, and the training dataset, generally, models with 30 billion parameters or more that are capable of multilingual dialogue and emergent generation and logical development above a certain level require more than 70GB of dedicated GPU memory when providing inference services using a GPU. Furthermore, during the training process, intermediate values ​​from gradient calculations that are backpropagated to the model parameter set must be stored, so the memory capacity required is typically 2 to 2.5 times that of the inference service. Based on inference services, as of late 2023 to early 2024, using NVIDIA's A100 and H100 GPUs used when building AI data centers, it barely fits on one GPU or requires two or more. When attempting to use models with 100 billion parameters or more, such as ChatGPT, two to four or more GPUs are required. Such language models are being extended to support not only prompt input and output in the form of natural language text, but also various media formats such as images and videos, and as a result, the size of the model and the GPU specifications required for computation are increasing day by day. Here, GPU (Graphics Processing Unit) refers to a graphics processing unit, which is an electronic circuit that performs mathematical calculations at high speed.

[0003] A foundation model is a massive language model that has been initially trained by injecting a large amount of natural language text. Such a foundation model has learned the grammar, vocabulary, and expressions of the language structure of the input text, as well as the facts and knowledge contained in the input text. By performing additional training on a specific dataset, it can be used in actual commercial services by providing more detailed background knowledge to answer questions in specific fields and themes, or by adding traits of a virtual personality expressed by the prompt. This process is called fine-tuning. Even with the same foundation model, there are various fine-tuning techniques and results depending on what kind of neural network layers are duplicated, transformed, iterated, and randomized, in what form new neural network layers are added, and in what way and with what additional datasets such neural network layers are trained.

[0004] In conventional large-scale cloud and data center environments, when numerous customers attempt to provide services using many models finely tuned based on the same underlying model, individually loading these models onto GPUs results in duplicate memory usage by loading pre-training layers containing the same parameter values ​​for each model and user. However, this duplicate loading, while varying depending on the fine-tuning techniques used, results in a significant amount of unnecessary memory space being wasted, as the memory space occupied by the pre-training layer and the fine-tuning layer differs by more than 10 times in large models.

[0005] Prior to this invention, there is Published Patent No. 10-2022-0064806 (Method and Apparatus for Allocating GPUs to Software Packages), but it only includes the steps of: checking package information for each of multiple software packages and GPU status information used by the multiple software packages for multiple terminals connected to a network entity; determining a change in the GPU usage profile based on the package information and GPU status information when at least one of the multiple software packages' workloads is changed; determining the number of packages to process the workload, the GPU IDs and GPU usage locations to be assigned to the packages based on the package information and GPU status information; updating the GPU usage profile according to the determined GPU IDs and GPU usage locations; and allocating GPUs to the multiple software packages based on the updated GPU usage profile. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Republic of Korea Published Patent No. 10-2022-0064806 [Overview of the project] [Problems that the invention aims to solve]

[0007] To solve the aforementioned problems, the present invention aims to provide a multi-node cluster-based inference method and system for multi-deep learning models that can serve diverse versions of finely tuned language models with far fewer GPU resources. This is achieved by separating the pre-training layer and the fine-tuning layer of a language model to which fine-tuning is applied, sharing the pre-training layer, but deploying only the fine-tuning layer on a GPU, and further utilizing container-level GPU partitioning virtualization techniques to enable multiple models to share a single GPU for their fine-tuning layers. [Means for solving the problem]

[0008] As a means to solve the aforementioned technical problems, according to an embodiment of the present invention, a multi-node cluster-based inference method by GPU-separated allocation of pre-training layers and fine-tuning layers of a multi-deep learning model includes the steps of: an endpoint unit of an inference server receiving input values ​​from a client; the endpoint unit of the inference server distributing the received input values ​​and transmitting the first input values ​​to a first computing node including a container loaded with the neural network bundle of the first step; the first container of the first computing node using the received first input values ​​as input and performing calculations by the neural network layer of a GPU to generate a first output value; and the first container selecting a container loaded with the neural network bundle of the next step and transmitting the first output value to a container that executes the next step or to a computing node including a container that executes the next step, wherein the GPU includes at least one GPU, meaning a physical GPU or a partitioned virtualized GPU fragment, and the GPU separately stores at least one pre-training layer and at least one fine-tuning layer of a model to which fine-tuning is applied by partitioned virtualization techniques.

[0009] A multi-node cluster-based inference system using GPU-separated allocation for pre-training and fine-tuning layers of a multi-deep learning model includes an endpoint unit that receives input values ​​from a client, distributes the input values ​​and transmits the distributed input values ​​to compute nodes containing a container loaded with the neural network bundle of the first step, receives output values ​​from the container and transmits the final output value to the client, and a server control unit that receives input values ​​or output values, performs calculations by the neural network layers of the GPU to generate output values, selects a container loaded with the neural network bundle of the next step, transmits the output value to a container that executes the next step or to another compute node containing a container that executes the next step, and transmits the final output value to the endpoint unit as the final output value when the container loaded with the neural network bundle of the last step generates output values, wherein the GPU includes at least one GPU, meaning a physical GPU or a partitioned virtualized GPU fragment, and the GPU is characterized in that at least one pre-training layer and at least one fine-tuning layer of the model to which fine-tuning is applied by partitioned virtualization techniques are stored separately. [Effects of the Invention]

[0010] According to the present invention, the pre-training layer and the fine-tuning layer of a model to which fine-tuning of the GPU is applied can be stored separately by using a GPU partitioning virtualization technique.

[0011] Furthermore, calculations can be performed sequentially using multiple GPUs, where layers are stored separately based on input values ​​received from the client.

[0012] Furthermore, the number of computations currently waiting and processing, as well as the estimated time required, can be tracked for each GPU, allowing for the selection of containers containing neural network layers that are in the correct order. [Brief explanation of the drawing]

[0013] [Figure 1] This figure illustrates a multi-node cluster-based inference method using GPU-separated allocation for the pre-training and fine-tuning layers of a multi-deep learning model according to an embodiment of the present invention. [Figure 2] This figure illustrates a multi-node cluster-based inference method using GPU-separated allocation for the pre-training and fine-tuning layers of a multi-deep learning model according to an embodiment of the present invention. [Figure 3] This is a diagram illustrating the configuration of a multi-node cluster-based inference system with GPU-separated allocation for the pre-training and fine-tuning layers of a multi-deep learning model according to an embodiment of the present invention. [Figure 4] This is a diagram showing the configuration of the inference framework unit according to an embodiment of the present invention. [Figure 5] This figure illustrates an embodiment of the present invention illustrating GPU isolation and allocation. [Figure 6] This figure illustrates an embodiment of the present invention illustrating GPU isolation and allocation. [Figure 7] This figure illustrates an embodiment of the present invention illustrating GPU isolation and allocation. [Figure 8] This figure illustrates an embodiment of the present invention illustrating GPU isolation and allocation. [Modes for carrying out the invention]

[0014] The specific structural or functional descriptions of the embodiments of the present invention disclosed herein are merely illustrative for the purpose of illustrating the embodiments of the present invention, and embodiments of the present invention can be carried out in a variety of forms and are not limited to the embodiments described herein.

[0015] Since embodiments according to the concept of the present invention can be subject to various modifications and can have various forms, embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit embodiments according to the concept of the present invention to a specific disclosed form, but includes all modifications, equivalents, or alternatives included in the spirit and technical scope of the present invention.

[0016] The terms used herein are merely used to explain specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Terms such as "including" or "having" in this specification are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and it should be understood that the presence or addition possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof are not precluded in advance.

[0017] The deep learning models described herein refer mainly to language models, but the objects that can be handled by the techniques and systems of the present invention are applicable to all types of deep learning models composed of multiple neural network layers, each of which can be classified for pre-training and fine-tuning purposes, and are not limited to language models. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings of this specification.

[0018] FIG. 1 and FIG. 2 are diagrams for explaining a multi-node cluster-based inference method by GPU separation allocation of a pre-training layer and a fine-tuning layer of a multi deep learning model according to an embodiment of the present invention.

[0019] As shown in FIGS. 1 and 2, the endpoint unit 210 of the inference server 200 receives an input value from the client 100 (S101), and distributes the input value received by the endpoint unit 210 of the inference server 200, and transmits the first input value to the first computing node 300a including the container loaded with the neural network bundle of the first step (S103). The input value may be attached with the client's first unique request ID and the fine-tuning model ID, and can be used to query the container list of the next step. The endpoint unit 210 can distribute the input value received from the client 100 by using round-robin or a separate load distribution algorithm that utilizes the performance metrics periodically updated or provided in real time by the container control unit 321 of each computing node 300.

[0020] Using the first input value received by the first container 320a of the first computing node 300a as an input, an operation is performed by the neural network layer of the GPU to generate a first output value (S105). If conversion of the received first input value is required, the first container 320a can convert it to be operable in the neural network layer stored in the GPU unit 323. Also, the first container 320a can perform calculations (operations) by the neural network layer by utilizing the GPU acceleration function through at least one GPU included in the GPU unit 323 for a set of embedding vectors that are the received input value or the converted input value to generate an output value. At this time, the generated output value can be expressed in the form of vectors, matrices, and other numerical information as an operation result. The generated output value can be converted if conversion is required, and when the output value is generated, the first container 320a selects the container loaded with the neural network bundle of the next step, and transmits the first output value to the container that executes the next step or the computing node including the container that executes the next step (S107).

[0021] The container control unit 321 of container 320 can track the number and duration of currently waiting and processing operations for each GPU in the GPU unit 323 within the same container 320 via performance metrics and performance measurement mechanisms, and store the number and estimated duration of operations as scheduling hints. The container control unit 321 can also receive a list of containers that can perform the next step from the server control unit 220. Based on the list of containers that can perform the next step received from the server control unit 220, the container control unit 321 can query scheduling hints via API / RPC requests, or retrieve and query scheduling hints stored in a separate database or cache server. The container control unit 321 can select the container to perform the next step according to a routing policy based on the container list and scheduling hints as described above. The routing policy can apply a variety of scheduling methods, including at least one of round-robin and a method of selecting the GPU with the fewest waiting operations.

[0022] The container control unit 321 selects a container to execute the next step, and if the second computing node 300b is loaded with the neural network bundle for the next step, the second container 320b of the second computing node 300b takes the received first output value as input and performs calculations using the neural network layer of the GPU to generate a second output value (S201). This is repeated until the nth output value is generated from the nth container 320n of the nth computing node 300n, which contains the container loaded with the neural network bundle for the final step (S203), and the nth container 320n transmits the nth output value as the final output value to the endpoint unit 210 (S205). The endpoint unit 210 transmits the final output value to the client 100 (S207).

[0023] Figure 3 is a diagram illustrating the configuration of a multi-node cluster-based inference system with GPU-separated allocation for the pre-training and fine-tuning layers of a multi-deep learning model according to an embodiment of the present invention.

[0024] As shown in Figure 3, the multi-node cluster-based inference system 10, which uses GPU-separated allocation for the pre-training and fine-tuning layers of a multi-deep learning model, consists of a client 100 and an inference server 200.

[0025] The inference server 200 consists of an endpoint unit 210, a server control unit 220, and at least one computing node 300.

[0026] The endpoint unit 210 consists of pre-generated inference endpoints having at least one fixed address, and receives input values ​​from the client 100 via the inference endpoints. The endpoint unit 210 can also distribute the received input values ​​and transmit them to the computing nodes 300, which include containers loaded with the neural network bundles of the first step. The input values ​​may be accompanied by the client's initial unique request ID and fine-tuning model ID, which can be used to query the container list for the next step. In one embodiment, if the container loaded with the neural network bundles of the first step corresponding to the first input value distributed by the endpoint unit 210 is the first computing node 300a, the endpoint unit 210 transmits the first input value to the first computing node 300a. The endpoint unit 210 can distribute the input values ​​received from the client 100 using round-robin or a separate load balancing algorithm that utilizes performance metrics provided periodically by the container control unit 321 of each computing node 300 or in real time.

[0027] When the endpoint unit 210 receives the final output value from the container 320, it can transmit the final output value to the client 100 via the corresponding client connection channel. When the endpoint unit 210 transmits the final output value to the client 100, it routes the traffic to the container control unit 321 of the container carrying the neural network bundle of the first step to process the response traffic. Also, when the endpoint unit 210 transmits the final output value to the client 100, it issues the final output value in a separate message queue so that the container control unit 321 of the container carrying the neural network bundle of the first step responds.

[0028] The server control unit 220 controls the entire configuration of the inference server 200 and communicates to the container 320 of the computing node 300 a list of containers that can perform the next steps.

[0029] The computing node 300 consists of a node control unit 310 and a container 320, and the inference server 200 may include at least one or more computing nodes 300.

[0030] The node control unit 310 generates at least one or more containers 320 on the computing node 300, can receive instructions from the server control unit 220 to interrupt or delete the execution of running containers 320, and performs performance and status monitoring.

[0031] The container 320 consists of a container control unit 321, an inference framework unit 322, and a GPU unit 323, and one computing node 300 may include at least one or more containers 320.

[0032] The container control unit 321 receives input values ​​from the endpoint unit 210 and transmits these input values ​​to the inference framework unit 322. The container control unit 321 also receives output values ​​from other computation nodes 300 or other containers 320 and transmits these as input values ​​to the inference framework unit 322. When the container control unit 321 receives output values ​​from the inference framework unit 322, it selects a container 320 to execute the next step and transmits the output values ​​to the container control unit 321 of the container 320 that executes the next step or to the computation node 300 that contains the container 320 that executes the next step. When transmitting output values, the container control unit 321 can route the requests of multiple clients sequentially through all neural network bundles so that individual fine-tuning model instances and requests from multiple clients are not mixed up, and each client's request is computed using the fine-tuning model instance that was requested.

[0033] The container control unit 321 tracks the number and duration of currently waiting and processing operations for each GPU in the GPU unit 323 within the same container 320 via the performance metrics and performance measurement mechanism of the inference framework unit 322, and can save the number and estimated duration of operations as scheduling hints. The container control unit 321 continuously updates the scheduling hints. The container control unit 321 can also receive a list of containers that can execute the next step from the server control unit 220. At this time, the container control unit 321 can call a dedicated API that can connect to the internal network of the inference server 200. Furthermore, when the container control unit 321 receives input values ​​from the endpoint unit 210, it can also receive step-by-step container list information and continuously transmit it together until the overall calculation for the corresponding client request is completed. In addition, the container control unit 321 can read the container list data from a separate database or cache server using each model instance and each neural network bundle as keys.

[0034] The container control unit 321 can query scheduling hints via API / RPC requests based on the list of containers that can perform the next step received from the server control unit 220, or it can retrieve scheduling hints stored in a separate database or cache server and query them. The container control unit 321 can select the containers to perform the next step according to the routing policy based on the container list and scheduling hints as described above. The routing policy can apply a variety of scheduling methods, including at least one of round-robin or a method of selecting the GPU with the fewest waiting operations.

[0035] If the container control unit 321 determines that there is no container 320 to execute the next step, it transmits the last output value received from the inference framework unit 322 to the endpoint unit 210 as the final output value.

[0036] The inference framework unit 322 consists of an additional metadata management module 331, an input value conversion module 332, an arithmetic processing module 333, and an output value conversion module 334. Figure 4 is a diagram showing the configuration of the inference framework unit according to an embodiment of the present invention.

[0037] The additional metadata management module 331 recognizes the additional metadata and, based on at least one of the following methods—patching by forking, API, and interface—selects a pre-training layer and a fine-tuning layer in a single model and can load (save) the pre-training layer and the fine-tuning layer in separate locations on at least one GPU constituting the GPU unit 323. The additional metadata management module 331 can load the model parameter set into GPU memory, and a single model may generally be divided into fixed-capacity sections on a neural network layer basis, taking into account the GPU capacity configuration, and saved in the form of a snapshot of the parameter set. In this case, the model is composed of neural network layers and is applicable to all types of deep learning models in which each neural network layer can be divided for pre-training and fine-tuning purposes. Since the pre-training layer and the fine-tuning layer can contain additional metadata, the additional metadata management module 331 can recognize the additional metadata contained in each layer and load one or more models separately onto each GPU. The additional metadata refers to data in the form of a constant list or binary flag that can determine whether each neural network layer is a pre-training layer or a fine-tuning layer.

[0038] Figures 5 to 8 are diagrams illustrating GPU isolation allocation according to embodiments of the present invention. Figure 5 shows the areas occupied by the pre-training layer 501 and the fine-tuning layer 505 within the memory space 503 for each GPU device. In Figure 5, as one embodiment, a model with a total capacity of 100GB is shown, using two 80GB GPUs, with the pre-training layer 501 being 90GB and the fine-tuning layer 505 being 10GB.

[0039] Generally, since models are simply stacked without separating and stacking their layers, if the number of models is N and the total memory required for the pre-training and fine-tuning layers of one model is M, then simply stacking them would require a total of NM of memory. In one embodiment, if the memory required for a model is 100GB (M), and 50 (N) individual models are simply stacked, then a total of 5,000GB (M*N) of GPU memory would be required.

[0040] However, by separating and stacking (storing) the model layers, and removing the overlap by stacking only the fine-tuning layers individually while sharing the pre-training layers, the required GPU memory can be significantly reduced. If the number of models is N, the space ratio occupied by the pre-training layers is (0≦p≦1), and the total memory required for the pre-training and fine-tuning layers is M, then stacking while sharing the pre-training layers requires Mp+NM(1-p)=M{(1-N)p+N} capacity of memory. In one embodiment, if the memory required for a model is 100GB (M), the size of the pre-training layer is 90GB (p=0.9), and 50 (N) individual models are stacked while sharing the pre-training layers, only 590GB is required.

[0041] From the above example, it can be seen that 5,000GB of GPU memory is needed if simply stacked, while 590GB of memory is needed if stacked separately. If we consider the A100 GPU with an 80GB capacity as a base, 63 modules would be needed to achieve 5,000GB of capacity, while 8 modules would be needed to achieve 590GB of capacity, making the separated stacking method significantly more economical.

[0042] When pre-training layers and fine-tuning layers are loaded separately, only one copy of the pre-training layers can be loaded into the GPU memory, while multiple fine-tuning layers can be loaded into individual GPU memories. Referring to Figure 6, this shows a case where two GPU memories 601 are shared, with pre-training layers 603 loaded into them, and fine-tuning layers 607 are loaded into GPUs 605 allocated separately for the user and the result model, with the pre-training layers using the shared GPU and only the fine-tuning layers being separated and loaded into individual GPU memories.

[0043] However, since there is still a possibility of wasted memory on the GPU where the fine-tuning layers are stored, container-level GPU virtualization techniques can be applied to improve the system so that at least one or more fine-tuning layers are served simultaneously on a single GPU. Referring to Figure 7, two GPU memories 701 loaded with pre-training layers 703 are shared, and container-level GPU partitioned virtualization is applied to load multiple fine-tuning layers 707 onto one GPU and then onto the partitioned virtualization applied GPU 705 memory. Referring to Figure 8, to further increase the efficiency of separated loading, the pre-training layers 803 can be divided and loaded onto two GPU memories 801, and after loading the pre-training layers 803, the fine-tuning layers 807 can be additionally loaded into the remaining space in the GPU memory 801. Container-level GPU partitioned virtualization can be applied to load multiple fine-tuning layers 807 onto one GPU and then onto the partitioned virtualization applied GPU 805 memory.

[0044] As shown in Figure 8, it is possible to minimize idle space and store it by dividing it into a pre-training layer and a fine-tuning layer. However, in reality, if there is only one set of pre-training layer GPUs for a very large number of individual model instances, the user requests for all individual models will be concentrated on one, which may cause bandwidth and latency issues. Therefore, when actually implementing this, it is necessary to measure bandwidth (number of requests processed per second, token generation rate for prompt responses) and latency (time taken per request) considering factors such as the performance of the actual model and GPUs, and the network performance of the inference server 200. Then, it is necessary to find the point at which performance attenuation occurs due to the number of copies of the pre-training layer GPU set, and apply a load balancing mechanism by making an appropriate compromise between cost (number of GPUs) and performance.

[0045] The input value conversion module 332 can convert input values ​​received from the container control unit 321 so that they can be processed in the neural network layer stored in the GPU unit 323, if conversion of the input values ​​is required. The input value conversion module 332 can convert a prompt string given as an input value into a set of embedding vectors using a tokenizer module. The input value conversion module 332 may also include modules for serializing and deserializing the input values ​​received from the container control unit 321 so that they can be transmitted over the network. The input value conversion module 332 can selectively apply a variety of compression and serialization techniques to the input values, ranging from uncompressed and Huffman coding-based compression algorithms to vector and matrix compression algorithms that consider sparsity, depending on the model characteristics, the hardware specifications of each computation node, and administrator settings. The input value conversion module 332 can also apply additional symmetric and asymmetric encryption depending on the network configuration and security requirements, but is not necessarily limited to these.

[0046] The arithmetic processing module 333 can perform calculations (operations) using a neural network layer, utilizing the GPU acceleration function, on a set of embedding vectors, which are input values ​​received from the container control unit 321 or input values ​​converted by the input value conversion module 332, via at least one GPU included in the GPU unit 323, and generate an output value. The output value generated at this time can be expressed as a vector, matrix, or other numerical information as a result of the calculation. The arithmetic processing module 333 can convert the generated output value using the output value conversion module 334, and can transmit the generated output value or the output value converted by the output value conversion module 334 to the container control unit 321. When the arithmetic processing module 333 transmits the generated output value or the output value converted by the output value conversion module 334 to the container control unit 321, it can transmit at high speed not only through common networks such as Ethernet and TCP sockets, but also through high-speed transmission technologies such as InfiniBand-based RDMA (GPUDirect Access) and Ethernet-based RDMA (RoCE). Furthermore, the arithmetic processing module 333 can transmit output values ​​via a shared memory with the container control unit 321 using a separate memory copy-free (zero-copy) method.

[0047] The output value conversion module 334 can convert the output value generated by the arithmetic processing module 333 if conversion of the output value is necessary. The output value conversion module 334 may also include modules for serializing and deserializing the output value generated by the arithmetic processing module 333 so that it can be transmitted over the network. The output value conversion module 334 can selectively apply a variety of compression and serialization techniques, ranging from uncompressed and Huffman coding-based compression algorithms to vector and matrix compression algorithms that consider sparsity, depending on the model characteristics, the hardware specifications of each computing node, and administrator settings, to convert the output value. Furthermore, the output value conversion module 334 can apply additional symmetric and asymmetric encryption depending on the network configuration and security requirements, but is not necessarily limited to these.

[0048] The GPU unit 323 includes at least one GPU and may consist of multiple physical GPUs or multiple partitioned virtualized GPU fragments. The GPU (Graphic Processing Unit) means a graphics processing unit, and at least one pre-trained layer and at least one fine-tuning layer of the model to which fine-tuning is applied by partitioned virtualization techniques may be stored separately. In this case, the pre-trained layer means a model that has undergone primary training, and the fine-tuning layer may mean a model that has undergone additional training on a specific dataset and possesses more detailed background knowledge for a specific field and subject. In this invention, the explanation is based on a very large language model to which fine-tuning is applied, but the present invention can be applied to any model to which fine-tuning is applied. Furthermore, in this invention, among such fine-tuning techniques, it is possible to handle cases in which the pre-trained layer and the fine-tuning layer can be explicitly distinguished by selectively retraining only some layers, generating additional layers, or duplicating and retraining existing layers, rather than retraining all neural network layers of the pre-trained model equally.

[0049] The pre-trained and fine-tuning layers stored on the GPU may include additional metadata. This additional metadata refers to data in the form of a constant list or binary flag that allows determination of whether each neural network layer is a pre-trained or fine-tuning layer. The additional metadata can be recorded in the pre-trained and fine-tuning layers using a markup language (such as JSON or YAML) as a separate file from the parameter set, using a built-in key-value metadata field supported by ONNX standard IR version 10 or later, or using a similar user-defined field in other modeling technology formats. The additional metadata may also include data that allows the pre-trained and fine-tuning layers to be updated and distributed independently, including format information for output and input values, including the size of the matrix or vector and the representation format of individual elements, the version of each layer set, and a unique ID.

[0050] Although the invention has been described with reference to the embodiments illustrated in the drawings, these are merely illustrative, and a person with ordinary skill in the art will understand that a variety of modifications and equivalent other embodiments are possible. Therefore, the true scope of technical protection of the invention must be determined by the technical idea of ​​the appended claims. [Explanation of symbols]

[0051] 10 Inference Systems 100 clients 200 inference servers 210 Endpoint Section 220 Server Control Unit 300 compute nodes 300a First Computation Node 300b Second computing node 300n n computing node 310 Node Control Unit 320 containers 320a Container No. 1 320b Container No. 2 320n n container 321 Container Control Unit 322 Inference Framework Section 323 GPU section 331 Additional Metadata Management Module 332 Input Value Conversion Module 333 Arithmetic Processing Module 334 Output Value Conversion Module 501 Pre-trained tier 503 Memory space 505 Fine adjustment layer 601 GPU memory 603 Pre-training layer 607 Fine adjustment layer 701 GPU memory 703 Pre-training layer 707 Fine adjustment layer 801 GPU memory 803 Pre-training layer 807 Fine adjustment layer

Claims

1. In a multi-node cluster-based inference method using GPU-separated allocation for the pre-training and fine-tuning layers of a multi-deep learning model, The inference server's endpoint receives input values ​​from the client; The inference server's endpoint distributes the received input values ​​and transmits the first input values ​​to a first computing node containing a container that loads the neural network bundle from the first step; The first container of the first computing node receives a first input value and performs calculations using the neural network layer of the GPU to generate a first output value; The first container selects a container loaded with the neural network bundle for the next step, and transmits the first output value to a container that performs the next step or to a compute node that contains a container that performs the next step; If the second computing node is loaded with the neural network bundle for the next step, the second container of the second computing node takes the received first output value as input and performs calculations using the neural network layer of the GPU to generate a second output value; The step is repeated until the nth output value is generated from the nth container of the nth computing node, which contains the container loaded with the neural network bundle from the last step; The nth container transmits the nth output value as the final output value to the endpoint unit; and The endpoint unit includes a step of communicating the final output value to the client, The aforementioned GPU includes at least one GPU, meaning a physical GPU or a partitioned virtualized GPU fragment. The aforementioned GPU separately stores at least one pre-training layer and at least one fine-tuning layer of a model to which fine-tuning is applied using a partitioned virtualization technique. The aforementioned computing node includes an inference framework unit, and the inference framework unit includes an additional metadata management module. The aforementioned additional metadata management module loads the model parameter set, which is stored in a snapshot form of the parameter set in a fixed capacity with regard to the GPU capacity configuration for each neural network layer, into the GPU memory, recognizes the additional metadata contained in each layer of the model, divides it among the GPUs, and stores one or more models separately. The aforementioned GPU is characterized by either using a shared GPU for the pre-training layer and loading only the fine-tuning layer into separate GPU memory, or using a shared GPU for the pre-training layer and loading a large number of fine-tuning layers into a single GPU, thereby enabling a multi-node cluster-based inference method for the pre-training layer and fine-tuning layer of a multi-deep learning model through GPU-separated allocation.

2. In the step in which the first container selects a container loaded with the neural network bundle for the next step and transmits the first output value to a container that performs the next step or to a computing node that contains a container that performs the next step, The first container tracks the number of operations currently waiting and processing and the estimated time required for each GPU in the first GPU section of the same container via the performance metrics and performance measurement mechanisms of the inference framework constituting the first container, stores the number of operations and the estimated time required as scheduling hints, receives a list of containers that can perform the next step from the server control unit of the inference server, and selects a container to perform the next step based on the scheduling hints and the container list, characterized in that it is a multi-node cluster-based inference method by GPU isolation allocation for pre-training layers and fine-tuning layers of a multi-deep learning model according to claim 1.

3. The aforementioned pre-training layer and fine-tuning layer include additional metadata, The aforementioned additional metadata is A multi-node cluster-based inference method for pre-training and fine-tuning layers of a multi-deep learning model according to claim 1, characterized in that the data is in the form of a constant list or binary flag that can determine whether each neural network layer is a pre-training layer or a fine-tuning layer, and includes format information of output and input values ​​including the size of a matrix or vector and the representation format of individual elements, the version of each layer set and a unique ID, so that the pre-training layers and fine-tuning layers can be updated and distributed independently.

4. The first container includes the first inference framework section, The multi-node cluster-based inference method for a multi-deep learning model by GPU-separated allocation of pre-training layers and fine-tuning layers according to claim 1, characterized in that the first inference framework unit recognizes additional metadata, selects pre-training layers and fine-tuning layers in a single model, and loads them onto at least one GPU of the first GPU unit.

5. An endpoint unit that receives input values ​​from the client, distributes the input values, transmits the distributed input values ​​to a computing node containing a container loaded with the neural network bundle from the first step, receives output values ​​from the container, and transmits the final output value to the client; At least one compute node that receives input or output values, performs calculations using a GPU's neural network layer to generate output values, selects a container loaded with the neural network bundle for the next step, transmits the output values ​​to the container that performs the next step or to other compute nodes that contain the container that performs the next step, and, when the container loaded with the neural network bundle for the final step generates an output value, transmits this as the final output value to the endpoint; and Includes a server control unit that communicates a list of containers that can perform the next steps to the compute nodes, The aforementioned GPU includes at least one GPU, meaning a physical GPU or a partitioned virtualized GPU fragment. The aforementioned GPU stores separately at least one pre-training layer and at least one fine-tuning layer of the model to which fine-tuning is applied using a partitioned virtualization technique. The aforementioned computing node includes an inference framework unit, and the inference framework unit includes an additional metadata management module. The aforementioned additional metadata management module loads the model parameter set, which is stored in a snapshot form of the parameter set, into GPU memory, taking into account the GPU capacity configuration for each neural network layer. It then recognizes the additional metadata contained in each layer of the model and divides it among the GPUs to store one or more models separately. The aforementioned GPU is characterized by either using a shared GPU for the pre-training layer and loading only the fine-tuning layer into separate GPU memory, or using a shared GPU for the pre-training layer and loading a large number of fine-tuning layers into a single GPU, thereby enabling a multi-node cluster-based inference system through GPU-separated allocation of pre-training and fine-tuning layers for a multi-deep learning model.

6. The container tracks the number of currently waiting and processing operations and the estimated time required for each GPU in the GPU section of the same container via the performance metrics and performance measurement mechanisms of the inference framework constituting the container, saves the number of operations and the estimated time required as scheduling hints, receives a list of containers that can perform the next step from the server control unit of the inference server, and selects a container to perform the next step based on the scheduling hints and the container list, characterized in that it is a multi-node cluster-based inference system with GPU isolation allocation for pre-training and fine-tuning layers of a multi-deep learning model according to claim 5.

7. The aforementioned pre-training layer and fine-tuning layer include additional metadata, The aforementioned additional metadata is A multi-node cluster-based inference system for the pre-training and fine-tuning layers of a multi-deep learning model according to claim 5, characterized in that the data is in the form of a constant list or binary flag that can determine whether each neural network layer is a pre-training layer or a fine-tuning layer, and includes format information of output and input values ​​including the size of the matrix or vector and the representation format of individual elements, the version of each layer set and a unique ID, so that the pre-training layers and fine-tuning layers are updated and distributed independently.

8. The container includes an inference framework section, The inference framework unit recognizes additional metadata and selects pre-training layers and fine-tuning layers in a single model, and then divides and loads them onto at least one GPU in the GPU unit, characterized in that it is a multi-node cluster-based inference system with GPU-separated allocation of pre-training layers and fine-tuning layers of a multi-deep learning model according to claim 5.

Images