Server for performing stepwise learning on model parameters in order to implement multilingual LLM, and operating method thereof

Abstract

According to various embodiments, a server for performing stepwise learning on model parameters in order to implement a multilingual LLM comprises a memory, a communication module and a processor, wherein the processor can be configured to: generate a token set for learning in which an additional token set is merged with a predetermined token set by using a pre-learned tokenizer; perform subword-based embedding initialization on the additional token set; in each predetermined step, learn a plurality of fourth parameter groups used for learning an additional input embedding value of the additional token set and / or a plurality of fifth parameter groups used for learning an additional output embedding value of the additional token set step in a state in which a plurality of first parameter groups used for learning a transformer layer of the multilingual LLM, a plurality of second parameter groups used for learning a conventional input embedding value of the predetermined token set, and a plurality of third parameter groups used learning a conventional output embedding value of the predetermined token set are fixed; in each predetermined step, learn at least one from among the plurality of third parameter groups, the plurality of fourth parameter groups and the plurality of fifth parameter groups in a state in which the plurality of first parameter groups and the plurality of second parameter groups are fixed; and in each predetermined step, learn at least one from among the plurality of first parameter groups to the plurality of fifth parameter groups in a state in which the plurality of first parameter groups are not fixed. Other embodiments are also possible.

Description

A server and its operating method for performing step-by-step learning of model parameters to implement a multilingual LLM.

[0001] Various embodiments of the present disclosure relate to a server and its operating method that perform step-by-step learning of model parameters to implement a multilingual LLM.

[0002] Large Language Models (LLMs) are artificial intelligence models capable of understanding and generating human language by learning from large amounts of text data. These models are driving advancements in natural language processing (NLP) technology and are used in a variety of applications, including text generation, translation, summarization, question-answering, and sentiment analysis. At the heart of large language models are deep learning-based neural networks with billions of parameters, which effectively learn the grammatical structure and contextual meaning of text.

[0003] Large-scale language models can utilize tokenizers to process text data. Tokenizers are used to break text into smaller units called tokens, which are then used as input for the model. Tokenizers can typically be broken down into units at the word, subword, or character level. In particular, algorithms such as Byte Pair Encoding (BPE) utilize frequent text patterns to enable efficient tokenization. These tokenization techniques can contribute to improving the model's computational efficiency while preserving the text's meaning.

[0004] Large-scale language models typically undergo two stages: pretraining and fine-tuning. Pretraining involves learning common language patterns using large amounts of unsupervised learning data, while fine-tuning optimizes the model for a specific application or task.

[0005] Recently, the development of larger and more sophisticated models has led to a rapid increase in computational costs and memory usage. To address this, technologies such as distributed learning, quantization, and model compression are being introduced, and various approaches are being studied to improve performance while maintaining model efficiency. The advancement of these large-scale language models significantly expands the ability of artificial intelligence to understand and utilize text data, driving innovative changes in various industries.

[0006] Various embodiments of the present disclosure are intended to propose an efficient and effective method for expanding the vocabulary of a large-scale language model, and provide a method for training a language model by performing subword-based embedding initialization for the vocabulary of an expanded language and fixing a specific parameter group step by step so that a language model centered on a specific language (e.g., English) can be extended to other languages.

[0007] According to various embodiments, a server for performing step-by-step learning for model parameters to realize a multilingual LLM includes a memory, a communication module, and a processor, wherein the processor generates a learning token set by merging an additional token set with a predetermined token set using a pre-learned tokenizer, performs subword-based embedding initialization on the additional token set, and trains at least one of a plurality of fourth parameter groups used for learning additional input embedding values ​​of the additional token set or a plurality of fifth parameter groups used for learning additional output embedding values ​​of the additional token set in a predetermined step-by-step manner while fixing a plurality of first parameter groups used for learning a transformer layer of the multilingual LLM, a plurality of second parameter groups used for learning existing input embedding values ​​of the predetermined token set, and a plurality of third parameter groups used for learning existing output embedding values ​​of the predetermined token set, and while fixing the plurality of first parameter groups and the plurality of second parameter groups, trains at least one of the plurality of third parameter groups, the plurality of fourth parameter groups, or the plurality of fifth parameter groups in a predetermined step-by-step manner. It can be set to learn at least one of the parameter groups, and to learn at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups.

[0008] According to various embodiments, a method of operating a server that performs step-by-step learning for model parameters to realize a multilingual LLM comprises: generating a learning token set by merging an additional token set with a predetermined token set using a pre-learned tokenizer, and performing subword-based embedding initialization on the additional token set; fixing a plurality of first parameter groups used for learning a transformer layer of a multilingual LLM, a plurality of second parameter groups used for learning existing input embedding values ​​of the predetermined token set, and a plurality of third parameter groups used for learning existing output embedding values ​​of the predetermined token set, and training at least one of a plurality of fourth parameter groups used for learning additional input embedding values ​​of the additional token set or a plurality of fifth parameter groups used for learning additional output embedding values ​​of the additional token set for each predetermined step; fixing the plurality of first parameter groups and the plurality of second parameter groups, and training at least one of the plurality of third parameter groups, the plurality of fourth parameter groups, or the plurality of fifth parameter groups for each predetermined step; It may include an operation of learning at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups.

[0009] The present disclosure can provide an effect of improving the processing performance of a multilingual large-scale language model (LLM) for a non-English language such as Korean by efficiently and effectively expanding the vocabulary of the model through parameter fixation, subword-based embedding initialization for input embedding / output embedding of additional tokens, and step-by-step parameter learning.

[0010] FIG. 1 illustrates a block diagram of a user device and a server according to various embodiments of the present disclosure.

[0011] FIG. 2 is a flowchart illustrating an operation in which a server performs step-by-step learning on model parameters of a multilingual LLM according to various embodiments.

[0012] FIG. 3 is a diagram illustrating an embodiment in which a server performs step-by-step learning on model parameters of a multilingual LLM according to various embodiments.

[0013] FIG. 4 is a diagram illustrating an embodiment of predicting an output embedding for an input embedding in a process of learning only the input embedding of an additional token of a multilingual LLM according to various embodiments.

[0014] FIG. 5 is a diagram illustrating the performance of a multilingual LLM after completing step-by-step learning of model parameters of the multilingual LLM according to various embodiments.

[0015] Hereinafter, various embodiments of the present document will be described with reference to the attached drawings. It should be understood that the embodiments and the terms used therein are not intended to limit the technology described in the present document to a specific embodiment, but rather include various modifications, equivalents, and / or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B" or "at least one of A and / or B" may include all possible combinations of the items listed together. Expressions such as "first," "second," "first," or "second," may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0016] In this document, "configured to" may be used interchangeably with, for example, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to," either in hardware or software. In some contexts, the phrase "a device configured to" may mean that the device is "capable of" doing something together with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in a memory device.

[0017] A user device or electronic device according to various embodiments of the present document may include, for example, at least one of a smartphone, a tablet PC, a desktop PC, a laptop PC, a netbook computer, a workstation, and a server.

[0018] Referring to FIG. 1, a user device (100) and a server (101) in various embodiments are described. The user device (100) may include a communication module (110), a processor (120), a memory (130), and a display (140). In some embodiments, the user device (100) may omit at least one of the components or additionally include other components.

[0019] The communication module (110) can establish communication between, for example, the user device (100) and an external device (e.g., a first external electronic device (102), a second external electronic device (104), or a server (101)). For example, the communication module (110) can be connected to a network (180) via wireless communication or wired communication to communicate with the external device (e.g., a second external electronic device (104) or a server (101)).

[0020] The wireless communication may include, for example, cellular communication using at least one of LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications). In one embodiment, the wireless communication may include, for example, at least one of WiFi (wireless fidelity), Bluetooth, Bluetooth low energy (BLE), Zigbee, near field communication (NFC), Magnetic Secure Transmission, radio frequency (RF), or body area network (BAN). In one embodiment, the wireless communication may include GNSS. The GNSS may be, for example, GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), Beidou Navigation Satellite System (hereinafter "Beidou"), or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS." Wired communication may include at least one of, for example, USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).The network (180) may include at least one of a telecommunications network, for example, a computer network (e.g., a LAN or WAN), the Internet, or a telephone network.

[0021] The processor (120) may include one or more of a central processing unit, an application processor, or a communication processor (CP). The processor (120) may, for example, perform operations or data processing related to control and / or communication of at least one other component of the user device (100).

[0022] The memory (130) may include volatile and / or non-volatile memory. The memory (130) may store, for example, commands or data related to at least one other component of the user device (100).

[0023] The display (140) may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. The display (140) may, for example, display various contents (e.g., text, images, videos, icons, and / or symbols) to the user. The display (160) may include a touch screen and may receive touch, gesture, proximity, or hovering inputs using, for example, an electronic pen or a part of the user's body.

[0024] Each of the first and second external electronic devices (102, 104) may be the same or a different type of device as the user device (100). According to various embodiments, all or part of the operations executed in the user device (100) may be executed in another one or more electronic devices (e.g., electronic devices (102, 104) or server (101). According to one embodiment, when the user device (100) needs to perform a certain function or service automatically or upon request, the user device (100) may request at least some functions related thereto from another device (e.g., electronic devices (102, 104) or server (101)) instead of executing the function or service by itself or in addition. The other electronic device (e.g., electronic devices (102, 104) or server (101)) may execute the requested function or additional function and transmit the result to the user device (100). The user device (100) may process the received result as is or additionally to provide the requested function or service. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0025] The server (101) may include a communication module (111), a processor (121), and a memory (131). In some embodiments, the server (101) may omit at least one of the components or additionally include other components. The communication module (111), the processor (121), and the memory (131) may perform the same functions as the communication module (110), the processor (120), and the memory (130) within the user device (100), respectively.

[0026]

[0027] FIG. 2 is a flowchart illustrating an operation in which a server (e.g., server (101) of FIG. 1) performs step-by-step learning on model parameters of a multilingual LLM according to various embodiments.

[0028] FIG. 3 is a diagram illustrating an embodiment in which a server (101) performs step-by-step learning on model parameters of a multilingual LLM according to various embodiments.

[0029] FIG. 4 is a diagram illustrating an embodiment of predicting an output embedding for an input embedding in a process of learning only the input embedding of an additional token of a multilingual LLM according to various embodiments.

[0030] According to various embodiments, the server (101) may operate an application composed of a plurality of execution screens or a website composed of a plurality of web pages, communicate with a user device (e.g., an electronic device (100, 102, 104) of FIG. 1) (e.g., a PC, a laptop, a smartphone, etc.) through a network (162, 164), process a request received from the user device (100) regarding the application or the web page, and transmit the requested information to the user device (100). The server (101) may transmit a source code that enables each execution screen of a dedicated application or website to be displayed on the user device (100), and the user device (101) may receive the source code and display an execution screen requested by a user of the user device (100) through the dedicated application or a web browser. According to one embodiment, the server (101) may include the same types of components as the components of the electronic device (100) of FIG. 1. According to one embodiment, the components referred to as user devices (100) in the present disclosure may refer to a user account that accesses a platform provided by the server (101) through the user device.

[0031]

[0032] In operation 201, according to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1) may generate a learning token set by merging an additional token set with a predetermined token set using a pre-trained tokenizer for use in learning a multilingual LLM, and perform subword-based embeddings initialization on the additional token set.

[0033] According to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1) may generate a learning token set to be used for learning a multilingual LLM by merging an additional token set with a predetermined token set.

[0034] In one embodiment, the predetermined token set may be a token set for vocabulary of a specific language (e.g., English) that has been previously learned by a multilingual LLM. For example, with respect to the English sentence "Hello, the weather is nice today.", the server (101) may use a pre-learned tokenizer to tokenize the English sentence into eight tokens, '_Hello', ',', '_the', '_weather', '_is', '_nice', '_today', and '.', and include them in the predetermined token set.

[0035] In one embodiment, the additional token set may be a token set for vocabulary of another language (e.g., Korean) that will be newly learned by the multilingual LLM. For example, with respect to the Korean sentence "Hello, the weather is nice today," the server (101) may use a pre-trained tokenizer to tokenize the Korean sentence into nine tokens: "_안", "녕", "하세요", ",", "_오늘은", "_날가", "_좋", "네", and ".", and include them in the additional token set.

[0036] According to one embodiment, the server (101) may pre-train a tokenizer based on tokens that appear more than a predetermined frequency within a specific corpus and tokens selected by an administrator of the server (101). At this time, the method for training the tokenizer may include a whitespace tokenizer, a basic regular expression tokenizer, a BPE (Byte Pair Encoding), a Wordpiece, or a SentencePiece-based learning method, and various embodiments according to the present disclosure are not limited to the above-described examples and may utilize various learning methods.

[0037] According to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1 ) may perform subword-based embedding initialization for an additional token set. Through subword-based embedding initialization, the server (101) may decompose the vocabulary of a language to be newly learned into smaller units, thereby enabling the multilingual LLM to more accurately understand and learn the characteristics of the new language. Here, the embedding initialization for a token may refer to a process of setting an initial value of an embedding vector of the token.

[0038] According to one embodiment, the server (101) may initialize the input embedding value of the first token of the additional token set to the average embedding value of at least one token corresponding to a subword of the first token among a predetermined token set as an input embedding initialization process of the additional token set. For example, the server (101) may set the input embedding initial value of the first token (e.g., 'do') of the additional token set to the average embedding value of each of three tokens (e.g., 'ha', 'se', 'yo') corresponding to the subword of the first token among the predetermined token set.

[0039] According to one embodiment, the server (101) may initialize the output embedding value of the first token of the additional token set to the output embedding value of the first subword token of the first token as an output embedding initialization process of the additional token set. For example, the server (101) may set the output embedding initialization value of the first token (e.g., "do") of the additional token set to the output embedding value of the first subword token (e.g., "ha") of the first token.

[0040]

[0041] In operation 203, according to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1) may fix a plurality of first parameter groups used for learning a transformer layer of a multilingual LLM, a plurality of second parameter groups used for learning existing input embedding values ​​of a predetermined token set, and a plurality of third parameter groups used for learning existing output embedding values ​​of a predetermined token set, and may train at least one of a plurality of fourth parameter groups used for learning additional input embedding values ​​of an additional token set or a plurality of fifth parameter groups used for learning additional output embedding values ​​of an additional token set, at each predetermined step. Here, fixing a parameter group means excluding a parameter group from learning, which may mean not updating a weight of the parameter group.

[0042] According to one embodiment, referring to FIG. 3, the server (101) may fix a plurality of first parameter groups (310) used for training a transformer layer of a multilingual LLM (301), a plurality of second parameter groups (320) used for training existing input embedding values ​​of a predetermined token set, and a plurality of third parameter groups (330) used for training existing output embedding values ​​of a predetermined token set before training an additional token set to the multilingual LLM (301). The existing input embedding values ​​of the predetermined token set may represent the embedding values ​​of the predetermined token set that have been trained as input embedding values ​​for the multilingual LLM (301) before training a new language to the multilingual LLM (301), and the existing output embedding values ​​of the predetermined token set may represent the embedding values ​​of the predetermined token set that have been trained as output embedding values ​​for the multilingual LLM (301) before training a new language to the multilingual LLM (301).

[0043] According to one embodiment, the server (101) can train the plurality of fourth parameter groups (340) while fixing the plurality of first parameter groups (310), the plurality of second parameter groups (320), the plurality of third parameter groups (330), and the plurality of fifth parameter groups (350). For example, referring to FIG. 3, as a first learning step of the multilingual LLM (301), the server (101) can train only the plurality of fourth parameter groups (340) while fixing the plurality of first parameter groups (310), the plurality of second parameter groups (320), the plurality of third parameter groups (330), and the plurality of fifth parameter groups (350). Through the first learning step, the multilingual LLM (301) can train only the plurality of fourth parameter groups (340) while fixing all other model parameters, thereby enabling the model to recognize the new token and establish an initial alignment between the input and output tokens, and can lay the foundation for the new additional token to be naturally integrated into the model.

[0044] According to one embodiment, the server (101) can train the multilingual LLM (301) to predict the additional input embedding value of a specific token among the additional token set as the additional output embedding value of the first subword token of the specific token by performing an output embedding initialization process of the additional token set. For example, referring to FIG. 4, as a first learning step of the multilingual LLM (301), while fixing a plurality of first parameter groups (310), a plurality of second parameter groups (320), a plurality of third parameter groups (330), and a plurality of fifth parameter groups (350), the multilingual LLM (301) can train the multilingual LLM (301) to output the additional output embedding value of the first subword token (e.g., 'ha') of the specific token when receiving an additional input embedding value (341) of a specific token (e.g., 'do') as an additional token set. In other words, the multilingual LLM (301) can be trained such that the last hidden representation for predicting the additional output embedding value of a specific additional token (e.g., 'do') in the first learning stage of the multilingual LLM (301) is aligned with the additional output embedding value of the first subword token of the specific additional token (e.g., 'ha').

[0045] According to one embodiment, the server (101) can train the plurality of fifth parameter groups (350) while fixing the plurality of first parameter groups (310), the plurality of second parameter groups (320), the plurality of third parameter groups (330), and the plurality of fourth parameter groups (340). For example, referring to FIG. 3, in a second learning step following the first learning step of the multilingual LLM (301), the server (101) can train only the plurality of fifth parameter groups (350) while fixing the plurality of first parameter groups (310), the plurality of second parameter groups (320), the plurality of third parameter groups (330), and the plurality of fourth parameter groups (340). Through the second learning step, the multilingual LLM (301) can adjust the additional output embeddings so that the model can accurately generate the new additional tokens in various contexts by fixing all other model parameters and learning only the additional output embeddings of the new additional tokens.

[0046] According to one embodiment, the server (101) can train a plurality of fourth parameter groups (340) and a plurality of fifth parameter groups (350) while fixing a plurality of first parameter groups (310), a plurality of second parameter groups (320), and a plurality of third parameter groups (330). For example, referring to FIG. 3, as a third learning step following the second learning step of the multilingual LLM (301), the server (101) can train a plurality of fourth parameter groups (340) and a plurality of fifth parameter groups (350) while fixing a plurality of first parameter groups (310), a plurality of second parameter groups (320), and a plurality of third parameter groups (330). Multilingual LLM (301) can improve the alignment between the additional input and additional output embeddings by simultaneously learning the additional input embedding and additional output embedding of the new additional token through the third learning stage, so that the new additional token can be used consistently in both input and output.

[0047]

[0048] In operation 205, according to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1) may train at least one of a plurality of third parameter groups, a plurality of fourth parameter groups, or a plurality of fifth parameter groups in predetermined steps while fixing a plurality of first parameter groups and a plurality of second parameter groups.

[0049] According to one embodiment, the server (101) can train a plurality of third parameter groups (330) and a plurality of fifth parameter groups (350) while fixing a plurality of first parameter groups (310), a plurality of second parameter groups (320), and a plurality of fourth parameter groups (340). For example, referring to FIG. 3, as a fourth learning step following the third learning step of the multilingual LLM (301), the server (101) can train a plurality of third parameter groups (330) and a plurality of fifth parameter groups (350) while fixing a plurality of first parameter groups (310), a plurality of second parameter groups (320), and a plurality of fourth parameter groups (340). Multilingual LLM (301) can adjust the scale (logit scale) between existing vocabulary and newly added vocabulary by learning the existing output embedding of a predetermined token set corresponding to the existing vocabulary and the additional output embedding of an additional token set corresponding to the newly added vocabulary together through the fourth learning stage.

[0050] According to one embodiment, the server (101) can train a plurality of third parameter groups (330), a plurality of fourth parameter groups (340), and a plurality of fifth parameter groups (350) while keeping a plurality of first parameter groups (310) and a plurality of second parameter groups (320) fixed. For example, referring to FIG. 3, in the fifth learning step following the fourth learning step of the multilingual LLM (301), the server (101) can train a plurality of third parameter groups (330), a plurality of fourth parameter groups (340), and a plurality of fifth parameter groups (350) while keeping a plurality of first parameter groups (310) and a plurality of second parameter groups (320) fixed. Multilingual LLM (301) can optimize the expanded vocabulary to work smoothly in the overall language understanding and generation process of the model by learning additional input embeddings of additional token sets and output embeddings of the entire vocabulary (existing vocabulary and additional vocabulary) together through the fifth learning stage.

[0051]

[0052] In operation 207, according to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1) can learn at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups.

[0053] According to one embodiment, the server (101) can train a plurality of first parameter groups (310), a plurality of second parameter groups (320), a plurality of third parameter groups (330), a plurality of fourth parameter groups (340), and a plurality of fifth parameter groups (350) without fixing a plurality of first parameter groups (310). For example, referring to FIG. 3, in a sixth learning step following the fifth learning step of the multilingual LLM (301), the server (101) can train a plurality of first parameter groups (310), a plurality of second parameter groups (320), a plurality of third parameter groups (330), a plurality of fourth parameter groups (340), and a plurality of fifth parameter groups (350) without fixing a plurality of first parameter groups (310). Multilingual LLM (301) can learn all parameters of the model through the 6th learning stage, thereby adjusting not only the input and output embeddings but also the internal layers, thereby optimizing the overall performance of the model.

[0054] According to one embodiment, the server (101) can train the plurality of first parameter groups (310) while fixing the plurality of second parameter groups (320) to the plurality of fifth parameter groups (350) without fixing the plurality of first parameter groups (310). For example, referring to FIG. 3, in the seventh learning step following the sixth learning step of the multilingual LLM (301), the server (101) can train only the plurality of first parameter groups (310) while fixing the plurality of second parameter groups (320) to the plurality of fifth parameter groups (350) without fixing the plurality of first parameter groups (310). Through the seventh learning step, the multilingual LLM (301) can adjust so that new vocabulary is reflected in detail in the internal processing of the model by learning only the parameters of the internal transformer layer of the model.

[0055] According to various embodiments, the server (101) (e.g., the processor (121) of FIG. 1) may terminate the corresponding learning step and perform the next learning step if the training loss for the learning data (corpus) satisfies a predetermined condition in each learning step (e.g., the first to seventh learning steps). Specifically, the server (101) may set a maximum learning step (e.g., 400 global steps) regarding the maximum number of parameter updates for each learning step of the multilingual LLM (301), and may terminate the corresponding learning step when the number of parameter updates in each learning step of the multilingual LLM (301) reaches the maximum learning step or, even if it does not reach the maximum learning step, it is determined that the training loss has converged (i.e., the training loss has not improved during a specific step). Meanwhile, the server (101) may terminate model learning if the learning of the model is completed in the seventh learning step.

[0056]

[0057] FIG. 5 is a diagram illustrating the performance of a multilingual LLM after completing step-by-step learning of model parameters of the multilingual LLM according to various embodiments.

[0058] According to various embodiments, referring to FIG. 5, multilingual LLMs learned according to the present disclosure (e.g., multilingual LLM (301) of FIG. 3), pre-trained (PT) models (e.g., EEVE-Korean-10.8B-v1.0 model, EEVE-Korean-2.8B-v1.0 model) and fine-tuned (FT) models (e.g., EEVE-Korean-Instruct-10.8B model, EEVE-Korean-Instruct-2.8B model) show similar or slightly higher performance on English tasks than other models, while showing much higher performance on Korean tasks than other models, compared to models focused on Korean tasks (e.g., OPEN-SOLAR-KO-10.7B, etc.).

[0059] In other words, it can be seen that the multilingual LLM (301) learned according to the present disclosure has significantly improved Korean task performance while maintaining the English task performance of the existing English-centered learned LLM, and it can be seen that it maintains high performance through efficient vocabulary expansion even in a small-sized model (2.8B).

[0060]

[0061] According to various embodiments, a server for performing step-by-step learning for model parameters to realize a multilingual LLM includes a memory, a communication module, and a processor, wherein the processor generates a learning token set by merging an additional token set with a predetermined token set using a pre-learned tokenizer, performs subword-based embedding initialization on the additional token set, and trains at least one of a plurality of fourth parameter groups used for learning additional input embedding values ​​of the additional token set or a plurality of fifth parameter groups used for learning additional output embedding values ​​of the additional token set in a predetermined step-by-step manner while fixing a plurality of first parameter groups used for learning a transformer layer of the multilingual LLM, a plurality of second parameter groups used for learning existing input embedding values ​​of the predetermined token set, and a plurality of third parameter groups used for learning existing output embedding values ​​of the predetermined token set, and while fixing the plurality of first parameter groups and the plurality of second parameter groups, trains at least one of the plurality of third parameter groups, the plurality of fourth parameter groups, or the plurality of fifth parameter groups in a predetermined step-by-step manner. It can be set to learn at least one of the parameter groups, and to learn at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups.

[0062] According to various embodiments, the processor may be configured to initialize, as an input embedding initialization process of the additional token set, an input embedding value of a first token of the additional token set to an average embedding value of at least one token corresponding to a subword of the first token among the predetermined token sets, and, as an output embedding initialization process of the additional token set, to initialize an output embedding value of the first token of the additional token set to an output embedding value of a first subword token of the first token.

[0063] According to various embodiments, the processor may be configured to train the plurality of fourth parameter groups while fixing the plurality of first parameter groups, the plurality of second parameter groups, the plurality of third parameter groups, and the plurality of fifth parameter groups as a first learning step of the multilingual LLM.

[0064] According to various embodiments, the processor may be configured to train the plurality of fifth parameter groups while fixing the plurality of first parameter groups, the plurality of second parameter groups, the plurality of third parameter groups, and the plurality of fourth parameter groups as a second learning step of the multilingual LLM.

[0065] According to various embodiments, the processor may be configured to train the plurality of fourth parameter groups and the plurality of fifth parameter groups while fixing the plurality of first parameter groups, the plurality of second parameter groups, and the plurality of third parameter groups as a third learning step of the multilingual LLM.

[0066] According to various embodiments, the processor may be configured to train the plurality of third parameter groups and the plurality of fifth parameter groups in a state where the plurality of first parameter groups, the plurality of second parameter groups, and the plurality of fourth parameter groups are fixed as a fourth learning step of the multilingual LLM, train the plurality of third parameter groups, the plurality of fourth parameter groups, and the plurality of fifth parameter groups in a state where the plurality of first parameter groups and the plurality of second parameter groups are fixed, train the plurality of third parameter groups, the plurality of fourth parameter groups, and the plurality of fifth parameter groups in a state where the plurality of first parameter groups are not fixed as a sixth learning step of the multilingual LLM, train the plurality of first parameter groups, the plurality of second parameter groups, the plurality of third parameter groups, the plurality of fourth parameter groups, and the plurality of fifth parameter groups, and train the plurality of first parameter groups in a state where the plurality of first parameter groups are not fixed and the plurality of second parameter groups to the plurality of fifth parameter groups are fixed, and train only the plurality of first parameter groups in a state where the plurality of first parameter groups are not fixed and the plurality of second parameter groups to the plurality of fifth parameter groups are fixed as a seventh learning step of the multilingual LLM.

[0067] According to various embodiments, a method of operating a server that performs step-by-step learning for model parameters to realize a multilingual LLM comprises: generating a learning token set by merging an additional token set with a predetermined token set using a pre-learned tokenizer, and performing subword-based embedding initialization on the additional token set; fixing a plurality of first parameter groups used for learning a transformer layer of a multilingual LLM, a plurality of second parameter groups used for learning existing input embedding values ​​of the predetermined token set, and a plurality of third parameter groups used for learning existing output embedding values ​​of the predetermined token set, and training at least one of a plurality of fourth parameter groups used for learning additional input embedding values ​​of the additional token set or a plurality of fifth parameter groups used for learning additional output embedding values ​​of the additional token set for each predetermined step; fixing the plurality of first parameter groups and the plurality of second parameter groups, and training at least one of the plurality of third parameter groups, the plurality of fourth parameter groups, or the plurality of fifth parameter groups for each predetermined step; It may include an operation of learning at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups.

[0068]

[0069] The term "module" or "part" used in this document includes a unit composed of hardware, software, or firmware, and can be used interchangeably with terms such as logic, logic block, component, or circuit, for example. The "module" or "part" can be an integrally configured component or a minimum unit or a part thereof that performs one or more functions. The "module" or "part" can be implemented mechanically or electronically, and can include, for example, an ASIC (application-specific integrated circuit) chip, FPGAs (field-programmable gate arrays), or a programmable logic device, known or to be developed in the future, that performs certain operations, and can be executed by the processor (120). At least a part of the device (e.g., modules or functions thereof) or method (e.g., operations) according to various embodiments can be implemented as instructions stored in a computer-readable storage medium (e.g., memory (130)) in the form of a program module. When the above command is executed by a processor (e.g., processor (120)), the processor can perform a function corresponding to the command. The computer-readable recording medium may include a hard disk, a floppy disk, a magnetic medium (e.g., a magnetic tape), an optical recording medium (e.g., a CD-ROM, a DVD, a magneto-optical medium (e.g., a floptical disk), a built-in memory, etc. The command may include a code generated by a compiler or a code executable by an interpreter. A module or program module according to various embodiments may include at least one or more of the above-described components, some of which may be omitted, or other components may be further included. Operations performed by a module, a program module, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0070] The embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content, and do not limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be interpreted to include all modifications or various other embodiments based on the technical concepts of the present disclosure.

Claims

1. In a server that performs step-by-step learning of model parameters to realize multilingual LLM, memory, Communication module, and Contains a processor, The above processor, Generate a training token set by merging an additional token set with a pre-defined token set using a pre-trained tokenizer, and perform subword-based embedding initialization on the additional token set. While fixing a plurality of first parameter groups used for learning a transformer layer of a multilingual LLM, a plurality of second parameter groups used for learning existing input embedding values ​​of the predetermined token set, and a plurality of third parameter groups used for learning existing output embedding values ​​of the predetermined token set, at least one of a plurality of fourth parameter groups used for learning additional input embedding values ​​of the additional token set or a plurality of fifth parameter groups used for learning additional output embedding values ​​of the additional token set is trained at each predetermined step, While fixing the plurality of first parameter groups and the plurality of second parameter groups, learning at least one of the plurality of third parameter groups, the plurality of fourth parameter groups, or the plurality of fifth parameter groups in predetermined steps, and Set to learn at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups. Server.

2. In paragraph 1, The above processor, As an input embedding initialization process of the additional token set, the input embedding value of the first token of the additional token set is initialized to the average embedding value of at least one token corresponding to a subword of the first token among the predetermined token sets, and As an output embedding initialization process of the additional token set, the output embedding value of the first token of the additional token set is set to be initialized to the output embedding value of the first subword token of the first token. Server.

3. In paragraph 1, The above processor, As a first learning step of the multilingual LLM, the plurality of first parameter groups, the plurality of second parameter groups, the plurality of third parameter groups, and the plurality of fifth parameter groups are fixed, and the plurality of fourth parameter groups are set to be learned. Server.

4. In paragraph 3, The above processor, As a second learning step of the multilingual LLM, the plurality of first parameter groups, the plurality of second parameter groups, the plurality of third parameter groups, and the plurality of fourth parameter groups are fixed, and the plurality of fifth parameter groups are set to be learned. Server.

5. In paragraph 4, The above processor, As a third learning step of the multilingual LLM, the plurality of first parameter groups, the plurality of second parameter groups, and the plurality of third parameter groups are fixed, and the plurality of fourth parameter groups and the plurality of fifth parameter groups are set to be learned. Server.

6. In paragraph 5, The above processor, As a fourth learning step of the multilingual LLM, while fixing the plurality of first parameter groups, the plurality of second parameter groups, and the plurality of fourth parameter groups, the plurality of third parameter groups and the plurality of fifth parameter groups are learned, As a fifth learning step of the multilingual LLM, while fixing the plurality of first parameter groups and the plurality of second parameter groups, the plurality of third parameter groups, the plurality of fourth parameter groups, and the plurality of fifth parameter groups are learned, As a sixth learning step of the multilingual LLM, while not fixing the plurality of first parameter groups, the plurality of second parameter groups, the plurality of third parameter groups, the plurality of fourth parameter groups, and the plurality of fifth parameter groups are learned, and As the seventh learning step of the multilingual LLM, it is set to learn only the plurality of first parameter groups while fixing the plurality of second parameter groups to the plurality of fifth parameter groups without fixing the plurality of first parameter groups. Server.

7. In the operating method of a server that performs step-by-step learning of model parameters to realize multilingual LLM, An operation of generating a training token set by merging an additional token set with a pre-determined token set using a pre-trained tokenizer, and performing subword-based embedding initialization on the additional token set; An operation of training at least one of a plurality of fourth parameter groups used for training additional input embedding values ​​of the additional token set or a plurality of fifth parameter groups used for training additional output embedding values ​​of the additional token set, while fixing a plurality of first parameter groups used for training a transformer layer of a multilingual LLM, a plurality of second parameter groups used for training existing input embedding values ​​of the predetermined token set, and a plurality of third parameter groups used for training existing output embedding values ​​of the predetermined token set, at each predetermined step. An operation of learning at least one of the plurality of third parameter groups, the plurality of fourth parameter groups, or the plurality of fifth parameter groups in predetermined steps while fixing the plurality of first parameter groups and the plurality of second parameter groups, and An operation of learning at least one of the plurality of first parameter groups to the plurality of fifth parameter groups in predetermined steps without fixing the plurality of first parameter groups, How the server works.

Images