Artificial intelligence device, data augmentation method of artificial intelligence device, and non-volatile recording medium

Abstract

A data augmentation method according to one embodiment of the present disclosure comprises the steps of: acquiring a text data set for training of a large language model; acquiring a context length indicating the number of tokens included in one context; and generating, from the acquired text data set, a plurality of overlapping contexts having the context length, wherein at least two of the plurality of overlapping contexts can share one or more tokens with each other.

Description

Artificial intelligence device, data augmentation method of artificial intelligence device, and non-volatile recording medium

[0001] The present disclosure relates to a method for augmenting a data set for training a large language model.

[0002] A Large Language Model (LLM) is an artificial intelligence (AI) model that can learn from large amounts of data to understand and generate text in a human-like manner.

[0003] Large-scale language models are used as a fundamental technology to understand intent from natural language and generate text in various application fields, including coding assistants for code recommendations, code explanations, documentation writing, and test code writing, AI Contact Centers (AICC) such as customer service bots, and more.

[0004] Large-scale language models are trained in several stages, depending on the scale and type of data. For example, large-scale language models can be trained in an initial stage using extremely large data sets from various fields, such as natural language text, source code data, and mathematical formula data. A mid-stage training process uses large-scale source code data written in a specific programming language. A later stage uses small-scale, in-house source code data.

[0005] When using large language models to generate outputs such as code recommendations and explanations, there are cases where the training data set is insufficient. This insufficient training data leads to poor performance of the large language model.

[0006] Specifically, the problem arises that the model parameters of the giant language model do not change sufficiently, or the diversity of the source code data is insufficient, resulting in learning biased towards the training data.

[0007] The purpose of the present disclosure may be to enhance the performance of a large-scale language model by augmenting a data set for training the large-scale language model.

[0008] The purpose of the present disclosure may be to augment data to enable fine tuning of a large language model for a data domain lacking in the construction of a data set for a large language model.

[0009] The purpose of the present disclosure may be to increase the performance of a large-scale language model by securing diversity in a training data set for learning the large-scale language model.

[0010] An object of the present disclosure may be to efficiently use memory by determining a storage method for contexts used in learning a large language model.

[0011] A method for augmenting data according to one embodiment of the present disclosure comprises the steps of: obtaining a text data set for training a large language model; obtaining a context length indicating the number of tokens included in one context; and generating a plurality of nested contexts having the context length from the obtained text data set, wherein at least two of the plurality of nested contexts can share one or more tokens with each other.

[0012] An artificial intelligence device according to one embodiment of the present disclosure comprises: a memory storing a text data set for learning a large language model; and one or more processors for obtaining a context length indicating the number of tokens included in one context and generating a plurality of nested contexts having the context length from the obtained text data set, wherein at least two of the plurality of nested contexts can share one or more tokens with each other.

[0013] A computer-readable non-volatile recording medium having recorded thereon a program for performing a data augmentation method according to an embodiment of the present disclosure, the augmentation method comprising: obtaining a text data set for learning a large language model; obtaining a context length indicating the number of tokens included in one context; and generating a plurality of nested contexts having the context length from the obtained text data set, wherein at least two of the plurality of nested contexts can share one or more tokens with each other.

[0014] According to an embodiment of the present disclosure, the performance of a large language model can be significantly improved by augmenting a data set for learning the large language model.

[0015] According to an embodiment of the present disclosure, the performance of a large language model can be improved by securing diversity in a training data set for learning a large language model.

[0016] According to an embodiment of the present disclosure, memory can be used efficiently as the storage method for contexts used in learning a large language model is determined.

[0017] FIG. 1 is a block diagram illustrating components of an artificial intelligence device according to one embodiment of the present disclosure.

[0018] FIG. 2 is a diagram for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0019] FIG. 3a is a diagram illustrating text files used for learning a large language model, FIG. 3b is a diagram illustrating a method for generating contexts by packing text files according to a conventional technique, and FIG. 3c is a diagram illustrating a process for predicting the next token for each context according to a conventional technique.

[0020] FIG. 4 is a flowchart for explaining an operation method of an artificial intelligence device according to an embodiment of the present disclosure.

[0021] FIGS. 5A and 5B are diagrams illustrating a process of generating multiple contexts from a large text file using a sliding window method according to one embodiment of the present disclosure.

[0022] FIG. 6 is a drawing illustrating an indexing method according to one embodiment of the present disclosure.

[0023] FIG. 7 is a drawing illustrating a random sampling method according to one embodiment of the present disclosure.

[0024] Figure 8 is a flowchart illustrating a method for determining a context storage method based on the capacity of memory.

[0025] FIG. 9 is a diagram illustrating a process for determining a context generation method based on the type of text data set according to one embodiment of the present disclosure.

[0026] Figures 10a and 10b are drawings explaining the process of inserting spaces to prevent them from being cut off by function or class unit when packing to create a context.

[0027] FIG. 11 is a diagram comparing the performance of learning a large language model using a non-overlapping context generation method according to a prior art and the performance of learning a large language model using a nested context generation method according to an embodiment of the present disclosure.

[0028] FIG. 12a is a diagram comparing the performance of learning a first large language model using a non-overlapping context generation method according to a prior art and the performance of learning a first large language model using a nested context generation method according to an embodiment of the present disclosure.

[0029] FIG. 12b is a diagram comparing the performance of learning a second giant language model using a non-overlapping context generation method according to a prior art and the performance of learning a second giant language model using a nested context generation method according to an embodiment of the present disclosure.

[0030] FIGS. 13a to 13c are drawings showing the diversity of contexts generated when using a method for generating overlapping contexts according to an embodiment of the present disclosure.

[0031] Artificial intelligence refers to a field that studies artificial intelligence or the methodologies for creating it, and machine learning refers to a field that defines various problems in the field of artificial intelligence and studies the methodologies for solving them.

[0032] Machine learning is sometimes defined as an algorithm that improves its performance on a task through continuous experience.

[0033] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities that is composed of artificial neurons (nodes) that form a network through the combination of synapses.

[0034] An artificial neural network can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates the output values.

[0035] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0036] Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0037] The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

[0038] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0039] Supervised learning refers to a method of training an artificial neural network given labels for training data. The labels can refer to the correct answer (or result value) that the artificial neural network must infer when training data is input to the artificial neural network.

[0040] Unsupervised learning can refer to a method of training an artificial neural network without being given labels for the training data.

[0041] Reinforcement learning can refer to a learning method that teaches an agent defined in an environment to select an action or action sequence that maximizes the cumulative reward in each state.

[0042] Among artificial neural networks, machine learning implemented with a deep neural network (DNN) that includes multiple hidden layers is also called deep learning, and deep learning is a part of machine learning.

[0043] Hereinafter, machine learning is used to mean deep learning.

[0044] FIG. 1 is a block diagram illustrating components of an artificial intelligence device according to one embodiment of the present disclosure.

[0045] The artificial intelligence device (100) can be implemented as a fixed device or a movable device, such as a TV, a projector, a mobile phone, a smart phone, a desktop computer, a laptop, a digital broadcasting terminal, a PDA (personal digital assistant), a PMP (portable multimedia player), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, digital signage, a robot, a vehicle, etc.

[0046] Referring to FIG. 1, an artificial intelligence device (100) may include a communication interface (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180).

[0047] The communication interface (110) can transmit and receive data with external devices such as other artificial intelligence devices or AI servers (200) using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

[0048] Communication technologies used by the communication interface (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0049] The input interface (120) can obtain various types of data.

[0050] The input interface (120) may include a camera (121) for capturing images, a microphone (122) for receiving audio signals, and a user input interface (123) for receiving information from a user.

[0051] By treating the camera (121) or microphone (122) as a sensor, the signal obtained from the camera (121) or microphone (122) can be called sensing data or sensor information.

[0052] The input interface (120) can acquire input data to be used when obtaining output using learning data and a learning model for model learning. The input interface (120) can also acquire raw input data, in which case the processor (180) or learning processor (130) can extract input features as preprocessing for the input data.

[0053] The camera (121) processes image frames, such as still images or moving images, obtained by the image sensor in video call mode or shooting mode. The processed image frames can be displayed on the display (151) or stored in the memory (170).

[0054] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application program) being performed by the artificial intelligence device (100). Meanwhile, various noise removal algorithms can be applied to the microphone (122) to remove noise generated during the process of receiving external acoustic signals.

[0055] The user input interface (123) is for receiving information from a user. When information is input through the user input interface (123), the processor (180) can control the operation of the artificial intelligence device (100) to correspond to the input information.

[0056] The user input interface (123) may include a mechanical input means (or a mechanical key, for example, a button located on the front / rear or side of the artificial intelligence device (100), a dome switch, a jog wheel, a jog switch, etc.) and a touch input means.

[0057] As an example, the touch input means may be composed of virtual keys, soft keys, or visual keys displayed on a touch screen through software processing, or may be composed of touch keys placed on a part other than the touch screen.

[0058] The learning processor (130) can train a model composed of an artificial neural network using learning data. The trained artificial neural network can be referred to as a learning model. The learning model can be used to infer result values ​​for new input data other than the learning data, and the inferred values ​​can be used as a basis for judgment to perform a certain action.

[0059] The running processor (130) can perform AI processing together with the running processor (240) of the AI ​​server (200).

[0060] The running processor (130) may include a memory integrated or implemented in the artificial intelligence device (100). The running processor (130) may also be implemented using a memory (170), an external memory directly coupled to the artificial intelligence device (100), or a memory maintained in an external device.

[0061] The sensor (140) can obtain at least one of internal information of the artificial intelligence device (100), information about the surrounding environment of the artificial intelligence device (100), and user information by using various sensors.

[0062] The sensor (140) may include one or more of a proximity sensor, a light sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar sensor, and a radar sensor.

[0063] The output interface (150) can generate output related to visual, auditory, or tactile sensations.

[0064] The output interface (150) may include a display (151) that outputs images, an audio output interface (152) that outputs audio, a haptic device (153) that outputs tactile information, and a light output interface (154) that outputs light.

[0065] The display (151) displays (outputs) information processed in the artificial intelligence device (100). For example, the display (151) may display execution screen information of an application program running in the artificial intelligence device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0066] The display (151) can be implemented as a touch screen by forming a mutual layer structure with the touch sensor or forming an integral structure. The touch screen can function as a user input interface (123) that provides an input interface between the artificial intelligence device (100) and the user, and at the same time, provide an output interface between the artificial intelligence device (100) and the user.

[0067] The audio output interface (152) can output audio data received from the communication interface (110) or stored in the memory (170) in a call signal reception mode, call mode, recording mode, voice recognition mode, broadcast reception mode, etc.

[0068] The audio output interface (152) may include at least one of a receiver, a speaker, and a buzzer.

[0069] The haptic device (153) generates various tactile effects that can be felt by the user. A representative example of the tactile effect generated by the haptic device (153) may be vibration.

[0070] The light output interface (154) outputs a signal to notify the occurrence of an event using light from a light source of the artificial intelligence device (100). Examples of events occurring in the artificial intelligence device (100) may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, and information reception through an application.

[0071] The memory (170) can store data that supports various functions of the artificial intelligence device (100). For example, the memory (170) can store input data, learning data, learning models, learning history, etc. obtained from the input interface (120).

[0072] The processor (180) can determine at least one executable operation of the artificial intelligence device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0073] The processor (180) can control components of the artificial intelligence device (100) to perform determined operations.

[0074] To this end, the processor (180) can request, retrieve, receive or utilize data from the running processor (130) or memory (170), and control components of the artificial intelligence device (100) to execute at least one of the executable operations, a predicted operation or an operation determined to be desirable.

[0075] When the processor (180) requires connection to an external device to perform a determined operation, it can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

[0076] The processor (180) can obtain intent information for user input and determine the user's requirements based on the obtained intent information.

[0077] The processor (180) can obtain intent information corresponding to the user input by using at least one of a STT (Speech To Text) engine for converting voice input into a string or a natural language processing (NLP) engine for obtaining intent information of natural language.

[0078] At least one of the STT engine or the NLP engine may be configured with an artificial neural network, at least in part, trained according to a machine learning algorithm. Furthermore, at least one of the STT engine or the NLP engine may be trained by the learning processor (130), the learning processor (240) of the AI ​​server (200), or through distributed processing thereof.

[0079] The processor (180) can collect history information including the operation details of the artificial intelligence device (100) or the user's feedback on the operation, and store the information in the memory (170) or the learning processor (130), or transmit the information to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0080] The processor (180) can control at least some of the components of the artificial intelligence device (100) to run an application program stored in the memory (170).

[0081] The processor (180) can operate two or more of the components included in the artificial intelligence device (100) in combination to drive the application program.

[0082] FIG. 2 is a diagram for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0083] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0084] The AI ​​server (200) may be composed of multiple servers and perform distributed processing, and may be defined as a 5G network device. The AI ​​server (200) may be included as part of the artificial intelligence device (100) and may perform at least a portion of the AI ​​processing.

[0085] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0086] The communication interface (210) can transmit and receive data with an external device such as an artificial intelligence device (100).

[0087] The memory (230) may include a model memory (231). The model memory (231) may store a model (or artificial neural network, 231a) being learned or learned through the learning processor (240).

[0088] A learning processor (240) can train an artificial neural network (231a) using learning data. The learning model can be used while mounted on the AI ​​server (200) of the artificial neural network, or can be mounted on an external device such as an artificial intelligence device (100).

[0089] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0090] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0091] FIG. 3a is a diagram illustrating text files used for learning a large language model, FIG. 3b is a diagram illustrating a method for generating contexts by packing text files according to a conventional technique, and FIG. 3c is a diagram illustrating a process for predicting the next token for each context according to a conventional technique.

[0092] Referring to FIG. 3A, a first text file (text file 1) may include first to third tokens (t1, t2, t3) sequentially connected. A token may be composed of one or more words or one or more spacing units.

[0093] The second text file (text file 2) may include sequentially connected fourth to sixth tokens (t4, t5, t6), and the third text file (text file 3) may include sequentially connected seventh to tenth tokens (t7, t8, t9, t10).

[0094] The first to third text files (text files 1, 2, and 3) may be related to each other or may be unrelated. If the first to third text files (text files 1, 2, and 3) are related to each other, each text file may be a portion of the source code corresponding to a single program.

[0095] The first to third text files (text files 1, 2, 3) can be concatenated into one to create a large text file (300), as illustrated in FIG. 3b. The process of concatenating the first to third text files (text files 1, 2, 3) into one can be referred to as packing.

[0096] A large text file (300) can be divided into fixed-length, non-overlapping contexts. A context may be referred to as a chunk. A chunk may be a group of tokens.

[0097] Contexts with fixed lengths (Context C1, Context C2) can be input to a large language model and used for training the large language model. The fixed length can be the number of tokens.

[0098] Referring to FIG. 3b, when the fixed length is 5, a large text file (300) may be composed of two contexts (Context C1, Context C2). Each of the two contexts (Context C1, Context C2) may not have overlapping tokens. That is, the tokens (t1 to t5) constituting the first context (Context C1) and the tokens (t6 to t10) constituting the second context (Context C2) may not overlap each other.

[0099] Referring to Fig. 3c, if the length of the context (l) is 5, each context can have l-1 instances of the problem of predicting the next token. For example, Context C1, which has a length of 5 and consists of 5 tokens, can have 4 instances (5-1=4).

[0100] That is, the first context (Context C1) can have an instance of a problem predicting the token that follows the first token (t1), an instance of a problem predicting the token that follows the first token (t1) and the second token (t2), an instance of a problem predicting the token that follows the first token (t1), the second token (t2), and the third token (t3), and an instance of a problem predicting the token that follows the first token (t1), the second token (t2), the third token (t3), and the fourth token (t4).

[0101] The method of generating multiple contexts without overlapping tokens can be referred to as a non-overlapping context generation method. A non-overlapping context generation method can be a method of generating multiple contexts without sharing tokens with each other.

[0102] The non-overlapping context generation method may cause problems such as insufficient variation in the model parameters of the giant language model when only a small data set exists, or insufficient diversity in the source code data, which may lead to bias in the training data and thus poor learning performance of the giant language model.

[0103] In order to solve this problem, an embodiment of the present disclosure aims to train a large language model using an overlapping context generation method in which two adjacent contexts are generated with overlapping tokens.

[0104] The method for creating an overlapping context may be a method for creating a context in which two adjacent contexts have overlapping tokens. More specifically, the method for creating an overlapping context may be a method for creating a context in which each of the two adjacent contexts has a certain number of overlapping tokens.

[0105] FIG. 4 is a flowchart for explaining an operation method of an artificial intelligence device according to an embodiment of the present disclosure.

[0106] Hereinafter, one or more processors (180) may be provided.

[0107] The processor (180) of the artificial intelligence device (100) can obtain a text data set for learning a large language model (S401).

[0108] The text data set is stored in memory (170) and can be input data used for training a large language model.

[0109] In one embodiment, the processor (180) can receive a text data set from an external device via a communication interface (110).

[0110] In another embodiment, the processor (180) may receive a text data set from an AI server (200).

[0111] In one embodiment, the text data set may include multiple text files. Each of the multiple text files may include one or more tokens. Each text file may include multiple tokens, as illustrated in FIG. 3A.

[0112] Each text file can be either source code or natural language.

[0113] A text data set may be a large text file consisting of multiple text files concatenated into one. For example, the text data set may be a large text file (300) as illustrated in FIG. 3b.

[0114] The processor (180) can obtain the context length and the overlapping token length between adjacent contexts (S403).

[0115] The context length can be the number of tokens contained in a single context. If the number of tokens is 5, the context length can be 5.

[0116] The overlapping token length can be the number of overlapping tokens between adjacent contexts. For example, if the number of overlapping tokens between adjacent contexts is 2, the overlapping token length can be 2.

[0117] The length of non-overlapping tokens can be referred to as the stride. The stride can indicate the degree to which tokens do not overlap. A smaller stride value can result in longer overlapping tokens, while a larger stride value can result in shorter overlapping tokens.

[0118] In one embodiment, the processor (180) may receive user input signals for setting the context length and the nested token length, respectively, through the user input interface (123).

[0119] In another embodiment, each of the context length and nested token length may be set to a default.

[0120] The processor (180) can generate a context data set including a plurality of nested contexts from a text data set based on a context length and a nested token length (S405), and can store the generated context data set (S407).

[0121] The processor (180) can train a large language model using the generated context data set. That is, the generated multiple overlapping contexts can be used as a training data set for the large language model.

[0122] In one embodiment, the processor (180) may use only a portion of the total text data set to generate overlapping contexts.

[0123] For example, the processor (180) may use only source codes among the text data set whose number of views is greater than a preset number to create a nested context.

[0124] As another example, the processor (180) may use only source codes of the same type (JAVA, C) among the text data sets to create a nested context.

[0125] As another example, the processor (180) may use only source codes with the same purpose among the text data set to create a nested context.

[0126] The processor (180) can generate a context data set including a plurality of overlapping contexts from a text data set based on a context length and an overlapping token length through a sliding window method.

[0127] The sliding window method can be used to create multiple nested contexts by moving the window according to the nested token length when generating a context with a fixed context length from a large text file. The window can refer to a set of consecutive tokens that are truncated to the context length.

[0128] FIGS. 5A and 5B are diagrams illustrating a process of generating multiple contexts from a large text file using a sliding window method according to one embodiment of the present disclosure.

[0129] Referring to FIG. 5a, a large text file (300), which is an example of a text data set, is illustrated.

[0130] In Figure 5a, it is assumed that the length of the context is 5 and the length of the overlapping token is 2. In this case, the stride can be 3.

[0131] The sliding window method may be a method of generating overlapping contexts by sequentially sliding the number of overlapping tokens between adjacent contexts so that the number of overlapping tokens between adjacent contexts is equal to the overlapping token length. Here, the number of overlapping tokens between adjacent contexts may be 2.

[0132] That is, the processor (180) can generate a third nested context (Context C3) including the first to fifth tokens (t1 to t5) from the large text file (300), and can generate a fourth nested context (Context C4) in which the third nested context (Context C3) and the fourth token (t4) and the fifth token (t5) overlap. The processor (180) can generate a plurality of nested contexts in this sliding window manner.

[0133] Next, Figure 5b is described.

[0134] In Figure 5b, it is assumed that the length of the context is 5 and the length of the overlapping token is 4. In this case, the stride can be 1.

[0135] The sliding window method may be a method of generating overlapping contexts by sequentially sliding the number of overlapping tokens between adjacent contexts so that the length of the overlapping tokens is equal to the number of overlapping tokens between adjacent contexts. Here, the number of overlapping tokens between adjacent contexts may be 4.

[0136] That is, the processor (180) can generate a fifth nested context (Context C5) including the first to fifth tokens (t1 to t5) from a large text file (300), and can generate a sixth nested context (Context C6) in which the fifth nested context (Context C5) and the second to sixth tokens (t2) to t6 overlap.

[0137] The processor (180) can create multiple overlapping contexts in this sliding window manner and store the multiple overlapping contexts created in the memory (170).

[0138] A large text file (300) and a plurality of nested contexts generated according to a sliding window method can be stored in a memory (170). The method by which a large text file (300) and a plurality of nested contexts generated according to a sliding window method are stored in a memory (170) can be referred to as a first storage method.

[0139] Under the first storage method, the storage space of the memory (170) can be set to a maximum of O(nl). n is the number of tokens included in the large text file (300), and l may be the context length. The storage space of the memory (170) can increase in proportion to the product of the number of tokens (n) and the context length (l).

[0140] In the case of the first storage method, since there is no separate context extraction process, learning of a large language model can be performed quickly, but efficiency may be reduced in a situation where the capacity of the memory (170) is insufficient.

[0141] For efficient use of memory (170), a second storage method based on an indexing method and a third storage method based on a random sampling method may be used.

[0142] FIG. 6 is a drawing illustrating an indexing method according to one embodiment of the present disclosure.

[0143] The indexing method may be a method of generating a plurality of indexes corresponding to the start token of each of a plurality of nested contexts generated from a large text file, and storing only the generated plurality of indexes and the large text file (300) in memory (170).

[0144] The processor (180) may generate a plurality of indexes corresponding to the start token of each of the plurality of nested contexts. Each of the plurality of nested contexts may be a context generated using a sliding window method.

[0145] As in Fig. 5b, when the length of the context is 5 and the length of the nested token is 4, six nested contexts can be generated according to the sliding window method. The processor (180) can generate an index matching the start token of each of the six nested contexts. For example, the processor (180) can generate an index matching the first token (t1), which is the start token of the fifth nested context (Context C5), as 1, and can generate an index matching the second token (t2), which is the start token of the sixth nested context (Context C6), as 2.

[0146] In this manner, the processor (180) can generate a plurality of indices [1,2,3,4,5,6] matching each of the six overlapping contexts. Each index can be referred to as a starting index.

[0147] The number of indexes generated (m) may be smaller than the number of tokens (n) contained in the large text file (300).

[0148] The processor (180) can store only the generated multiple indexes [1,2,3,4,5,6] and a large text file (300) in the memory (170).

[0149] In another embodiment, the processor (180) may obtain the start token of each of the multiple nested contexts as an index. For example, the processor (180) may generate the first token (t1), which is the start token of the fifth nested context (Context C5), as an index, and may generate the second token (t2), which is the start token of the sixth nested context (Context C6), as an index. A large text file (300) and multiple start tokens may be stored in the memory (170).

[0150] When a large text file (300) and a plurality of indexes generated according to an indexing method are stored in a memory (170), the storage space (O(n)) of the memory (170) can be set to a space corresponding to a maximum of 2n tokens. n may be the number of tokens included in the large text file (300). This is because when each of the plurality of indexes is a start token of a context, the number of indexes (m) cannot be greater than n.

[0151] FIG. 7 is a drawing illustrating a random sampling method according to one embodiment of the present disclosure.

[0152] The random sampling method may be a method of randomly sampling contexts having a context length from a large text file (300). Specifically, the random sampling method may be a method of extracting multiple overlapping contexts having a certain context length but having random starting tokens from the large text file (300).

[0153] In the case of a random sampling method, the storage space (O(n)) of the memory (170) can be set to the amount of space corresponding to n tokens. In the case of a random sampling method, only a large text file (300), which is usually an original data set, is stored, and when training a large language model, a starting token is randomly selected, and a context matching the selected starting token is generated.

[0154] Referring to FIG. 7, under a random sampling method, the memory (170) may only store a large text file (300) containing a plurality of stored tokens (t1 to t10). The context length is assumed to be 5.

[0155] When context generation for learning a large language model is required, the processor (180) can randomly sample starting positions (1, 5, 6) corresponding to starting tokens (t1, t5, t6). Each of the starting positions (1, 5, 6) may be an index representing each of the starting tokens (t1, t5, t6).

[0156] The processor (180) can extract a seventh nested context (Context 7), an eighth nested context (Context 8), and a ninth nested context (Context 9) corresponding to each of the randomly sampled starting positions (1, 5, 6) from the large text file (300). The processor (180) can train a large language model using the extracted nested contexts.

[0157] Figure 8 is a flowchart illustrating a method for determining a context storage method based on the capacity of memory.

[0158] Referring to FIG. 8, the processor (180) of the artificial intelligence device (100) can acquire the capacity of the memory (170) (S801).

[0159] In one embodiment, the capacity of the memory (170) may represent the storage space required when a large language model is trained.

[0160] The processor (180) can determine a context storage method based on the capacity of the memory (S803), and can use the memory (170) according to the determined context storage method (S805).

[0161] In one embodiment, the context storage method may include a first storage method, a second storage method, and a third storage method.

[0162] The first storage method may be a method in which a large text file (300) and a plurality of nested contexts generated according to a sliding window method are stored in a memory (170). Under the first storage method, the storage space of the memory (170) may increase in proportion to the product of the number of tokens (n) included in the large text file (300) and the context length (l), which is the length of the nested context.

[0163] The second storage method is based on the indexing method described above, and may be a method in which a large text file (300) and multiple indexes are stored in memory (170).

[0164] Under the second storage method, the processor (180) can extract multiple overlapping contexts using multiple indices during training of the large language model.

[0165] Under the second storage method, if the index is a start token, the start tokens do not overlap with each other, so the storage space may not exceed twice the size of the original data set (a large text file) at most.

[0166] The third storage method is based on the random sampling method described above, and may be a method in which only large text files (300) are stored in the memory (170).

[0167] The processor (180) can use the third storage method as the capacity of the memory (170) is smaller, and can use the first storage method as the capacity of the memory (170) is larger.

[0168] During training of a large language model, the processor (180) may use the first storage method when the capacity of the memory (170) is greater than or equal to the first reference capacity, may use the second storage method when the capacity of the memory (170) is less than the first reference capacity and greater than or equal to the second reference capacity, and may use the third storage method when the capacity of the memory (170) is less than the second reference capacity.

[0169] FIG. 9 is a diagram illustrating a process for determining a context generation method based on the type of text data set according to one embodiment of the present disclosure.

[0170] FIG. 9 may be an example of a process performed between steps S401 and S403 of FIG. 4.

[0171] The processor (180) can determine whether the type of the acquired text data set is source code (S402).

[0172] The type of text dataset can be either source code for programming or natural language.

[0173] If the type of the acquired text data set is source code, the processor (180) can perform step S403 of FIG. 4 and subsequent steps.

[0174] That is, when the type of text data set is source code, the processor (180) can create multiple nested contexts through a nested context creation method.

[0175] If the type of the acquired text data set is not source code, the processor (180) can create contexts through a non-overlapping context creation method (S901).

[0176] That is, when the type of text data set is natural language, the processor (180) can apply a non-overlapping context generation method in which multiple contexts are generated without overlapping tokens.

[0177] Source code should not be broken down into functions or classes, allowing large language models to better understand the context. Large language models can more accurately learn the logical flow of source code when contexts share some tokens.

[0178] In this way, according to embodiments of the present disclosure, a context generation method can be selectively applied depending on the type of text data set. Accordingly, the learning performance of a large-scale language model can be improved.

[0179] Figures 10a and 10b are drawings explaining the process of inserting spaces to prevent them from being cut off by function or class unit when packing to create a context.

[0180] Referring to Figure 10a, a portion of the multiply() function may be truncated in the source code of [context 1]. As a portion of the multiply() function is truncated, the large language model is likely to lose context.

[0181] The processor (180) can create a context by inserting padding (space) to prevent the code from being cut into function or class units during the packing process of creating the context.

[0182] In an embodiment of the present disclosure, when the entire function cannot be included due to the context length, as illustrated in FIG. 10b, padding (space, 1000) is inserted after return x+y in the source code of [context 1] to prevent the multiply() function from being truncated. Accordingly, a natural flow of code can be maintained, which can assist in the learning of a large-scale language model in understanding the context.

[0183] FIG. 11 is a diagram comparing the performance of learning a large language model using a non-overlapping context generation method according to a prior art and the performance of learning a large language model using a nested context generation method according to an embodiment of the present disclosure.

[0184] The large language model used in Fig. 11 may be the known EXAONE model.

[0185] Referring to Figure 11, the horizontal axis represents the number of steps, representing the number of times the parameters of the large language model are updated, and the vertical axis represents the problem-solving rate. The problem-solving rate (pass@1) can be the rate at which a problem is solved when all unit tests pass.

[0186] When learning a giant language model by increasing steps or epochs through a non-overlapping context generation method according to a prior art, a first graph (1110) showing the problem solving ratio of the giant language model and when learning a giant language model through a nested context generation method according to an embodiment of the present disclosure, a second graph (1120) showing the problem solving ratio of the giant language model are shown.

[0187] As the number of steps or epochs increases, it can be confirmed that the problem-solving rate is better when the giant language model is trained using the nested context generation method than when the giant language model is trained using the non-nested context generation method.

[0188] FIG. 12a is a diagram comparing the performance of learning a first large language model using a non-overlapping context generation method according to a prior art and the performance of learning a first large language model using a nested context generation method according to an embodiment of the present disclosure.

[0189] The first large language model used in Fig. 12a may be the well-known EXAONE model. NOC stands for Non-Overlapping Context and may indicate a non-overlapping context creation method. OC stands for Overlapping Context and may indicate an overlapping context creation method.

[0190] Referring to Fig. 12a, when the same number of parameters (8.8B) is used, it can be confirmed that the problem solving rate (pass@1) of the case where learning is performed using the overlapping context creation method improves as the training step increases compared to the problem solving rate of the case where learning is performed using the non-overlapping context creation method.

[0191] FIG. 12b is a diagram comparing the performance of learning a second giant language model using a non-overlapping context generation method according to a prior art and the performance of learning a second giant language model using a nested context generation method according to an embodiment of the present disclosure.

[0192] The second large language model used in Fig. 12b may be the known code Llama model.

[0193] Referring to Fig. 12b, when the same number of parameters (7B) is used, it can be confirmed that the problem solving rate (pass@1) in the case of learning using the overlapping context creation method improves as the epoch increases compared to the problem solving rate in the case of learning using the non-overlapping context creation method.

[0194] FIGS. 13a to 13c are drawings showing the diversity of contexts generated when using a method for generating overlapping contexts according to an embodiment of the present disclosure.

[0195] Hereinafter, the unique ratio may be the ratio of the number of non-overlapping prediction problems to the number of premise prediction problems.

[0196] As the unique ratio increases, context diversity is secured, allowing large language models to learn more flexibly.

[0197] Fig. 13a is a diagram showing the unique ratio according to stride, Fig. 13b is a diagram showing the unique ratio according to context length, and Fig. 13c is a diagram showing the unique ratio when a random sampling method is applied.

[0198] As shown in Figures 13a to 13c, when the overlapping context creation method is used, a high unique ratio is obtained, confirming that the diversity of learning data is greatly secured.

[0199] A method for augmenting data according to one embodiment of the present disclosure comprises the steps of: obtaining a text data set for training a large language model; obtaining a context length indicating the number of tokens included in one context; and generating a plurality of nested contexts having the context length from the obtained text data set, wherein at least two of the plurality of nested contexts can share one or more tokens with each other.

[0200] The augmentation method may further include a step of obtaining an overlap token length indicating the number of tokens overlapping between two adjacent contexts, and the generating step may include a step of generating a plurality of overlap contexts from the obtained text data set based on the context length and the overlap token length.

[0201] The above generating step may include a step of generating a plurality of overlapping contexts while moving a window to match the overlapping token length when generating a context having the context length from the text data set.

[0202] The above augmentation method may further include a step of storing the generated plurality of overlapping contexts.

[0203] The above augmentation method may further include a step of storing a plurality of indices corresponding to each of the plurality of overlapping contexts.

[0204] Each of the above multiple indices can represent the start token of each nested context.

[0205] The generating step may include a step of generating the plurality of overlapping contexts having the context length by randomly sampling a starting token from the obtained text data set.

[0206] The augmentation method may further include a step of determining one of a plurality of context storage methods based on a capacity of a memory, wherein the plurality of context storage methods may include a first storage method for storing the text data set and the generated plurality of nested contexts in the memory, a second storage method for storing a plurality of indexes corresponding to each of the text data set and the plurality of nested contexts, and a third storage method for storing only the text data set.

[0207] The augmentation method may further include a step of determining a context generation method based on the type of the text data set, and the context generation method may include a nested context generation method for generating the plurality of nested contexts and a non-nested context generation method for generating the plurality of contexts that do not share tokens with each other.

[0208] The type of the text data set may be either source code or natural language, and the determining step may include, when the type of the text data set is the source code, a step of determining the context creation method as the nested context creation method, and when the type of the text data set is the natural language, a step of determining the context creation method as the non-nested context creation method.

[0209] A portion of the total proportion of the above text data set is used to create the plurality of overlapping contexts, and the portion may be determined based on data having a preset number of views or more, data of the same type, or data having the same purpose.

[0210] The above generating step may include a step of performing a padding operation to prevent the text data set from being truncated into function or class units when the text data set is composed of source codes.

[0211] The functions of the elements disclosed in the present invention may be implemented using circuits or processing circuits including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), existing circuits, and / or combinations thereof. A processor may be defined as a processing circuit or circuits including transistors and other circuits.

[0212] In the present invention, the circuits, units, or means may be hardware designed or programmed to perform the specified functions. The hardware may be the hardware disclosed in the present invention or other known hardware programmed or configured to perform the specified functions. If the hardware is a processor, which may be considered a type of circuit, the circuits, units, or means may be a combination of hardware and software, and the software may constitute the hardware and / or the processor.

[0213] The above-described present disclosure can be implemented as computer-readable code on a program-recorded medium. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. In addition, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. In the method of data augmentation, A step of obtaining a text data set for training a large language model; A step of obtaining a context length indicating the number of tokens included in one context; and A step of generating a plurality of overlapping contexts having the context length from the obtained text data set, At least two of the above nested contexts share one or more tokens with each other. How to augment data.

2. In paragraph 1, Further comprising the step of obtaining an overlap token length indicating the number of tokens overlapping between two adjacent contexts, The above generating steps are A step of generating a plurality of nested contexts from the obtained text data set based on the context length and the nested token length. How to augment data.

3. In paragraph 2, The above generating steps are When generating a context having the context length from the text data set, a step of generating the plurality of overlapping contexts while moving a window according to the overlapping token length is included. How to augment data.

4. In paragraph 3, Further comprising a step of storing the plurality of generated nested contexts How to augment data.

5. In paragraph 1, Further comprising a step of storing a plurality of indexes corresponding to each of the plurality of nested contexts. How to augment data.

6. In paragraph 5, Each of the above multiple indexes Indicates the start token of each nested context. How to augment data.

7. In paragraph 1, The above generating steps are A step of generating the plurality of overlapping contexts having the context length by randomly sampling a starting token from the obtained text data set. How to augment data.

8. In paragraph 1, Further comprising a step of determining one of a plurality of context storage methods based on the capacity of the memory, The above multiple context storage methods are A first storage method for storing the text data set and the generated plurality of nested contexts in the memory, a second storage method for storing a plurality of indexes corresponding to each of the text data set and the plurality of nested contexts, and a third storage method for storing only the text data set. How to augment data.

9. In paragraph 1, Further comprising a step of determining a context generation method based on the type of the above text data set, The above context creation method is A method for creating a nested context that creates the above multiple nested contexts, and A non-overlapping context creation method that creates multiple contexts that do not share tokens with each other. How to augment data.

10. In paragraph 9, The type of the above text data set is Either source code or natural language, The above decision steps are If the type of the above text data set is the above source code, a step of determining the context creation method as the nested context creation method; and If the type of the above text data set is the natural language, a step of determining the context creation method as the non-overlapping context creation method is included. How to augment data.

11. In paragraph 1, A portion of the total percentage of the above text data set is used to generate the plurality of overlapping contexts, Some of the above ratios It is determined based on data that has a number of views exceeding a preset number of times, data of the same type, or data with the same purpose. How to augment data.

12. In paragraph 1, The above generating steps are If the above text data set is composed of source codes, a step of performing padding to prevent it from being truncated by function or class unit is included. How to augment data.

13. In artificial intelligence devices, Memory for storing text data sets for training large language models; and Comprising one or more processors that obtain a context length indicating the number of tokens included in one context and generate a plurality of nested contexts having the context length from the obtained text data set, At least two of the above nested contexts share one or more tokens with each other. Artificial intelligence devices.

14. In paragraph 13, One or more of the above processors Obtain the overlap token length, which represents the number of tokens overlapping between two adjacent contexts, Generating multiple nested contexts from the obtained text data set based on the context length and the nested token length. Artificial intelligence devices.

15. In paragraph 14, One or more of the above processors When generating a context having the context length from the text data set, a plurality of overlapping contexts are generated while moving the window to match the overlapping token length. Artificial intelligence devices.

16. In paragraph 15, One or more of the above processors Storing the above-mentioned multiple nested contexts in the memory Artificial intelligence devices.

17. In paragraph 13, One or more of the above processors Store a plurality of indexes corresponding to each of the above multiple nested contexts in the memory, Each of the above multiple indexes Indicates the start token of each nested context. Artificial intelligence devices.

18. In paragraph 13, One or more of the above processors Generating the plurality of overlapping contexts having the context length by randomly sampling a starting token from the obtained text data set. Artificial intelligence devices.

19. In paragraph 13, One or more of the above processors Determine one of multiple context storage methods based on the capacity of the above memory, The above multiple context storage methods are A first storage method for storing the text data set and the generated plurality of nested contexts in the memory, a second storage method for storing a plurality of indexes corresponding to each of the text data set and the plurality of nested contexts, and a third storage method for storing only the text data set. Artificial intelligence devices.

20. A computer-readable non-volatile recording medium having recorded thereon a program for performing a method of augmenting data, The above augmentation method A step of obtaining a text data set for training a large language model; A step of obtaining a context length indicating the number of tokens included in one context; and A step of generating a plurality of overlapping contexts having the context length from the obtained text data set, At least two of the above nested contexts share one or more tokens with each other. Non-volatile recording media.

Images