Model training method and device associated with wsd

Abstract

The present invention relates to a model learning method associated with WSD. The model learning method associated with WSD comprises the steps of: generating a batch including a first set of training data associated with a first meaning of a word and a second set of training data associated with a second meaning of a word; and generating a model that performs WSD for a word by performing fine-tuning on a pre-trained language model based on the generated batch.

Description

Technology Field

[0001] The present invention relates to a model learning method and apparatus associated with WSD, and more specifically, to a model learning method and apparatus associated with WSD for solving the problems of existing WSD algorithms. Background Technology

[0003] Word Sense Disambiguation (WSD) can refer to determining which meaning of a word is activated by its use in a specific context, or machine learning models for this purpose. In other words, WSD can be used to assess the overall context of a sentence and, based on this, determine the clear meaning of each word.

[0004] Generally, WSD models face the problem of increased inference time and cost because they perform inference by considering not only the input sentence but also a vast amount of additional information. Furthermore, there is a disadvantage in that model performance degrades as training is conducted with limited data due to the difficulty in securing a sufficient amount of labeled data to train the WSD model. Additionally, there is a limitation in that it cannot consider the meanings of idioms and phrasal verbs.

[0005] delete Prior art literature

[65535] Registered Patent Publication 10-0574887 B1 The problem to be solved

[0006] The present invention provides a model learning method associated with WSD for solving the above-mentioned problems, a computer program stored on a computer-readable medium, a computer-readable medium storing the computer program, and a device (system). means of solving the problem

[0008] The present invention may be implemented in various ways, including a method, an apparatus (system), a computer program stored on a computer-readable medium, or a computer-readable medium on which a computer program is stored.

[0009] According to one embodiment of the present invention, a model learning method associated with WSD performed by at least one processor comprises the steps of: generating a batch including a first set of training data associated with a first meaning of a word and a second set of training data associated with a second meaning of a word; and generating a model that performs WSD for a word by performing fine-tuning on a pre-trained language model based on the generated batch.

[0010] According to one embodiment of the present invention, the step of generating a model that performs WSD includes the step of generating a model that performs WSD on words by performing fine-tuning on a pre-trained language model using margin-based loss and mask language modeling loss.

[0011] According to one embodiment of the present invention, the step of generating a model that performs WSD on words using a margin-based loss and a mask language modeling loss includes the step of generating a model that performs WSD on words using a linear combination of the margin-based loss and the mask language modeling loss.

[0012] According to one embodiment of the present invention, margin-based loss is It consists of. Here, is the loss value, and X represents the batch, represents the language model, and i and j represent the indices of the training data, represents the distance between the embedding vectors of the first training data and the second training data, and , , , , represents the training parameters.

[0013] According to one embodiment of the present invention, mask language modeling loss is It consists of. Here, is the loss value, and X represents the batch, represents a language model, represents a corrupted input that is masked on X with a certain probability, i represents an index of the training data, and t represents an index of a token included in the training data, and Is Indicates whether it is a damaged token, Is Represents the parameters for, represents a variable for sampling training data.

[0014] According to one embodiment of the present invention, the step of generating a model that performs WSD on words using a linear combination of margin-based loss and mask language modeling loss includes the step of performing backpropagation so that the total loss value of the linear combination of margin-based loss and mask language modeling loss becomes smaller.

[0015] According to one embodiment of the present invention, the language model is a DeBERTa (Decoding-enhanced BERT with Disentangled Attention) model.

[0016] A computer program stored on a computer-readable recording medium is provided to execute the above-described method according to one embodiment of the present invention on a computer.

[0017] A computing device according to one embodiment of the present invention includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory. The at least one program includes instructions for generating a batch comprising a first set of training data associated with a first meaning of a word and a second set of training data associated with a second meaning of a word, and for generating a model that performs WSD on a word by performing fine-tuning on a pre-trained language model based on the generated batch. Effects of the invention

[0019] In various embodiments of the present invention, when training a WSD model that generates a batch corresponding to each word is performed, all data corresponding to all meanings can be effectively extracted, and independent multi-task training for each word can be performed. Furthermore, when multi-task training is performed in this manner, the generated WSD model can perform WSD in a specific context using only a single input sentence.

[0020] In various embodiments of the present invention, the WSD model performs calculations without distinguishing between the support set and the query set, thereby enabling learning that utilizes all of the training data, and thus enables learning without loss of information. Additionally, when the margin is satisfied, learning may not be performed to make the distance between data points further or closer, thus increasing learning efficiency.

[0021] In various embodiments of the present invention, when training a WSD model using a linear combination of margin-based loss and mask language modeling loss, there is an advantage in that additional pre-training effects occur by applying the mask language modeling loss to an additional corpus. In addition, by continuously training the pre-training task together, the WSD model can maintain the function of restoring masked inputs, and the phenomenon of overfitting occurring for words constituting the batch being trained can be prevented.

[0022] In various embodiments of the present invention, when weights are optimized using both the SWA algorithm and the GreedySoup algorithm, the generalization performance of the final WSD model can be significantly improved.

[0023] In various embodiments of the present invention, time complexity can be significantly reduced by training a WSD model through labeling of tokens constituting idioms, thereby effectively preventing the possibility of system overload.

[0024] In various embodiments of the present invention, by performing training using both weak label data and pseudo-label data, a second model can be generated so that no decrease in accuracy occurs even when weak label data is used. Additionally, labeling data can be effectively collected even when only weak label data is used by utilizing the first model.

[0025] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present invention pertains (referred to as "person skilled in the art") from the description in the claims. Brief explanation of the drawing

[0027] Embodiments of the present invention will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto. FIG. 1 is a diagram showing an example of a batch being generated for creating a model that performs WSD according to one embodiment of the present invention. FIG. 2 is a diagram showing an example of a loss function for training a WSD model according to one embodiment of the present invention. FIG. 3 is a diagram showing an example in which a generalization according to one embodiment of the present invention is performed. FIG. 4 is a diagram showing an example of labeling that enables effective processing of idioms according to one embodiment of the present invention. FIG. 5 is a diagram showing an example of generating data for creating a WSD model according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of a model learning method associated with WSD according to one embodiment of the present invention. FIG. 7 is a diagram showing an example of a model learning method associated with WSD for idiom processing according to one embodiment of the present invention. FIG. 8 is a diagram illustrating an example of a model learning method associated with weak label data-based WSD according to an embodiment of the present invention. FIG. 9 is a block diagram showing the internal configuration of a computing device according to one embodiment of the present invention. Specific details for implementing the invention

[0028] Hereinafter, specific details for implementing the present invention will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the essence of the present invention.

[0029] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0030] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms, and these embodiments are provided merely to make the present invention complete and to fully inform those skilled in the art of the scope of the invention.

[0031] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in the present invention; however, these terms may vary depending on the intent of those skilled in the relevant field, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the present invention.

[0032] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0033] In the present invention, terms such as “comprising,” “comprising,” etc. may indicate the presence of features, steps, operations, elements, and / or components, but do not exclude the addition of one or more other functions, steps, operations, elements, components, and / or combinations thereof.

[0034] In the present invention, where a specific component is described as being "combined," "combined," "connected," or "reacting" with any other component, the specific component may be directly combined, combined, and / or connected, or react with the other component, but is not limited thereto. For example, one or more intermediate components may exist between the specific component and the other component. Additionally, in the present invention, "and / or" may include each of the one or more listed items or a combination of at least some of the one or more items.

[0035] In the present invention, terms such as "first," "second," etc. are used to distinguish a specific component from another component, and the components described above are not limited by these terms. For example, the "first" component may be an element of the same or similar form as the "second" component.

[0036] In the present invention, "WSD (Word Sense Disambiguation)" may refer to determining which meaning of a word is activated by the use of the word in a specific context, and the WSD model may refer to a machine learning model that performs WSD.

[0037] In the present invention, "model" refers to a machine learning model and / or an artificial neural network model and may include any model used to infer an answer for a given input. According to one embodiment, the model may include a deep neural network comprising an input layer, a plurality of hidden layers, and an output layer. Here, each layer may include one or more nodes. Additionally, the model may include weights associated with a plurality of nodes included in the model. Here, the weights may include any parameters associated with the artificial neural network model.

[0038] In the present invention, "few-shot learning" may refer to a method of training an artificial neural network model that has good performance with only a small amount of data.

[0039] In the present invention, the "language model" may be an artificial neural network model for assigning or calculating probabilities for a sequence of words, and may include, for example, a BERT (Bidirectional Encoder Representations from Transformers)-based model as a pre-trained model.

[0040] FIG. 1 is a diagram illustrating an example in which a batch (120) is generated to create a model for performing WSD according to an embodiment of the present invention. As illustrated, the training data (110) may include data corresponding to each word and sense. For example, the data may be sentence data, and the sentence data may consist of a plurality of tokens corresponding to each word included in the sentence. That is, the training data (110) may include a plurality of sentence data distinguished by the meaning of the word in a specific context.

[0041] According to one embodiment, a batch (120) may be generated that includes a first set of training data associated with a first meaning of a word and a second set of training data associated with a second meaning of a word. That is, a batch (120) may be generated to perform fine tuning on a pre-trained language model (130). Here, the batch (120) is a set of at least a portion of the training data (110), and a batch (120) and / or batch set corresponding to each word may be generated. Accordingly, the language model (130) may be trained in an independent multi-task learning manner for each word based on the batch set corresponding to each word. Here, the language model (130) may be a BERT (Bidirectional Encoder Representations from Transformers) based model, and may include, for example, a DeBERTa (Decoding-enhanced BERT with Disentangled Attention) model.

[0042] According to one embodiment, in order to include data having various combinations of meanings in a single batch (120), a sense index may be randomly selected to extract the data included in the batch (120). For example, if the batch (120) may include data corresponding to K meanings, K meanings among K+a meanings may be randomly selected, and data corresponding to the selected meanings may be included in the batch (120). In this case, the value of a may be a value for adjusting the degree to which the sense index is randomly selected and the degree to which data overlaps. With such a configuration, when training a WSD model that generates a batch (120) corresponding to each word is performed, all data corresponding to all meanings can be effectively extracted, and independent multi-task training for each word can be performed. Furthermore, when multi-task training is performed in this manner, the generated WSD model can perform WSD in a specific context using only one input sentence.

[0043] FIG. 2 is a diagram showing an example of a loss function for training a WSD model (210) according to an embodiment of the present invention. Generally, existing WSD models can be trained using a few-shot learning method to achieve sufficient performance even for the sense of words with a small amount of data. However, the general few-shot learning method is a method of training by dividing the batch set into a support set and a query set, and since it only learns the distance between the support set and the query set, there is a problem in that it does not utilize all information for training.

[0044] According to one embodiment, to solve the problem described above, the WSD model (210) according to the present invention may be trained using metric-based few-shot learning. For example, the WSD model (210) may be trained using a margin-based loss (or loss function) (212) such as the following Equation 1.

[0046]

[0048] Here, is the loss value, and X represents the batch, represents the language model, and i and j represent the indices of the training data, represents the distance between the embedding vectors of the first training data and the second training data, and , , , , can represent trainable parameters. As shown in Equation 1, since multi-task learning is performed where each word is a single task, adjusting the margin for each word It can be added and used for learning.

[0049] According to one embodiment, the WSD model (210) can be trained to reduce the distance between embedding vectors of data having the same meaning and to increase the distance between embedding vectors of data having different meanings based on a margin-based loss (212). Additionally, even if the WSD model (210) has data having the same meaning If the distance is less than or equal to the distance, it does not learn to make the distance smaller, or If the distance is greater than or equal to the above, the model may not learn to make the distance further apart. With this configuration, the WSD model (210) can perform calculations without distinguishing between the support set and the query set, thereby enabling learning that utilizes all of the training data, and thus can perform learning without loss of information. Additionally, if the margin is satisfied, the model may not learn to make the distance between data further apart or closer apart, thus increasing the learning efficiency.

[0050] Generally, when fine-tuning is performed on a language model pre-trained with a vast amount of data, there is a problem where information contained in the parameters of the pre-trained language model is lost or overfitting occurs.

[0051] To solve such problems, the WSD model (210) according to the present invention can be trained using a linear combination of a margin-based loss (212) and a mask language modeling loss (214). For example, the WSD model (210) can be trained using a linear combination of a margin-based loss (212) and a mask language modeling loss (214) as shown in Equation 2 below.

[0053]

[0055] Here, is the total loss function, and represents a margin-based loss (212), can represent the mask language modeling loss (214). Here, the mask language modeling loss (214) can be constructed as shown in the following mathematical formula 3.

[0057]

[0059] Here, is the loss value, and X represents the batch, represents a language model, represents a corrupted input that is masked in X with a certain probability, i represents an index of the training data, and t represents the index of a token included in the training data, and Is Indicates whether it is a damaged token, Is Represents the parameters for, can represent a variable for sampling training data.

[0060] According to one embodiment, the mask language modeling loss (214) is in a function used when pre-training a language model. It may be a form with added. Here, The loss value of the mask language modeling loss (214) can be calculated only when it is 1, and data sampling can be performed accordingly. In this way, when data sampling is performed, the usage of GPU VRAM can be reduced, and accordingly, a larger batch size can be used for training. In addition, the performance of the WSD model (210) can be improved by observing more distance pairs through this mask language modeling loss (214).

[0061] According to one embodiment, back-propagation can be performed so that the total loss value (220) of the linear combination of the margin-based loss (212) and the mask language modeling loss (214) becomes smaller. With this configuration, when training the WSD model (210) using the linear combination of the margin-based loss (212) and the mask language modeling loss (214), there is an advantage of additional pre-training effects by applying the mask language modeling loss (214) to an additional corpus. In addition, the WSD model (210) can maintain the function of restoring masked inputs by continuously training the pre-training task together, and can prevent the phenomenon of overfitting occurring for the words constituting the batch in which training is performed.

[0062] FIG. 3 is a diagram illustrating an example in which generalization is performed according to an embodiment of the present invention. According to an embodiment, generalization may refer to a method of learning a model that minimizes errors on new samples not present in the training data. As illustrated, a plurality of models (310) may be created, and a Stochastic Weight Averaging (SWA) algorithm (320) and a GreedySoup algorithm (330) may be applied to each model to improve the generalization performance of the final model. Here, the SWA algorithm (320) may refer to an algorithm for averaging the weights of each model, and the GreedySoup algorithm (330) may refer to an algorithm for averaging the weights of a plurality of models by combining them.

[0063] According to one embodiment, the SWA algorithm (320) may refer to an algorithm that averages weights while continuing model training at a learning rate below a predetermined threshold after the model training has progressed beyond a predetermined threshold. For example, when the SWA algorithm (320) is used, the weights of the first model (310_1) may be averaged, the weights of the second model (310_2) may be averaged, and the weights of the nth model (310_n) may be averaged. When weights are averaged in this way, the performance of each model may be improved.

[0064] According to one embodiment, a single WSD model can be generated using a plurality of learned models (310). For example, the weights of each model can be averaged based on the GreedySoup algorithm (330) to determine the weights of the final WSD model. In this case, some of the weights of the plurality of learned models (310) may be selected and averaged. For example, after combining and averaging the weights, the performance of the WSD model may be measured, and the combined weights may be selected and averaged only when the performance of the WSD model has improved. With this configuration, when weights are optimized using both the SWA algorithm (320) and the GreedySoup algorithm (330), the generalization performance of the final WSD model can be significantly improved.

[0065] In FIG. 3, both the SWA algorithm (320) and the GreedySoup algorithm (330) are used for generalization of the WSD model, but are not limited thereto. The SWA algorithm (320) and the GreedySoup algorithm (330) may be used optionally.

[0066] FIG. 4 is a diagram illustrating an example of labeling that enables effective processing of idioms according to an embodiment of the present invention. According to an embodiment, an idiom may refer to a word formed by combining two or more words to function as a single word in terms of syntax. For example, an idiom k (410) may be composed of a first token (420_1), a second token (420_2), an nth token (420_n), etc., representing each word constituting the idiom k (410).

[0067] Generally, conventional WSD models assume that the tokens constituting an idiom are adjacent to each other and perform WSD on the idiom using the embeddings of adjacent tokens. However, using this method fails to account for idioms composed of non-adjacent tokens, and there is a problem of increased time complexity for calculating idiom embeddings.

[0068] For example, any idiom k (410) has n tokens ({ Consists of})(420), and for any sentence x, Among the tokens of sentence x If defined as the index of tokens that match, It can be expressed as follows. That is, the number of combinations for calculating the embedding of an idiom for any sentence x can be constructed as shown in the following mathematical formula 4.

[0070]

[0072] To solve the problem of increasing time complexity in this way, each token included in the idiom can be labeled with the meaning of the idiom. For example, if the idiom k (410) is composed of a first token (420_1), a second token (420_2), an nth token (420_n), etc., representing each word that makes up the idiom, the tokens (420) can be labeled as idiom k (430).

[0073] In other words, when idiom data consisting of multiple tokens including a first token and a second token is extracted, the sense of the idiom can be labeled on the first word corresponding to the first token, and the sense of the idiom can be labeled on the second word corresponding to the second token. Then, a model that performs WSD for the first word and the second word can be generated by performing fine-tuning on a pre-trained language model based on the first batch generated to correspond to the first word and the second batch generated to correspond to the second word. When labeling is performed in this manner, the number of combinations for calculating the embedding of the idiom for any sentence x can be configured as shown in the following mathematical formula 5.

[0075]

[0077] With this configuration, the time complexity can be significantly reduced by training the WSD model through labeling the tokens that constitute the idiom, thereby effectively preventing the possibility of system overload.

[0078] FIG. 5 is a diagram illustrating an example of generating data for creating a WSD model according to an embodiment of the present invention. Generally, labeling all tokens included in a sentence for WSD model training is costly and inefficient. Therefore, it is important to perform training using weakly labeled data in which only some of the tokens constituting the sentence are labeled. However, when using weakly labeled data, there is a problem in that incorrect embeddings are generated for unlabeled tokens, which may lead to an accuracy drop.

[0079] To solve these problems, a total of three stages of WSD model training may be performed. First, stage 1 may be a process of creating a first model (510) that performs WSD using fully labeled data (512). That is, a first model (510) that performs WSD on words may be created using fully labeled data (512) in which multiple tokens constituting a sentence are all labeled. As described above, the first model (510) may be trained using a linear combination of margin-based loss and mask language modeling loss, and may be a model to which the SWA algorithm and / or Greedy Soup algorithm is applied.

[0080] Stage 2 may be a process of extracting pseudo-labeled data (516) by providing weak labeled data (514), in which some of the multiple tokens constituting a sentence are labeled, to the generated first model (510). That is, the first model (510) may generate labels for unlabeled tokens among the tokens included in the weak labeled data (514) so ​​that pseudo-labeled data (516) can be extracted. Here, the pseudo-labeled data (516) may include estimated labeling information for unlabeled tokens included in the weak labeled data (514).

[0081] Stage 3 may be a process of generating a second model (520) that performs WSD on words using weak label data (514) and pseudo-label data (516). As described above, the second model (520) may be trained using a linear combination of margin-based loss and mask language modeling loss, and may be a model to which the SWA algorithm and the Greedy Soup algorithm are applied.

[0082] According to one embodiment, the margin-based loss used to generate the second model (520) may include a variable for applying a penalty to the pseudo-label data (516). For example, the loss function for generating the second model (520) may be configured as shown in the following Equation 6.

[0084]

[0086] For example, if at least one of the first data and second data input into the margin-based loss is pseudo-label data (516), the variable ( The value of ) can be determined to be a value of 1 or less. That is, the pseudo-label data (516) inevitably contains noise, and when the pseudo-label data (516) is input, the effect of the noise can be reduced by applying a penalty to the pseudo-label data (516).

[0087] With this configuration, a second model (520) can be generated so that a decrease in accuracy does not occur even when using weak label data (514) by performing training using both weak label data (514) and pseudo-label data (516). Additionally, labeling data can be effectively collected even when using only weak label data (514) by using the first model (510).

[0088] FIG. 6 is a diagram illustrating an example of a model learning method (600) associated with WSD according to an embodiment of the present invention. The model learning method (600) associated with WSD may be performed by at least one processor (e.g., at least one processor of a computing device). The model learning method (600) associated with WSD may be initiated by the processor generating a batch comprising a first set of training data associated with a first meaning of a word and a second set of training data associated with a second meaning of a word (S610).

[0089] The processor can generate a model that performs WSD on words by performing fine-tuning on a pre-trained language model based on the generated batch (S620). For example, the processor can generate a model that performs WSD on words by performing fine-tuning on a pre-trained language model using margin-based loss and mask language modeling loss. In this case, the processor can generate a model that performs WSD on words using a linear combination of margin-based loss and mask language modeling loss. Additionally, the processor can perform backpropagation to minimize the total loss value of the linear combination of margin-based loss and mask language modeling loss.

[0090] FIG. 7 is a diagram illustrating an example of a model learning method (700) associated with WSD for idiom processing according to an embodiment of the present invention. The model learning method (700) associated with WSD for idiom processing may be performed by at least one processor (e.g., at least one processor of a computing device). The model learning method (700) associated with WSD for idiom processing may be initiated by the processor extracting idiom data composed of a plurality of tokens including a first token and a second token (S710).

[0091] According to one embodiment, the processor can label the meaning of an idiom to a first word corresponding to a first token and label the meaning of an idiom to a second word corresponding to a second token (S720). Then, the processor can create a model that performs WSD on the first word and the second word by performing fine-tuning on a pre-trained language model based on a first batch generated to correspond to the first word and a second batch generated to correspond to the second word (S730).

[0092] The processor can generate a model that performs WSD on the first and second words by fine-tuning a pre-trained language model using margin-based loss and mask language modeling loss. For example, the processor can generate a model that performs WSD on the first and second words using a linear combination of margin-based loss and mask language modeling loss. Additionally, the processor can perform backpropagation to minimize the total loss value of the linear combination of margin-based loss and mask language modeling loss.

[0093] FIG. 8 is a diagram illustrating an example of a model learning method (800) associated with weak label data-based WSD according to an embodiment of the present invention. The model learning method (800) associated with weak label data-based WSD may be performed by at least one processor (e.g., at least one processor of a computing device). The model learning method (800) associated with weak label data-based WSD may be initiated by the processor creating a first model that performs WSD for words using fully labeled data in which a plurality of tokens constituting a sentence are all labeled (S810).

[0094] The processor can extract pseudo-label data by providing weak label data, in which some of the multiple tokens constituting a sentence are labeled, to the generated first model (S820). Here, the pseudo-label data may include estimated labeling information for unlabeled tokens included in the weak label data. Then, the processor can generate a second model that performs WSD on words using the weak label data and the pseudo-label data (S830).

[0095] According to one embodiment, the processor may generate a second model that performs WSD on words by performing fine-tuning on a pre-trained language model using margin-based loss and mask language modeling loss. For example, the processor may generate a second model that performs WSD on words using a linear combination of margin-based loss and mask language modeling loss.

[0096] According to one embodiment, a processor may generate a second model that performs WSD on words using a margin-based loss including a variable for imposing a penalty on pseudo-label data and a mask language modeling loss. For example, if at least one of the first data and the second data input to the margin-based loss is pseudo-label data, the processor may determine the value of the variable to be 1 or less.

[0097] FIG. 9 is a block diagram showing the internal configuration of a computing device (900) according to an embodiment of the present invention. According to an embodiment, the computing device (900) may include a memory (910), a processor (920), a communication module (930), and an input / output interface (940). As shown in FIG. 9, the computing device (900) may be configured to communicate information and / or data through a network using the communication module (930).

[0098] The memory (910) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (910) may include a permanent mass storage device such as random access memory (RAM), read-only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. As another example, a permanent mass storage device such as ROM, SSD, flash memory, disk drive, etc. may be included in the computing device (900) as a separate permanent storage device distinct from the memory. Additionally, an operating system and at least one program code may be stored in the memory (910).

[0099] These software components may be loaded from a computer-readable recording medium separate from memory (910). This separate computer-readable recording medium may include a recording medium that can be directly connected to this computing device (900), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into memory (910) via a communication module (930) rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (910) based on a computer program installed by files provided through the communication module (930) by developers or a file distribution system that distributes installation files for applications.

[0100] The processor (920) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to a user terminal (not shown) or another external system by memory (910) or a communication module (930).

[0101] The communication module (930) may provide a configuration or function for a user terminal (not shown) and a computing device (900) to communicate with each other via a network, and may provide a configuration or function for the computing device (900) to communicate with an external system (e.g., a separate cloud system). For example, control signals, commands, data, etc. provided under the control of the processor (920) of the computing device (900) may be transmitted to the user terminal and / or the external system through the communication module (930) and the network, and through the communication module of the user terminal and / or the external system.

[0102] Additionally, the input / output interface (940) of the computing device (900) may be a means for interfacing with a device (not shown) for input or output that is connected to the computing device (900) or that the computing device (900) may include. In FIG. 9, the input / output interface (940) is shown as an element configured separately from the processor (920), but is not limited thereto, and the input / output interface (940) may be configured to be included in the processor (920). The computing device (900) may include more components than those shown in FIG. 9. However, there is no need to clearly illustrate most of the prior art components.

[0103] The processor (920) of the computing device (900) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems.

[0104] The methods and / or various embodiments described above may be realized in digital electronic circuits, computer hardware, firmware, software, and / or combinations thereof. Various embodiments of the present invention may be executed by a data processing device, for example, one or more programmable processors and / or one or more computing devices, or implemented as a computer program stored on a computer-readable recording medium and / or a computer program stored on a computer-readable recording medium. The computer program described above may be written in any form of programming language, including a compiled language or an interpreted language, and may be distributed in any form, such as a standalone program, a module, a subroutine, etc. The computer program may be distributed through a single computing device, a plurality of computing devices connected through the same network, and / or a plurality of computing devices distributed to be connected through a plurality of different networks.

[0105] The above-described methods and / or various embodiments may be performed by one or more processors configured to execute one or more computer programs that process, store, and / or manage any functions, functions, etc. by operating based on input data or generating output data. For example, the methods and / or various embodiments of the present invention may be performed by special-purpose logic circuits such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and an apparatus and / or system for performing the methods and / or embodiments of the present invention may be implemented as a special-purpose logic circuit such as an FPGA or an ASIC.

[0106] One or more processors executing a computer program may include one or more processors of a general-purpose or special-purpose microprocessor and / or any type of digital computing device. The processor may receive instructions and / or data from each of read-only memory and random access memory, or receive instructions and / or data from read-only memory and random access memory. In the present invention, components of a computing device performing the methods and / or embodiments may include one or more processors for executing instructions and one or more memory devices for storing instructions and / or data.

[0107] According to one embodiment, a computing device may exchange data with one or more mass storage devices for storing data. For example, the computing device may receive and / or receive data from a magnetic disc or an optical disc, and may transfer data to a magnetic disc or an optical disc. A computer-readable storage medium suitable for storing instructions and / or data associated with a computer program may include, but is not limited to, any form of non-volatile memory including semiconductor memory devices such as EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable PROM), and flash memory devices. For example, a computer-readable storage medium may include magnetic discs such as internal hard disks or removable disks, optical magnetic discs, CD-ROMs, and DVD-ROMs.

[0108] To provide interaction with a user, the computing device may include, but is not limited to, a display device for providing or displaying information to the user (e.g., CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), etc.) and a pointing device (e.g., keyboard, mouse, trackball, etc.) on which the user can provide input and / or commands, etc. on the computing device. That is, the computing device may further include any other type of device for providing interaction with the user. For example, the computing device may provide any form of sensory feedback to the user for interaction with the user, including visual feedback, auditory feedback and / or tactile feedback. In this regard, the user may provide input to the computing device through various gestures such as visual, vocal, and motion.

[0109] In the present invention, various embodiments may be implemented in a computing system comprising backend components (e.g., data servers), middleware components (e.g., application servers), and / or frontend components. In this case, the components may be interconnected by any form or medium of digital data communication, such as a communication network. For example, the communication network may include a Local Area Network (LAN), a Wide Area Network (WAN), etc.

[0110] A computing device based on the exemplary embodiments described herein may be implemented using hardware and / or software configured to interact with a user, including a user device, a user interface (UI) device, a user terminal, or a client device. For example, the computing device may include a portable computing device such as a laptop computer. Additionally or alternatively, the computing device may include, but is not limited to, Personal Digital Assistants (PDAs), tablet PCs, game consoles, wearable devices, Internet of Things (IoT) devices, Virtual Reality (VR) devices, Augmented Reality (AR) devices, etc. The computing device may further include other types of devices configured to interact with a user. Additionally, the computing device may include a portable communication device suitable for wireless communication over a network such as a mobile communication network (e.g., a mobile phone, a smartphone, a wireless cellular phone, etc.). A computing device may be configured to communicate wirelessly with a network server using wireless communication technologies and / or protocols such as radio frequency (RF), microwave frequency (MWF) and / or infrared frequency (IRF).

[0111] Various embodiments of the present invention, including specific structural and functional details, are exemplary. Accordingly, the embodiments of the present invention are not limited to those described above and may be implemented in various other forms. Furthermore, the terms used in the present invention are intended to describe some embodiments and are not to be interpreted as limiting the embodiments. For example, singular words and the above may be interpreted to include plural forms unless the context clearly indicates otherwise.

[0112] In this invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which such concepts belong. Furthermore, commonly used terms, such as those defined in advance, should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology.

[0113] Although the present invention has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the invention as understood by a person skilled in the art to which the invention pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification. Explanation of the symbols

[0115] 110: Training data 120: Batch 130: Language Model

Claims

Claim 1 A model training method for Word Sense Disambiguation (WSD) performed by at least one processor, comprising: a step of generating a batch including a first set of training data associated with a first sense of a word and a second set of training data associated with a second sense of a word; and a step of generating a model that performs WSD for said word by performing fine-tuning on a pre-trained language model based on said generated batch; wherein the step of generating a model that performs WSD comprises a step of generating a model that performs WSD for said word by performing fine-tuning on said pre-trained language model using a margin-based loss and a mask language modeling loss; and wherein the margin-based loss It consists of, and here, is the loss value, and X represents the above arrangement, represents the above language model, and i and j represent the indices of the training data, represents the distance between the embedding vectors of the first training data and the second training data, and , , , , is a model training method associated with WSD representing trainable parameters. Claim 2 delete Claim 3 A method for training a model associated with WSD, comprising: in claim 1, the step of generating a model that performs WSD on the word using the margin-based loss and the mask language modeling loss, wherein the model generates a model that performs WSD on the word using a linear combination of the margin-based loss and the mask language modeling loss. Claim 4 delete Claim 5 In claim 1, the mask language modeling loss is It consists of, and here, is the loss value, and X represents the above arrangement, represents the above language model, and represents a corrupted input that is masked in X with a certain probability, i represents an index of the training data, and t represents the index of a token included in the training data, and Is Indicates whether it is a damaged token, Is Represents the parameters for, A model training method associated with WSD, representing a variable for sampling training data. Claim 6 A method for training a model associated with WSD according to claim 1, wherein the step of generating a model that performs WSD on the word using a linear combination of the margin-based loss and the mask language modeling loss comprises the step of performing back-propagation so that the total loss value of the linear combination of the margin-based loss and the mask language modeling loss becomes smaller. Claim 7 In claim 1, the language model is a WSD-related model training method in which the language model is a DeBERTa (Decoding-enhanced BERT with Disentangled Attention) model. Claim 8 A computer program stored on a computer-readable recording medium for executing a method according to any one of paragraphs 1, 3, 5 through 7 on a computer. Claim 9 A computing device comprising: a communication module; a memory; and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program comprises instructions for generating a batch including a first set of training data associated with a first meaning of a word and a second set of training data associated with a second meaning of a word, generating a model that performs WSD on the word by performing fine-tuning on a pre-trained language model based on the generated batch, and generating a model that performs WSD on the word by performing fine-tuning on the pre-trained language model using a margin-based loss and a mask language modeling loss, wherein the margin-based loss It consists of, and here, is the loss value, and X represents the above arrangement, represents the above language model, and i and j represent the indices of the training data, represents the distance between the embedding vectors of the first training data and the second training data, and , , , , is a computing device representing training parameters. Claim 10 delete Claim 11 In claim 9, the computing device further comprises instructions for generating a model that performs WSD on the word using a linear combination of the margin-based loss and the mask language modeling loss, wherein at least one program. Claim 12 delete Claim 13 In paragraph 9, the above mask language modeling loss is It consists of, and here, is the loss value, and X represents the above arrangement, represents the above language model, and represents a corrupted input that is masked with a certain probability on X, i represents an index of the training data, and t represents an index of a token included in the training data, and Is Indicates whether it is a damaged token, Is Represents the parameters for, A computing device representing a variable for sampling training data. Claim 14 In claim 9, the computing device further comprises instructions for performing backpropagation such that the total loss value of the linear combination of the margin-based loss and the mask language modeling loss is reduced. Claim 15 In claim 9, the above language model is a DeBERTa (Decoding-enhanced BERT with Disentangled Attention) model, a computing device.

Images