Task-aware knowledge distillation apparatus and method for performing task-aware knowledge distillation based on dynamic token selection and dynamic token integration

Abstract

The present invention relates to a task awareness-based knowledge distillation apparatus and method for performing task awareness-based knowledge distillation based on dynamic token selection and dynamic token integration. According to one embodiment of the present invention, the task awareness-based knowledge distillation apparatus may include: a dynamic token selection unit that learns a relationship diagram for each of the general tokens from a teacher model to generate a positive mask and a sequence token based on context tokens and token weights; a dynamic token integration unit that integrates the context tokens and the general tokens using local residual adjustment and global residual adjustment to generate a task awareness token embedding; a loss function processing unit that calculates a margin loss for the task awareness token embedding based on the sequence token, aligns the task awareness token embedding and the student model embedding through mean squared error, and learns the task loss to output a fine-tuned task awareness token; and a knowledge distillation processing unit that performs knowledge distillation of the student model using the task awareness token.

Description

Technology Field

[0001] The present invention relates to a task-aware knowledge distillation apparatus and method for performing task-aware knowledge distillation based on dynamic token selection and dynamic token integration. Specifically, it relates to a technique for performing task-aware knowledge distillation from a teacher model to a student model by generating task-aware token embeddings, which are task-aware representations, from a teacher model based on dynamic token selection and dynamic token integration. Background Technology

[0002] Knowledge distillation has emerged as an important technique for compressing large-scale language models into a deployable version.

[0003] Existing approaches focus on conveying knowledge through language units of various lengths (e.g., tokens, sentences, and sequences).

[0004] However, modern language models often fail to capture the essential hierarchical attention mechanisms they utilize.

[0005] Conventional technology has a problem in representing subspaces because different attentions in the representation space lead to entangled representations at the granularity level.

[0006] Recent developments have sought sophisticated approaches to knowledge transfer between teacher and student models, introducing various techniques such as adjusting token levels for contrastive learning, mimicking sequence-level knowledge transfer through others, or addressing the quality of distillation; however, there is a problem in that the independence of processing at each level is compromised.

[0007] In summary, in conventional multi-segment distillation, length-based distillation using tokens, words, and sequences makes it difficult to leverage the benefits of segmentation because attention scores are mostly similar, and difficulties exist in detailed distillation due to overlapping attention patterns. Prior art literature

[0008] Korean Published Patent No. 10-2024-0074690, "Method and system for pre-training a vision transformer through knowledge distillation, and a vision transformer pre-trained thereby" Korean Published Patent No. 10-2024-0138038, "Method for pre-training a vision-language transformer and an artificial intelligence system including a vision-language transformer pre-trained thereby" Korean Published Patent No. 10-2024-0163867, "Data generation device for transfer learning that improves the performance of knowledge distillation from a teacher model to a student model and a method for improving the performance of knowledge distillation by the same" The problem to be solved

[0009] The present invention aims to provide a task-aware knowledge distillation apparatus and method that performs task-aware knowledge distillation from a teacher model to a student model by generating task-aware token embeddings, which are task-aware representations, in a teacher model based on dynamic token selection and dynamic token integration.

[0010] The present invention aims to provide a task recognition-based knowledge distillation apparatus and method that generates task recognition tokens, which are task-specific representations, in real time from a teacher model and effectively distills knowledge from the task recognition tokens into a structurally simple student model.

[0011] The present invention aims to perform knowledge distillation by internalizing task recognition representations within a lightweight structure based on information transmitted from the teacher model, without a token selection and integration structure, as task recognition tokens, which are task-specific representations, are generated in real-time in the teacher model. means of solving the problem

[0012] According to one embodiment of the present invention, a task recognition-based knowledge distillation device may include a dynamic token selection unit that learns a relationship diagram for each of the general tokens from a teacher model to generate a positive mask and a sequence token based on context tokens and token weights; a dynamic token integration unit that integrates the context tokens and the general tokens using local residual adjustment and global residual adjustment to generate a task recognition token embedding; a loss function processing unit that calculates a margin loss for the task recognition token embedding based on the sequence token, aligns the task recognition token embedding and the student model embedding through mean squared error, and learns the task loss to output a fine-tuned task recognition token; and a knowledge distillation processing unit that performs knowledge distillation of the student model using the task recognition token.

[0013] The dynamic token selection unit calculates cross-multi-head attention between a first sentence and a second sentence constituting each of the general tokens to derive a first attention result for the first sentence and a second attention result for the second sentence, merges the first attention result and the second attention result to generate a merged token, and generates context tokens by applying self-attention, residual linking, and layer normalization to the merged token.

[0014] The dynamic token selection unit calculates a token weight that emphasizes a selected token among the context tokens in a downstream task by calculating the relative importance of each of the context tokens as a scalar vector, and can generate a positive mask representing a criterion value for selecting a positive token using the calculated token weight.

[0015] The above positive mask can be used to sort the calculated token weights in descending order, set a threshold based on an index where the cumulative weight is greater than or equal to the total sum of the preset probe value, and select tokens among the calculated token weights that are greater than or equal to the threshold as the positive tokens.

[0016] The dynamic token selection unit calculates a token weight that emphasizes the selected token among the context tokens in a downstream task by calculating the relative importance of each of the context tokens as a scalar vector, and generates a sequence token as a core vector representing the overall meaning of the sentence by generating a sequence level vector by multiplying the context tokens and the token weights to reflect the token-specific weights and calculating the weighted sum of the context tokens.

[0017] The dynamic token integration unit can determine a positive token in general tokens through the positive mask, perform local residual adjustment based on the positive token in integrated tokens formed by integrating the context tokens and general tokens, and perform global residual adjustment based on the positive token in the sequence token.

[0018] The dynamic token integration unit can generate the task recognition token embedding by normalizing the locally and globally adjusted information through adjustment with local scale parameters in the local residual adjustment, fine-tuning centered on the positive token, and adjustment with global scale parameters while maintaining consistency with sequence level information in the global residual adjustment.

[0019] The loss function processing unit can calculate the margin loss by learning the distance difference between positive tokens and negative tokens based on the sequence tokens, align representation similarity by aligning the task recognition token embeddings of the teacher model and the embeddings of the student model through mean squared error, and calculate the task loss by predicting the correct answer based on the downstream task.

[0020] A task recognition-based knowledge distillation method according to an embodiment of the present invention may include: a step in a dynamic token selection unit of learning a relationship diagram for each of the general tokens from a teacher model to generate a positive mask and a sequence token based on context tokens and token weights; a step in a dynamic token integration unit of integrating the context tokens and the general tokens using local residual adjustment and global residual adjustment to generate a task recognition token embedding; a step in a loss function processing unit of calculating a margin loss for the task recognition token embedding based on the sequence token, aligning the task recognition token embedding and the student model embedding through mean squared error, and learning the task loss to output a fine-tuned task recognition token; and a step in a knowledge distillation processing unit of performing knowledge distillation of the student model using the task recognition token.

[0021] The step of generating positive masks and sequence tokens based on context tokens and token weights by learning a relationship map for each of the general tokens from the above-mentioned teacher model comprises: calculating cross-multihead attention between a first sentence and a second sentence constituting each of the general tokens to derive a first attention result for the first sentence and a second attention result for the second sentence; merging the first attention result and the second attention result to generate a merged token; and generating the context tokens by applying self-attention, residual linking, and layer normalization to the merged token; calculating token weights that emphasize the selected token among the context tokens in a downstream task by computing the relative importance of each of the context tokens as a scalar vector; generating the positive mask representing a criterion value for selecting a positive token using the calculated token weights; and computing the relative importance of each of the context tokens as a scalar vector The method may include the step of calculating a token weight that highlights a selected token among the context tokens in a downstream task, and generating a sequence level vector by multiplying the context tokens and the token weights to reflect the token-specific weights and calculating the weighted sum of the context tokens to generate the sequence token as a core vector representing the overall meaning of the sentence. Effects of the invention

[0022] The present invention can provide a task-aware knowledge distillation apparatus and method that performs task-aware knowledge distillation from a teacher model to a student model by generating task-aware token embeddings, which are task-aware representations, in a teacher model based on dynamic token selection and dynamic token integration.

[0023] The present invention can provide a task recognition-based knowledge distillation apparatus and method that generates task recognition tokens, which are task-specific representations, in real time from a teacher model and effectively distills knowledge from the task recognition tokens into a structurally simple student model.

[0024] As the present invention generates task recognition tokens, which are task-specific representations, in real time in the teacher model, the student model can perform knowledge distillation by internalizing task recognition representations within a lightweight structure based on information transmitted from the teacher model without a token selection and integration structure. Brief explanation of the drawing

[0025] FIGS. 1 and 2 are drawings illustrating a task recognition-based knowledge distillation apparatus according to an embodiment of the present invention. FIG. 3 is a diagram illustrating dynamic token selection of a task recognition-based knowledge distillation method according to an embodiment of the present invention. FIG. 4 is a diagram illustrating dynamic token integration of a task recognition-based knowledge distillation method according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the task recognition-based knowledge distillation procedure of a task recognition-based knowledge distillation method according to an embodiment of the present invention. Specific details for implementing the invention

[0026] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0027] The embodiments and the terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of said embodiments.

[0028] In describing various embodiments below, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the invention, such detailed description will be omitted.

[0029] Furthermore, the terms described below are defined considering their functions in various embodiments, and these may vary depending on the intentions or practices of the user or operator. Therefore, their definitions should be based on the content throughout this specification.

[0030] In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0031] A singular expression may include a plural expression unless the context clearly indicates otherwise.

[0032] In this document, expressions such as "A or B" or "at least one of A and / or B" may include all possible combinations of the items listed together.

[0033] Expressions such as "first," "second," "first," or "second" may modify the corresponding components regardless of order or importance, and are used merely to distinguish one component from another without limiting the components.

[0034] Where it is stated that a certain (e.g., first) component is "(functionally or telecommunicationally) connected" or "connected" to another (e.g., second) component, said certain component may be directly connected to said other component or connected through another component (e.g., third component).

[0035] In this specification, "configured to" may be used interchangeably with, depending on the context, for example, in hardware or software, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to."

[0036] In some situations, the expression "device configured to..." may mean that the device is "able to..." together with other devices or parts.

[0037] For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a general-purpose processor capable of performing said operations by executing one or more software programs stored in a memory device (e.g., a CPU or an application processor).

[0038] Also, the term 'or' means an inclusive or rather an exclusive or.

[0039] That is, unless otherwise noted or is not clear from the context, the expression 'x uses a or b' means any one of the natural inclusive permutations.

[0040] Terms such as '..bu', '..gi' used below refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or a combination of hardware and software.

[0042] FIGS. 1 and 2 are drawings illustrating a task recognition-based knowledge distillation apparatus according to an embodiment of the present invention.

[0043] FIG. 1 illustrates the components of a task recognition-based knowledge distillation apparatus according to an embodiment of the present invention.

[0044] Referring to FIG. 1, a task recognition-based knowledge distillation device (100) according to one embodiment of the present invention includes a dynamic token selection unit (110), a dynamic token integration unit (120), a loss function processing unit (130), and a knowledge distillation processing unit (140).

[0045] For example, a task recognition-based knowledge distillation device (100) according to one embodiment of the present invention may be a device that performs a dynamic token selection and integration-based knowledge distillation method for a task-specific natural language processing task.

[0046] For example, the dynamic token selection unit (110) can learn the relationship diagram for each of the general tokens from the teacher model to generate context tokens, positive masks based on token weights, and sequence tokens.

[0047] A dynamic token selection unit (110) according to an embodiment of the present invention can calculate a cross-multi-head attention between a first sentence and a second sentence constituting each of the general tokens to derive a first attention result for the first sentence and a second attention result for the second sentence.

[0048] For example, the dynamic token selection unit (110) can generate a merged token by merging the first attention result and the second attention result, and generate context tokens by applying self-attention, residual linking, and layer normalization to the merged token.

[0049] According to one embodiment of the present invention, the dynamic token selection unit (110) calculates a token weight that emphasizes a selected token among the context tokens in a downstream task by calculating the relative importance of each of the context tokens as a scalar vector, and can generate the positive mask representing a criterion value for selecting a positive token using the calculated token weight.

[0050] For example, a positive mask can be used to sort the calculated token weights in descending order, set a threshold based on an index where the cumulative weight is greater than or equal to the total sum of the preset probe value, and select tokens with calculated token weights greater than or equal to the threshold as positive tokens.

[0051] According to one embodiment of the present invention, the dynamic token selection unit (110) calculates a token weight that emphasizes a selected token among the context tokens in a downstream task by calculating the relative importance of each context token as a scalar vector, and calculates the weighted sum of the context tokens by multiplying the context tokens and the token weights to reflect the weight per token, thereby generating a sequence level vector and generating a sequence token as a core vector representing the overall meaning of the sentence.

[0052] For example, the dynamic token integration unit (120) can generate a task recognition token embedding by integrating the context tokens and the general tokens using local residual adjustment and global residual adjustment.

[0053] According to one embodiment of the present invention, the dynamic token integration unit (120) can determine a positive token in general tokens through the positive mask, perform local residual adjustment based on the positive token in the integrated tokens formed by integrating context tokens and general tokens, and perform global residual adjustment based on the positive token in the sequence token.

[0054] For example, the dynamic token integration unit (120) can generate a task recognition token embedding by normalizing the local and global adjusted information by adjusting with local scale parameters in local residual adjustment, performing fine-tuning centered on positive tokens, and adjusting with global scale parameters while maintaining consistency with sequence level information in global residual adjustment.

[0055] According to one embodiment of the present invention, the loss function processing unit (130) calculates a margin loss for the task recognition token embedding based on the sequence token, aligns the task recognition token embedding and the student model embedding through mean squared error, and learns the task loss to output a fine-tuned task recognition token.

[0056] For example, the loss function processing unit (130) can calculate the margin loss by learning the distance difference between positive tokens and negative tokens based on sequence tokens, align the representation similarity by aligning the teacher model's task recognition token embeddings and the student model's embeddings through mean squared error, and calculate the task loss by predicting the correct answer based on the downstream task.

[0057] According to one embodiment of the present invention, the knowledge distillation processing unit (140) can perform knowledge distillation of the student model using a task recognition token.

[0058] For example, the knowledge distillation processing unit (140) can perform task recognition-based knowledge distillation within a lightweight structure based on information received from the teacher model in the student model.

[0059] Accordingly, the present invention can provide a task-aware knowledge distillation apparatus and method that generates task-aware token embeddings, which are task-aware representations, from a teacher model based on dynamic token selection and dynamic token integration, and performs task-aware knowledge distillation from a teacher model to a student model.

[0060] FIG. 2 illustrates the components of a task recognition-based knowledge distillation device according to an embodiment of the present invention.

[0061] Accordingly, the present invention can provide a task recognition-based knowledge distillation apparatus and method that generates task recognition tokens, which are task-specific representations, in real time from a teacher model and effectively distills knowledge from the task recognition tokens into a structurally simple student model.

[0062] Figure 2 specifically describes a configuration in which a task recognition-based knowledge distillation device according to an embodiment of the present invention generates a task recognition embedding using a dynamic token selection unit and a dynamic token integration unit in a teacher model, and the student model internalizes a task recognition representation within a lightweight structure based on information received from the teacher model.

[0063] Referring to FIG. 2, a task recognition-based knowledge distillation device (200) according to one embodiment of the present invention includes a dynamic token selection unit (210), a dynamic token integration unit (220), a loss function processing unit (230), and a knowledge distillation processing unit (240).

[0064] According to one embodiment of the present invention, the dynamic token selection unit (210) can extract general tokens, which are the last hidden state, through the encoder of the teacher model, and generate context tokens by calculating the interaction between the question and the context of the first sentence and the second sentence constituting each of the extracted general tokens as cross-attention and self-multi-head attention.

[0065] The dynamic token selection unit (210) generates token weights by calculating the importance of each token using z-score normalization and sigmoid.

[0066] The dynamic token selection unit (210) generates a positive mask that selects a top token as a positive token based on accumulated weights.

[0067] The dynamic token selection unit (210) can generate a sequence token as a weighted average vector from the selected token and weight.

[0068] The dynamic token integration unit (220) combines a general token and a context token using a GRU style gate (gated recurrent unit, GRU) mechanism.

[0069] GRU provides a filter function that automatically adjusts which information from input values ​​to remember and which to forget.

[0070] That is, the dynamic token integration unit (220) controls the ratio of general tokens and context tokens when integrating general tokens and context tokens.

[0071] The dynamic token integration unit (220) performs local residual adjustment centered on the integration token and global residual adjustment centered on the sequence token to generate a refined task-aware token embedding.

[0072] The student model uses the same encoder as the teacher model and performs distillation learning by utilizing task recognition tokens generated by the teacher model, without including adapter structures such as the dynamic token selection unit (210) and the dynamic token integration unit (220).

[0073] The loss function processing unit (230) can learn the distance difference between positive tokens and negative tokens using a sequence token as an anchor, and learn to exclude tokens similar to negative tokens as target tokens for task recognition token embedding.

[0074] The loss function processing unit (230) calculates the mean square error (MSE) loss function and aligns the task recognition token embeddings of the teacher model and the student model through the mean square error.

[0075] That is, the loss function processing unit (230) can directly align expression similarities to perform feature level knowledge distillation.

[0076] The loss function processing unit (230) processes the task loss by predicting the correct answer based on the original downstream task, such as classification or span prediction, and calculating the loss.

[0077] The knowledge distillation learning result of a language model based on a task recognition-based knowledge distillation device (200) according to one embodiment of the present invention can be compared with the prior art as shown in Table 1 below.

[0078] [Table 1]

[0079]

[0080] Referring to Table 1, the performance difference (Diff) can be identified by comparing the ratio of the prediction that exactly matches the correct answer (Exact Match, EM) and the ratio of words that overlap with the correct answer (F1) by deriving training results for various language models.

[0081] EM scores are accepted as correct only when they exactly match the correct answer, and F1 scores are accepted for partial answers.

[0082] When comparing the present invention (Ours) with the prior art (Base), the performance regarding knowledge distillation accuracy in most language models is improved and increased.

[0083] In particular, for the Mobile BERT language model corresponding to the mobile dataset, EM shows a score difference of 56.43 and F1 shows 53.75, which indicates that there is the greatest improvement for the mobile dataset.

[0084] That is, knowledge distillation according to one embodiment of the present invention provides more effective improvement performance for mobile datasets and demonstrates that it can be specialized for language models based on mobile datasets with limited data resources.

[0086] FIG. 3 is a diagram illustrating dynamic token selection of a task recognition-based knowledge distillation method according to an embodiment of the present invention.

[0087] FIG. 3 illustrates a configuration in which a dynamic token selection unit of a task recognition-based knowledge distillation method according to an embodiment of the present invention generates a context token, which is a context information enhancement embedding generated through question and context interaction, self-attention, and normalization processes.

[0088] Referring to FIG. 3, a dynamic token selection unit (300) according to an embodiment of the present invention receives general tokens as input in step (S301) and identifies a first sentence and a second sentence constituting each of the general tokens as a question and context in step (S302).

[0089] In step (S303), the dynamic token selection unit (300) performs cross-multi-head attention calculation between question contexts.

[0090] Multi-head attention calculation is derived from the query, key, and value and can be represented in the form of two attention results (question_to_context_attn and context_to_question_attn).

[0091] In step (S305), the dynamic token selection unit (300) generates merged tokens by weighting the two attention results with a learnable parameter alpha.

[0092] Additionally, the dynamic token selection unit (300) generates context tokens, token weight operations, sequence tokens, and a positive mask in step (S306) by applying self-attention to merged tokens and performing residual connection and layer normalization.

[0093] The dynamic token selection unit (300) reflects deep semantic interaction between context and question to preserve token-level information useful in downstream tasks and can be utilized for subsequent token weight calculation, sequence token calculation, and knowledge distillation learning.

[0094] Token weights are scalar value vectors representing the relative importance of each token in the range [0, 1], which highlight important tokens in downstream tasks.

[0095] Token weights are linearly transformed for context tokens, adjusted to a distribution of mean (0) and variance (1) in Z-score normalization, and all values ​​are converted into importance scores between (0, 1) through the application of a sigmoid function.

[0096] Token weights are used as weights for each token when calculating sequence-level representations, can be utilized as criteria for selecting positive tokens, and can also be used as XOR-style loss weights during contrastive learning.

[0097] Sequence tokens are sequence-level embeddings that compress the overall meaning of a sentence, generated by weighting the context tokens and then applying L2 normalization.

[0098] Sequence tokens are multiplied by context tokens and token weights to reflect token-specific weights, and a sequence-level vector of the form (B, 1, H) is generated from the calculation of the weighted sum of all tokens, after which it is refined into a directional representation through L2 normalization. Here, L2 normalization refers to data normalization in artificial intelligence learning.

[0099] Sequence tokens are core vectors representing the overall meaning of a sentence, are used for calculating global residuals in the dynamic token integration unit, and serve as reference points in margin loss to contribute to token-level shrinkage learning.

[0100] The positive mask can be a Boolean mask tensor that marks the token deemed most important within each batch as 1.

[0101] The positive mask sorts the token weights in descending order and sets a threshold based on the minimum index where the accumulated weight is greater than or equal to the product of the positive probe and the total sum.

[0102] The positive mask is used to select positive tokens by setting tokens above a threshold value in the token weight to "1".

[0103] The positive mask is used to highlight positive tokens in the local and global residual calculations of the token integration section.

[0104] The positive mask can be used as a criterion for separating positive and negative tokens in the calculation of the margin loss function.

[0105] Positive masks can be used as an element to enhance information selectivity and focus during the learning process.

[0107] FIG. 4 is a diagram illustrating dynamic token integration of a task recognition-based knowledge distillation method according to an embodiment of the present invention.

[0108] FIG. 4 illustrates a method in which a dynamic token integration unit of a task recognition-based knowledge distillation method according to an embodiment of the present invention integrates a context token and a general token to finally generate a task recognition token embedding.

[0109] Referring to FIG. 4, a dynamic token integration unit (400) according to an embodiment of the present invention performs local residual adjustment in step (S401), performs global residual adjustment in step (S402), and generates a task recognition token in step (S403).

[0110] A dynamic token integration unit (400) according to one embodiment of the present invention can integrate information precisely by combining a GRU-style gate mechanism with local and global residual adjustment and normalization.

[0111] A dynamic token fusion unit (400) according to one embodiment of the present invention prepares to combine information between context-aware tokens and general tokens through input fusion, and uses the two representations simply connected as input to a GRU-style operation.

[0112] The dynamic token integration unit (400) performs a gate operation to softly determine how much context information to reflect, similar to GRU.

[0113] The gate operation controls how much of the candidate value is reflected in the update, and controls how much of the normal token is ignored and how much the context is focused during the reset.

[0114] The dynamic token integration unit (400) generates an update vector by combining existing information and new information to generate candidate embeddings, applies a reset to a general token, and then combines it with the context to pass through non-linear activation.

[0115] The dynamic token integration unit (400) updates the group style by utilizing an update gate to appropriately mix context-aware information and general information to update the group style, and updates the group style in a form in which context-based candidates are partially inserted.

[0116] The dynamic token integration unit (400) corrects how much the integrated token information differs from the general token only for positive tokens, by adjusting with local scale parameters and performing local residual adjustment that performs fine-tuning centered on positive tokens.

[0117] In step (S401), the dynamic token integration unit (400) determines positive tokens from general tokens using a positive mask and performs local residual adjustment to extract positive tokens from integrated tokens.

[0118] In step (S402), the dynamic token integration unit (400) performs global residual adjustment to extract positive tokens from sequence tokens.

[0119] Global residual adjustment corrects how well each token matches the sequence token representing the entire context, while maintaining consistency with sequence-level information, and adjusts using the global scale parameter.

[0120] In step (S403), the dynamic token integration unit (400) normalizes local and global coordinated information to generate a stable task recognition embedding, and generates a task recognition token embedding as a representative ready to be delivered to the next task layer for the final integration of the residual structure.

[0122] FIG. 5 is a diagram illustrating the task recognition-based knowledge distillation procedure of a task recognition-based knowledge distillation method according to an embodiment of the present invention.

[0123] FIG. 5 illustrates a procedure in which a task-cognitive-based knowledge distillation method according to an embodiment of the present invention performs task-cognitive-based knowledge distillation from a teacher model to a student model according to a task-cognitive-based knowledge distillation procedure.

[0124] For example, a task-cognition-based knowledge distillation method can be referred to as a dynamic token selection and integration-based knowledge distillation method for task-specific natural language processing tasks.

[0125] Referring to FIG. 5, in step (S501), the task recognition-based knowledge distillation method according to an embodiment of the present invention generates context tokens, a positive mask, and a sequence token from general tokens of a teacher model.

[0126] That is, the task recognition-based knowledge distillation method according to one embodiment of the present invention can generate positive masks and sequence tokens based on context tokens and token weights by learning a relationship diagram for each of the general tokens from a teacher model.

[0127] In step (S502), the task recognition-based knowledge distillation method according to an embodiment of the present invention can generate a task recognition token embedding by integrating context tokens and general tokens.

[0128] That is, the task awareness-based knowledge distillation method according to one embodiment of the present invention can generate a task awareness token embedding by integrating context tokens and general tokens using local residual adjustment and global residual adjustment.

[0129] In step (S503), the task recognition-based knowledge distillation method according to an embodiment of the present invention learns and fine-tunes the margin loss, mean error squared loss, and task loss to output a task recognition token.

[0130] That is, the task recognition-based knowledge distillation method according to one embodiment of the present invention calculates a margin loss for a task recognition token embedding based on a sequence token, aligns the task recognition token embedding and the student model embedding through mean squared error, and learns the task loss to output a fine-tuned task recognition token.

[0131] In step (S504), the task recognition-based knowledge distillation method according to an embodiment of the present invention performs knowledge distillation of a student model using a task recognition token.

[0132] That is, the task recognition-based knowledge distillation method according to one embodiment of the present invention can perform knowledge distillation of the student model using a task recognition token.

[0133] Accordingly, as the present invention generates task recognition tokens, which are task-specific representations, in real time in the teacher model, the student model can perform knowledge distillation by internalizing task recognition representations within a lightweight structure based on information transmitted from the teacher model without a token selection and integration structure.

[0135] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0136] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0137] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0138] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0139] 100: Task Awareness-Based Knowledge Distillation Device 110: Dynamic Token Selection Unit 120: Dynamic Token Integration Unit 130: Loss Function Processing Unit 140: Knowledge Distillation Processing Unit

Claims

Claim 1 A dynamic token selection unit that learns a relationship map for each general token from a teacher model to generate context tokens from the general tokens, calculates token weights representing the relative importance of each context token, generates a positive mask for selecting tokens above a threshold value as positive tokens based on the token weights, and generates a sequence token as a sequence-level embedding representing the overall meaning of a sentence using the context tokens and the token weights; a dynamic token integration unit that integrates the context tokens and the general tokens to generate an integrated token and generates a task recognition token embedding based on the integrated token; a loss including a mean squared error that calculates a margin loss for the task recognition token embedding based on the distance difference between the positive token and the negative token based on the sequence token and reduces the difference between the task recognition token embedding and the student model embedding generated in the student model corresponding to the general tokens, and based on the difference between the predicted value based on the task recognition token embedding and the correct answer for the downstream task A task recognition-based knowledge distillation device characterized by comprising: a loss function processing unit that learns a calculated task loss and outputs a task recognition token; and a knowledge distillation processing unit that performs knowledge distillation of the student model by using the task recognition token as a reference to align the embedding of the student model so that it corresponds to the task recognition token. Claim 2 A task recognition-based knowledge distillation device according to claim 1, wherein the dynamic token selection unit calculates cross-multi-head attention between a first sentence and a second sentence constituting each of the general tokens to derive a first attention result for the first sentence and a second attention result for the second sentence, merges the first attention result and the second attention result to generate a merged token, and generates context tokens by applying self-attention, residual linking, and layer normalization to the merged token. Claim 3 A task recognition-based knowledge distillation apparatus according to claim 1, wherein the dynamic token selection unit calculates a scalar value representing relative importance by applying a non-linear function after normalizing the linear transformation result of each of the context tokens, generates a vector of the scalar values ​​to calculate a token weight that emphasizes the selected token among the context tokens in the downstream task, and generates a positive mask representing a criterion value for selecting the positive token using the calculated token weight. Claim 4 A task recognition-based knowledge distillation device according to claim 1, characterized in that the positive mask is used to sort the calculated token weights in descending order, set the threshold value based on an index where the cumulative weight is greater than or equal to the total sum of the preset probe value, and select tokens that are greater than or equal to the threshold value among the calculated token weights as the positive tokens. Claim 5 A task-cognition-based knowledge distillation device according to claim 1, wherein the dynamic token selection unit calculates a scalar value representing relative importance by applying a non-linear function after normalizing the linear transformation result of each of the context tokens, generates a vector of the scalar values ​​to calculate the token weight that emphasizes the selected token among the context tokens in the downstream task, calculates the weighted sum of the context tokens by multiplying the context tokens and the token weights to reflect the token-specific weight, generates a sequence-level vector as the sequence-level embedding, and generates the sequence token as a core vector representing the overall meaning of the sentence. Claim 6 A task recognition-based knowledge distillation apparatus according to claim 1, wherein the dynamic token integration unit determines the positive token through the positive mask in the general tokens, performs local residual adjustment based on the positive token in the integrated tokens formed by integrating the context tokens and the general tokens, and performs global residual adjustment based on the positive token in the sequence tokens. Claim 7 A task recognition-based knowledge distillation device according to claim 6, wherein the dynamic token integration unit adjusts with local scale parameters in the local residual adjustment, performs fine-tuning centered on the positive token, and adjusts with global scale parameters while maintaining consistency with sequence level information in the global residual adjustment to normalize the locally and globally adjusted information and generate the task recognition token embedding. Claim 8 A task recognition-based knowledge distillation device according to claim 6, wherein the loss function processing unit aligns representation similarity by aligning the task recognition token embeddings of the teacher model and the embeddings of the student model through mean squared error. Claim 9 A step of learning a relationship map for each general token from a teacher model to generate context tokens from said general tokens, calculating token weights representing the relative importance of each of said context tokens, generating a positive mask for selecting tokens above a threshold value as positive tokens based on said token weights, and generating a sequence token as a sequence-level embedding representing the overall meaning of a sentence using said context tokens and said token weights; a step in a dynamic token integration unit to integrate said context tokens and said general tokens to generate an integrated token, and generating a task awareness token embedding based on said integrated token; a step in a loss function processing unit to calculate a margin loss for said task awareness token embedding based on the distance difference between said positive tokens and negative tokens relative to said sequence tokens, and to calculate a loss including a mean squared error that reduces the difference between said task awareness token embedding and the student model embedding generated in the student model corresponding to said general tokens, and a predicted value based on said task awareness token embedding and downstream A task recognition-based knowledge distillation method characterized by comprising: a step of outputting a task recognition token by learning a task loss calculated based on the difference between correct answers for a task; and a step of performing knowledge distillation of the student model in a knowledge distillation processing unit, wherein the student model’s embeddings are aligned and learned to correspond to the task recognition tokens using the task recognition tokens as a reference. Claim 10 In claim 9, the step of generating context tokens from the general tokens by learning a relationship map for each of the general tokens from the teacher model, calculating token weights representing the relative importance of each of the context tokens, and generating a sequence token as a sequence-level embedding representing the overall meaning of a sentence using the context tokens and the token weights, and a positive mask for selecting a token greater than or equal to a threshold value as a positive token based on the token weights, is to generate the context tokens by calculating cross-multihead attention between a first sentence and a second sentence constituting each of the general tokens to derive a first attention result for the first sentence and a second attention result for the second sentence, merging the first attention result and the second attention result to generate a merged token, and generating the context tokens by applying self-attention, residual linking, and layer normalization to the merged token; and normalizing the linear transformation result of each of the context tokens and then a non-linear function A step of calculating a scalar value representing relative importance by applying it, generating a vector of the scalar values ​​to calculate a token weight that emphasizes a selected token among the context tokens in the downstream task, and generating a positive mask representing a criterion value for selecting the positive token using the calculated token weight;A task-cognition-based knowledge distillation method characterized by including the step of normalizing the linear transformation results of each of the context tokens and applying a non-linear function to calculate a scalar value representing relative importance, generating a vector of the scalar values ​​to calculate a token weight that emphasizes a selected token among the context tokens in the downstream task, multiplying the context tokens and the token weights to calculate a weighted sum of the context tokens by reflecting the token-specific weights, generating a sequence-level vector as a sequence-level embedding, and generating the sequence token as a core vector representing the overall meaning of the sentence.

Images