Storage device including a transformer model, control device for the transformer model, and control method thereof

The control method for transformer models using singular value decomposition addresses computational and oversmoothing issues, enabling efficient processing of large-scale data by managing computational load and preserving high-frequency information.

KR103044497B1Active Publication Date: 2026-11-06KOREA ADVANCED INST OF SCI & TECH +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020250147952
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-10-14
Publication Date
2026-11-06
Estimated Expiration
2045-10-14

AI Technical Summary

Technical Problem

Conventional transformer models face challenges with exponentially increasing computational demands and oversmoothing issues when processing large-scale data, leading to a neglect of high-frequency information and difficulty in distinguishing token differences.

Method used

A control method and device for transformer models that incorporate singular value decomposition, utilizing embedding, left and right singular vector matrices, and orthogonality loss to manage computational load and prevent oversmoothing, with the aid of artificial neural networks for matrix derivation.

Benefits of technology

The solution effectively controls computational volume and prevents oversmoothing, ensuring efficient processing of large-scale data without exponential increases, maintaining high-frequency information integrity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025114742684-PAT00036_ABST
    Figure 112025114742684-PAT00036_ABST
Patent Text Reader

Abstract

A control method for a transformer model includes a computation step. The computation step includes a step for deriving a left singular vector matrix, a step for deriving a singular value matrix, a step for deriving a right singular vector matrix, and a step for deriving an output matrix. In the step for deriving the left singular vector matrix, a left singular vector matrix is ​​derived from an input embedding matrix. In the step for deriving the singular value matrix, a singular value matrix is ​​derived from the input embedding matrix, a plurality of singular value generation matrices, a plurality of coefficients, and a plurality of basis polynomial functions. In the step for deriving the right singular vector matrix, a right singular vector matrix is ​​derived from the input embedding matrix. In the step for deriving the output matrix, an output matrix is ​​derived through the left singular vector matrix, the singular value matrix, the right singular vector matrix, and the input embedding matrix.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a storage device, a control device, and a control method. More specifically, the invention relates to a storage device comprising a transformer model simulating singular value decomposition, a control device for controlling said transformer model, and a method for controlling said transformer model. Background Technology

[0002] Transformer models can be one of the deep learning-based artificial intelligence models. Transformer models can be trained based on a self-attention mechanism. Transformer models can be utilized in conversational AI, image generation AI, and speech generation AI.

[0003] The self-attention mechanism of conventional transformer models, when input with n tokens, n 2 It may require computations proportional to the input information. Therefore, in a transformer model, if the input information becomes long, the amount of computation increases exponentially, making it difficult to process large-scale data.

[0004] Furthermore, in conventional transformer models, oversmoothing can occur during the iterative learning process. Due to oversmoothing, low-frequency information is emphasized and high-frequency information is ignored, which can make it difficult to distinguish detailed differences between tokens. Prior art literature

[65535] (Patent Document 0001) KR 10-2024-0133384 A(Patent Document 0002) KR 10-2019-0059033 A(Patent Document 0003) KR 10-2674551 B The problem to be solved

[0005] The objective of the present invention is to provide a storage device including a transformer model that simulates singular value decomposition so that the amount of computation does not increase exponentially with the amount of input information and prevents oversmoothing, a control device for controlling the transformer model, and a method for controlling the transformer model. means of solving the problem

[0006] A control method for a transformer model according to an embodiment of the present invention may include an embedding step, a computation step, and a data output step. In the embedding step, an input value may be embedded to derive an input embedding matrix. In the computation step, an output matrix may be derived through the input embedding matrix. In the data output step, an output value may be derived through the output matrix. The computation step may include a left singular vector matrix derivation step, a singular value matrix derivation step, a right singular vector matrix derivation step, and an output matrix derivation step. In the left singular vector matrix derivation step, a left singular vector matrix may be derived from the input embedding matrix. In the singular value matrix derivation step, a singular value matrix may be derived from the input embedding matrix, a plurality of singular value generating matrices, a plurality of coefficients, and a plurality of basis polynomial functions. The plurality of singular value generating matrices may be diagonal matrices. In the right singular vector matrix derivation step, a right singular vector matrix may be derived from the input embedding matrix. In the above output matrix derivation step, the output matrix can be derived through the left singular vector matrix, the singular value matrix, the right singular vector matrix, and the input embedding matrix.

[0007] In one embodiment of the present invention, an artificial neural network may be used in the step of deriving the left singular vector matrix, the step of deriving the singular value matrix, and the step of deriving the right singular vector matrix.

[0008] In one embodiment of the present invention, the singular value matrix can be derived as shown in Equation 1 below. The plurality of singular value generating matrices may be diagonal matrices.

[0009] [Mathematical Formula 1]

[0010]

[0011] Here, Σ is the singular value matrix, and θ i is the coefficient of the i-th term, and T i (x) is the basis polynomial function of the i-th term, K is the number of basis polynomial functions, and Σ g is the singular value generating matrix.

[0012] In one embodiment of the present invention, the left singular vector matrix can be derived through the following Equation 2. The right singular vector matrix can be derived through the following Equation 3.

[0013] [Mathematical Formula 2]

[0014]

[0015] [Mathematical Formula 3]

[0016]

[0017] Here, U is the left singular vector matrix, V is the right singular vector matrix, ρ(A) is the softmax function for matrix A, X is the input embedding matrix, and W U is the weight matrix for deriving the left singular vector matrix, and W V is a weight matrix for deriving the right singular vector matrix, and T is a symbol representing the transpose of a matrix.

[0018] In one embodiment of the present invention, each of the plurality of singular value generation matrices can be derived through the following mathematical formula 4.

[0019] [Mathematical Formula 4]

[0020]

[0021] Here, diag(A) is the diagonal matrix generating function for matrix A, σ(A) is the sigmoid function for matrix A, X is the input embedding matrix, and W Σ is a weight matrix for deriving the singular value generation matrix.

[0022] In one embodiment of the present invention, each of the plurality of basis polynomial functions may be a Jacobi basis polynomial function.

[0023] In one embodiment of the present invention, in the output matrix derivation step, the output matrix can be derived through the following mathematical formula 5.

[0024] [Mathematical Formula 5]

[0025]

[0026] Here, OA is the output matrix, U is the left singular vector matrix, Σ is the singular value matrix, and V T is the transposed right singular vector matrix, X is the input embedding matrix, and W val is a weight matrix for deriving values.

[0027] In one embodiment of the present invention, in the output matrix derivation step, the left singular vector matrix and the singular value matrix can be multiplied through element-wise multiplication.

[0028] In one embodiment of the present invention, the control method of the transformer model may further include a training data input step, a score loss value derivation step, an orthogonality loss value derivation step, and a weight modification step. In the training data input step, training input data including the input value and the correct answer for the input value may be input. In the score loss value derivation step, the output value and the correct answer are compared to derive a score loss value for the output value, and the score loss value may be a value that increases when the difference between the output value and the correct answer increases. In the orthogonality loss value derivation step, the orthogonality loss value of the left singular vector matrix and the right singular vector matrix is ​​derived, and the orthogonality loss value is 0 when the left singular vector matrix and the right singular vector matrix are orthogonal, and may be greater than 0 when at least one of the left singular vector matrix and the right singular vector matrix is ​​not orthogonal. In the weight modification step, the weights of the artificial neural network may be modified through the score loss value and the orthogonality loss value.

[0029] In one embodiment of the present invention, the orthogonality loss value can be derived through the following mathematical formula 6.

[0030] [Mathematical Formula 6]

[0031]

[0032] Here, here L o is the orthogonality loss value, n is the number of tokens, ∥∥ is the norm symbol, U is the left singular vector matrix, V is the right singular vector matrix, and T is the symbol representing the transpose of a matrix, and I is the identity matrix.

[0033] In one embodiment of the present invention, the orthogonality measure of the left singular vector matrix and the right singular vector matrix may be 0 or greater and 0.5 or less. The orthogonality measure may be derived through the following mathematical formula 7.

[0034] [Mathematical Formula 7]

[0035]

[0036] Here, O(A) is the orthogonality measure of matrix A, and T is the symbol representing the transpose of a matrix, and I is the identity matrix.

[0037] A control device according to one embodiment of the present invention may include a control unit that controls a transformer model. The control unit may derive an input embedding matrix by embedding input values. The control unit may derive a left singular vector matrix and a right singular vector matrix from the input embedding matrix. The control unit may derive a singular value matrix through the input embedding matrix and a plurality of Jacobi basis polynomial functions. The control unit may derive an output matrix through the input embedding matrix, the left singular vector matrix, the singular value matrix, and the right singular vector matrix. The control unit may derive an output value through the output matrix.

[0038] In one embodiment of the present invention, the control unit can derive the left singular vector matrix, the singular value matrix, and the right singular vector matrix through an artificial neural network.

[0039] In one embodiment of the present invention, the control unit may derive the singular value matrix through a plurality of singular value generation matrices. The plurality of singular value generation matrices may be diagonal matrices.

[0040] In one embodiment of the present invention, training input data including the input value and the correct answer for the input value may be input to the control unit. The control unit compares the output value with the correct answer to derive a score loss value for the output value, and the score loss value may be a value that increases when the difference between the output value and the correct answer increases. The control unit derives an orthogonality loss value through the left singular vector matrix and the right singular vector matrix, and the orthogonality loss value may be 0 when the left singular vector matrix and the right singular vector matrix are orthogonal, and may be a value greater than 0 when at least one of the left singular vector matrix and the right singular vector matrix is ​​not orthogonal. The control unit may modify the weights of the artificial neural network through the score loss value and the orthogonality loss value.

[0041] In one embodiment of the present invention, the orthogonality measure of the left singular vector matrix and the right singular vector matrix may be 0 or greater and 0.5 or less. The orthogonality measure may be derived through the following mathematical formula 8.

[0042] [Mathematical Formula 8]

[0043]

[0044] Here, O(A) is the orthogonality measure of matrix A, and T is the symbol representing the transpose of a matrix, and I is the identity matrix.

[0045] A storage device according to one embodiment of the present invention may include a transformer model. The transformer model can derive an input embedding matrix by embedding input values. The transformer model can derive a left singular vector matrix and a right singular vector matrix from the input embedding matrix. The transformer model can derive a singular value matrix from the input embedding matrix and a plurality of Jacobi basis matrix. The transformer model can derive an output matrix through the input embedding matrix, the left singular vector matrix, the singular value matrix, and the right singular vector matrix. The transformer model can derive an output value through the output matrix.

[0046] In one embodiment of the present invention, the transformer model can derive the left singular vector matrix, the singular value matrix, and the right singular vector matrix through an artificial neural network.

[0047] In one embodiment of the present invention, the transformer model can derive the singular value matrix through a plurality of singular value generating matrices. The plurality of singular value generating matrices may be diagonal matrices.

[0048] In one embodiment of the present invention, training input data including the input value and the correct answer for the input value may be input to the transformer model. The transformer model derives a score loss value for the output value by comparing the output value with the correct answer, and the score loss value may be a value that increases when the difference between the output value and the correct answer increases. The transformer model derives an orthogonality loss value through the left singular vector matrix and the right singular vector matrix, and the orthogonality loss value may be 0 when the left singular vector matrix and the right singular vector matrix are orthogonal, and may be a value greater than 0 when at least one of the left singular vector matrix and the right singular vector matrix is ​​not orthogonal. The transformer model may modify the weights of the artificial neural network through the score loss value and the orthogonality loss value. Effects of the invention

[0049] According to one embodiment of the present invention, a storage device including a transformer model that prevents an oversmoothing phenomenon and prevents an exponential increase in computational volume depending on the amount of input information, and a control device for controlling said transformer model may be provided.

[0050] In one embodiment of the present invention, a control method for a transformer model is provided in which the amount of computation does not increase exponentially depending on the amount of input information and the oversmoothing phenomenon is prevented. Brief explanation of the drawing

[0051] FIGS. 1 and FIGS. 2 are drawings illustrating an exemplary control device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an exemplary control system according to one embodiment of the present invention. FIG. 4 is a diagram illustrating an artificial neural network according to one embodiment of the present invention. FIG. 5 illustrates an exemplary method for controlling a transformer model according to one embodiment of the present invention. Figure 6 is a graph showing the normalized magnitude according to frequency of the embodiments and comparative examples of the present invention. Figure 7 is a graph showing the cosine similarity according to the Layer Index of the embodiments and comparative examples of the present invention. Specific details for implementing the invention

[0052] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0053] In the drawings, the proportions and dimensions of the components are exaggerated for the effective illustration of the technical content. "And / or" includes all one or more combinations that the associated components may define.

[0054] Terms such as "include" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0055] FIG. 1 is a diagram illustrating an exemplary control device (10) according to one embodiment of the present invention.

[0056] Referring to FIG. 1, the control device (10) may include a storage device (100) and a control unit (200). The control device (10) may be an electronic device. The control device (10) may be a terminal such as a computer, a smartphone, and a tablet PC. The control device (10) may be a remote computing device such as a cloud server and an edge server.

[0057] Meanwhile, the configuration of the control device (10) is not limited to the above. For example, the control device (10) may further include an input / output unit. Data may be input and output through the input / output unit. If the control device (10) is a terminal, the input / output unit may include input devices such as a keyboard and a mouse, and output devices such as a monitor. If the control device (10) is a remote computing device, the input / output unit may include a communication module. Additionally, the control device (10) may be a separate device other than a terminal or a remote computing device.

[0058] The storage device (100) may include a Transformer model (TM). The storage device (100) may be a device for storing the Transformer model (TM). The storage device (100) may be a magnetic recording device such as a hard disk, SSD, and flash memory. The storage device (100) may be an optical recording device such as a CD, DVD, and Blu-ray. Meanwhile, the configuration of the storage device (100) is not limited to the above. The storage device (100) is merely a device for storing the Transformer model (TM), and the configuration of the storage device may not be limited.

[0059] The transformer model (TM) may be implemented as software including at least one of a program and an algorithm stored in a storage device (100). Meanwhile, the configuration of the transformer model (TM) is not limited to the above. The transformer model (TM) may be implemented as hardware, and the transformer model (TM) may be implemented as a combination of software and hardware.

[0060] The Transformer model (TM) may include an artificial neural network. The Transformer model (TM) may operate through the weights of the artificial neural network. The configuration of the artificial neural network may be described through a separate drawing.

[0061] The transformer model (TM) of the present invention may be an attention-based graph filter (AGF) transformer model (TM) based on graph signal processing (GSP).

[0062] Data may be input into a Transformer model (TM). The input data may include input values. The input data may be training input data that includes input values ​​and correct answers for the input values. The input data may be inference input data that includes input values ​​but does not include correct answers for the input values.

[0063] The transformer model (TM) can embed input values. The transformer model (TM) can embed input values ​​to derive an input embedding matrix (X).

[0064] The transformer model (TM) can derive an output matrix (OA) through an input embedding matrix (X). The transformer model (TM) can derive a left singular vector matrix (U), a singular value matrix (Σ), and a right singular vector matrix (V) through the input embedding matrix (X).

[0065] The transformer model (TM) is a weight matrix (W) for deriving a left singular vector matrix from an input embedding matrix (X). U After multiplying by ) and substituting this into the softmax function (ρ(A)), the left singular vector matrix (U) can be derived.

[0066] The Transformer model (TM) consists of multiple singular value generating matrices (Σ g ) and multiple coefficients (θ i ), and multiple basis polynomial functions (T i The singular value matrix (Σ) can be derived through (x). The transformer model (TM) derives the weight matrix (W) for deriving the singular value generation matrix from the input embedding matrix (X). Σ After multiplying by ), sequentially substitute into the sigmoid function (σ(A)) and the diagonal matrix generating function (diag(A)) to obtain multiple singular value generating matrices (Σ gEach of these can be derived.

[0067] The transformer model (TM) is a weight matrix (W) for deriving a right singular vector matrix from an input embedding matrix (X). V After multiplying by ), it can be transposed. Then, by substituting the result into the softmax function (ρ(A)), the transpose matrix (V) of the left singular vector matrix T ) can be derived.

[0068] The Transformer model (TM) comprises a left singular vector matrix (U), a singular value matrix (Σ), a right singular vector matrix (V), an input embedding matrix (X), and a weight matrix (W) for deriving values. val The output matrix (OA) can be derived by multiplying ) together.

[0069] A Transformer Model (TM) can derive output data through an output matrix (OA). When training input data is input, the Transformer Model (TM) can derive training output data that includes output values ​​and correct answers for the input values. When inference input data is input, the Transformer Model (TM) can derive inference output data that includes output values ​​but does not include correct answers for the input values.

[0070] The Transformer Model (TM) can be trained using training output data. The Transformer Model (TM) compares the output values ​​of the training output data with the correct answers to determine the score loss value (L s ) can be derived. The Transformer model (TM) derives the orthogonality loss value (L) of the left singular vector matrix and the right singular vector matrix. o ) can be derived. The Transformer model (TM) can derive the score loss value (L s ) and orthogonality loss value (L o The weights of an artificial neural network can be modified by considering ).

[0071] The transformer model(TM) can provide output values ​​to the user.

[0072] Meanwhile, the detailed control method of the Transformer Model(TM) can be explained through a separate drawing.

[0073] The control unit (200) can control the transformer model (TM). The control unit (200) may include a computing device and memory. The computing device may include a CPU and a GPU, etc. The memory may include volatile memory such as DRAM and SRAM. Meanwhile, the configuration of the control unit (200) is not limited to the above.

[0074] The control unit (200) can input data into the transformer model (TM).

[0075] The control unit (200) can embed input values. The control unit (200) can embed input values ​​to derive an input embedding matrix (X).

[0076] The control unit (200) can derive an output matrix (OA) through an input embedding matrix (X). The control unit (200) can derive a left singular vector matrix (U), a singular value matrix (Σ), and a right singular vector matrix (V) through the input embedding matrix (X).

[0077] The control unit (200) is a weight matrix (W) for deriving a left singular vector matrix from an input embedding matrix (X). U After multiplying by ) and substituting this into the softmax function (ρ(A)), the left singular vector matrix (U) can be derived.

[0078] The control unit (200) has a plurality of singular value generation matrices (Σ g ) and multiple coefficients (θ i ), and multiple basis polynomial functions (T i A singular value matrix (Σ) can be derived through (x). The control unit (200) derives a weight matrix (W) for deriving a singular value generation matrix from the input embedding matrix (X). Σ After multiplying by ), sequentially substitute into the sigmoid function (σ(A)) and the diagonal matrix generating function (diag(A)) to obtain multiple singular value generating matrices (Σg Each of these can be derived.

[0079] The control unit (200) is a weight matrix (W) for deriving a right singular vector matrix from an input embedding matrix (X). V After multiplying by ), it can be transposed. Then, by substituting the result into the softmax function (ρ(A)), the transpose matrix (V) of the left singular vector matrix T ) can be derived.

[0080] The control unit (200) comprises a left singular vector matrix (U), a singular value matrix (Σ), a right singular vector matrix (V), an input embedding matrix (X), and a weight matrix (W) for deriving values. val The output matrix (OA) can be derived by multiplying ) together.

[0081] The control unit (200) can derive output data through an output matrix (OA). When training input data is input, the control unit (200) can derive training output data including an output value and a correct answer for the input value. When inference input data is input, the control unit (200) can derive inference output data including an output value and not including a correct answer for the input value.

[0082] The control unit (200) can train the Transformer model (TM) using the training output data. The control unit (200) compares the output value of the training output data with the correct answer to obtain a score loss value (L). s ) can be derived. The control unit (200) derives the orthogonality loss value (L) of the left singular vector matrix and the right singular vector matrix. o ) can be derived. The control unit (200) derives the score loss value (L s ) and orthogonality loss value (L o The weights of an artificial neural network can be modified by considering ).

[0083] The control unit (200) can provide an output value to the user through the transformer model (TM).

[0084] Meanwhile, a detailed method for the control unit (200) to control the transformer model (TM) can be explained through a separate drawing.

[0085] FIG. 2 is a diagram illustrating an exemplary control device (10-1) according to one embodiment of the present invention.

[0086] Referring to FIG. 2, the transformer model (TM) may be included in the control unit (200-1). The transformer model (TM) may be implemented in hardware including at least one of a spiking neural network processor (SNN), a neural processing unit (NPU), and a digital signal processor (DSP).

[0087] Meanwhile, the transformer model (TM) is not limited to being contained in the storage device (100-1) or the control unit (200-1). A part of the transformer model (TM) may be contained in the storage device (100-1), and another part of the transformer model (TM) may be contained in the control unit (200-1). For example, information such as the weights and biases of the transformer model (TM) may be contained in the storage device (100-1), and the artificial neural network of the transformer model (TM) may be implemented in hardware.

[0088] The control unit (200-1) can control the transformer model (TM) in the same way as the control unit (200) of FIG. 1.

[0089] FIG. 3 is a diagram illustrating an exemplary control system (1) according to one embodiment of the present invention.

[0090] Referring to FIG. 3, the control system (1) may include a control device (10-2) and a server (20). A transformer model (TM) may be included in the server (20).

[0091] The control unit (10-2) can communicate with the server (20). The control unit (200-2) can control the transformer model (TM) through communication with the server (20). The control unit (200-2) can control the transformer model (TM) in the same way as the control unit (200) of FIG. 1.

[0092] FIG. 4 is a diagram illustrating an artificial neural network (ANN) according to one embodiment of the present invention.

[0093] Referring to FIG. 4, the artificial neural network (ANN) may include multiple neurons. The multiple neurons may form multiple hidden layers. Multiple input variables may be input into the artificial neural network (ANN), and multiple output variables may be output.

[0094] Multiple neurons may have different weights and biases. Multiple neurons may produce output variables corresponding to input variables. Meanwhile, FIG. 4 is merely an exemplary structure of an artificial neural network (ANN), and the number of hidden layers, the number of neurons, and the connection method may not be limited thereto.

[0095] FIG. 5 illustrates an exemplary method (S10) for controlling a transformer model according to one embodiment of the present invention.

[0096] Referring to FIG. 5, the control method (S10) of the transformer model may include a data input step (S100), an embedding step (S200), a computation step (S300), a data output step (S400), a learning step (S500), and an inference value output step (S600). The control method (S10) of the transformer model may be implemented through the control device (10, 10-1, 10-2) of FIG. 1, FIG. 2, or FIG. 3.

[0097] The data input step (S100) may include a training data input step (S110) and an inference data input step (S120). In the data input step (S100), input data containing input values ​​may be input to the Transformer model (TM). The training data input step (S110) and the inference data input step (S120) may be performed optionally. Depending on the type of input data input to the Transformer model (TM), the training data input step (S110) or the inference data input step (S120) may be performed.

[0098] In the training data input step (S110), training input data including input values ​​and correct answers for the input values ​​may be input. The training input data may be data for training a Transformer model (TM). When the training data input step (S110) is performed, a subsequent training step (S500) may be performed so that the Transformer model (TM) can be trained.

[0099] In the inference data input step (S120), inference input data including an input value may be input. The inference input data may be data that does not include the correct answer. The inference input data may be data intended to provide an output value to the user. When the inference data input step (S120) is performed, the inference value output step (S600) may subsequently be performed to provide an output value to the user.

[0100] In the embedding step (S200), input values ​​are embedded to derive an input embedding matrix (X). The input embedding matrix (X) may be a matrix for mapping multiple tokens into a vector space. Each row of the input embedding matrix (X) may represent a vector corresponding to a single token, and each column of the input embedding matrix (X) may represent the coordinate value of the token in each dimension. The number of rows of the input embedding matrix (X) may be the number of tokens (n), and the number of columns may be the number of embedding dimensions (d). The input embedding matrix (X) may be an (n-xd) matrix.

[0101] In the embedding step (S200), the input value can be separated into multiple tokens. For example, if a natural language sentence is provided as the input value, each word can be separated into tokens. In the embedding step (S200), the input value is decomposed into multiple tokens, and a token sequence can be obtained. A token ID can be set for each of the multiple tokens.

[0102] In the embedding step (S200), an input embedding matrix (X) can be derived through an embedding matrix (E). The embedding matrix (E) may be a matrix in which vectors corresponding to each token ID are stored. The embedding matrix (E) may be a weight matrix stored in a transformer model (TM). In the embedding step (S200), the input embedding matrix (X) can be derived by selecting and inputting a vector corresponding to the token ID of each of the multiple tokens from the embedding matrix (E).

[0103] The operation step (S300) may include a left singular vector matrix derivation step (S310), a singular value matrix derivation step (S320), a right singular vector matrix derivation step (S330), and an output matrix derivation step (S340). In the operation step (S300), an output matrix (OA) may be derived through an input embedding matrix (X).

[0104] The operation step (S300) can be repeated. For example, after the operation step (S300) is completed, the operation step (S300) can be performed again on the output matrix (OA). The output matrix (OA) is substituted into the input embedding matrix (X), and the operation step (S300) is repeated so that the result of the transformer model (TM) can be improved.

[0105] In the step of deriving the left singular vector matrix (S310), the left singular vector matrix (U) can be derived. The left singular vector matrix (U) may be a matrix corresponding to the left singular vector matrix derived by singular value decomposition of the self-attention matrix (H). The left singular vector matrix (U) may emulate an orthogonal matrix. Therefore, the left singular vector matrix (U) may have high orthogonality. In this specification, orthogonality may be a measure indicating the degree to which a matrix is ​​orthogonal.

[0106] In this specification, the orthogonality of a matrix can be derived through an orthogonality determination operation such as Formula 1 below.

[0107] [Formula 1]

[0108]

[0109] Here, O(A) is the orthogonality measure of matrix A, and T is a symbol representing the transpose of a matrix, and I can be the identity matrix.

[0110] Referring to Equation 1, the orthogonality of a matrix can be determined through an orthogonality determination operation. The orthogonality of a matrix may be greater as the orthogonality measure approaches 0. In one embodiment of the present invention, the orthogonality measure (O(U)) of the left singular vector matrix may be approximately 0.3. The orthogonality measure (O(U)) of the left singular vector matrix may be 0 or greater and 0.5 or less. If the orthogonality measure (O(U)) of the left singular vector matrix is ​​greater than 0.5, the performance of the transformer model (TM) may be degraded. More preferably, the orthogonality measure (O(U)) of the left singular vector matrix may be 0.1 or greater and 0.5 or less. If the orthogonality measure (O(U)) of the left singular vector matrix is ​​less than 0.1, the performance of the transformer model (TM) may be degraded.

[0111] Meanwhile, Formula 1 is merely an exemplary construction, and the criteria for determining orthogonality are not limited to the above content.

[0112] An artificial neural network may be used in the step of deriving the left singular vector matrix (S310). The left singular vector matrix (U) may be derived through an artificial neural network. The left singular vector matrix (U) may be a matrix that mimics the left singular vector matrix derived by singular value decomposition of the self-attention matrix (H).

[0113] The number of rows of the left singular vector matrix (U) can be the number of tokens (n), and the number of columns can be the number of embedding dimensions (d). The left singular vector matrix (U) can be an (nХd) matrix.

[0114] The left singular vector matrix (U) can be derived through the left singular vector derivation operation as shown in Equation 2 below.

[0115] <Equation 2>

[0116]

[0117] Here, U is the left singular vector matrix, ρ(A) is the softmax function for matrix A, X is the input embedding matrix, and W Ucan be a weight matrix for deriving the left singular vector matrix.

[0118] Referring to Equation 2, the weight matrix (W) for deriving the left singular vector matrix for the input embedding matrix (X) U After being multiplied, it is substituted into a softmax function (ρ(A)) to derive a left singular vector matrix (U). The softmax function (ρ(A)) may be a normalization function. In one embodiment of the present invention, the softmax function (ρ(A)) may be replaced with another normalization function.

[0119] In one embodiment of the present invention, the left singular vector derivation operation can be performed iteratively. The left singular vector (U) derived through the left singular vector derivation operation is substituted into the input embedding matrix (X), and the left singular vector derivation operation can be performed iteratively. In the present invention, the left singular vector matrix (U) is derived through an artificial neural network, so the time and cost required to obtain the left singular vector matrix (U) can be reduced.

[0120] In the singular value matrix derivation step (S320), a singular value matrix (Σ) can be derived. The singular value matrix (Σ) may be a matrix corresponding to the singular value matrix derived by singular value decomposition of the self-attention matrix (H). The singular value matrix (Σ) may be formed through multiple diagonal matrices.

[0121] The number of rows of the singular value matrix (Σ) can be the number of multiple diagonal matrices. The number of multiple diagonal matrices can be equal to the number of tokens (n). Each row of the singular value matrix (Σ) can be formed through each of the multiple diagonal matrices.

[0122] The number of columns of the singular value matrix (Σ) can be the number of embedding dimensions (d). The singular value matrix (Σ) can be an (nХd) matrix.

[0123] An artificial neural network may be used in the singular value matrix derivation step (S320). The singular value matrix (Σ) may be derived through an artificial neural network. Each row of the singular value matrix (Σ) may represent the singular value matrix derived by singular value decomposition of the self-attention matrix (H).

[0124] The singular value matrix (Σ) can be derived through a singular value derivation operation as shown in Equation 3 below.

[0125] [Equation 3]

[0126]

[0127] Here, Σ is the singular value matrix, and θ i is the coefficient of the i-th term, and T i (x) is the basis polynomial function of the i-th term, K is the number of multiple basis polynomial functions, and Σ g can be a singular value generating matrix.

[0128] Referring to Equation 3, the singular value matrix (Σ) is a plurality of singular value generating matrices (Σ g ) and multiple coefficients (θ i ), and multiple basis polynomial functions (T i It can be derived through (x)). The singular value matrix (Σ) is a plurality of singular value generating matrices (Σ g ) are multiple coefficients (θ i ) and multiple basis polynomial functions (T i It may be a matrix derived by graphic filter processing through (x). In one embodiment of the present invention, the number (K) of multiple basis polynomial functions may be 4 or more and 10 or less. If the number (K) of multiple basis polynomial functions is less than 4 or greater than 10, the performance of the transformer model (TM) may decrease. The number (K) of multiple basis polynomial functions is not limited to the above.

[0129] Multiple singular value generating matrices (Σ g Each of ) can be a diagonal matrix. Multiple singular value generating matrices (Σ g) can correspond to multiple tokens. Multiple singular value generating matrices (Σ g The number of ) can be equal to the number of tokens (n).

[0130] Multiple singular value generating matrices (Σ g Each of ) can be derived through an artificial neural network. Multiple singular value generating matrices (Σ g Each of ) can be a (dХd) matrix. Multiple singular value generating matrices (Σ g Each of ) can have d effective elements arranged along the diagonal of the (dХd) matrix. Multiple singular value generating matrices (Σ g The d valid elements of ) can form one row of the singular value matrix (Σ). Multiple singular value generating matrices (Σ g Each row of the singular value matrix (Σ) can be formed through each of the following.

[0131] Equation 3 may be an equation based on graph signal processing (GSP). Through Equation 3, the transformer model (TM) can be processed by attention-based graph filter (AGF) to become an attention-based graph filter (AGF) transformer model (TM).

[0132] Multiple singular value generating matrices (Σ g Each of these can be derived through a singular value generation matrix derivation operation as shown in Equation 4 below.

[0133] [Equation 4]

[0134]

[0135] Here, diag(A) is the diagonal matrix generating function for matrix A, σ(A) is the sigmoid function for matrix A, X is the input embedding matrix, and W Σ is a weight matrix for deriving the singular value generation matrix.

[0136] Referring to Equation 4, multiple singular value generating matrices (Σ gEach of ) is a weight matrix (W) for deriving a singular value generation matrix for the input embedding matrix (X). Σ After being multiplied by ), it can be derived by sequentially substituting into the sigmoid function (σ(A)) and the diagonal matrix generating function (diag(A)).

[0137] The sigmoid function (σ(A)) may be an activation function. The activation function may be a function whose result is between 0 and 1. In one embodiment of the present invention, the sigmoid function (σ(A)) may be replaced with another activation function.

[0138] The diagonal matrix generating function (diag(A)) can be a function that converts an input matrix into a diagonal matrix.

[0139] Multiple basis polynomial functions (T i Each of (x)) can be a Jacobi basis polynomial function. Multiple basis polynomial functions (T i Since each of (x) is a Jacobi basis polynomial function, basis polynomial functions of different degrees can be orthogonal to each other. Therefore, basis polynomial functions of different degrees can not influence each other and can not interfere with each other.

[0140] Multiple coefficients (θ i Each of the basis polynomial functions (T i It can be a coefficient of (x). Multiple coefficients (θ i The singular value matrix (Σ) can be adjusted by adjusting the size of each. Multiple coefficients (θ i ) among high-frequency basis polynomial functions (T i If the coefficient of (x) increases, it may be more significantly affected by high-frequency information, and multiple coefficients (θ i ) among low-frequency basis polynomial functions (T i If the coefficient of (x) increases, it may be more significantly affected by low-frequency information.

[0141] Multiple coefficients (θ iThe size of each can be adjusted through the weights of the artificial neural network. Consequently, multiple coefficients (θ i Each size is adjusted through a learning process, so that the influence of high-frequency and low-frequency information can be controlled, and thus the oversmoothing phenomenon can be reduced.

[0142] In the step of deriving the right singular vector matrix (S330), the right singular vector matrix (V) can be derived. The right singular vector matrix (V) may be a matrix corresponding to the right singular vector matrix derived by singular value decomposition of the self-attention matrix (H). The right singular vector matrix (V) can emulate an orthogonal matrix. Therefore, the right singular vector matrix (V) may have high orthogonality.

[0143] In one embodiment of the present invention, the orthogonality measure (O(V)) of the right singular vector matrix may be approximately 0.3. The orthogonality measure (O(V)) of the right singular vector matrix may be 0 or greater and 0.5 or less. If the orthogonality measure (O(V)) of the right singular vector matrix is ​​greater than 0.5, the performance of the transformer model (TM) may be degraded. More preferably, the orthogonality measure (O(V)) of the right singular vector matrix may be 0.1 or greater and 0.5 or less. If the orthogonality measure (O(V)) of the right singular vector matrix is ​​less than 0.1, the performance of the transformer model (TM) may be degraded.

[0144] An artificial neural network may be used in the step of deriving the right singular vector matrix (S330). The right singular vector matrix (V) may be derived through an artificial neural network. The right singular vector matrix (V) may be a matrix that mimics the right singular vector matrix derived by singular value decomposition of the self-attention matrix (H).

[0145] The number of rows of the right singular vector matrix (V) can be the number of tokens (n), and the number of columns can be the number of embedding dimensions (d). The right singular vector matrix (V) can be an (n-xd) matrix.

[0146] The right singular vector matrix (V) can be derived through the right singular vector derivation operation as shown in Equation 5 below.

[0147] [Formula 5]

[0148]

[0149] Here, V is the right singular vector matrix, and T is a symbol representing the transpose of a matrix, ρ(A) is the softmax function for matrix A, X is the input embedding matrix, and W V is a weight matrix for deriving the right singular vector matrix.

[0150] Referring to Equation 5, the weight matrix (W) for deriving the right singular vector matrix for the input embedding matrix (X) V After being multiplied by ), it can be transposed. Then, the result is substituted into the softmax function (ρ(A)) to obtain the transpose matrix (V) of the left singular vector matrix. T ) can be derived. In the right singular vector matrix derivation step (S330), the transposed right singular vector matrix (V T ) can be derived. The transposed right singular vector matrix (V T ) can be a (dХn) matrix.

[0151] In one embodiment of the present invention, the right singular vector derivation operation can be performed iteratively. The right singular vector (V) derived through the right singular vector derivation operation is substituted into the input embedding matrix (X), and the right singular vector derivation operation can be performed iteratively. In the present invention, the right singular vector matrix (V) is derived through an artificial neural network, so the time and cost required to obtain the right singular vector matrix (V) can be reduced.

[0152] In the output matrix derivation step (S340), an output matrix (OA) can be derived. In the output matrix derivation step (S340), the output matrix (OA) can be derived through an output matrix (OA) derivation operation as shown in Equation 6 below.

[0153] [Equation 6]

[0154]

[0155] Here, OA is the output matrix, U is the left singular vector matrix, Σ is the singular value matrix, and V T is the transposed right singular vector matrix, X is the input embedding matrix, and W val is a weight matrix for deriving values.

[0156] Referring to Equation 6, in the output matrix derivation step (S340), Equation 6 can be calculated in the following order. First, a first-order operation is performed to derive the input embedding matrix (X) and the weight matrix (W) for deriving the value. val ) can be multiplied together. The weight matrix (W) for deriving the value val ) can be a (dХd) matrix. Therefore, the input embedding matrix (X) and the weight matrix (W) for deriving values val The value obtained by multiplying ) together is the first-order operation value (XW val ) can be. The value of the first operation can be an (n-xd) matrix. The complexity of the operation is O(nd 2 It can be.

[0157] Then, a second operation is performed so that the left singular vector matrix (U) and the singular value matrix (Σ) can be multiplied together. The product of the left singular vector matrix (U) and the singular value matrix (Σ) can be the second operation value (UΣ). The second operation value can be an (n-xd) matrix. Since the left singular vector matrix (U) and the singular value matrix (Σ) are (n-xd) matrices, the product of the left singular vector matrix (U) and the singular value matrix (Σ) can be derived simply through element-wise multiplication. Consequently, the complexity of the operation can be O(nd).

[0158] In addition, a third-order operation is performed, and the transposed right singular vector matrix (V T ) and 1st operation value (XWval ) can be multiplied together. The transposed right singular vector matrix (V T ) and 1st operation value (XW val The value obtained by multiplying ) together is the cubic operation value (V T XW val It can be ). 3rd order operation value (V T XW val ) can be a (dХd) matrix. The complexity of the corresponding operation is O(nd 2 It can be.

[0159] Finally, a fourth operation is performed, and the second operation value (UΣ) and the third operation value (V T XW val ) are multiplied together to derive the output matrix (OA). The output matrix (OA) can be an (n-xd) matrix. The complexity of this operation is O(nd 2 It can be.

[0160] Consequently, the complexity of the operation with the highest complexity in the output matrix derivation step (S340) is O(nd 2 ) may be. Therefore, as the complexity of operations increases linearly as the number of tokens (n) increases, the amount of operations may not increase exponentially even when the number of tokens (n) increases.

[0161] Meanwhile, the order of operations is not limited to the above. For example, the order of second-order and third-order operations can be reversed.

[0162] Meanwhile, in Equation 5, the value obtained by multiplying the left singular vector matrix (U), the singular value matrix (Σ), and the right singular vector matrix (V) (UΣV) can correspond to the self-attention matrix (H). In the present invention, the self-attention matrix (H) can be derived through the left singular vector matrix (U), the singular value matrix (Σ), and the right singular vector matrix (V).

[0163] In the data output step (S400), output values ​​can be derived through the output matrix (OA). In the data output step (S400), output data can be derived. Depending on the type of data input in the data input step (S100), the output data derived in the data output step (S400) may vary.

[0164] When the learning data input step (S110) is performed and learning input data including an input value and a correct answer for the input value is input, learning output data may be derived in the data output step (S400). The learning output data may include an output value and a correct answer for the input value. When learning output data is derived, the learning step (S500) may be performed.

[0165] When the inference data input step (S120) is performed and inference input data is input that includes an input value but does not include a correct answer for the input value, inference output data may be derived in the data output step (S400). The inference output data may include an output value but may not include a correct answer for the input value. When inference output data is derived, the inference value output step (S600) may be performed.

[0166] The learning step (S500) may include a score loss value derivation step (S510), an orthogonality loss value derivation step (S520), and a weight modification step (S530). In the learning step (S500), the Transformer model (TM) can be trained using the learning output data.

[0167] In the score loss value derivation step (S510), the output value of the training output data and the correct answer can be compared with each other. In the score loss value derivation step (S610), the score loss value (L s ) can be derived. Score loss value (L s ) can be a value that increases when the difference between the output value and the correct answer increases. For example, the score loss value (L s) can be derived through at least one of loss functions such as mean squared error (MSE), mean absolute error (MAE), and cross-entropy loss between the output value and the correct answer.

[0168] In the step of deriving orthogonality loss values ​​(S520), the orthogonality loss values ​​(L of the left singular vector matrix and the right singular vector matrix) o ) can be derived. Orthogonality loss value (L o ) may be a value that increases when the orthogonality of the left singular vector matrix (U) and the right singular vector matrix (V) decreases.

[0169] Orthogonality loss value (L o ) may be a value derived through the following Equation 7.

[0170] [Equation 7]

[0171]

[0172] Here, here L o is the orthogonality loss value, n is the number of tokens, ∥∥ is the norm symbol, U is the left singular vector matrix, V is the right singular vector matrix, and T is a symbol representing the transpose of a matrix, and I can be the identity matrix.

[0173] Referring to Equation 7, the orthogonality loss value (L o ) can be 0 if the left singular vector matrix (U) and the right singular vector matrix (V) are orthogonal. Meanwhile, the orthogonality loss value (L o ) can be greater than 0 if the left singular vector matrix (U) and the right singular vector matrix (V) are not orthogonal. Orthogonality loss value (L o ) may be a loss function that determines the orthogonality of the left singular vector matrix (U) and the right singular vector matrix (V).

[0174] In the weight modification step (S530), the weights of the artificial neural network may be modified. In the weight modification step (S530), a weight matrix, for example, an embedding matrix (E), and a weight matrix (W) for deriving a left singular vector matrix are used. U ), weight matrix (W) for deriving the singular value generation matrix Σ ), weight matrix for deriving the right singular vector matrix (W V It may include a weight matrix for multiple coefficients, etc.

[0175] In the weight modification step (S530), the total loss value (L tot The weight matrices can be modified so that ) becomes smaller. The total loss value (L tot ) can be derived through the following Equation 8.

[0176] [Equation 8]

[0177]

[0178] Here, L tot is the total loss value, and L s is the score loss value, and L o is the orthogonality loss value, and γ can be the coefficient of the orthogonality loss value.

[0179] Referring to Equation 8, in the weight modification step (S530), the score loss value (L s ) and orthogonality loss value (L o ) is considered, and the weight matrix can be modified. As a result, the Transformer model (TM) can be trained and improved. In one embodiment of the present invention, the coefficient (γ) of the orthogonality loss value is 10 -4 More than 10 -1 Lee Ha-il It is possible.

[0180] The coefficient (γ) of the orthogonality loss value is 10 -1If it exceeds, the orthogonality measure of the left singular vector matrix (O(U)) and the orthogonality measure of the right singular vector matrix (O(V)) become excessively small, and the performance of the Transformer model (TM) may degrade. The coefficient (γ) of the orthogonality loss value is 10 -4 If it is less than that, the orthogonality measure (O(U)) of the left singular vector matrix and the orthogonality measure (O(V)) of the right singular vector matrix become excessively large, and the performance of the transformer model (TM) may be degraded. In the inference value output step (S600), the output value is provided to the user, and the product through the transformer model (TM) can be provided to the user.

[0181] Below, a transformer model (TM) according to an experimental example of the present invention is described.

[0182] <Experimental Example 1>

[0183] In Experimental Example 1, the performance of the embodiment of the present invention and a plurality of comparative examples is compared. The embodiment of the present invention may be the attention-based graph filter (AGF) transformer model (TM) of the present invention. In Experimental Example 1, the performance of the embodiment and a plurality of comparative examples is compared through UEA Time Series Classification results and LRA benchmark results.

[0184] Table 1 shows the UEA Time Series Classification results of the examples and a plurality of comparative examples.

[0185]

[0186]

[0187] Referring to Table 1, the attention-based graph filter (AGF) transformer model of the present invention demonstrates higher accuracy than a number of comparative examples in EthanolConcentration (EC), FaceDetection (FD), Heartbeat (HB), JapaneseVowels (JV), PEMS-SF, SelfRegulation SCP1 (SRSCP1), SelfRegulation SCP2 (SRSCP2), SpokenArabicDigits (SAD), and UWaveGesture Library (UWGL). Additionally, the average accuracy of the attention-based graph filter (AGF) transformer model of the present invention may be higher than the average accuracy of other models.

[0188] Table 2 shows the LRA benchmark results of the examples and a plurality of comparative examples.

[0189] Table 2

[0190]

[0191] Referring to Table 2, the attention-based graph filter (AGF) transformer model of the present invention demonstrates higher accuracy than a plurality of comparative examples in ListOps and Retrieval. Additionally, the average accuracy of the attention-based graph filter (AGF) transformer model of the present invention may be higher than the average accuracy of other models.

[0192] Table 3 shows the LRA benchmark results for the computation time and memory usage of the examples and a plurality of comparative examples.

[0193] Table 3

[0194]

[0195] Referring to Table 3, the attention-based graph filter (AGF) transformer model of the present invention has the least computation time in Text (4K). Additionally, the computation time and memory usage of the attention-based graph filter (AGF) transformer model of the present invention may be less than that of the transformer model, performer transformer model, and reformer transformer model.

[0196] Table 4 shows the UEA Time Series Classification results of the examples and a plurality of comparative examples.

[0197] Table 4

[0198]

[0199] Referring to Table 4, the attention-based graph filter (AGF) transformer model of the present invention is compared with a plurality of comparative examples (H) in EthanolConcentration (EC), FaceDetection (FD), HandWrighting (HW), Heartbeat (HB), PEMS-SF, and UWaveGesture Library (UWGL). UVT , H SVD It shows higher accuracy than ).

[0200] Multiple comparative examples (H UVT , H SVD ) may be a transformer model simulating singular value decomposition. Meanwhile, comparative examples (H UVT , H SVD ) only simulates singular value decomposition and may not perform attention-based graph filtering (AGF). Therefore, through Table 4, it can be confirmed that the performance of the Transformer model (TM) is improved with attention-based graph filtering (AGF).

[0201] <Experimental Example 2>

[0202] In Experimental Example 2, an embodiment of the present invention and a comparative example are compared with each other. An embodiment of the present invention is set as a DeiT-small model to which the attention-based graph filter (AGF) of the present invention is applied. A comparative example is set as a DeiT-small model to which the attention-based graph filter (AGF) of the present invention is not applied.

[0203] Table 5 shows the ImageNet-100 and ImageNet-1K test results of the examples and comparative examples.

[0204] Table 5

[0205]

[0206] In Table 5, it was confirmed that the embodiment of the present invention (+AGF) showed superior performance compared to the comparative example (DeiT-small).

[0207] Figure 6 is a graph showing the normalized magnitude according to frequency of the embodiments and comparative examples of the present invention.

[0208] Referring to FIG. 6, it was confirmed that in the embodiment of the present invention (Deit+AGF), the normalized magnitude of the high-frequency signal is greater than in the comparative example (DieT). Therefore, it was confirmed that the influence of the high-frequency signal is increased by applying an attention-based graph filter (AGF), and the oversmoothing phenomenon is reduced.

[0209] Figure 7 is a graph showing the cosine similarity according to the Layer Index of the embodiments and comparative examples of the present invention.

[0210] Referring to FIG. 7, it was confirmed that in the embodiment of the present invention (Deit-small+AGF), when the Layer Index is 8 or higher, the cosine similarity is lower than that of the comparative example (DeiT-small). Therefore, it was confirmed that the oversmoothing phenomenon is reduced by applying an attention-based graph filter (AGF).

[0211] <Experimental Example 3>

[0212] In Experimental Example 3, embodiments of the present invention were compared with one another. In Experimental Example 3, the number (K) of multiple basis polynomial functions was set differently.

[0213] Table 6 shows the UEA Time Series Classification results of the embodiments of the present invention.

[0214] Table 6

[0215]

[0216] Referring to Table 6, it is confirmed that JapaneseVowels (JV) and PEMS-SF have excellent performance when the number of multiple basis polynomial functions (K) is set to 4. It is confirmed that EthanolConcentration (EC) has excellent performance when the number of multiple basis polynomial functions (K) is set to 6. It is confirmed that FaceDetection (FD) and S SelfRegulation SCP1 (SRSCP1) have excellent performance when the number of multiple basis polynomial functions (K) is set to 9. It is confirmed that UWaveGesture Library (UWGL) has excellent performance when the number of multiple basis polynomial functions (K) is set to 10. The number of multiple basis polynomial functions (K) can be set to an appropriate value depending on the application of the Transformer model (TM).

[0217] <Experimental Example 4>

[0218] In Experimental Example 4, embodiments of the present invention were compared with one another. In Experimental Example 4, the coefficient (γ) of the orthogonality loss value was set differently.

[0219] Table 7 shows the LRA benchmark results of the embodiments of the present invention.

[0220] Table 7

[0221]

[0222] Referring to Table 7, the coefficient (γ) of the orthogonality loss value is 1x10 -1It is confirmed that ListOps and Retrieval performance are excellent when set to this value. The coefficient (γ) of the orthogonality loss value is 1x10 -2 It is confirmed that image performance is excellent when set to this value. The coefficient (γ) of the orthogonality loss value is 1x10 -3 It is confirmed that Text and Pathfinder performance is excellent when set to this. The coefficient (γ) of the orthogonality loss value is 1x10 -4 It is confirmed that Pathfinder performance is excellent when set to this value. The coefficient (γ) of the orthogonality loss value can be set to an appropriate value depending on the application of the transformer model (TM).

[0223] Although the invention has been described with reference to exemplary embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as set forth in the following claims. Furthermore, the exemplary embodiments disclosed in the invention are not intended to limit the technical spirit of the invention, and all technical spirits within the scope of the following claims and their equivalents should be interpreted as being included within the scope of the rights of the invention. Explanation of the symbols

[0224] 10: Control unit 100: Storage unit 200: Control Unit TM: Transformer Model ANN: Artificial Neural Network

Claims

Claim 1 A data input step including an inference data input step in which inference input data containing input values ​​is input to a transformer model; an embedding step in which the input values ​​are embedded by the transformer model to derive an input embedding matrix; an operation step in which an output matrix is ​​derived through the input embedding matrix by the transformer model; a data output step in which an output value is derived through the output matrix by the transformer model; and an inference value output step in which the output value is provided to a user by the transformer model, wherein the operation step comprises: a left singular vector matrix derivation step in which a left singular vector matrix is ​​derived from the input embedding matrix by the transformer model; and a singular value matrix derivation step in which a plurality of singular value generating matrices are derived from the input embedding matrix by the transformer model, and a singular value matrix is ​​derived through the plurality of singular value generating matrices, a plurality of coefficients, and a plurality of basis polynomial functions, wherein the plurality of singular value generating matrices are diagonal matrices. A method for controlling a transformer model comprising: a right singular vector matrix derivation step in which a right singular vector matrix is ​​derived from an input embedding matrix due to the transformer model; and an output matrix derivation step in which an output matrix is ​​derived through a left singular vector matrix, a singular value matrix, a right singular vector matrix, and an input embedding matrix due to the transformer model, wherein the transformer model includes an artificial neural network, and the artificial neural network is used in the left singular vector matrix derivation step, the singular value matrix derivation step, and the right singular vector matrix derivation step. Claim 2 delete Claim 3 In claim 1, the singular value matrix is ​​a control method for a transformer model derived through the following mathematical formula 1. [Mathematical Formula 1] Here, Σ is the singular value matrix, and θ i is the coefficient of the i-th term, and T i (x) is the basis polynomial function of the i-th term, K is the number of basis polynomial functions, and Σ g is the singular value generating matrix. Claim 4 A control method for a transformer model according to claim 3, wherein the left singular vector matrix is ​​derived through the following Equation 2 and the right singular vector matrix is ​​derived through the following Equation 3. [Equation 2] [Mathematical Formula 3] Here, U is the left singular vector matrix, V is the right singular vector matrix, ρ(A) is the softmax function for matrix A, X is the input embedding matrix, and W U is the weight matrix for deriving the left singular vector matrix, and W V is a weight matrix for deriving the right singular vector matrix, and T is a symbol representing the transpose of a matrix. Claim 5 In claim 3, each of the plurality of singular value generation matrices is a control method for a transformer model derived through the following Equation 4. [Equation 4] Here, diag(A) is the diagonal matrix generating function for matrix A, σ(A) is the sigmoid function for matrix A, X is the input embedding matrix, and W Σ is a weight matrix for deriving the singular value generation matrix. Claim 6 A method for controlling a transformer model according to claim 5, wherein each of the plurality of basis polynomial functions is a Jacobi basis polynomial function. Claim 7 In claim 6, the control method of a transformer model wherein, in the output matrix derivation step, the output matrix is ​​derived through the following mathematical formula 5.[Mathematical Formula 5] Here, OA is the output matrix, U is the left singular vector matrix, Σ is the singular value matrix, and V T is the transposed right singular vector matrix, X is the input embedding matrix, and W val is a weight matrix for deriving values. Claim 8 A control method for a transformer model according to claim 7, wherein in the output matrix derivation step, the left singular vector matrix and the singular value matrix are multiplied through element-wise multiplication. Claim 9 In claim 3, the method further includes a step for deriving a score loss value, a step for deriving an orthogonality loss value, and a step for modifying weights, and further includes a learning step in which the transformer model is trained; the data input step further includes a learning data input step in which learning input data including the input value and the correct answer for the input value is input to the transformer model; when the learning data input step is performed, learning output data including the output value and the correct answer is derived in the data output step; in the step for deriving a score loss value, the output value of the learning output data is compared with the correct answer to derive a score loss value for the output value of the learning output data, wherein the score loss value is a value that increases when the difference between the output value of the learning output data and the correct answer increases; in the step for deriving an orthogonality loss value, an orthogonality loss value of the left singular vector matrix and the right singular vector matrix is ​​derived, wherein the orthogonality loss value is 0 when the left singular vector matrix and the right singular vector matrix are orthogonal, and is greater than 0 when at least one of the left singular vector matrix and the right singular vector matrix is ​​not orthogonal, and the weight modification A method for controlling a transformer model in which the weights of the artificial neural network are modified through the score loss value and the orthogonality loss value in the step. Claim 10 In claim 9, the above orthogonality loss value is a control method for a transformer model derived through the following mathematical formula 6.[Mathematical Formula 6] Here, here L o is the orthogonality loss value, n is the number of tokens, ∥∥ is the norm symbol, U is the left singular vector matrix, V is the right singular vector matrix, and T is the symbol representing the transpose of a matrix, and I is the identity matrix. Claim 11 In claim 3, the orthogonality measure of the left singular vector matrix and the right singular vector matrix is ​​0 or greater and 0.5 or less, and the orthogonality measure is a control method for a transformer model derived through the following Equation 7. [Equation 7] Here, O(A) is the orthogonality measure of matrix A, and T is the symbol representing the transpose of a matrix, and I is the identity matrix. Claim 12 A control device comprising a control unit for controlling a transformer model including an artificial neural network, wherein the control unit can embedding input values ​​to derive an input embedding matrix, derive a left singular vector matrix and a right singular vector matrix from the input embedding matrix, derive a plurality of diagonal singular value generating matrices from the input embedding matrix, derive a singular value matrix through the plurality of singular value generating matrices, a plurality of coefficients, and a plurality of basis polynomial functions, derive an output matrix through the input embedding matrix, the left singular vector matrix, the singular value matrix, and the right singular vector matrix, and derive an output value through the output matrix, wherein the control unit derives the left singular vector matrix, the singular value matrix, and the right singular vector matrix through the artificial neural network. Claim 13 delete Claim 14 delete Claim 15 In claim 12, the control unit may receive learning input data including the input value and the correct answer for the input value, and when the learning input data is received, the control unit may derive learning output data including the output value and the correct answer, the control unit derives a score loss value for the output value of the learning output data by comparing the output value of the learning output data with the correct answer, wherein the score loss value is a value that increases when the difference between the output value of the learning output data and the correct answer increases, and the control unit derives an orthogonality loss value of the left singular vector matrix and the right singular vector matrix, wherein the orthogonality loss value is 0 when the left singular vector matrix and the right singular vector matrix are orthogonal, and is a value greater than 0 when at least one of the left singular vector matrix and the right singular vector matrix is ​​not orthogonal, and the control unit modifies the weights of the artificial neural network through the score loss value and the orthogonality loss value. Claim 16 In claim 12, the orthogonality measure of the left singular vector matrix and the right singular vector matrix is ​​0 or greater and 0.5 or less, and the orthogonality measure is a control device of a transformer model derived through the following mathematical formula 8.[Mathematical Formula 8] Here, O(A) is the orthogonality measure of matrix A, and T is the symbol representing the transpose of a matrix, and I is the identity matrix. Claim 17 A storage device comprising a transformer model, wherein the transformer model embeds input values ​​to derive an input embedding matrix, derives a left singular vector matrix and a right singular vector matrix from the input embedding matrix, derives a plurality of diagonal singular value generating matrices from the input embedding matrix, derives a singular value matrix through the plurality of singular value generating matrices, a plurality of coefficients, and a plurality of basis polynomial functions, derives an output matrix through the input embedding matrix, the left singular vector matrix, the singular value matrix, and the right singular vector matrix, and derives an output value through the output matrix, wherein the transformer model derives the left singular vector matrix, the singular value matrix, and the right singular vector matrix through an artificial neural network. Claim 18 delete Claim 19 delete Claim 20 In claim 17, training input data including the input value and the correct answer for the input value may be input to the transformer model, the transformer model compares the output value with the correct answer to derive a score loss value for the output value, the score loss value is a value that increases when the difference between the output value and the correct answer increases, the transformer model derives an orthogonality loss value through the left singular vector matrix and the right singular vector matrix, the orthogonality loss value is 0 when the left singular vector matrix and the right singular vector matrix are orthogonal, and is a value greater than 0 when at least one of the left singular vector matrix and the right singular vector matrix is ​​not orthogonal, and the transformer model modifies the weights of the artificial neural network through the score loss value and the orthogonality loss value.

Citation Information

Patent Citations

  • Fine dust sensing method and fine dust sensing device performing the same

    KR102700249B1

  • Fine-tuning diffusion-based generative neural networks using singular value decompositions for text-to-image generation

    US20250225700A1

  • Learning devices and learning methods

    KR1020230162115A

  • Computing system and non-transitory storage medium including attention-based artificial intelligence model

    KR1020250113726A