Tensor loss compression technique using neural network-based tensor-train decomposition

Abstract

A tensor loss compression technique using neural network-based tensor-train decomposition is disclosed. The tensor loss compression method performed by a computer system, according to one embodiment, may comprise the steps of: inputting, into a neural network-based tensor train (TT) decomposition model, a mode index of a target item to be restored; and obtaining, through the neural network-based tensor train decomposition model, a tensor train core from the mode index of the target item to be restored.

Description

A lossy tensor compression technique using neural network-based tensor-train decomposition.

[0001] The description below is about tensor lossy compression techniques.

[0002]

[0003] Tensors are high-dimensional arrays, and the cost of storing a tensor is proportional to the number of entries in the tensor, which is the product of the lengths of each dimension. As tensors become more dimensional or the length of one dimension increases, the cost of storing them also increases. Therefore, techniques for compressing tensors are essential.

[0004] Among the techniques currently used to compress large matrices and tensors, Tensor-Train (TT) decomposition is attracting attention. However, TT decomposition suffers from low accuracy due to limited expressive power. Deep learning models demonstrate outstanding performance due to their ability to approximate arbitrary functions, and among them, recurrent neural networks are widely used for problems involving permutations as input.

[0005]

[0006] By applying tensor train decomposition and recurrent neural networks, a tensor lossy compression method and system can be provided that compresses tensors to a small size while exhibiting low errors when restored.

[0007]

[0008] A tensor lossy compression method performed by a computer system may include a step of inputting a mode index of a target item to be restored into a neural network-based tensor train (TT) decomposition model; and a step of obtaining a tensor train core from the mode index of the target item to be restored through the neural network-based tensor train decomposition model.

[0009] The above neural network-based tensor train decomposition model may be configured for tensor train decomposition using a recurrent neural network.

[0010] The above neural network-based tensor train decomposition model may be constructed through the steps of: searching for an embedding in an embedding layer to encode each mode index; obtaining a hidden embedding for the searched embedding through an LSTM layer; generating a tensor train core from the obtained hidden embedding through a linear layer; and learning to return an approximate item value through a product of the generated tensor train cores.

[0011] The above obtaining step may include a step of exploring compressed data composed of parameters and a rearrangement function of the neural network-based tensor train decomposition model.

[0012] The obtaining step may include a step of initializing parameters and a rearrangement function of the neural network-based tensor train decomposition model, rearranging and folding the order of input tensors to generate a tensor with a dimension greater than a preset standard, and updating the parameters and the rearrangement function of the neural network-based tensor train decomposition model to minimize an approximation error.

[0013] The obtaining step may include a step of performing a process of folding the input tensor into a tensor of a dimension greater than or equal to a preset standard in order to reduce the compression size of the tensor.

[0014] The obtaining step may include the adjusting step may include a step of rearranging the tensor before performing the folding process.

[0015] The obtaining step may include a step of rearranging the mode indices of the input tensor so that items having similar values ​​are placed close to each other by sharing the mode indices of the folded tensor.

[0016] A computer program stored in a computer-readable storage medium for executing a tensor lossy compression method performed by a computer system, wherein the tensor lossy compression method may include a step of inputting a mode index of a target item to be restored into a neural network-based tensor-train (TT) decomposition model; and a step of obtaining a tensor-train core from the mode index of the target item to be restored through the neural network-based tensor-train decomposition model.

[0017] In a computer system, a memory; and a processor connected to the memory and configured to execute at least one command stored in the memory, wherein the processor inputs a mode index of a target item to be restored into a neural network-based tensor train (TT) decomposition model, and obtains a tensor train core from the mode index of the target item to be restored through the neural network-based tensor train decomposition model.

[0018]

[0019] The present invention can improve performance in both compression size and restoration error compared to conventional techniques. Furthermore, the algorithm's compression time linearly increases with the number of tensor elements, demonstrating excellent scalability. Furthermore, a case study utilizing a traffic data set confirmed that the rearranged results, in a mode that includes location information, are consistent with geographical information.

[0020]

[0021] FIG. 1 is a diagram for explaining a neural network-based tensor train decomposition operation in one embodiment.

[0022] FIG. 2 is a diagram for explaining a tensor folding operation in one embodiment.

[0023] FIG. 3 is a diagram illustrating a computer system in one embodiment.

[0024] FIG. 4 is a diagram for explaining a neural network-based tensor train decomposition method in one embodiment.

[0025] Figure 5 is an algorithm for explaining the overall operation of tensor lossy compression in one embodiment.

[0026] FIG. 6 is an algorithm for describing an LSTM-based compression model that generalizes tensor train decomposition in one embodiment.

[0027] FIG. 7 is an algorithm for explaining an operation of updating a rearrangement function in one embodiment.

[0028]

[0029] Hereinafter, the detailed description will be given with reference to the attached drawings.

[0030]

[0031] Lossy compression algorithms require small compressed data. Furthermore, the error when reconstructing the original tensor from the compressed data must be small. Compression algorithms are designed to maximize these two goals. Furthermore, compression algorithms must be highly scalable, allowing them to operate on large tensors. Scalability is considered high when the compression time is proportional to the number of elements in the tensor.

[0032] Existing tensor lossy compression algorithms require assumptions about tensors to compress them effectively. For example, tensor decomposition techniques assume that the input tensor can be represented with low error in a low-rank structure defined by the algorithm. Video and image compression techniques require the assumption that the values ​​stored in the tensor vary smoothly along the axes. Some algorithms specialize in sparse matrices where most of the elements are zero. However, many tensor data representing real-world data do not adhere to these assumptions. Therefore, there is a need for a general algorithm that achieves good compression performance without requiring tensor assumptions.

[0033] In this work, we describe a robust lossy compression algorithm that can compress even large tensors while achieving small compressed sizes and low restoration errors, without requiring any assumptions about the tensors.

[0034] FIG. 1 is a diagram for explaining a neural network-based tensor train decomposition operation in one embodiment.

[0035] A computer system can provide neural network-based tensor train decomposition. The computer system can generalize tensor train decomposition using neural networks to improve expressiveness and mitigate limitations due to low-rank assumptions. The computer system can construct a neural network-based tensor train decomposition model for tensor train decomposition using neural networks. Unlike conventional tensor train decomposition, where the tensor core matrix is ​​fixed by the index of each dimension, the computer system uses a recurrent neural network to obtain the matrix, making it dependent on the indices of other modes of the entries. The neural network-based tensor train decomposition model can approximate the input tensor with a limited number of parameters, even if the input tensor has a high-dimensional structure. Furthermore, the computer system can reduce the space required for the neural network-based tensor train decomposition model by folding the input tensor into a high-dimensional tensor. This further reduces the number of parameters required for the neural network-based tensor train decomposition model. Finally, the computer system can rearrange the mode indices of the input tensor to reveal patterns that can be exploited by neural network-based tensor train decomposition models for improved approximation.

[0036] To help you understand the explanation of tensor lossy compression, let's start with the basic concepts.

[0037] Matrices and tensors: size N1× Given a real-valued matrix M of N2, the entry in the ith row and jth column is denoted as M(i, j). The dimension of the tensor represents the number of modes. Let X be N1× … × N d Let X be a d-dimensional tensor of size (i 1, …, i d ) is the item at position X(i 1, …, i d ) is displayed.

[0038] Tensor slicing and rearrangement: mode-j index i ∈ [Nj ], represents the ith slice of X following the jth mode. That is, , where :k is all possible mode-k indices (i.e., [N k ] represents the index of the mode index. Accordingly, we consider rearranging the mode index. A set of rearrangement functions Let X be a tensor rearranged in log. Here, each is a one-to-one correspondence function from the set of mode-i indices to itself. In (i 1, …, i d ) The th item is X It corresponds to the second item.

[0039] Frobenius norm: Frobenius norm of is defined as the square root of the sum of the squares of all items.

[0040]

[0041] Next, we will explain Tensor-Train Decomposition (TTD). Tensor-Train Decomposition converts a given d-th order tensor X into a d-th tensor called the tensor train core. , and each item of X is approximated as follows.

[0042]

[0043] Here, for all k And follows the second mode is the ith slice. At this time, r0 and r d is always set to 1. In the embodiment, for simplicity, all other tensor train ranks (i.e., r1, …, r d-1 ) are integrated into a single value denoted by R. A representative optimization algorithm for tensor train decomposition is TT-SVD, which has a specified accuracy. About Satisfying The goal is to obtain: is an approximate tensor obtained by tensor train decomposition. In TT-SVD, the truncated SVD is applied after reconstructing the tensor into a matrix. Tensor train decomposition is naturally used as a tensor lossy compression algorithm, and the compressed result is an entry of the tensor train (TT) core tensor. The number of these entries is , where N represents the maximum mode length.

[0044] Additionally, we provide a formal definition of the tensor lossy compression problem as follows.

[0045]

[0046] Compressed data D is a set of parameters of a neural network-based tensor train decomposition model. and a set of rearrangement functions It consists of.

[0047] Referring to Figure 5, an algorithm is provided to explain the overall operation of tensor lossy compression. First, the computer system uses a neural network-based tensor train decomposition model. and rearrangement function initializes. Next, the computer system rearranges and folds X to create higher-dimensional tensors. The computer system then updates the model parameters and rearrangement function to minimize the approximation error. The computer system repeats this process until convergence is reached, i.e., until the approximation error no longer exhibits significant changes. The output of the compression process is based on the logarithmic time approximation of each tensor entry. and Optimize.

[0048] To simplify the explanation, we assume that the input tensor X is already correctly aligned. Then, the rearrangement function We will explain in detail how to initialize and update .

[0049] A computer system can provide a neural network-based tensor train decomposition model for accurately approximating tensors. The computer system can accurately approximate tensor elements with a small number of parameters through the neural network-based tensor train decomposition model. The computer system proposes a neural network-based tensor train decomposition model that integrates recurrent neural networks and tensor train decomposition.

[0050] Instead of learning the tensor decomposition core directly as a free variable, we learn the (i) of the tensor 1, …, i d ) For each item, a tensor train core is obtained as the output of a neural network-based tensor train decomposition model that takes the mode index of the item as input. The neural network, denoted as , is trained to approximate the items as follows:

[0051]

[0052] Here, T1∈R 1×R , T2∈R R×R , … , T d-1 ∈R R×R , and T d ∈R R×1 silver It is a tensor decomposition core generated by .

[0053] For detailed operation of the neural network-based tensor train decomposition model, refer to Algorithm 2 of Fig. 6 and Fig. 1. The neural network-based tensor train decomposition model can be composed of an embedding layer, an LSTM, and a linear layer. Each mode index i k To encode, the neural network-based tensor train decomposition model first creates an embedding layer E k Embedding e in (lines 1 and 2) k can be explored. Then, the explored embedding e k Feed the LSTM layer and the hidden embedding h for 1 ≤ k ≤ d (row 3).k can be obtained. The hidden embedding h obtained using the linear layer (lines 4-7) k From Tensor Train Core T k After generating the tensor train core T k The product of can be returned as an approximate item value (row 8).

[0054] The computer system uses a model of recurrent neural network structure to allow for dependencies between tensor train cores and mode indices. However, in the neural network-based tensor train decomposition model, each tensor train core T k It depends not only on the mode-k index of the target item (as in TTD), but also on the mode-j index for all j ≤ k. In the examples, we have illustrated the use of LSTM, but GRU and Scaled Dot-product Attention are also applicable.

[0055] Neural network-based tensor train decomposition model It has the following advantages over the existing tensor train decomposition (TTD):

[0056] Contextuality: In neural network-based tensor train decomposition models, each tensor train core depends not only on the current mode index but also on all previous mode indices. For example, consider approximating X(2, 1, 2) and X(1, 2, 2). In neural network-based tensor train decomposition models, the tensor train core used for the third mode is different in both cases. However, in tensor train decomposition, the same tensor train core is used in both cases because the third mode index is the same. Neural network-based tensor train decomposition models are contextual and nonlinear, which allows them to model tensors that tensor train decomposition cannot easily approximate even with more parameters. This improved expressiveness reduces the reliance on structural assumptions about the input tensor.

[0057] Nonlinearity: Neural network-based tensor train decomposition models incorporate the nonlinear operations introduced by LSTM layers, whereas tensor train decomposition does not. This contributes to the enhanced expressiveness of neural network-based tensor train decomposition models, enabling them to better approximate tensor entries.

[0058] Simplicity: The neural network-based tensor train decomposition model shares parameters for different modes (specifically, W and b in line 6 of Algorithm 2), making it more concise with fewer parameters than tensor train decomposition. On the other hand, tensor train decomposition requires a unique tensor train core for each mode.

[0059] Space Complexity Analysis: In Theorem 1, we present the compressed output size of a neural network-based tensor train decomposition model. The hidden dimension of the LSTM and the rank of the tensor train cores are denoted by h and R.

[0060] Summary 1 (Size of compressed output of neural network-based tensor train decomposition model). The size of the compressed output (i.e., the number of parameters) of the neural network-based tensor train decomposition model is , and if h and R are treated as constants, It becomes.

[0061] Proof. The embedding layer is parameters. LSTM and fully connected layers have O(h 2 +hR 2 ) has parameters. Therefore, the total size is am.

[0062] Optimization method for: Problem 1, i.e., Parameters of a neural network-based tensor train decomposition model using mini-batch gradient descent to minimize the loss function Update. and rearrangement function Update alternately. After updating, the optimizer (e.g., Adam) is reinitialized, as the loss surface changes after the rearrangement.

[0063] The computer system can provide a folding technique for a lightweight neural network-based tensor train decomposition model. According to Theorem 1, The goal is to further reduce the compressed output size of a neural network-based tensor train decomposition model proportional to the input tensor. The main idea is to fold the input tensor into a higher-order tensor with a smaller mode length while maintaining the same number of entries. A neural network-based tensor train decomposition model The goal is to fit the folded tensor rather than the input tensor after the array process.

[0064] Tensor train matrix format: Folding techniques are inspired by the tensor train matrix format, which aims to fold matrices into tensors to reduce the number of parameters in the tensor train decomposition. and Given a matrix A of size N × M, let matrix A be of size d-th order tensor of is folded (see the example of d = 3 in Fig. 2). As follows, each entry of matrix A is a tensor is mapped to the item.

[0065]

[0066] Here, i for each 1 ≤ k ≤ d k ∈ [n k ] and j k ∈ |m k |is and The impact of folding techniques on neural network-based tensor train decomposition models is described below.

[0067] Tensor train tensor format: The tensor train matrix format can be naturally extended to tensors. The tensor train matrix format is extended to tensors by folding the input tensor into higher-order tensors with smaller mode lengths. This process is called the tensor train tensor format. Size N1 ×… ×N d Given a tensor X (where, ), size X d'-order tensor of Fold it. Then X and The mapping between the items is as follows:

[0068]

[0069] Here, for all k ∈ {1, … , d} and l ∈ {1, … , d'}, i k,l ∈ [n k,l ]am.

[0070] We choose the new dimension d' such that the folded tensor has a higher dimension than the input tensor (i.e., d' > d), while d'=O(logN max ) while maintaining N max represents the maximum mode length in the input tensor X. This is usually possible because in real tensors the mode lengths are usually much larger than the dimensions. For example, a 4th-order tensor of size 256×256×256×256 can be folded into an 8th-order tensor with each mode having length 16. It may not always be possible to construct a folded tensor that satisfies the above criteria while having the same number of entries as the input tensor. In such cases, the folded tensor may contain extra entries, which are ignored. For real tensors, we initially multiply 2 by n for all k ∈ {1, …, d} and l ∈ {1, …, d'}. k,land modify some of them using integers up to 5 to ensure that the input tensor and the folded tensor have similar numbers of entries. For example, for the PEMS-SF dataset (a 3rd order tensor of size 963×144×440), the values ​​assigned in the form of a d × d' matrix are as follows.

[0071]

[0072] This becomes a 10th-order tensor with dimensions 8×8×8×8×8×20×4×4×4×2. Note that they are close to 963, 144 and 440 respectively.

[0073] Space Complexity Analysis: We analyze the impact of folding on the number of parameters (i.e., the compressed output size). For all k ∈ [d] and l ∈ [d'], we simply According to Theorem 1, the number of parameters of the original neural network-based tensor train decomposition model of X is as follows.

[0074]

[0075] If h and R are treated as constants.

[0076] The number of parameters of the tensor train decomposition model based on neural networks is as follows.

[0077]

[0078] Here, Therefore, in the original tensor neural network-based tensor train decomposition model, O(N1+…+N d ) is significantly smaller than N, where N max is the maximum mode length of X. Therefore, for any e < 1, d' = Considering R and h, the space complexity becomes as follows according to Theorem 1.

[0079]

[0080] Computer systems can provide rearrangement techniques to better fit the folded tensor. Essentially, the mode indices of the input tensor are rearranged before folding, so that items with similar values ​​share mode indices in the folded tensor and are placed close to each other. This arrangement A neural network-based tensor train decomposition model that is more effective at fitting folded tensors because it generates tensor train cores based on the mode indices of target items that serve as inputs to the model. Improves the ability of

[0081] In the example in Figure 2, the closer two items are to each other in the original tensor, the more indices they tend to share in the folded tensor. Specifically, items in the black region share only the first mode index in the folded tensor. Adjacent items in the gray region share both the first and second mode indices in the folded tensor. A neural network-based tensor train decomposition model In , the k-th tensor train core T of equation (3) is used to approximate two items. k It is important to note that the first k indices are identical. Consequently, two tensor train cores are shared for items in the gray area. Therefore, the closer two items are to the original tensor, More input and tensor train cores are likely to share these items. Due to this property, placing similar items close to each other is more likely to This helps to more accurately approximate the items. It rearranges the positions by reordering the mode indices in the input tensor. As shown in the example in Figure 2, the gray area contains more similar items after the rearrangement.

[0082] Mode index rearrangement is It is achieved by learning the rearrangement of functions. The computer system uses a neural network-based tensor train decomposition model. and Update alternately. Next Describes the initialization and update procedures. It is worth noting that mode index alignment is related to increasing smoothness.

[0083] Order initialization: A set of rearrangement functions using a surrogate loss function. Initialize . For all k, change the order of the mode-k indices (i.e., Optimization) is formulated as follows.

[0084]

[0085] Here, is the ith slice of X following the kth mode. Note that if we minimize Equation (6), adjacent slices are similar.

[0086] The problem in equation (6) can be reduced to the Metric Traveling Salesman Problem. Assume that each node represents a slice of the tensor and that each pair of nodes forms an edge with a weight equal to the Frobenius norm of the difference between the slices. Then, equation (6) can be minimized using the optimal solution of the TSP problem on the resulting entire graph. However, the computation is NP-hard, so instead, a two-approximate solution is obtained based on the fact that the Frobenius norm satisfies the triangle inequality. Then, the edge with the largest weight in the obtained solution is deleted. For each ith node of the resulting path, Set to .

[0087] Order update based on (Algorithm 3 in Fig. 7): Neural network-based tensor train decomposition model After updating, the updated and a rearranged set that rearranges the order based on the loss function of problem 1. is updated. This update step is described in Algorithm 3. Using the tensor rearranged in X It is shown as a star. For each kth mode, the mode-k index (lines 17-18) Consider disjoint candidate pairs. The process of obtaining candidate pairs is described below. For each pair of mode indices (i, i'), we find the corresponding slice and Consider the case of reducing the loss function in problem 1. and Change the value of (lines 22-24). Since the pairs are disjoint, we use the GPU to calculate the change in loss. can be updated in parallel.

[0088] If you change each pair in the above process It is structured so that similar slices tend to be located nearby. We find these pairs using locality-sensitivity hashing (LSH) on Euclidean distance. We sample half of the indices in each mode and vectorize the corresponding slices into points in a high-dimensional space. We project the vectorized slices onto random vectors (lines 6-10) and evenly divide the projected points into buckets. Then, we repeatedly select two points from the same bucket. Assuming the corresponding mode indices are i1 and i2, we have two pairs (i1, i2). 1) and (i1 1, i2) are added as candidate pairs, where represents the XOR operation (lines 17-18). This approach aims to find indices corresponding to similar slices. The remaining mode indices are randomly paired (lines 19-21).

[0089] In this example, we theoretically analyze the compressed output size, item restoration speed, and compression speed. Simply, the input tensor All mode magnitudes are powers of 2 (i.e., n for all l ∈ {1, …, d} and k ∈ {1, …, d'} l,k ∈{1, 2}). Let N represent the maximum mode size of X. max We use h and R to represent the hidden dimension of the model and the tensor train rank.

[0090] Size of compressed output: In Theorem 2, we present the space complexity of the output generated by the neural network-based tensor train decomposition model. The complexity is the size of the original tensor. is much lower than that of tensor train decomposition, especially when R is large. and CP decomposition (CPD) It should be noted that it may be lower.

[0091] Summary 2 (Size of compressed output). Compressed output generated by Algorithm 1. The size of am.

[0092] Proof. Our model is Sharing embedding layers in various modes The largest mod size is 2 d Because of this The embedding layer of is O(h2 d ) parameters. Since the number of parameters in each linear layer is proportional to the product of the input dimension and the output dimension, the number of parameters in the LSTM and fully connected layers is O(h 2 + hR 2 ) for each mode i. The number of all possible orderings is N i ! is. So to save one of them, bits are required. Therefore, the total size of the compressed output is am.

[0093] Restoration speed: Another important aspect of a compression algorithm is the restoration speed. Theorem 3 formalizes the restoration speed for the output of a neural network-based tensor train decomposition model. The complexity is O(dR) of TTD. 2 ) or higher than O(dR) of CPD, but only logarithms are recorded in the mode length.

[0094] Theorem 3 (Restoration Speed). Output of Algorithm 1 Given the input tensor entries (i.e., Algorithm 2) Approximating the value It takes time.

[0095] Proof. For each item of X, It takes O(d) time to obtain the mode index in Eq. (4) for all i kl Since is between 1 and d and l must be computed to be between 1 and d' = O(logNmax) Computing the mode index in takes O(dlogNmax) time. Processing the input through the embedding layer and LSTM takes O(h 2 logN max ) time is required. If the computational order is optimized, the computational time complexity of a hierarchically fully connected tensor train core is O(hR 2 logN max ) and the time complexity of the computational multiplication of the tensor train core is O(R 2 logN max ) is. Therefore, the total time complexity of approximating each item is am.

[0096] Compression Speed: Analyzes the speed of the compression process.

[0097] Theorem 4 (Compression speed). The time complexity of Algorithm 1 with T update steps is and here is the number of tensor entries.

[0098] Proof sketch. To initialize all rearrangement functions of It takes time. and To update once (i.e. 4-5 rows) It takes time.

[0099] Connection to actual execution time: We measure the actual execution time for compression and reconstruction to determine the time complexity. In reality, it corresponds to the number of items. Since the term is much larger and grows much faster than all other terms, the compression time increases almost linearly.

[0100] Memory requirements: The complexity of the memory space required for compression does not exceed the memory requirements for the mini-batch, compressed output, and rearrange functions.

[0101] Theorem 5 (Memory requirements for compression). Algorithm 1 has It requires memory space, where B is the number of tensor entries in a mini-batch.

[0102] FIG. 3 is a diagram illustrating a computer system in one embodiment.

[0103] The computer system (300) may include at least one of an interface module (310), a memory (320), or a processor (330). In some embodiments, at least one of the components of the computer system (300) may be omitted, and at least one other component may be added. In some embodiments, at least two of the components of the computer system (300) may be implemented as a single integrated circuit.

[0104] The interface module (310) may provide an interface for the computer system (300). In one embodiment, the interface module (310) includes a communication module, which may communicate with an external device. The communication module may establish a communication channel between the computer system (300) and the external device, and may communicate with the external device through the communication channel. The communication module may include at least one of a wired communication module and a wireless communication module. The wired communication module may be connected to the external device via a wired connection, enabling wired communication. The wireless communication module may include at least one of a short-range communication module and a long-range communication module. The short-range communication module may communicate with the external device via a short-range communication method. The long-range communication module may communicate with the external device via a long-range communication method. Here, the long-range communication module may communicate with the external device via a wireless network. In another embodiment, the interface module (310) may include at least one of an input module and an output module. The input module may input a signal to be used for at least one component of the computer system (300). The input module may include at least one of an input device configured to allow a user to directly input a signal into the computer system (300), a sensor device configured to sense the surrounding environment and generate a signal, or a camera module configured to capture an image and generate image data. The output module may include at least one of a display module for visually displaying information or an audio module for outputting information as an audio signal.

[0105] The memory (320) may store various data used by at least one component of the computer system (300). For example, the memory (320) may include at least one of volatile memory and non-volatile memory. The data may include at least one program and input data or output data associated therewith. The program may be stored in the memory (320) as software including at least one command.

[0106] The processor (330) can control at least one component of the computer system (300) by executing a program in the memory (320). Through this, the processor (330) can perform data processing or calculations. At this time, the processor (330) can execute instructions stored in the memory (320).

[0107] The processor (330) inputs the mode index of the target item to be restored into a neural network-based Tensor-Train (TT) decomposition model, and obtains a Tensor-Train core from the mode index of the target item to be restored through the neural network-based Tensor-Train (TT) decomposition model. The processor (330) may perform a process of folding the tensor into a tensor with a dimension greater than a preset standard in order to reduce the compression size of the tensor. The processor (330) may rearrange the tensor before folding it.

[0108] FIG. 4 is a diagram for explaining a neural network-based tensor train decomposition method in one embodiment.

[0109] The computer system can perform neural network-based tensor train decomposition (410), tensor folding (420), and tensor rearrangement (430) operations.

[0110] In a neural network-based tensor train decomposition (410), a computer system can use a recurrent neural network (e.g., LSTM) to improve the expressiveness of tensor train decomposition. The computer system can configure a recurrent neural network-based tensor train decomposition model. The computer system can input the mode index (positional permutation / arrangement) of the target item (element) into the neural network-based tensor train decomposition model. The computer system can output a matrix constituting the tensor train decomposition through the recurrent neural network-based tensor train decomposition model. Accordingly, the compressed tensor can be viewed as a recurrent neural network. At this time, since each matrix constituting the tensor train decomposition is determined by considering the position of the target item in multiple dimensions, the item can be restored more accurately than in a general tensor train decomposition.

[0111] Additionally, in tensor folding (420), the computer system can perform a process of folding the tensor into a higher-dimensional tensor to further reduce the compression size. The number of parameters of the neural network-based tensor train decomposition model increases in proportion to the sum of the dimensions of the tensor. Folding the tensor into a higher-dimensional tensor can further reduce the sum of the dimensions of the tensor. For example, a two-dimensional matrix with a side length of 32 can be folded into a five-dimensional matrix with a side length of 4, reducing the sum of the dimensions from 64 to 20.

[0112] Finally, in tensor rearrangement (430), the computer system can rearrange the tensor before folding to further improve the reconstruction accuracy of the neural network-based tensor train decomposition model. The closer a specific item is located in the tensor before folding, the more similar the positional permutation in the folded tensor used as the input of the neural network-based tensor train decomposition model becomes. Therefore, the matrices output by the neural network-based tensor train decomposition model become similar, and the restored values ​​become similar. Therefore, rearranging the tensor so that similar values ​​are located close together also reduces the model's reconstruction error. In addition to the parameters of the neural network-based tensor train decomposition model, the arrangement information that restores the rearranged tensor to the original tensor is also considered as a result of tensor compression.

[0113] In practice, tensors can be utilized in any field that uses them. For example, meteorological data measured at air quality monitoring stations, stock market records, and video feature data can all be expressed as tensors. Because these tensors continue to grow in size over time, compression techniques can be useful. This can reduce network traffic when transmitting tensor data over the Internet, and it can also enable the storage of large tensors on small memory devices like smartphones and laptops.

[0114] The parameters of artificial neural networks are primarily expressed as tensors. Large language models, recently highlighted for their outstanding performance, require hundreds of gigabytes of memory to store all parameters, based on OpenAI's chatGPT. This size is beyond the capacity of most electronic devices used by ordinary users. If these large artificial neural networks could be compressed, stored on devices with small memory, and only the necessary portions restored, the computational load on central servers and internet traffic could be significantly reduced.

[0115] Companies that can apply this technology include companies with large tensor data, such as Netflix, Meta, YouTube, and Twitch; companies that produce electronic devices with small memory, such as Samsung, Apple, and Huawei; and companies that provide services using large artificial neural networks, such as Google and OpenAI.

[0116]

[0117] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0118] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0119] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0120] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0121] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a tensor lossy compression method performed by a computer system, A step of inputting the mode index of the target item to be restored into a neural network-based Tensor-Train (TT) decomposition model; and A step of obtaining a tensor train core from the mode index of the target item to be restored through the tensor train decomposition model based on the neural network. A tensor lossy compression method including .

2. In paragraph 1, A tensor loss compression method characterized in that the above neural network-based tensor train decomposition model is configured for tensor train decomposition using a recurrent neural network.

3. In paragraph 2, The above neural network-based tensor train decomposition model is, A step of searching the embeddings in the embedding layer to encode each mode index; A step of obtaining a hidden embedding for the above-mentioned searched embedding through an LSTM layer; A step of generating a tensor train core from the obtained hidden embedding through a linear layer; and A step of learning to return approximate item values through the product of the above-generated tensor train cores. A tensor lossy compression method characterized by being constructed through .

4. In paragraph 1, The above acquisition steps are: A step of exploring compressed data composed of parameters and rearrangement functions of the above neural network-based tensor train decomposition model. A tensor lossy compression method including .

5. In paragraph 1, The above acquisition steps are: A step of initializing the parameters and rearrangement function of the tensor train decomposition model based on the neural network, rearranging and folding the order of the input tensor to generate a tensor with a dimension greater than a preset standard, and updating the parameters and rearrangement function of the tensor train decomposition model based on the neural network to minimize the approximation error. A tensor lossy compression method including .

6. In paragraph 5, The above acquisition steps are: A step of folding the input tensor into a tensor with a dimension greater than a preset standard to reduce the compressed size of the tensor. A tensor lossy compression method further comprising:

7. In paragraph 6, The above acquisition steps are: A step of rearranging the tensor before performing the above folding process. A tensor lossy compression method including .

8. In paragraph 7, The above acquisition steps are: A step of rearranging the mode indices of the input tensor so that items with similar values are placed close to each other by sharing the mode indices of the folded tensor. A tensor lossy compression method including .

9. A computer program stored in a computer-readable storage medium for executing a tensor lossy compression method performed by a computer system, The above tensor lossy compression method is, A step of inputting the mode index of the target item to be restored into a neural network-based Tensor-Train (TT) decomposition model; and A step of obtaining a tensor train core from the mode index of the target item to be restored through the tensor train decomposition model based on the neural network. A computer program stored on a computer-readable storage medium containing .

10. In computer systems, memory; and A processor connected to the memory and configured to execute at least one instruction stored in the memory, The above processor, The mode index of the target item to be restored is input into the neural network-based Tensor-Train (TT) decomposition model, Obtaining a tensor train core from the mode index of the target item to be restored through the tensor train decomposition model based on the neural network above. A computer system characterized by:

Images