Method and apparatus for lossy compression of sparse reorderable multi-dimensional data

Abstract

The present invention relates to an apparatus for lossy compression, the apparatus being driven by at least one processor, and comprising: a reordering unit for reordering an input matrix to be self-similar; a position encoding unit for recursively decomposing the reordered matrix to encode the position of a target component as a sequence; and an approximation model trained through a predetermined number of parameters to infer the value of the target component from the position sequence of the target component.

Description

Method and device for lossy compression of rearrangeable sparse multidimensional data

[0001] The present disclosure relates to lossy compression techniques.

[0002] A sparse matrix is ​​a matrix with a significantly smaller number of non-zero entries than the total number of elements. Despite their sparse non-zero entries, these sparse matrices are massive, consisting of billions of rows and columns. Consequently, they require significant storage space and generate massive traffic. Therefore, compression techniques are essential for efficiently handling large sparse matrices derived from real-world data.

[0003] Efficient compression techniques are required to handle sparse matrices and their generalized form, sparse multidimensional data (tensors), in the training and application of AI models. Recently, due to potential privacy risks, there has been a growing demand for processing large amounts of data on mobile devices rather than sending them to the cloud or servers. Compression techniques are also becoming increasingly important for handling sparse matrices on memory-constrained smartphones and IoT devices.

[0004] While various lossy compression techniques have been studied to date, existing lossy compression techniques have limitations: they produce outputs whose size is linear with the size of the input matrix, regardless of the number of non-zero elements. Consequently, they fail to fully utilize data sparsity. Therefore, a new lossy compression technique is needed that achieves higher compression ratios while reducing approximation errors compared to existing lossy compression techniques.

[0005] The present disclosure relates to a method and apparatus of lossy compression for sparse reorderable multi-dimensional data.

[0006] According to one embodiment, a lossy compression device driven by at least one processor comprises a rearrangement unit for rearranging an input matrix to have self-similarity, a position encoding unit for recursively decomposing the rearranged matrix to encode positions of target components into a sequence, and an approximate model trained to infer values ​​of target components from the sequence of positions of the target components through a predetermined number of parameters.

[0007] The above rearrangement unit can select two similar rows or two similar columns based on the similarity of rows or columns constituting the input matrix, and generate the rearranged matrix by positioning the selected pair of rows or pair of columns close together.

[0008] The above position encoding unit can generate the position sequence by sequentially encoding the position of the target component at each decomposition step while repeating the process of decomposing the rearranged matrix into partitions, which are sub-matrices, selecting a partition where the target component is located, and decomposing the selected partition.

[0009] If the rearranged matrix is ​​a square matrix, the above position encoding unit recursively decomposes the rearranged matrix into 2x2 partitions, and if the rearranged matrix is ​​a non-square matrix, the position encoding unit recursively decomposes the rearranged matrix into 2x2 partitions and then decomposes it into 1x2 partitions or 2x1 partitions.

[0010] The parameters of the above approximate model can be updated through training that minimizes the approximation error between the approximated value and the actual value from the position sequence of the random component.

[0011] The above approximate model can generate matrices from the position sequence through the above-described number of parameters, and infer the component values ​​corresponding to the position sequence through the Kronecker product of the generated matrices.

[0012] The above approximate model may be a model implemented with LSTM (Long Short Term Memory) cells of the recurrent neural network series.

[0013] The above input matrix may be a sparse matrix.

[0014] A method of operating a lossy compression device driven by at least one processor according to one embodiment, comprising the steps of: training an approximate model to infer a value of a target component from a sequence encoding a position of the target component in an input matrix through a predetermined number of parameters; and obtaining the approximate model composed of the predetermined number of parameters.

[0015] The step of training the above approximate model may include a process of rearranging the input matrix to have self-similarity, recursively decomposing the rearranged matrix to encode the positions of target components into a sequence, and then updating the parameters so that the approximation error between the approximated value and the actual value from the sequence of positions of the target components is minimized.

[0016] The step of training the above approximate model may include decomposing the rearranged matrix into partitions, which are sub-matrices, selecting a partition where the target component is located, and decomposing the selected partition, while sequentially encoding the position of the target component at each decomposition step to generate the position sequence.

[0017] If the rearranged matrix is ​​a square matrix, the rearranged matrix is ​​recursively decomposed into 2x2 partitions, and if the rearranged matrix is ​​a non-square matrix, the rearranged matrix is ​​recursively decomposed into 2x2 partitions, and then can be decomposed into 1x2 partitions or 2x1 partitions.

[0018] The above approximate model can generate matrices from the position sequence through the above-described number of parameters, and infer the component values ​​corresponding to the position sequence through the Kronecker product of the generated matrices.

[0019] The above approximate model may be a model implemented with LSTM (Long Short Term Memory) cells of the recurrent neural network series.

[0020] The above input matrix may be a sparse matrix.

[0021] A lossy compression device according to one embodiment comprises a processor that repeats training by executing instructions to rearrange an input matrix or an input tensor to have self-similarity, recursively decompose the rearranged matrix or tensor to encode the positions of target components into a sequence, and update parameters of an approximation model so that an approximation error between an approximated value and an actual value from the sequence of positions of the target components is minimized.

[0022] The processor may provide the trained approximate model or a certain number of parameters constituting the approximate model as a result of compressing the input matrix or the input tensor.

[0023] The above processor can slice the input matrix or the input tensor and rearrange the slices so that similar slices become closer by changing the order of the slices.

[0024] The above approximate model can generate matrices from an input position sequence through a certain number of parameters, and infer component values ​​corresponding to the input position sequence through the Kronecker product of the generated matrices.

[0025] The above approximate model may be a model implemented with LSTM (Long Short Term Memory) cells of the recurrent neural network series.

[0026] According to the present disclosure, it is possible to compress sparse matrices or tensors of various sizes into a constant size, thereby providing a high compression ratio and restoring original data from the compression result with a low error.

[0027] According to the present disclosure, the approximation error of an input matrix or tensor can be quickly obtained by accessing only non-zero target components, thereby enabling efficient calculation by utilizing sparsity and linear-time training.

[0028] According to the present disclosure, each component of an input matrix or tensor can be approximated in sublinear time using a trained approximation model.

[0029] According to the present disclosure, since it is non-rearrangeable data, it can provide superior compression performance compared to existing technologies even if the original positions of rows and columns are stored.

[0030] FIG. 1 is a diagram conceptually illustrating a lossy compression device for rare data according to one embodiment.

[0031] Figure 2 is a block diagram of a lossy compression device according to one embodiment.

[0032] FIG. 3 and FIG. 4 are each drawings illustrating rearrangement according to one embodiment.

[0033] FIG. 5 and FIG. 6 are diagrams each illustrating matrix decomposition according to one embodiment.

[0034] FIG. 7 is a diagram illustrating the operation of an approximate model according to one embodiment.

[0035] FIG. 8 is a diagram illustrating tensor compression according to one embodiment.

[0036] Figure 9 is a flowchart of a lossy compression method according to one embodiment.

[0037] Figure 10 is a configuration diagram of a computing device according to one embodiment.

[0038] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of explanation, and similar parts are designated with similar reference numerals throughout the specification.

[0039] In the description, drawing symbols and names are attached for convenience of explanation, and the devices are not necessarily limited to drawing symbols or names.

[0040] In the description, when a part is said to "include" a component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated. Furthermore, terms such as "part," "unit," and "module" used in the specification mean a unit that processes at least one function or operation, which may be implemented using hardware, software, or a combination of hardware and software.

[0041] In the description, expressions described in the singular may be interpreted as either singular or plural, unless explicitly stated as "one" or "single." Terms containing ordinal numbers, such as "first," "second," etc., may be used to describe various components, but the components are not limited by these terms. These terms may be used to distinguish one component from another.

[0042] In the flowchart described with reference to the drawing, the order of operations may be changed, several operations may be merged, some operations may be decomposed, and certain operations may not be performed.

[0043] FIG. 1 is a diagram conceptually illustrating a lossy compression device for rare data according to one embodiment.

[0044] Referring to Fig. 1, a sparse matrix (10) refers to a matrix in which the number of non-zero entries is much smaller than the total number of elements. Data from various domains, such as the relationship between users and items in e-commerce, the relationship between friends in social networks, and the relationship between users and clicked advertisements in search engines, can be stored and utilized as a sparse matrix. Sparse matrix compression is utilized in many application problems. For example, an e-commerce system can confirm the purchase history of each buyer by lossily compressing a sparse matrix and then restoring it.

[0045] Sparse matrices generated from real-world data can be massive, with billions of rows and columns. Therefore, compression techniques are essential for efficiently handling large sparse matrices. For example, an e-commerce purchase history consisting of 100 million buyers and 100 million products would result in a matrix with 100 trillion elements, equal to the product of the total number of buyers and the total number of products.

[0046] Conventional lossy compression techniques, even for sparse matrix compression, require space linear to the size of the input matrix, limiting their ability to fully utilize data sparsity. Recently, AI models that utilize sparse matrices, such as recommendation systems and graph neural networks, have been widely studied. However, GPUs, commonly used for training AI models, have relatively limited memory. Training AI models under these memory constraints requires training data compression. Furthermore, due to potential privacy risks, there is a growing demand for processing large amounts of data on mobile devices rather than sending them to the cloud or servers. Sparse matrix compression is also becoming increasingly important for handling sparse matrices on memory-constrained smartphones and IoT devices.

[0047] The lossy compression device (100) of the present disclosure can compress sparse data of arbitrary size that can be rearranged into a constant size. Here, the sparse data includes a two-dimensional sparse matrix (10), a tensor (20) such as a multi-layer graph, and may also include high-dimensional data that can be rearranged. In the following, the lossy compression method of the present disclosure will be described mainly using a sparse matrix as an example of sparse data, but this can also be applied to high-dimensional data including tensors.

[0048] In fact, a sparse matrix can have its rows and columns rearranged arbitrarily. For example, in a user-item matrix constructed based on e-commerce data, it doesn't matter which user or item is next to which user or item. Based on this rearrangeable characteristic of a sparse matrix, a lossy compression device (100) can rearrange the rows and columns so that a sparse matrix of any size can have self-similarity. Here, self-similarity refers to the property that when a portion of an object is enlarged, a pattern similar to the entire object appears.

[0049] Self-similarity is difficult to detect because it is hidden in a large amount of data. However, the lossy compression device (100) can detect hidden self-similarity by rearranging the order of rows and columns of a sparse matrix. The lossy compression device (100) recursively decomposes a given matrix into four submatrices, and based on which submatrices the target component is located at each decomposition step, each element of the matrix can be expressed in the form of a sequence. If self-similarity exists, elements expressed in similar sequences have similar values. An approximate model is trained by using the sequence obtained for each element as input to infer the value of the actual element. The approximate model performs accurate inference based on the self-similarity of the matrix and can reconstruct the original data from the compression result with a low error.

[0050] A lossy compression device (100) can encode the positions of matrix elements into a sequence through a process of recursively decomposing a matrix rearranged to have self-similarity. Here, a non-zero element can be called a target element, and the positions of the target elements can be encoded through recursive decomposition.

[0051] The lossy compression device (100) includes an approximate model Θ that generalizes the Kronecker product, and can restore component values ​​corresponding to a position sequence through an approximate model composed of a constant number of parameters.

[0052] FIG. 2 is a block diagram of a lossy compression device according to one embodiment, FIGS. 3 and 4 are each diagrams explaining rearrangement according to one embodiment, FIGS. 5 and 6 are each diagrams explaining matrix decomposition according to one embodiment, and FIG. 7 is a diagram explaining the operation of an approximate model according to one embodiment.

[0053] Referring to FIG. 2, the lossy compression device (100) is a computing device driven by at least one processor, and can provide the lossy compression method of the present disclosure through a computer program including instructions executed by the processor. The lossy compression device (100) compresses sparse data of any size that can be rearranged into a constant size, and a sparse matrix will be described as an example. The method described below can also be utilized for compressing high-dimensional data that can be rearranged, including tensors.

[0054] A lossy compression device (100) may include a rearrangement unit (110) that rearranges rows or columns of an original matrix A, a position encoding unit (130) that recursively decomposes the rearranged matrix and encodes the positions of components into a sequence, and an approximate model (150) that infers component values ​​from the position sequence. Here, the rearrangement unit (110), the position encoding unit (130), and the approximate model (150) are divided to separately explain the operation of the lossy compression device (100), and they do not necessarily need to be implemented separately, and they do not need to be implemented together in a physically separate single device. The approximate model (150) or the lossy compression device (100) including the same may be called NeuKron.

[0055] The approximate model (150) is a generalized model of the Kronecker product to compress the original matrix, and can generate matrices from a position sequence through a certain number of parameters (up to k), and infer the component values ​​corresponding to the position sequence through the Kronecker product of the generated matrices. Here, a non-zero component can be called a target component, and the position of the target component can be input into the approximate model (150), and the value of the target component can be inferred. Naturally, if the position of a 0 component is input into the approximate model (150), it can be inferred as a value close to 0.

[0056] The approximate model (150) is implemented as an auto-regressive sequence model, and can be implemented based on, for example, a Long Short Term Memory (LSTM) model of the recurrent neural network (RNN) series. The approximate model (150) can be implemented as, for example, LSTM cells that individually process the input position sequence with a certain number of parameters.

[0057] Matrix A can be approximated by the Kronecker product of matrices generated in the approximation model (150). The components of matrix A are a ij can be approximated by the (i,j)th element of the Kronecker product of the matrices generated in the approximate model (150). The loss function for all elements can be calculated by obtaining the approximate values ​​of the corresponding elements from the position sequence of non-zero element components. Here, the loss function is the loss function for the given matrix A and the approximated matrix , and the process of rearranging the matrix and updating the parameters of the approximate model (150) can be repeated so that the approximation error of the given matrix calculated with the loss function is minimized. The approximate model can express the loss function when all data values ​​are 0 as a single variable T. The loss function can be calculated by adding the error between the actual value and the approximate value calculated only for the target components after excluding the part related to the location of the target components from T. Since the above process can be performed by accessing only the target components, the update of the approximate model (150) takes time proportional to the number of non-zero components, not the size of the input matrix. In addition, the approximate value of each component can be searched in logarithmic time.

[0058] Since the approximate matrix inferred by the approximate model (150) is approximated from a position sequence encoded by recursive decomposition, the approximation error between the original matrix and the approximate matrix is ​​minimized as the self-similarity of the original matrix increases. Accordingly, the lossy compression device (100) can optimize a certain number of parameters by repeating the process of rearranging the matrix and calculating the approximation error.

[0059] First, the rearrangement unit (110) rearranges rows and columns so that a sparse matrix of arbitrary size can have self-similarity. The rearrangement method may vary.

[0060] For example, the rearrangement unit (110) estimates the similarity of rows and columns and rearranges them so that similar rows / columns are located nearby. The similarity of rows / columns can be calculated based on the Jaccard similarity of non-zero column / row indices, and, for example, the min-hashing method can be used to quickly estimate the similarity.

[0061] The method of rearranging the positions may vary. For example, the rearrangement unit (110) may sample a pair of matched rows or a pair of columns based on the estimated similarity, and may arrange the matched pair of rows or a pair of columns so that they are closer together by swapping the positions of one row / column among the sampled pairs with the surrounding rows / columns of the other row / column. Meanwhile, the rearrangement unit (110) may randomly match the remaining unmatched rows / columns and sample the pairs as described above.

[0062] The rearrangement unit (110) can repeatedly rearrange the rows and columns of a rearrangeable matrix until the approximation error is optimized. At this time, the rearrangement unit (110) can create a matrix with self-similarity by slicing the matrix into rows and columns, and rearranging the order of the slices (rows or columns) so that similar slices become closer. In the case of a tensor, the rearrangement unit (110) can slice the tensor by dimension, and rearrange the slices so that similar slices become closer by rearranging the order of the slices. If the approximation error increases through the rearrangement, the rearrangement unit (110) may not change the sampled pairs or perform additional rearrangement. That is, the rearrangement unit (110) can repeatedly rearrange the matrix until the approximation error converges, and as a result, reduce the restoration error of the approximation model (150).

[0063] Referring to FIG. 3, the rearrangement unit (110) can find a pair (e.g., C3, C11) of similar values ​​for row indices in the columns of a given matrix. The rearrangement unit (110) can arrange the matrix in a pattern in which non-zero elements are arranged nearby by exchanging the positions of C10 and C3 near C11 to arrange the matched C3 and C11 nearby.

[0064] Referring to Figure 4, matrix A1 is a sparse matrix composed of user rows and item columns, and matrix A2 can be created by repeating the process of arranging similar rows and columns nearby. At this time, a pattern similar to the overall pattern of matrix A2 is created. This is repeated inside, which can be called self-similarity. Matrix A2 with self-similarity can be expressed as the Kronecker product of matrix A3 and matrix A4. For reference, the Kronecker product is a giant matrix created by multiplying each element of matrix A by matrix B, as in mathematical expression 1.

[0065] [Mathematical Formula 1]

[0066]

[0067] The position encoding unit (130) can recursively decompose the input matrix and encode the positions of the components into a sequence. The position encoding unit (130) can receive a rearranged matrix as input, but it is not necessary to input only a rearranged matrix.

[0068] To explain positional encoding, simply, In the input matrix A expressed as N=M=2 l Assume that if N=M is ​​2 l If it is not satisfied, zero-padding can be performed by adding rows and columns filled with 0. The position encoding unit (130) is a component a of matrix A. ijRecursively decompose matrix A in a top-down manner, encoding the positions (i, j) as a sequence of length l=log2M. That is, the positions (i, j) of an element can be encoded as l tuples.

[0069] Referring to Fig. 5, the position encoding unit (130) decomposes the matrix A5 into submatrices, i.e., 2x2 partitions, a ij Select a partition with . A 2x2 partition can be composed of four partitions: Top Left (TL), Top Right (TR), Bottom Left (BL), and Bottom Right (BR). Top left can be encoded as (1, 1), top right as (1, 2), bottom left as (2, 1), and bottom right as (2, 2). Referring to Equation 2, a square matrix , N=M=2 l In this case, the positions (i,j) are sequentially encoded through matrix decomposition, and the kth tuple (t(i,k), t(j,k)) of the position sequence can be encoded as in mathematical expression 2.

[0070] [Equation 2]

[0071]

[0072] The location encoding unit (130) is a ij Until only one remains, a ij Split the partition into 2x2 partitions, a ij Repeat the process of selecting and disassembling the partition that has a ij The locations selected as partitions with a ij is encoded as a sequence of positions.

[0073] For example, a of matrix A5 with l=3 3,4 If we encode the position (3,4) of a 3,4is located at the top left of matrix A5, is located at the bottom left among the partitions that decompose the top left partition, and is located at the top right among the partitions that decompose the bottom left partition. Therefore, a 3,4 can be encoded as a positional sequence of three tuples: (1, 1), (2, 2), (1, 2). Similarly, a 5,1 is located at the bottom left of matrix A5, is located at the top left among the partitions that decompose the bottom left partition, and is located at the top left among the partitions that decompose the top left partition. Therefore, a 5,1 The position (5,1) of can be encoded as the position sequence of (2, 1), (1, 1), (1, 1). a 8,5 is located at the bottom right of matrix A5, is located at the bottom left among the partitions that decompose the bottom right partition, and is located at the bottom left among the partitions that decompose the bottom left partition. Therefore, a 8,5 The position (8,5) can be encoded as the position sequence of (2, 2), (2, 1), (2, 1).

[0074] Referring to Fig. 6, matrix decomposition is possible even if the matrix is ​​not a square matrix like Fig. 5. The position encoding unit (130) decomposes a non-square matrix A6 of arbitrary size into 2x2 partitions, and a ij The process of selecting a partition and decomposing it is repeated. At this time, since it is a non-square matrix, the last partition can be a 1x2 or 2x1 partition.

[0075] Irregular matrix Let me explain with an example. l co Is l colFirst, to decompose a non-square matrix into 2x2 partitions, add rows or columns filled with 0 to the given matrix. It is extended to . If the rows and columns of the extended matrix have different sizes (e.g., l row <l col ), the location encoding unit (130) is a ij To encode the location of , the given matrix is ​​partitioned into 2x2 partitions l row Decompose it recursively, A partition having the size of is obtained. And the location encoding unit (130) The size of the partition is the same size, for example, 1x2 partition l row - l col can be decomposed recursively. A 1x2 partition can be decomposed into left and right. Positions (i,j) are sequentially encoded through matrix decomposition, and the k-th tuple (t row (i,k), (t col (j,k)) can be encoded as in mathematical expression 3.

[0076] [Equation 3]

[0077]

[0078] For example, a of the non-square matrix A6 2,3 If we encode the position (2,3), a 2,3 is located at the top left of matrix A6, at the bottom right among the partitions that decompose the top left partition, and at the left among the 1x2 partitions that decompose the bottom right. Therefore, a 2,3 can be encoded as a positional sequence of (1,1), (2,2), (0,1). Here, the position of the left side of the 1x2 partition can be defined as (0,1). Similarly, a of a non-square matrix A6 4,6The position (4,6) of is at the bottom right of matrix A6, is at the bottom left among the partitions that decompose the bottom right partition, and is at the right among the 1x2 partitions that decompose the bottom left. Therefore, a 4,6 can be encoded as a position sequence of (2,2), (2,1), (1,0). Here, the position on the right side of the 1x2 partition can be defined as (1,0).

[0079] Referring to Fig. 7, the approximate model (150) receives a position sequence of a target entry and, through a certain number of parameters, uses matrices (K) used to infer the value of the target entry. 1, K2, … ) are generated. Here, the approximation matrix of the given matrix can be considered as the generalized Kronecker product of the matrices generated by the approximation model (150). For example, the target component a 3,4 When the sequence (1, 1), (2, 2) encoding the location is input to the approximate model (150), matrices K generated through a certain number of parameters 1, K2, K3 are output. K 1, Approximation matrix created by the Kronecker product of K2 and K3 The value of the (3,4) component of the target component a 3,4 can be restored as an approximation. Here, the entire Kronecker product does not need to be calculated to obtain an approximation, but rather a product of the components corresponding to the input tuples in the matrix generated by each LSTM cell, i.e., K1(1,1), K2(2,2), K3(1,2). 3,4 The value of can be approximated. Meanwhile, a 3,4The last tuple (1, 2) in the sequence encoding the position of is not input to the approximate model, but can be used to restore the value through the Kronecker product. However, the approximate model (150) of Fig. 7 is an example, and the approximate model can be implemented as various forms of artificial intelligence models that can restore the component value from the position sequence of the target component through a certain number of parameters.

[0080] In this way, a certain number of parameters are updated by repeatedly rearranging the matrix, calculating the approximation error, and minimizing it using the gradient descent method so that the approximate value estimated by the approximate model (150) becomes closer to the original component value.

[0081] for example, , N=M=2 l Assume that the matrix is ​​compressed. Each LSTM cell of the approximate model (150) has a target component a ij Given the location (i,j), , … , can be generated. The internal configuration of the approximate model (150) can be designed in various ways. For example, the k-th LSTM cell receives the k-th tuple of the position sequence as input, and the hidden state of the k-th LSTM cell passes through a linear layer and activation (e.g., Softplus activation) to K k+1 can contribute to generating . Here, each component of K1 can be a separate learnable parameter. Approximation of the target component can be calculated from the (i,j)th component of the Kronecker product of the output matrices as in mathematical equation 4.

[0082] [Equation 4]

[0083]

[0084] In mathematical equation 4, is the (i,j)th component of the Kronecker product, and q can be a learnable parameter. Meanwhile, for non-square matrices, the approximation model (150) can be modified to generate matrices of size 1x2 in some LMTM cells.

[0085] FIG. 8 is a diagram illustrating tensor compression according to one embodiment.

[0086] Referring to FIG. 8, the lossy compression device (100A) can compress a sparse and rearrangeable tensor (20) in a manner similar to a sparse matrix.

[0087] D-dimensional tensor, Assuming that, the position encoding unit (130A) first zero-padding the tensor. It is expanded in size. The position encoding unit (130A) expands the expanded tensor to 2 for position encoding. D Recursively decompose into partitions l1 times, A partition having a size is obtained. The position encoding unit (130A) repeatedly decomposes the (D-1)-dimensional tensor recursively, resulting in a D-dimensional position (i1,...,i D ) can be obtained as a position sequence encoded.

[0088] An approximate model (150A) with a certain number of parameters can receive a position sequence as input and generate matrices generated by the parameters of each LSTM cell. The generated matrices (K 1, K 2, … ) can be approximated by the Kronecker product, and parameters of a certain size can be optimized for this purpose.

[0089] The lossy compression device (100A) can also repeat the process of rearranging the D-dimensional tensor, encoding the location, and approximating the target component so that the approximation error is minimized.

[0090] Training of the lossy compression device (100A) for tensor compression is similar to matrix compression, except that there may be differences in shingle calculation. Since D arbitrary bijective functions are used, each mode index has D-1 shingles in the corresponding function, excluding functions of the same mode. The pair of positions i1 and i2 are placed close to each other only if both the D-1 shingles of i1 and the shingles of index j are the same. The initial indices of the tensor are rearranged randomly.

[0091] Figure 9 is a flowchart of a lossy compression method according to one embodiment.

[0092] Referring to FIG. 9, the loss compression device (100) initializes a given matrix (S110). The loss compression device (100) can initialize the given matrix using a co-clustering algorithm.

[0093] The lossy compression device (100) estimates the similarity of rows or columns and selects two similar rows or two similar columns (S120).

[0094] The lossy compression device (100) rearranges the matrix by positioning a pair of selected rows or a pair of selected columns closer together through position exchange within the matrix (S130). The lossy compression device (100) rearranges the rows and columns to make the given matrix self-similar.

[0095] The lossy compression device (100) recursively decomposes the rearranged matrix and encodes the positions of non-zero elements at each decomposition step (S140). The lossy compression device (100) can recursively decompose the input matrix after expanding it by zero-padding it.

[0096] The loss compression device (100) trains the approximation model by updating the parameters of the approximation model (150) so that the loss function (approximation error) for all components is minimized by obtaining the approximate values ​​of the corresponding components from the position sequence of non-zero components (S150). The loss compression device (100) can calculate the approximate values ​​of the corresponding elements and the approximate error between the actual values, and train the parameters of the approximation model (150) using the gradient descent method so that the approximate error is minimized.

[0097] The loss compression device (100) determines whether the approximation error converges (S160) and repeats the process of rearranging the current matrix until the approximation error converges. The loss compression device (100) can alternately perform parameter training and matrix rearrangement.

[0098] The loss compression device (100) terminates the training of the approximate model (150) when the approximation error converges, and obtains an approximate model (150) composed of a certain number of parameters (S170). The trained approximate model (150) or a certain number of parameters constituting the same may be provided as a result of compression of a sparse matrix or a sparse tensor. The component values ​​of a given matrix may be restored through the approximate model (150). During the training of the approximate model (150), sequences encoding the positions of target components and the actual values ​​of the target components are required, so a space proportional to the number of non-zero components of the sparse matrix is ​​required. However, as a result, a sparse matrix of any size can be compressed to a constant size corresponding to the number of parameters (up to k).

[0099] Figure 10 is a configuration diagram of a computing device according to one embodiment.

[0100] Referring to FIG. 10, the lossy compression device (100) can be implemented as a computing device (200) operated by at least one processor.

[0101] A computing device (200) may include one or more processors (210), a memory (230) for loading a computer program to be executed by the processor (210), a storage device (250) for storing the computer program and various data, a communication interface (270), and a bus (290) connecting them. In addition, the computing device (200) may further include various components. The program may include instructions that cause the processor (210) to perform methods / operations according to various embodiments of the present disclosure when loaded into the memory (230). That is, the processor (210) may perform methods / operations according to various embodiments of the present disclosure by executing the instructions. An instruction is a series of computer-readable instructions grouped based on a function, and refers to a component of a computer program and something executed by a processor.

[0102] The processor (210) controls the overall operation of each component of the computing device (200). The processor (210) may be configured to include at least one of a Central Processing Unit (CPU), a Micro Processor Unit (MPU), a Micro Controller Unit (MCU), a Graphics Processing Unit (GPU), or any other type of processor well known in the art of the present disclosure. In addition, the processor (210) may perform operations for at least one computer program for executing methods / operations according to various embodiments of the present disclosure.

[0103] The memory (230) stores various data, commands, and / or information. The memory (230) can load one or more programs from the storage device (250) to execute methods / operations according to various embodiments of the present disclosure. The memory (230) may be implemented as volatile memory, such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0104] The storage device (250) can store a program non-temporarily. The storage device (250) can be configured to include a non-volatile memory such as a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0105] The communication interface (270) supports wired and wireless communication of the computing device (200). To this end, the communication interface (270) may be configured to include a communication module well known in the technical field of the present disclosure.

[0106] The bus (290) provides communication between components of the computing device (200). The bus (290) may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0107] The computer program includes instructions to be executed by the processor (210), and the processor (210) can, by executing the instructions, estimate the similarity of rows or columns in a given matrix to select two similar rows or two similar columns, rearrange the matrix by positioning a pair of selected rows or a pair of columns close together through position exchange within the matrix, and then recursively decompose the rearranged matrix to encode the position of each target element. In addition, the processor (210) can, by executing the instructions, update the parameters of the approximation model so that the approximation error of the matrix approximated from the position sequence of the target element is minimized, and repeat the process of rearranging the current matrix and updating the approximation model until the approximation error converges. The processor (210) can finally obtain an approximation model composed of the updated parameters, and restore the component values ​​of the given matrix through the approximation model.

[0108] In this way, the lossy compression device (100) can compress matrices or tensors of various sizes into a constant size, and as a result, provides a compression ratio that is 50 times better than existing technologies with similar approximation errors, and can provide an approximation error that is 10 times smaller than existing technologies that provide outputs of similar sizes. According to the present disclosure, 200 million video viewing histories corresponding to sparse matrices can be successfully compressed into a size of 10 KB, and the information loss due to compression is less than that of compressing into 1 GB using existing technologies.

[0109] The loss compression device (100) can calculate a loss function for an input matrix by accessing only non-zero target components, thereby enabling efficient calculation by utilizing the sparsity of the input matrix and linear-time training.

[0110] During training of the lossy compression device (100), a space proportional to the number of non-zero elements of the input matrix is ​​required, but as a result, compression of constant size can be provided.

[0111] The lossy compression device (100) can approximate each component of the input matrix in sublinear time using a trained approximation model (150).

[0112] Meanwhile, the lossy compression device (100) can reduce the approximation error by rearranging the input, and since it is data that cannot be rearranged, it can provide superior compression performance compared to existing technologies even if the original positions of rows and columns are stored.

[0113] The embodiments of the present disclosure described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present disclosure or a recording medium on which the program is recorded.

[0114] Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concepts of the present disclosure defined in the following claims also fall within the scope of the present disclosure.

Claims

1. A lossy compression device driven by at least one processor, A rearrangement unit that rearranges the input matrix so that it has self-similarity, A position encoding unit that recursively decomposes the rearranged matrix and encodes the position of the target element into a sequence, and An approximate model trained to infer the value of a target component from a position sequence of the target component through a certain number of parameters. A lossy compression device comprising:

2. In paragraph 1, The above rearrangement part A lossy compression device that selects two similar rows or two similar columns based on the similarity of rows or columns constituting the input matrix, and generates the rearranged matrix by positioning the selected pair of rows or pair of columns close together.

3. In paragraph 1, The above location encoding section A lossy compression device that generates the position sequence by sequentially encoding the position of the target component at each decomposition step while repeating the process of decomposing the rearranged matrix into partitions, which are sub-matrices, selecting a partition in which the target component is located, and decomposing the selected partition.

4. In paragraph 3, The above location encoding section If the above rearranged matrix is ​​a square matrix, recursively decompose the above rearranged matrix into 2x2 partitions, A lossy compression device that recursively decomposes the rearranged matrix into 2x2 partitions and then decomposes the rearranged matrix into 1x2 partitions or 2x1 partitions if the rearranged matrix is ​​a non-square matrix.

5. In paragraph 1, The parameters of the above approximate model are A lossy compression device that is updated through training to minimize the approximation error between the approximated value and the actual value from the positional sequence of random components.

6. In paragraph 1, The above approximate model is A lossy compression device that generates matrices from the position sequence through the above-mentioned predetermined number of parameters, and infers component values ​​corresponding to the position sequence through the Kronecker product of the generated matrices.

7. In paragraph 1, The above approximate model is A lossy compression device, a model implemented with LSTM (Long Short Term Memory) cells of the recurrent neural network family.

8. In paragraph 1, A lossy compression device, wherein the above input matrix is ​​a sparse matrix.

9. A method of operating a lossy compression device driven by at least one processor, A step of training an approximate model to infer the value of a target component from a sequence encoding the position of the target component in an input matrix through a certain number of parameters, and A step of obtaining the approximate model composed of the above-mentioned number of parameters. A method of operation, comprising:

10. In paragraph 9, The steps for training the above approximate model are An operating method comprising: rearranging the input matrix to have self-similarity, recursively decomposing the rearranged matrix to encode the positions of target elements into a sequence, and then repeating the process of updating the parameters so that the approximation error between the approximated value and the actual value from the sequence of positions of the target elements is minimized.

11. In paragraph 10, The steps for training the above approximate model are An operating method for generating the position sequence by sequentially encoding the position of the target component at each decomposition step while repeating the process of decomposing the rearranged matrix into partitions, which are sub-matrices, selecting a partition where the target component is located, and decomposing the selected partition.

12. In paragraph 11, If the above rearranged matrix is ​​a square matrix, the above rearranged matrix is ​​recursively decomposed into 2x2 partitions, A method of operation in which, if the above rearranged matrix is ​​a non-square matrix, the above rearranged matrix is ​​recursively decomposed into 2x2 partitions and then decomposed into 1x2 partitions or 2x1 partitions.

13. In paragraph 9, The above approximate model is An operating method for generating matrices from the position sequence through the above-mentioned predetermined number of parameters, and inferring component values ​​corresponding to the position sequence through the Kronecker product of the generated matrices.

14. In paragraph 9, The above approximate model is The operating method of a model implemented with LSTM (Long Short Term Memory) cells of the recurrent neural network series.

15. In paragraph 9, A method of operation in which the above input matrix is ​​a sparse matrix.

16. A processor that repeats training to rearrange an input matrix or input tensor to have self-similarity by executing commands, recursively decompose the rearranged matrix or tensor to encode the positions of target elements into a sequence, and update the parameters of an approximation model so that the approximation error between the approximated value and the actual value from the sequence of positions of the target elements is minimized. A lossy compression device comprising:

17. In paragraph 16, The above processor A lossy compression device that provides the trained approximate model or a certain number of parameters constituting the approximate model as a compression result of the input matrix or the input tensor.

18. In paragraph 16, The above processor A lossy compression device that slices the input matrix or the input tensor and rearranges the slices so that similar slices become closer while changing the order of the slices.

19. In paragraph 16, The above approximate model is A lossy compression device that generates matrices from an input position sequence through a certain number of parameters and infers component values ​​corresponding to the input position sequence through the Kronecker product of the generated matrices.

20. In paragraph 16, The above approximate model is A lossy compression device, a model implemented with LSTM (Long Short Term Memory) cells of the recurrent neural network family.

Images