Data compression method and system based on tensor transformation

Abstract

The invention discloses a data compression method and system based on tensor transformation, and the method comprises the steps: converting original data into tensor data of a first dimension according to the features and dimensions of the original data; performing dimension reduction processing on the tensor data of the first dimension through a tensor decomposition technology to obtain a plurality of pieces of tensor data of the second dimension; wherein the first dimension is greater than the second dimension; reconstructing the tensor data of the second dimension to obtain tensor representation of the second dimension of the original data; and based on dictionary coding, performing data coding and compression on the tensor representation of the second dimension to obtain compressed data corresponding to the original data. The tensor transformation technology is used in the data acquisition and compression process of the model, so that the data compression speed and effect are improved.

Description

Technical Field

[0001] The present application belongs to the field of medium- and long-term load forecasting for power grids, and specifically relates to a data compression method and system based on tensor transformation. Background Art

[0002] With the development of computer technology, the amount of data generated in the model operation process continues to grow. For example, in the process of deep learning model training, the amount of data generated is tens of GB or even hundreds of GB. This data has brought huge challenges in storage, transmission and analysis. Traditional data compression methods such as JPEG and MP3 can no longer meet existing needs, because these methods can only compress specific types of data and cannot adapt to various complex data structures.

[0003] In order to solve this problem, tensor transformation technology has been widely used in the field of data compression in recent years. A tensor is a multidimensional array that can describe complex data structures. Tensor transformation reduces the amount of data by reducing the dimension and compressing the data while retaining the main features of the data. This technology has achieved remarkable results in image processing, speech recognition, data analysis and other fields. However, there are still some problems with the existing tensor transformation technology. First, the computational complexity of some tensor transformation algorithms is high, resulting in slow data compression; second, some algorithms have unsatisfactory compression effects when processing complex data structures; finally, the existing tensor transformation algorithms lack targeted processing of the data collected during the model operation process. Therefore, how to improve the speed and effect of data compression collected during the model operation process of tensor transformation while ensuring data quality has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present application proposes a data compression method and system based on tensor transformation, which uses tensor transformation technology in the process of data acquisition and compression of the model to improve the speed and effect of data compression.

[0005] A first aspect of the present application provides a data compression method based on tensor transformation, the method comprising:

[0006] According to the characteristics and dimensions of the original data, the original data is converted into tensor data of the first dimension;

[0007] Performing dimensionality reduction processing on the tensor data of the first dimension by using tensor decomposition technology to obtain a plurality of tensor data of the second dimension; wherein the first dimension is greater than the second dimension;

[0008] Reconstructing the tensor data of the second dimension to obtain a tensor representation of the second dimension of the original data;

[0009] Based on dictionary coding, data encoding and compression are performed on the tensor representation of the second dimension to obtain compressed processed data corresponding to the original data.

[0010] The above scheme first converts the original data into high-dimensional, tensor-form data through the characteristics and dimensions of the original data to complete the conversion of the data structure. Then the complex high-dimensional tensor data is decomposed into multiple low-dimensional tensor data, and then the low-dimensional tensor data is reconstructed into a low-dimensional tensor representation of the original data through data reconstruction, so as to ensure the main characteristics of the original data while reducing the amount of data, which helps to improve the subsequent calculation rate. Finally, dictionary encoding is used to compress the second-dimensional tensor representation of the original data after dimensionality reduction, further reducing the amount of data and improving the speed and effect of data compression.

[0011] In a possible implementation method of the first aspect, the original data is converted into tensor data of the first dimension according to the characteristics and dimensions of the original data, specifically:

[0012] Pre-process the collected raw data according to the preset processing requirements;

[0013] Determine the corresponding tensor structure based on the characteristics and dimensions of the preprocessed raw data;

[0014] According to the tensor structure, the preprocessed raw data is converted into tensor data of the first dimension.

[0015] The above scheme first preprocesses the data to improve data quality, and then selects a suitable tensor structure to complete the tensor conversion according to the characteristics and dimensions of the preprocessed data, thereby obtaining high-dimensional tensor data that can retain the main features of the data, thereby extracting the main data features from the complex data structure for subsequent calculations.

[0016] In a possible implementation method of the first aspect, the tensor data of the second dimension is specifically:

[0017] tensor=W_1×…×W_n×VT×…×VT;

[0018] In the formula, tensor is the tensor data of the first dimension, W_n is the nth tensor data of the second dimension, n is the total number of tensor data of the second dimension, and VT is the tensor of the second dimension.

[0019] In a possible implementation method of the first aspect, the tensor data of the second dimension is reconstructed to obtain a tensor representation of the second dimension of the original data, specifically:

[0020] Based on the preset tensor coefficient of the second dimension, all the tensor data of the second dimension are reconstructed to obtain the tensor representation of the second dimension of the original data, so as to complete the dimensionality reduction of the original data.

[0021] In a possible implementation method of the first aspect, based on dictionary coding, data encoding and compression are performed on the tensor representation of the second dimension to obtain compressed processed data corresponding to the original data, specifically:

[0022] Determine the corresponding dictionary size and encoding method based on the characteristics of the original data;

[0023] According to the dictionary size and encoding method, a dictionary of the original data is constructed by a clustering algorithm;

[0024] Based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data.

[0025] The above scheme first selects a dictionary size and encoding method suitable for the original data to provide a higher data compression ratio to improve the compression effect. The tensor representation of the second dimension is encoded and compressed through the field, and the reduced-dimensional data is represented as a linear combination of elements in the dictionary to achieve further compression of the reduced-dimensional data and reduce the amount of data.

[0026] In a possible implementation method of the first aspect, data encoding and compression are sequentially performed on the tensor representation of the second dimension, specifically as follows:

[0027]

[0028] Wherein, compressed_data is the compressed data, x_i is the i-th element in the tensor representation of the second dimension, w_i is the dictionary element corresponding to x_i, sign(x_i) is the sign function of x_i, N is the total number of elements in the tensor representation of the second dimension, and γ is an adjustable parameter.

[0029] A second aspect of the present application provides a data compression system based on tensor transformation, the system comprising: a data conversion module, a data dimension reduction module, a data reconstruction module and a data compression module;

[0030] The data conversion module is used to convert the original data into tensor data of the first dimension according to the characteristics and dimensions of the original data;

[0031] The data dimension reduction module is used to perform dimension reduction processing on the tensor data of the first dimension by using the tensor decomposition technology to obtain a plurality of tensor data of the second dimension; wherein the first dimension is greater than the second dimension;

[0032] The data reconstruction module is used to reconstruct the tensor data of the second dimension to obtain a tensor representation of the second dimension of the original data;

[0033] The data compression module is used to perform data encoding and compression on the tensor representation of the second dimension based on dictionary encoding to obtain compressed processed data corresponding to the original data.

[0034] In a possible implementation of the second aspect, the data conversion module includes: a data structure transformation unit;

[0035] Among them, the data structure transformation unit is used to preprocess the collected original data according to preset processing requirements; determine the corresponding tensor structure according to the characteristics and dimensions of the preprocessed original data; and convert the preprocessed original data into tensor data of the first dimension according to the tensor structure.

[0036] In a possible implementation manner of the second aspect, the data reconstruction module includes a data reconstruction unit;

[0037] The data reconstruction unit is used to reconstruct all second-dimensional tensor data based on preset second-dimensional tensor coefficients to obtain a second-dimensional tensor representation of the original data, so as to complete dimensionality reduction of the original data.

[0038] In a possible implementation of the second aspect, the data compression module includes: a dictionary encoding unit;

[0039] Among them, the dictionary encoding unit is used to determine the corresponding dictionary size and encoding method according to the characteristics of the original data; construct a dictionary of the original data through a clustering algorithm according to the dictionary size and encoding method; based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a specific flow chart of a data compression method based on tensor transformation provided in a certain embodiment of the present application;

[0042] Figure 2 It is a specific structural diagram of a data compression system based on tensor transformation provided in a certain embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] It should be understood that the step numbers used in this article are only for the convenience of description and are not intended to limit the order in which the steps are executed.

[0045] First embodiment

[0046] Tensors can be understood as a collection of vectors and matrices, which can perform mathematical operations and transfer data between different layers. Therefore, tensors are often used as basic data structures in current machine learning. Tensor transformation technology is often used in the process of data compression to compress data and retain the main features of the data. However, when the data to be compressed is more complex during the model operation process, the high computational complexity of tensor transformation will cause the data compression speed to slow down, and it is difficult to improve the compression effect when facing more complex data structures. Therefore, how to transform complex data structures into simpler data structures to facilitate data compression when using tensor transformation for data compression is the main research problem of the embodiments of the present application.

[0047] like Figure 1 As shown, Figure 1 A specific flow chart of a data compression method based on tensor transformation is provided for a certain embodiment of the present application. The data compression method based on tensor transformation of this embodiment includes steps S1 to S4, which are described in detail as follows:

[0048] Step S1, converting the original data into tensor data of the first dimension according to the characteristics and dimensions of the original data.

[0049] In the embodiment of the present application, the model collects data in real time during the operation process, and the raw data collected here can be images, audio, video or other types of data. In the embodiment of the present application, image data is mainly used as an example. In the collection process, preprocessing techniques such as data cleaning and feature extraction are also used to improve data quality.

[0050] Exemplarily, according to actual processing requirements, preprocessing operations are performed on the collected image data, including data cleaning, feature extraction and data standardization.

[0051] Among them, data cleaning is to conduct necessary review and verification of the original image data to discover and correct identifiable errors in the data, improve data quality and availability, and ensure that the data can be better used in subsequent operations. The main contents of data cleaning include checking the consistency of the original image data, processing invalid values ​​and missing values, and deleting duplicate information. Feature extraction mainly extracts the main features for subsequent data compression from the original image data, including color features and shape features, and data standardization mainly unifies the data format to facilitate subsequent unified processing.

[0052] Then, according to the characteristics and dimensions of the preprocessed raw data, a suitable tensor structure is determined to convert the raw data into multi-dimensional data to obtain tensor data of the first dimension, wherein the first dimension is generally higher to more completely represent the main characteristics of the raw data and ensure the quality of the data.

[0053] For example, taking image data as an example, the RGB value of each pixel can be used as one dimension, and the number of rows and columns of the image data can be used as the other two dimensions to form a three-dimensional tensor. This three-dimensional tensor can be expressed as:

[0054] image=tensor(row,col,channel);

[0055] In the formula, image is the image data, tensor is a three-dimensional array (or three-dimensional tensor), channel represents the number of channels of the image (for example, for a color image, the number of channels is usually 3, corresponding to red, green, and blue, respectively), and row and col represent the number of rows and columns of the image data, respectively.

[0056] Step S2, performing dimensionality reduction processing on the tensor data of the first dimension by using tensor decomposition technology to obtain a plurality of tensor data of the second dimension.

[0057] In an embodiment of the present application, after obtaining high-dimensional tensor data of the first dimension, in order to reduce the amount of data to increase the rate of data compression, it is necessary to reduce the dimension of the tensor data of the first dimension through tensor decomposition technology to obtain multiple low-dimensional tensor data of the second dimension.

[0058] In the process of tensor decomposition, you can choose a suitable decomposition method according to the data structure characteristics. If the tensor is a symmetric tensor, you can use the symmetric tensor decomposition method to reduce the data dimension.

[0059] For example, taking symmetric tensor decomposition as an example, the following formula can be used for dimensionality reduction to obtain tensor data of the second dimension:

[0060] tensor=W_1×…×W_n×VT×…×VT;

[0061] In the formula, tensor is the tensor data of the first dimension, W_n is the nth tensor data of the second dimension, n is the total number of tensor data of the second dimension, VT is the tensor of the second dimension, and generally refers to the tensors of the second dimension obtained by decomposition except W_n.

[0062] Taking a three-dimensional symmetric image tensor as an example, this formula can decompose the three-dimensional symmetric image tensor into the product of multiple two-dimensional symmetric image tensors to achieve data dimensionality reduction.

[0063] Optionally, a core tensor decomposition method may be used to reduce the dimension of the tensor, and the specific method depends on the data characteristics and requirements of the tensor data of the first dimension.

[0064] Because the data after tensor decomposition is decomposed from a high-dimensional data into multiple low-dimensional data, the data volume is reduced by reducing the complexity of the data, while retaining the main features of the data, so the computational complexity of subsequent data compression is greatly reduced, and the data compression speed is improved. Moreover, the low-dimensional data structure is simpler than before the dimensionality reduction process, ensuring that the compression algorithm can achieve significant compression effects.

[0065] Optionally, during the dimensionality reduction process, an appropriate optimization algorithm such as gradient descent or Newton's method may be used to improve the dimensionality reduction effect.

[0066] Step S3, reconstructing the tensor data of the second dimension to obtain a tensor representation of the second dimension of the original data.

[0067] In an embodiment of the present application, the tensor data of the second dimension is reconstructed into a tensor representation of the original data to complete the dimensionality reduction of the original data.

[0068] Optionally, the embodiment of the present application uses tensor multiplication and addition operations to realize tensor reconstruction, specifically using the following formula:

[0069] reconstructed_tensor=W_1×V×…×W_n×VT;

[0070] In the formula, reconstructed_tensor is the tensor representation of the second dimension of the original data.

[0071] Based on this, a low-dimensional tensor representation of the original data is obtained, which reduces the complexity of the data while retaining the main features of the data.

[0072] Step S4: Based on dictionary coding, data encoding and compression are performed on the tensor representation of the second dimension to obtain compressed processed data corresponding to the original data.

[0073] In an embodiment of the present application, the reduced-dimensional data is compressed by dictionary coding to further reduce the amount of data.

[0074] Optionally, dictionary encoding may adopt LZ77, LZ78 or other types of dictionary encoding algorithms.

[0075] First, according to the characteristics of the original data, the corresponding dictionary size and encoding method are determined for the tensor representation of the second dimension after dimensionality reduction to improve the compression effect.

[0076] A dictionary is constructed for the original data according to the dictionary size and the encoding method, wherein each element of the dictionary corresponds to a feature in the original data.

[0077] Specifically, a K-means clustering algorithm or other clustering algorithms may be used to cluster the data, and each cluster center is used as a dictionary element.

[0078] Based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data.

[0079] Optionally, the embodiment of the present application uses Huffman coding to compress data. In some embodiments, LZW coding or other forms of coding methods can also be used for data compression, which is specifically selected according to the characteristics of the tensor representation of the second dimension and the size of the dictionary.

[0080] The dictionary encoding specifically adopts the following formula:

[0081]

[0082] Wherein, compressed_data is the compressed data, x_i is the i-th element in the tensor representation of the second dimension, w_i is the dictionary element corresponding to x_i, sign(x_i) is the sign function of x_i, N is the total number of elements in the tensor representation of the second dimension, and γ is an adjustable parameter between 0 and 1, which is used to adjust the compression ratio or compression effect in the data compression process.

[0083] The purpose of this formula is to treat the reduced-dimensional data as a linear combination of the elements in the dictionary and encode them using the signs and absolute values ​​of the elements to provide data support for subsequent data compression.

[0084] After data compression is completed, the compressed data can be stored on disk or transmitted to other devices according to actual needs. This can greatly reduce data storage and transmission costs without affecting data quality.

[0085] Furthermore, the data may be stored in binary format, text format or other suitable data storage formats. At the same time, in order to ensure the reliability and integrity of the data, an encryption algorithm may be used to encrypt the data to improve the security of the data.

[0086] Implementing the embodiments of the present application has the following beneficial effects:

[0087] The embodiment of the present application converts the original data into high-dimensional data in tensor form by first using the characteristics and dimensions of the original data to complete the conversion of the data structure. The complex high-dimensional tensor data is then decomposed into multiple low-dimensional tensor data, and the low-dimensional tensor data is reconstructed into a low-dimensional tensor representation of the original data through data reconstruction, so as to reduce the amount of data while ensuring the main characteristics of the original data, and improve the subsequent calculation rate by reducing the complexity of the data. Finally, dictionary coding is used to encode and compress the tensor representation of the second dimension of the original data after dimensionality reduction, further reducing the amount of data and improving the speed and effect of data compression. Moreover, the compressed data obtained can greatly reduce the storage and transmission costs in storage and transmission operations.

[0088] Second embodiment

[0089] Further, in order to execute the data compression system based on tensor transformation corresponding to the above method embodiment to achieve corresponding functions and technical effects, Figure 2 A structural diagram of a data compression system based on tensor transformation is provided. For ease of explanation, only the parts related to this embodiment are shown. The data compression system based on tensor transformation provided by the embodiment of the present application includes:

[0090] The data conversion module 201 is used to convert the original data into tensor data of the first dimension according to the characteristics and dimensions of the original data.

[0091] In an embodiment of the present application, the collected raw data is preprocessed according to preset processing requirements; the corresponding tensor structure is determined according to the characteristics and dimensions of the preprocessed raw data; and according to the tensor structure, the preprocessed raw data is converted into tensor data of the first dimension.

[0092] The data dimension reduction module 202 is used to perform dimension reduction processing on the tensor data of the first dimension by using tensor decomposition technology to obtain tensor data of a plurality of second dimensions; wherein the first dimension is greater than the second dimension.

[0093] The data reconstruction module 203 is used to reconstruct the tensor data of the second dimension to obtain a tensor representation of the second dimension of the original data.

[0094] In an embodiment of the present application, based on the preset tensor coefficient of the second dimension, all tensor data of the second dimension are reconstructed to obtain a tensor representation of the second dimension of the original data to complete the dimensionality reduction of the original data.

[0095] The data compression module 204 is used to perform data encoding and compression on the tensor representation of the second dimension based on dictionary encoding to obtain compressed processed data corresponding to the original data.

[0096] In an embodiment of the present application, the corresponding dictionary size and encoding method are determined according to the characteristics of the original data; a dictionary of the original data is constructed through a clustering algorithm according to the dictionary size and encoding method; based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data.

[0097] In some embodiments, the data conversion module 201 further includes:

[0098] The model collects data in real time during the operation process. The raw data collected here can be images, audio, video or other types of data. In the embodiments of the present application, image data is mainly used as an example. During the collection process, appropriate pre-processing techniques such as data cleaning and feature extraction are also used to improve data quality.

[0099] Exemplarily, according to actual processing requirements, preprocessing operations are performed on the collected image data, including data cleaning, feature extraction and data standardization.

[0100] Among them, data cleaning is to conduct necessary review and verification of the original image data to discover and correct identifiable errors in the data, improve data quality and availability, and ensure that the data can be better used in subsequent operations. The main contents of data cleaning include checking the consistency of the original image data, processing invalid values ​​and missing values, and deleting duplicate information. Feature extraction mainly extracts the main features for subsequent data compression from the original image data, including color features and shape features, and data standardization mainly unifies the data format to facilitate subsequent unified processing.

[0101] Then, according to the characteristics and dimensions of the preprocessed raw data, a suitable tensor structure is determined to convert the raw data into multi-dimensional data to obtain tensor data of the first dimension, wherein the first dimension is generally higher to more completely represent the main characteristics of the raw data and ensure the quality of the data.

[0102] For example, taking image data as an example, the RGB value of each pixel can be used as one dimension, and the number of rows and columns of the image data can be used as the other two dimensions to form a three-dimensional tensor. This three-dimensional tensor can be expressed as:

[0103] image=tensor(row,col,channel);

[0104] In the formula, image is the image data, tensor is a three-dimensional array (or three-dimensional tensor), channel represents the number of channels of the image (for example, for a color image, the number of channels is usually 3, corresponding to red, green, and blue, respectively), and row and col represent the number of rows and columns of the image data, respectively.

[0105] In some embodiments, the data dimension reduction module 202 further includes:

[0106] After obtaining high-dimensional first-dimensional tensor data, in order to reduce the data volume and improve the data compression rate, it is necessary to reduce the dimension of the first-dimensional tensor data through tensor decomposition technology to obtain multiple low-dimensional second-dimensional tensor data.

[0107] In the process of tensor decomposition, you can choose a suitable decomposition method according to the data structure characteristics. If the tensor is a symmetric tensor, you can use the symmetric tensor decomposition method to reduce the data dimension.

[0108] For example, taking symmetric tensor decomposition as an example, the following formula can be used for dimensionality reduction to obtain tensor data of the second dimension:

[0109] tensor=W_1×…×W_n×VT×…×VT;

[0110] In the formula, tensor is the tensor data of the first dimension, W_n is the nth tensor data of the second dimension, n is the total number of tensor data of the second dimension, VT is the tensor of the second dimension, and generally refers to the tensors of the second dimension obtained by decomposition except W_n.

[0111] Taking a three-dimensional symmetric image tensor as an example, this formula can decompose the three-dimensional symmetric image tensor into the product of multiple two-dimensional symmetric image tensors to achieve data dimensionality reduction.

[0112] Optionally, a core tensor decomposition method may be used to reduce the dimension of the tensor, and the specific method depends on the data characteristics and requirements of the tensor data of the first dimension.

[0113] Because the data after tensor decomposition is decomposed from a high-dimensional data into multiple low-dimensional data, the data volume is reduced by reducing the complexity of the data, while retaining the main features of the data, so the computational complexity of subsequent data compression is greatly reduced, and the data compression speed is improved. Moreover, the low-dimensional data structure is simpler than before the dimensionality reduction process, ensuring that the compression algorithm can achieve significant compression effects.

[0114] Optionally, during the dimensionality reduction process, an appropriate optimization algorithm such as gradient descent or Newton's method may be used to improve the dimensionality reduction effect.

[0115] In some embodiments, the data reconstruction module 203 further includes:

[0116] The tensor data of the second dimension is reconstructed into a tensor representation of the original data to complete the dimensionality reduction of the original data.

[0117] Optionally, the embodiment of the present application uses tensor multiplication and addition operations to realize tensor reconstruction, specifically using the following formula:

[0118] reconstructed_tensor=W_1×V×…×W_n×VT;

[0119] In the formula, reconstructed_tensor is the tensor representation of the second dimension of the original data.

[0120] Based on this, a low-dimensional tensor representation of the original data is obtained, which reduces the complexity of the data while retaining the main features of the data.

[0121] In some embodiments, the data compression module 204 further includes:

[0122] The reduced-dimensional data is compressed through dictionary encoding to further reduce the amount of data.

[0123] Optionally, dictionary encoding may adopt LZ77, LZ78 or other types of dictionary encoding algorithms.

[0124] First, according to the characteristics of the original data, the corresponding dictionary size and encoding method are determined for the tensor representation of the second dimension after dimensionality reduction to improve the compression effect.

[0125] A dictionary is constructed for the original data according to the dictionary size and the encoding method, wherein each element of the dictionary corresponds to a feature in the original data.

[0126] Specifically, a K-means clustering algorithm or other clustering algorithms may be used to cluster the data, and each cluster center is used as a dictionary element.

[0127] Based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data.

[0128] Optionally, the embodiment of the present application uses Huffman coding to compress data. In some embodiments, LZW coding or other forms of coding methods can also be used for data compression, which is specifically selected according to the features represented by the tensor of the second dimension and the size of the dictionary.

[0129] The dictionary encoding specifically adopts the following formula:

[0130]

[0131] Wherein, compressed_data is the compressed data, x_i is the i-th element in the tensor representation of the second dimension, w_i is the dictionary element corresponding to x_i, sign(x_i) is the sign function of x_i, N is the total number of elements in the tensor representation of the second dimension, and γ is an adjustable parameter between 0 and 1, which is used to adjust the compression ratio or compression effect in the data compression process.

[0132] The purpose of this formula is to treat the reduced-dimensional data as a linear combination of the elements in the dictionary and encode them using the signs and absolute values ​​of the elements to provide data support for subsequent data compression.

[0133] After data compression is completed, the compressed data can be stored on disk or transmitted to other devices according to actual needs. This can greatly reduce data storage and transmission costs without affecting data quality.

[0134] Furthermore, the data may be stored in binary format, text format or other suitable data storage formats. At the same time, in order to ensure the reliability and integrity of the data, an encryption algorithm may be used to encrypt the data to improve the security of the data.

[0135] Implementing the embodiments of the present application has the following beneficial effects:

[0136] The embodiment of the present application converts the original data into high-dimensional data in tensor form by first using the characteristics and dimensions of the original data to complete the conversion of the data structure. The complex high-dimensional tensor data is then decomposed into multiple low-dimensional tensor data, and the low-dimensional tensor data is reconstructed into a low-dimensional tensor representation of the original data through data reconstruction, so as to reduce the amount of data while ensuring the main characteristics of the original data, and improve the subsequent calculation rate by reducing the complexity of the data. Finally, dictionary coding is used to encode and compress the tensor representation of the second dimension of the original data after dimensionality reduction, further reducing the amount of data and improving the speed and effect of data compression. Moreover, the compressed data obtained can greatly reduce the storage and transmission costs in storage and transmission operations.

[0137] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A data compression method based on tensor transformation, characterized in that: include: According to the characteristics and dimensions of the original data, the original data is converted into tensor data of the first dimension; Performing dimensionality reduction processing on the tensor data of the first dimension by using tensor decomposition technology to obtain a plurality of tensor data of the second dimension; wherein the first dimension is greater than the second dimension; Reconstructing the tensor data of the second dimension to obtain a tensor representation of the second dimension of the original data; Based on dictionary coding, data encoding and compression are performed on the tensor representation of the second dimension to obtain compressed processed data corresponding to the original data.

2. The data compression method based on tensor transformation according to claim 1, characterized in that: According to the characteristics and dimensions of the original data, the original data is converted into tensor data of the first dimension, specifically: Pre-process the collected raw data according to the preset processing requirements; Determine the corresponding tensor structure based on the characteristics and dimensions of the preprocessed raw data; According to the tensor structure, the preprocessed raw data is converted into tensor data of the first dimension.

3. The data compression method based on tensor transformation according to claim 1, characterized in that: The tensor data of the second dimension is specifically: tensor=W_1×…×W_n×VT×…×VT; In the formula, tensor is the tensor data of the first dimension, W_n is the nth tensor data of the second dimension, n is the total number of tensor data of the second dimension, and VT is the tensor of the second dimension.

4. The data compression method based on tensor transformation according to claim 1, characterized in that: The reconstructing of the tensor data of the second dimension to obtain the tensor representation of the second dimension of the original data is specifically as follows: Based on the preset tensor coefficient of the second dimension, all the tensor data of the second dimension are reconstructed to obtain the tensor representation of the second dimension of the original data, so as to complete the dimensionality reduction of the original data.

5. The data compression method based on tensor transformation according to claim 1, characterized in that: The dictionary-based coding is used to encode and compress the tensor representation of the second dimension to obtain compressed data corresponding to the original data, specifically: Determine the corresponding dictionary size and encoding method based on the characteristics of the original data; According to the dictionary size and encoding method, a dictionary of the original data is constructed by a clustering algorithm; Based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data.

6. The data compression method based on tensor transformation according to claim 5, characterized in that: The data encoding and compression of the tensor representation of the second dimension are performed in sequence, specifically: Wherein, compressed_data is the compressed data, x_i is the i-th element in the tensor representation of the second dimension, w_i is the dictionary element corresponding to x_i, sign(x_i) is the sign function of x_i, N is the total number of elements in the tensor representation of the second dimension, and γ is an adjustable parameter.

7. A data compression system based on tensor transformation, characterized in that: include: Data conversion module, data dimension reduction module, data reconstruction module and data compression module; The data conversion module is used to convert the original data into tensor data of the first dimension according to the characteristics and dimensions of the original data; The data dimension reduction module is used to perform dimension reduction processing on the tensor data of the first dimension by using the tensor decomposition technology to obtain a plurality of tensor data of the second dimension; wherein the first dimension is greater than the second dimension; The data reconstruction module is used to reconstruct the tensor data of the second dimension to obtain a tensor representation of the second dimension of the original data; The data compression module is used to perform data encoding and compression on the tensor representation of the second dimension based on dictionary encoding to obtain compressed processed data corresponding to the original data.

8. The data compression system based on tensor transformation according to claim 7, characterized in that: The data conversion module includes: a data structure conversion unit; Among them, the data structure transformation unit is used to preprocess the collected original data according to preset processing requirements; determine the corresponding tensor structure according to the characteristics and dimensions of the preprocessed original data; and convert the preprocessed original data into tensor data of the first dimension according to the tensor structure.

9. The data compression system based on tensor transformation according to claim 7, characterized in that: The data reconstruction module includes a data reconstruction unit; The data reconstruction unit is used to reconstruct all second-dimensional tensor data based on preset second-dimensional tensor coefficients to obtain a second-dimensional tensor representation of the original data, so as to complete dimensionality reduction of the original data.

10. The data compression system based on tensor transformation according to claim 7, characterized in that: The data compression module includes: a dictionary encoding unit; Among them, the dictionary encoding unit is used to determine the corresponding dictionary size and encoding method according to the characteristics of the original data; construct a dictionary of the original data through a clustering algorithm according to the dictionary size and encoding method; based on the dictionary of the original data, the tensor representation of the second dimension is sequentially encoded and compressed to obtain compressed processed data corresponding to the original data.

Images