A method and system for improving data storage and query performance

Abstract

The present invention proposes a method and system for improving data storage and query performance. When a user creates a data table, they set compression parameters for the data table or the fields in the data table based on the data type stored in the data table or the data type stored in the fields in the data table. The compression parameters determine the compression algorithm corresponding to the data, as well as whether the compression algorithm uses a vector instruction set and which vector instruction set to use for optimization. The data is then segmented, and the corresponding compression method is determined for the segmented data based on its type and the compression parameters of the field or data table to which it belongs, and the data is compressed and stored. The present invention optimizes the compression algorithm using the CPU's built-in vector instruction set, simultaneously adopts multiple compression methods for the data in a single data table, and uses a more appropriate compression method to process the data stored in each field in the data table, thereby improving data storage and reading performance.

Description

Technical Field

[0001] The present invention relates to the field of data storage and query technology, and in particular to a method and system for improving data storage and query performance. Background Art

[0002] In the information age, as the amount of data continues to rise, taking the power grid data of a certain province as an example, the metering data alone reaches 50 billion rows per year. The large-scale data poses a great challenge to data storage. To solve the storage problem of large-scale data, generally speaking, various databases will compress the data and store it on the hard disk, but this method faces the problem of low compression and decompression efficiency.

[0003] On the other hand, existing databases face the problem of fast storage and query performance failing to keep up with the time requirements of real-time analysis. Currently, existing databases on the market cannot specify compression methods, and instead use the default compression method. Some databases implement the ability to specify compression methods for entire tables, but no database supports different compression methods for each field.

[0004] Therefore, there is currently no universally applicable method that can solve the problem of being unable to efficiently compress different fields of a data table. Summary of the Invention

[0005] In view of this, the present invention proposes a method for improving data storage and query performance, which is used to solve the problem of being unable to achieve efficient compression of different fields of a data table.

[0006] The technical solution of the present invention is achieved as follows:

[0007] In a first aspect, the present invention discloses a method for improving data storage and query performance, the method comprising:

[0008] S1, the user sets the compression parameters when creating a data table;

[0009] S2, uses the CPU's built-in vector instruction set to optimize the compression algorithm;

[0010] S3 divides the data of each field in the data table into multiple high-dimensional vectors, compresses different fields using different compression algorithms, obtains compressed data and stores it.

[0011] The present invention uses the above method to optimize the compression algorithm using the vector instruction set built into the CPU, and simultaneously adopts multiple compression methods for the data in a single data table, and uses a more appropriate compression method to process the data stored in each field in the data table, thereby improving the performance of data storage and reading.

[0012] Based on the above technical solution, preferably, step S1 specifically includes:

[0013] When a user creates a data table, he sets compression parameters for the data table or fields in the data table. The compression parameters include: which compression algorithm to use, whether to enable a vector instruction set, and which instruction set to enable.

[0014] The present invention sets compression parameters through the above method to set a corresponding optimized compression method for data, which is concise and easy to operate.

[0015] Based on the above technical solution, preferably, in the compression parameter content, the compression algorithm used specifically includes:

[0016] When creating a data table, the user sets the compression algorithm type in the compression parameters based on the data type stored in the data table or the data type stored in the fields in the data table. For example, if ID data is set to String type, the corresponding compression algorithm is the lz4 compression algorithm; the compression algorithm corresponding to time series data is the delta-of-delta compression algorithm; the compression algorithm corresponding to status data is the delta compression algorithm; and the compression algorithms for high-frequency data without special patterns include lzw and zstd.

[0017] The present invention uses different compression methods for different types of data through the above method, which greatly improves the data compression efficiency and further reduces the storage space required for the compressed data.

[0018] Based on the above technical solution, preferably, step S2 specifically includes:

[0019] When the compression parameters of the data table or the compression parameters of the fields in the data table specify enabling a vector instruction set, the compression algorithm to be used and the vector instruction set built into the CPU are determined according to the compression parameters, and the compression algorithm is optimized using the vector instruction set built into the CPU.

[0020] Based on the above technical solution, preferably, optimizing the compression algorithm using the vector instruction set built into the CPU specifically includes:

[0021] If the compression parameters determine that the LZ4 compression algorithm is to be used, the CPU's built-in vector instruction set is used for optimization. The vector instruction set includes the AVX256 instruction set, which is used to improve the overall compression speed and search performance.

[0022] If the compression parameters determine the use of the lzw and zstd compression algorithms, the CPU's built-in vector instruction set is used for optimization. The vector instruction set includes the SSE instruction set and the NEON instruction set to improve the overall compression speed and hash search performance.

[0023] The present invention uses the above method to optimize the specified compression algorithm using the vector instruction set specified by the compression parameters, thereby achieving a better optimization effect and improving the compression performance.

[0024] Based on the above technical solution, preferably, step S3 specifically includes:

[0025] S3-1, determine whether the data type stored in the segmented field is a String type. If so, compress the data using the compression algorithm specified in the compression parameters of the field in a memory pool and address length manner; if not, execute step S3-2;

[0026] S3-2, determine whether the segmented field has a compression parameter set. If so, compress it using the compression algorithm specified in the compression parameter of the field; if not, execute step S3-3;

[0027] S3-3, determining whether the data table to which the segmented field belongs has compression parameters set, and if so, using the compression algorithm specified in the compression parameters of the data table for compression; if not, using a preset compression algorithm for compression.

[0028] The present invention determines the compression parameters corresponding to the data through the above method, and determines the compression mode corresponding to the data according to the compression parameters, thereby improving the storage and reading efficiency of the data.

[0029] Based on the above technical solution, preferably, after step S3-3 is executed, the following steps are further included:

[0030] If the compressed data is not accessed within a preset time, the compressed data is written to the hard disk;

[0031] When a user makes a query, a corresponding compression algorithm is determined according to the compression parameters of the compressed data, and decompression is performed according to the compression algorithm, and the decompressed data is stored in a memory cache for use by the user.

[0032] The present invention stores compressed data in real time through the above method. When a user needs to use the data, the decompression method is determined according to the compression parameters and the data is decompressed, thereby improving the efficiency of the user in obtaining data in real time.

[0033] In a second aspect, the present invention discloses a system for improving data storage and query performance, the system comprising:

[0034] Parameter setting module: used for setting compression parameters when users create data tables, based on the data type stored in the data table or the data type stored in the fields in the data table. The compression parameters include: which compression algorithm to use, whether to enable the vector instruction set, and which instruction set to enable;

[0035] Compression algorithm optimization module: used to determine the compression algorithm to be used and the vector instruction set built into the CPU according to the compression parameters, and optimize the compression algorithm using the vector instruction set built into the CPU;

[0036] Data processing module: used to split the data of each field in the data table to obtain multiple high-dimensional vectors; compress the data using the compression algorithm corresponding to the data to obtain compressed data and store it; determine the corresponding compression algorithm based on the compression parameters of the compressed data, decompress it according to the compression algorithm, and store the decompressed data in the memory cache for user use.

[0037] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other via the bus; the memory stores a method program for improving data storage and query performance that can be executed by the processor, and the method program for improving data storage and query performance is configured to implement a method for improving data storage and query performance as described in the first aspect of the present invention.

[0038] In a fourth aspect of the present invention, a computer-readable storage medium is disclosed, on which is stored a method program for improving data storage and query performance. When the method program for improving data storage and query performance is executed, the method for improving data storage and query performance as described in the first aspect of the present invention is implemented.

[0039] The method of improving data storage and query performance of the present invention has the following beneficial effects compared with the prior art:

[0040] (1) By setting compression parameters and specifying different corresponding compression methods according to different data types, multiple compression methods can be implemented for data in a single data table, greatly improving the data compression efficiency and reducing the space required for compressed data storage;

[0041] (2) By enabling the CPU's built-in vector instruction set to optimize the compression algorithm, the performance of the compression algorithm is improved, thereby increasing the compression speed and data reading speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1This is a workflow diagram of a method for improving data storage and query performance according to the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] Example

[0046] The present invention provides a method for improving data storage and query performance. Figure 1 The processing steps are as follows:

[0047] In the first step, the user creates a data table and sets compression parameters for the table based on the data type stored in the table, or sets compression parameters for the fields in the table based on the data type stored in the fields. Go to the second step.

[0048] It should be understood that, based on the above technical solutions, the content of the compression parameters includes: which compression algorithm to use, whether to enable a vector instruction set, and which instruction set to enable.

[0049] It should be understood that, based on the above technical solution, the compression parameters specify the compression algorithm used: For ID data set to String type, the corresponding compression algorithm is lz4; for time series data, the corresponding compression algorithm is delta-of-delta; for status data, the corresponding compression algorithm is delta; and for high-frequency data with no particular pattern, the corresponding compression algorithms include lzw and zstd. Proceed to step 2.

[0050] For example, a user creates a user information data table containing multiple fields, each of which stores a type of data, including ID data, time series data, status data, and note data. The user can set compression parameters for the data table; use the lzw compression algorithm, enable the vector instruction set, and enable the NEON instruction set; and set compression parameters for each field.

[0051] Since each field stores different data types, different compression algorithms can be selected for compression processing to improve compression efficiency. For example, ID data is often of string type and has a very high repetition rate. This type uses a storage pool method, which only records the physical address and length of the ID. Therefore, the ID data is set to String type, and compression parameters are set for the field to which the ID data belongs: use the lz4 compression algorithm, enable the vector instruction set, and enable the AVX256 instruction set; the time of time series data is often continuous, so compression parameters are set for the field to which the time series data belongs: use the lzw compression algorithm, enable the vector instruction set, and enable the NEON instruction set; simple status data such as switches often uses the change upload method to transmit data back, and the changes are relatively rare. Therefore, compression parameters are set for the field to which the status data belongs: use the delta compression algorithm, do not enable the vector instruction set; compression parameters are not set for the field to which the remark data belongs.

[0052] The second step is to use the CPU's built-in vector instruction set to optimize the compression algorithm. Go to the third step.

[0053] It should be understood that, based on the above technical solution, when the compression parameters of the data table or the compression parameters of the fields in the data table specify the enabling of a vector instruction set, the compression algorithm to be used and the vector instruction set built into the CPU are determined according to the compression parameters, and the compression algorithm is optimized using the vector instruction set built into the CPU.

[0054] It should be understood that, based on the above technical solution, using the vector instruction set built into the CPU to optimize the compression algorithm specifically includes:

[0055] If the compression parameters determine that the LZ4 compression algorithm is to be used, the CPU's built-in vector instruction set is used for optimization. The vector instruction set includes the AVX256 instruction set, which is used to improve the overall compression speed and search performance.

[0056] If the compression parameters determine the use of the lzw and zstd compression algorithms, the CPU's built-in vector instruction set is used for optimization. The vector instruction set includes the SSE instruction set and the NEON instruction set to improve the overall compression speed and hash search performance.

[0057] For example, in the user information data table described above, the AVX256 instruction set is used to optimize the LZ4 compression algorithm based on the compression parameters of the fields belonging to the ID data. The most time-consuming part of the LZ4 compression algorithm is the hash lookup of historical data, which is used to determine whether the data has been stored and obtain the data pointer. After enabling the instruction set, the LZ4 compression algorithm will use the CPU's built-in AVX256 instruction set based on the compression parameters, improving hash lookup performance by 64-128 times and overall compression speed by 32-64 times, achieving the goal of improving the read and write speed of string type fields. Based on the compression parameters of the fields belonging to the time series data, the NEON instruction set is used to optimize the LZW compression algorithm. The core of the LZW algorithm is dictionary management. Improving dictionary query performance can effectively improve compression and decompression performance. After enabling the instruction set, the CPU's built-in NEON instruction set can be used based on the compression parameters to improve data dictionary lookup performance by dozens of times and the compression speed of random data by more than ten times. Based on the compression parameters of the fields belonging to the status data, no vector instruction set is used to optimize the delta compression algorithm.

[0058] The third step is to divide the data of each field in the data table into multiple high-dimensional vectors, compress different fields using different compression algorithms, obtain compressed data and store it.

[0059] It should be understood that, based on the above technical solution, the method of dividing the data of each field in the data table into multiple high-dimensional vectors is: dividing the data stored in each field in the data table into high-dimensional vectors of a fixed length, wherein the length includes 64 bits, 128 bits, 256 bits or 512 bits.

[0060] It should be understood that, based on the above technical solution, step S3 specifically includes:

[0061] S3-1, determine whether the data type stored in the segmented field is a String type. If so, compress the data using the compression algorithm specified in the compression parameters of the field in a memory pool and address length manner; if not, execute step S3-2;

[0062] S3-2, determine whether the segmented field has a compression parameter set. If so, compress it using the compression algorithm specified in the compression parameter of the field; if not, execute step S3-3;

[0063] S3-3, determining whether the data table to which the segmented field belongs has compression parameters set, and if so, using the compression algorithm specified in the compression parameters of the data table for compression; if not, using a preset compression algorithm for compression.

[0064] For example, the data in the above-mentioned user information data table are all split into 64-bit high-dimensional vectors; the split data type is determined, first determine whether the data type is a String type, where the ID data is a String type, then use the lz4 compression algorithm optimized by the AVX256 instruction set in a memory pool and address length manner to compress it; the time series data, status data and remark data are not of String type, and the next step of judgment is performed, where the fields to which the time series data and status data belong are both set with compression parameters, then the split time series data and status data are compressed according to the compression parameters of the fields to which they belong: the split time series data is compressed using the lzw compression algorithm optimized by the NEON instruction set, and the split status data is not compressed using any delta compression algorithm optimized by a vector instruction set; the field to which the remark data belongs does not have a compression parameter, and the next step of judgment is performed, where the data table to which the remark data belongs has a compression parameter, then the split remark data is compressed using the lzw compression algorithm optimized by the NEON instruction set according to the compression parameters of the data table.

[0065] It should be understood that, based on the above technical solution, after step S3-3 is executed, the following steps are further included:

[0066] If the compressed data is not accessed within a preset time, the compressed data is written to the hard disk;

[0067] When a user makes a query, a corresponding compression algorithm is determined according to the compression parameters of the compressed data, and decompression is performed according to the compression algorithm, and the decompressed data is stored in a memory cache for use by the user.

[0068] For example, after the compression of the above-mentioned user information data table is completed, if there is no access to the data table within one minute, the compressed data table will be written to the hard disk storage; when the user queries the data, the compression algorithm used for the compressed data table contained in the query result is determined according to the compression parameters of the user information data table, and the corresponding decompression method is used to decompress it, and the decompressed data is stored in the memory cache for user use.

[0069] The present invention also discloses a system for improving data storage and query performance, the system comprising:

[0070] Parameter setting module: used for setting compression parameters when users create data tables, based on the data type stored in the data table or the data type stored in the fields in the data table. The compression parameters include: which compression algorithm to use, whether to enable the vector instruction set, and which instruction set to enable;

[0071] Compression algorithm optimization module: used to determine the compression algorithm to be used and the vector instruction set built into the CPU according to the compression parameters, and optimize the compression algorithm using the vector instruction set built into the CPU;

[0072] Data processing module: used to split the data of each field in the data table to obtain multiple high-dimensional vectors; compress the data using the compression algorithm corresponding to the data to obtain compressed data and store it; determine the corresponding compression algorithm based on the compression parameters of the compressed data, decompress it according to the compression algorithm, and store the decompressed data in the memory cache for user use.

[0073] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory and communication interface communicate with each other via the bus; the memory stores a method program for improving data storage and query performance that can be executed by the processor, and the method program for improving data storage and query performance is configured to implement a method for improving data storage and query performance as described in an embodiment of the present invention.

[0074] The present invention also discloses a computer-readable storage medium, on which is stored a method program for improving data storage and query performance. When the method program for improving data storage and query performance is executed, a method for improving data storage and query performance as described in an embodiment of the present invention is implemented.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for improving data storage and query performance, characterized in that: The method for improving data storage and query performance comprises the following steps: S1. When a user creates a data table, he sets compression parameters for the data table or fields in the data table. The compression parameters include: which compression algorithm to use, whether to enable the vector instruction set, and which instruction set to enable. The compression algorithm to use specifically includes: when the user creates the data table, according to the data type stored in the data table, or the data type stored in the fields in the data table, the type of compression algorithm to use in the compression parameters, including: ID type data is set to String type, the corresponding compression algorithm is lz4 compression algorithm, the compression algorithm corresponding to time series data is delta-of-delta compression algorithm, the compression algorithm corresponding to status data is delta compression algorithm, and the compression algorithms for high-frequency data without special patterns include lzw and zstd. S2, when the compression parameters of the data table or the compression parameters of the fields in the data table specify the enablement of a vector instruction set, determining the compression algorithm to be used and the vector instruction set built into the CPU according to the compression parameters, and optimizing the compression algorithm using the vector instruction set built into the CPU; if the compression parameters determine that the lz4 compression algorithm to be used is used, optimizing is performed using the vector instruction set built into the CPU, the vector instruction set including: the AVX256 instruction set, which is used to improve the overall compression speed and search performance; if the compression parameters determine that the lzw and zstd compression algorithms to be used are used, optimizing is performed using the vector instruction set built into the CPU, the vector instruction set including: the SSE instruction set and the NEON instruction set, which are used to improve the overall compression speed and hash search performance; S3 divides the data of each field in the data table into multiple high-dimensional vectors, compresses different fields using different compression algorithms, obtains compressed data and stores it.

2. A method for improving data storage and query performance according to claim 1, characterized in that: The step S3 specifically includes: S3-1, determine whether the data type stored in the segmented field is a String type. If so, compress the data using the compression algorithm specified in the compression parameters of the field in a memory pool and address length manner; if not, execute step S3-2; S3-2, determine whether the segmented field has a compression parameter set. If so, compress it using the compression algorithm specified in the compression parameter of the field; if not, execute step S3-3; S3-3, determining whether the data table to which the segmented field belongs has compression parameters set, and if so, using the compression algorithm specified in the compression parameters of the data table for compression; if not, using a preset compression algorithm for compression.

3. A method for improving data storage and query performance according to claim 2, characterized in that: After step S3-3 is completed, the following steps are further included: If the compressed data is not accessed within a preset time, the compressed data is written to the hard disk; When a user makes a query, a corresponding compression algorithm is determined according to the compression parameters of the compressed data, and decompression is performed according to the compression algorithm, and the decompressed data is stored in a memory cache for use by the user.

4. A system for improving data storage and query performance, characterized in that: The system comprises: Parameter setting module: used to set compression parameters when a user creates a data table, based on the data type stored in the data table or the data type stored in the fields in the data table. The compression parameters include: which compression algorithm to use, whether to enable the vector instruction set, and which instruction set to enable. The compression algorithm to use specifically includes: when a user creates a data table, based on the data type stored in the data table or the data type stored in the fields in the data table, setting the type of compression algorithm to use in the compression parameters, including: the lz4 compression algorithm for ID data set to String type, the delta-of-delta compression algorithm for time series data, the delta compression algorithm for status data, and the lzw and zstd compression algorithms for high-frequency data without special patterns. Compression algorithm optimization module: used for, when the compression parameters of the data table or the compression parameters of the fields in the data table specify the enablement of a vector instruction set, determining the compression algorithm to be used and the vector instruction set built into the CPU according to the compression parameters, and using the vector instruction set built into the CPU to optimize the compression algorithm; when the compression parameters of the data table or the compression parameters of the fields in the data table specify the enablement of a vector instruction set, determining the compression algorithm to be used and the vector instruction set built into the CPU according to the compression parameters, and using the vector instruction set built into the CPU to optimize the compression algorithm; if the lz4 compression algorithm to be used is determined in the compression parameters, using the vector instruction set built into the CPU for optimization, the vector instruction set includes: AVX256 instruction set, for improving the overall compression speed and search performance; if the lzw and zstd compression algorithms to be used are determined in the compression parameters, using the vector instruction set built into the CPU for optimization, the vector instruction set includes: SSE instruction set, NEON instruction set, for improving the overall compression speed and hash search performance; Data processing module: used to split the data of each field in the data table to obtain multiple high-dimensional vectors; compress the data using the compression algorithm corresponding to the data to obtain compressed data and store it; determine the corresponding compression algorithm based on the compression parameters of the compressed data, decompress it according to the compression algorithm, and store the decompressed data in the memory cache for user use.

5. An electronic device, characterized in that: It includes at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores a method program for improving data storage and query performance that can be executed by the processor, and the method program for improving data storage and query performance is configured to implement a method for improving data storage and query performance as claimed in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The storage medium stores a method program for improving data storage and query performance, which, when executed, implements a method for improving data storage and query performance as claimed in any one of claims 1 to 3.

Images