A database system, a data processing method and an electronic device

Abstract

The application provides a database system, a data processing method and an electronic device, the database system comprises a storage module and a processing module, the processing module is connected with the storage module, the processing module is used for selecting a corresponding compression algorithm to segmentally compress each column of data according to the access frequency of each column of data of a data table, wherein the corresponding relationship between the access frequency and the compression algorithm is preset, and the storage module is used for segmentally compressing and saving each column of data of the data table. The method is applied to the database system. The database system, the data processing method and the electronic device provided by the embodiment of the application can realize the storage of big data on a single electronic device, and reduce the cost of big data storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a database system, a data processing method and an electronic device. BACKGROUND

[0002] With the rapid development of information technology, Internet technology and Internet of Things technology, the output of data in various industries is increasing exponentially, and the scale of data is becoming increasingly large.

[0003] The characteristics of large-scale and high-dimension of data pose high technical challenges to traditional database systems. For example, for traditional relational databases such as MySQL, Oracle, SQL Server, etc., they are typical row table designs. These databases organize and store data in units of data behaviors, and improve query speed through index tables. The advantage of this database design is that it is convenient for data insertion, deletion and modification operations under high real-time requirements, and the limitation is that the column number of the data table is limited due to the limitation of the scale of each row of data, which is only suitable for low-dimensional data. In order to solve the defect that the traditional relational database cannot effectively store high-dimensional big data, the NoSQL database system based on distributed storage list design appears, such as BigTable, HBase, etc. These new database systems all need to be built on large-scale distributed storage clusters, and often need dozens or even thousands of computing nodes to achieve ideal results. The establishment of such a large-scale computing cluster requires extremely high purchase and maintenance costs, and it is difficult to be popularized and applied in the daily use scenarios faced by numerous small and medium-sized enterprises and scientific research institutions. In addition, due to the use of distributed storage, the query often has high latency, which is difficult to adapt to application scenarios with high real-time requirements. SUMMARY

[0004] In view of the problems in the prior art, the embodiments of the present application provide a database system, a data processing method and an electronic device, which can at least partially solve the problems in the prior art.

[0005] In a first aspect, the present application provides a database system, comprising a storage module and a processing module, wherein:

[0006] The processing module is connected to the storage module;

[0007] The processing module is configured to select a corresponding compression algorithm for each column of data according to the access frequency of each column of data of the data table, and perform segmented compression on each column of data; wherein the correspondence between the access frequency and the compression algorithm is pre-set;

[0008] The storage module is configured to save each column of data of the data table after segmented compression.

[0009] Further, the storage module comprises a first-level storage unit, a second-level storage unit and a third-level storage unit, and the processing module is connected with the first-level storage unit, the second-level storage unit and the third-level storage unit respectively, wherein:

[0010] The first-level storage unit is used for storing a data table by using a redundancy mechanism.

[0011] The second-level storage unit is used for caching data stored by the first-level storage unit.

[0012] The third-level storage unit is used for caching data stored by the second-level storage unit.

[0013] Further, the database system provided by the embodiment of the present application further comprises a communication module connected with the processing module.

[0014] In the second aspect, the present application provides a data processing method of the database system according to any one of the above embodiments, comprising:

[0015] The processing module acquires to-be-stored data, and the to-be-stored data comprises a data table.

[0016] The processing module selects a corresponding compression algorithm according to the access frequency of each column data of the data table to perform segmented compression on each column data, wherein the corresponding relationship between the access frequency and the compression algorithm is preset.

[0017] The processing module stores each column data after segmented compression into the storage module.

[0018] Further, the selecting a corresponding compression algorithm according to the access frequency of each column data of the data table to perform segmented compression on each column data comprises:

[0019] The access frequency corresponding to each column data of the data table is acquired.

[0020] A corresponding compression algorithm is obtained according to the access frequency corresponding to each column data of the data table.

[0021] Each column data of the data table is segmented and compressed according to the compression algorithm corresponding to each column data of the data table.

[0022] Further, the segmented and compressed according to the compression algorithm corresponding to each column data of the data table comprises:

[0023] The segmented data length corresponding to the column data is obtained according to the data length of the column data and the number of processor threads.

[0024] If it is judged that the segment data length corresponding to the column data is less than the first preset value, the column data is segmented based on the first preset value to obtain each segment data corresponding to the column data and the segment number;

[0025] According to the compression algorithm corresponding to the column data, the segment data corresponding to the column data is compressed in parallel by the thread number.

[0026] Further, the segment compression of each column data of the data table according to the compression algorithm corresponding to the column data includes:

[0027] According to the data length of the column data and the processor thread number, the segment data length corresponding to the column data is obtained.

[0028] If it is judged that the segment data length corresponding to the column data is greater than or equal to the first preset value and less than the second preset value, the column data is segmented based on the segment data length to obtain each segment data corresponding to the column data.

[0029] According to the compression algorithm corresponding to the column data, the segment data corresponding to the column data is compressed in parallel by the thread number.

[0030] Further, the segment compression of each column data of the data table according to the compression algorithm corresponding to the column data includes:

[0031] According to the data length of the column data and the processor thread number, the segment data length corresponding to the column data is obtained.

[0032] If it is judged that the segment data length corresponding to the column data is greater than or equal to the second preset value, the column data is segmented based on the second preset value to obtain each segment data corresponding to the column data.

[0033] According to the compression algorithm corresponding to the column data, the segment data corresponding to the column data is compressed in parallel by the thread number.

[0034] Further, the data processing method provided by the embodiment of the present application further includes:

[0035] If the processing module judges that the data type of the column data in the data table is a preset type, the column data is de-duplicated to obtain de-duplicated column data.

[0036] The processing module hashes the de-duplicated column data according to the string length to obtain the hash value corresponding to each string data in the de-duplicated column data.

[0037] The processing module groups each string data in the deduplicated column data according to a hash value corresponding to the string data, and obtains a plurality of string groups; each string group includes string data having the same string length and the same hash value.

[0038] The processing module sorts the string data included in each string group, and obtains each sorted string group.

[0039] The processing module generates an index table of the column data according to an index value corresponding to the deduplicated column data, each sorted string group, and a hash value corresponding to each sorted string group; the index value corresponding to the deduplicated column data is obtained in advance.

[0040] Further, the data processing method provided by the embodiment of the present application further includes:

[0041] The processing module replaces the corresponding column data according to the index value corresponding to the deduplicated column data, and obtains query column data corresponding to the column data.

[0042] The processing module segments and compresses the query column data corresponding to the column data according to a compression algorithm corresponding to the column data, and obtains each segment of compressed data of the query column data corresponding to the column data. In a third aspect, the present application further provides a data processing method of a database system based on any of the above embodiments, including:

[0043] The processing module obtains a query request from a query input queue, the query request including a data table identifier, at least one field, and a query keyword;

[0044] The processing module queries the storage module according to the data table identifier and the at least one field, and obtains each segment of compressed data corresponding to each field in the at least one field; each field corresponds to a column data of the data table;

[0045] The processing module decompresses each segment of compressed data of the column data corresponding to each field in the at least one field according to a compression algorithm corresponding to each field, and obtains decompressed data corresponding to each field in the at least one field;

[0046] The processing module obtains a query result according to the query keyword and the decompressed data corresponding to each field in the at least one field.

[0047] Further, the decompressing each segment of compressed data of the column data corresponding to each field in the at least one field according to the compression algorithm corresponding to each field, and obtaining the decompressed data corresponding to each field in the at least one field includes:

[0048] If it is judged that the number of segments of the column data corresponding to the field is greater than the number of processor threads, then the column data corresponding to the field is decompressed in parallel using the number of processor threads according to the compression algorithm corresponding to the column data corresponding to the field;

[0049] If it is judged that the number of segments of the column data corresponding to the field is less than or equal to the number of processor threads, then the column data corresponding to the field is decompressed in parallel using the number of segments of the column data corresponding to the field according to the compression algorithm corresponding to the column data corresponding to the field.

[0050] Further, the data processing method provided by the embodiment of the present application further comprises:

[0051] If the processing module judges that the data type corresponding to the field is a preset type, then the index table of the column data corresponding to the field is obtained according to the field;

[0052] The query keyword is hashed according to the string length to obtain the hash value corresponding to the query keyword;

[0053] The processing module obtains the index value corresponding to the query keyword according to the hash value corresponding to the query keyword and the index table of the column data corresponding to the field;

[0054] The processing module obtains the query result corresponding to the query keyword according to the index value corresponding to the query keyword and the query column data corresponding to the column data corresponding to the field.

[0055] Further, before the query result corresponding to the query keyword is obtained according to the index value corresponding to the query keyword and the query column data corresponding to the column data corresponding to the field, the method further comprises:

[0056] The processing module decompresses each segment of compressed data of the query column data corresponding to the column data corresponding to the field according to the compression algorithm corresponding to the field to obtain the query column data corresponding to the column data corresponding to the field.

[0057] Further, the data processing method provided by the embodiment of the present application further comprises:

[0058] The processing module obtains the query request sent by each client;

[0059] The processing module puts each query request into a query input queue according to the order of receiving time of each query request.

[0060] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method according to any one of the above embodiments when executing the program.

[0061] In a fifth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the data processing method according to any one of the above embodiments.

[0062] The database system, the data processing method and the electronic device provided by the embodiments of the present application comprise a storage module and a processing module, the processing module is connected with the storage module, the processing module is used for selecting a corresponding compression algorithm to segmentally compress each column of data according to the access frequency of each column of data of a data table, and the storage module is used for segmentally compressing and saving each column of data of the data table, so that the storage of big data can be realized on a single electronic device, and the cost of big data storage is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort. In the drawings:

[0064] Figure 1 Fig. 1 is a structural schematic diagram of a database system provided by a first embodiment of the present application.

[0065] Figure 2 Fig. 2 is a structural schematic diagram of a database system provided by a second embodiment of the present application.

[0066] Figure 3 Fig. 3 is a structural schematic diagram of a database system provided by a third embodiment of the present application.

[0067] Figure 4 Fig. 4 is a flow schematic diagram of a data processing method provided by a fourth embodiment of the present application.

[0068] Figure 5 Fig. 5 is a flow schematic diagram of a data processing method provided by a fifth embodiment of the present application.

[0069] Figure 6 Fig. 6 is a flow schematic diagram of a data processing method provided by a sixth embodiment of the present application.

[0070] Figure 7 Fig. 7 is a flow schematic diagram of a data processing method provided by a seventh embodiment of the present application.

[0071] Figure 8 is a flowchart of a data processing method provided by an eighth embodiment of the present application.

[0072] Figure 9 is a flowchart of a data processing method provided by a ninth embodiment of the present application.

[0073] Figure 10 is a flowchart of a data processing method provided by a tenth embodiment of the present application.

[0074] Figure 11 is a flowchart of a data processing method provided by an eleventh embodiment of the present application.

[0075] Figure 12 is a flowchart of a data processing method provided by a twelfth embodiment of the present application.

[0076] Figure 13 is a flowchart of a data processing method provided by a thirteenth embodiment of the present application.

[0077] Figure 14 is a flowchart of a data processing method provided by a fourteenth embodiment of the present application.

[0078] Figure 15 is a schematic diagram of an entity structure of an electronic device provided by a fifteenth embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application are further described in detail below with reference to the drawings. Herein, the schematic embodiments of the present application and the descriptions thereof are used to explain the present application, but are not used as the limitation of the present application. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other at will.

[0080] The traditional relational database system adopts a row storage mode for data organization, each row is divided into a plurality of columns (fields), the number of which is limited, and the total amount of data that can be stored in each row is also limited. This type of database system can easily perform real-time modification operations such as insertion and deletion for small-scale (fewer rows) and low-dimensional (fewer columns) data tables, but it is difficult to handle data tables with millions or even hundreds of millions of rows and tens of thousands or even millions of columns. On the one hand, due to the limited amount of data that can be stored in each row, it is difficult to directly store high-dimensional data. Although high-dimensional data can be stored by dividing the data into several data tables according to the data dimension, this approach will cause the problem of needing to operate multiple data tables for a single query, greatly reducing the query efficiency. On the other hand, when the data size is large and the number of rows in the data table is large, the query time cost of the relational database is too high, especially for queries of variable-length strings, which are too slow to complete the regular query of large-scale databases within the time tolerance of the user.

[0081] In view of the above problems existing in the traditional relational database, the present application proposes a data organization method of the database, which is to store the data table by column, that is, to store the data table by column. Each column of data of the large-scale high-dimensional data is segmented and compressed according to the number of processor threads to realize high-density and high-reliability storage of high-dimensional data on an electronic device. And through hierarchical storage to meet the demand for fast access to data. The electronic device includes but is not limited to desktop computers, servers, notebook computers, etc.

[0082] Figure 1 is a structural schematic diagram of the database system provided by the first embodiment of the present application, as Figure 1 shown, the database system provided by the embodiment of the present application comprises a storage module 1 and a processing module 2, wherein:

[0083] The processing module 1 is connected to the storage module 2;

[0084] The processing module 1 is configured to select a corresponding compression algorithm to segment and compress each column of data according to the access frequency of each column of data of the data table; wherein the corresponding relationship between the access frequency and the compression algorithm is pre-set;

[0085] The storage module 2 is configured to segment and compress each column of data of the data table.

[0086] Specifically, the initial value of the access frequency of each column data in the data table is preset, and can be updated according to the actual access situation. For column data with high access frequency, a compression algorithm with relatively low compression rate but short compression and decompression time can be used, such as fast compression algorithms such as lz4, zstd, etc. For column data with low access frequency, a compression algorithm with relatively high compression rate but long compression and decompression time can be used, such as compression algorithms such as zlib, bzip2, lzma, etc. Generally, most column data has low frequency of use, and only a small number of column data is frequently used. Therefore, by using the above-mentioned fusion of multiple different compression algorithms to compress and store column data with different access frequencies, not only a high average compression rate is achieved to save storage space, but also the use demand of fast data access is met. The specific segmented compression processing process of the column data by the processing module 1 is described below, and will not be described here. The correspondence between the access frequency and the compression algorithm is preset, for example, when the access frequency is greater than the frequency threshold, it means that the access frequency of the column data is high, and fast compression algorithms such as lz4, zstd can be used; when the access frequency is less than or equal to the frequency threshold, it means that the access frequency of the column data is low, and compression algorithms such as zlib, bzip2, lzma can be used.

[0087] The database system provided by the embodiment of the present application comprises a storage module and a processing module, the processing module is connected with the storage module, the processing module is used for selecting a corresponding compression algorithm to segmentally compress each column data of a data table according to the access frequency of each column data, and the storage module is used for segmentally compressing and saving each column data of the data table. The database system can realize the storage of big data on a single electronic device, and reduces the cost of big data storage. In addition, different compression algorithms can be used to compress data with different access frequencies, so as to improve the access efficiency of the data while ensuring the data storage.

[0088] Figure 2 is a structural schematic diagram of the database system provided by the second embodiment of the present application, as Figure 2 shown, on the basis of the above-mentioned embodiments, further, the storage module 1 comprises a first-level storage unit 11, a second-level storage unit 12 and a third-level storage unit 13, and the processing module 2 is connected with the first-level storage unit 11, the second-level storage unit 12 and the third-level storage unit 13 respectively, wherein:

[0089] The first-level storage unit 11 is used for storing the data table by using a redundancy mechanism;

[0090] The second-level storage unit 12 is used for caching the data stored by the first-level storage unit;

[0091] The third-level storage unit 13 is used for caching the data stored by the second-level storage unit.

[0092] Specifically, the first-level storage unit 11 can adopt a mechanical hard disk to store all the data tables. A plurality of large-capacity mechanical hard disks can be used to build a disk array, and through a flexible disk redundancy mechanism, data loss is prevented and normal read and write operations are ensured in the case of simultaneous failure of multiple disks. The level of disk redundancy is selected according to the requirement of data reliability. The level of disk redundancy used can include RAID10, RAID50, RAID60 or equivalent levels. When the number of mechanical hard disks of the electronic device is small, such as 8, RAID10 mode can be used, that is, two hard disks are grouped into RAID1 (mirroring), and then each mirrored disk group is grouped into RAID0. When the number of mechanical hard disks of the electronic device is large, such as 60, RAID60 mode can be used, for example, 10 hard disks are grouped into RAID6 (two redundant), and then the obtained disk group is grouped into RAID0, so that when two hard disks in each group fail, data loss is prevented and data access can proceed normally. RAID50 mode can also be used, for example, 5 hard disks are grouped into RAID5 (one redundant), and then the obtained disk group is grouped into RAID0, so that when one hard disk in each group fails, data loss is prevented.

[0093] The second-level storage unit 12 can adopt a solid state disk to cache column data with high access frequency and / or recently accessed column data in the data table stored by the first-level storage unit 11. The solid state disk adopts a high-speed non-volatile memory host controller interface specification (NVME), so that the data not only has extremely high sequential access speed, but also has extremely high random access performance. For example, a super-speed NVME solid state disk supporting PCIE-3.0x8 or higher bandwidth standard is used. The column data with high access frequency in the data table is set according to actual needs, and the embodiments of the present application are not limited.

[0094] For example, when the access frequency of the column data of the data table is greater than the first cache threshold, it is determined that the access frequency of the column data is high, and the column data can be cached to the second-level storage unit 12. The recently accessed column data is set according to actual needs, and the embodiments of the present application are not limited. For example, the column data accessed within a week before the current time is taken as the recently accessed column data, which can be cached to the second-level storage unit 12.

[0095] The third-level storage unit 13 can adopt the memory of the electronic device to cache the data stored by the second-level storage unit, so that the access to the data can be performed at the speed of the memory, so as to improve the access efficiency of the data. The capacity of the third-level storage unit 13 is set according to actual needs, and the embodiments of the present application are not limited. For example, a memory greater than or equal to 512GB is used.

[0096] The three-level storage of the data by the first-level storage unit 11, the second-level storage unit 12 and the third-level storage unit 13 improves the data access speed under the premise of ensuring the reliability of data access. Compared with a mechanical hard disk, the access speed can be improved by more than 30 times, and the random access performance can be improved by more than 1,000 times.

[0097] Figure 3 is a structural schematic diagram of a database system provided by a third embodiment of the application, as Figure 3 shown, on the basis of the above embodiments, further, the database system provided by the embodiment of the application further comprises a communication module 3, the communication module 3 is connected with the processing module 1.

[0098] The communication module 3 can realize the communication of the database system with the client and other devices.

[0099] The communication module 3 can be realized based on the Thrift framework. The communication module 3 can provide a multi-language cross-platform remote calling interface, realize the remote access and cross-platform calling of the database system. According to the standard SQL structured query statement specification, the remote calling interfaces are encapsulated, so that the interfaces in the database system can complete the semantic analysis of the standard relational database statements and perform related query tasks. In addition, in order to ensure the communication with the client, the interface and the transmission format of the data remote access and query result transmission are standardized.

[0100] In view of the flexible and variable requirements in the use process of the database system, the communication module of the database system provided by the embodiment of the application has the characteristics of remote access, cross-platform calling, multi-language programming and structured query, so as to realize the separation of the database system server and the client, and meet the purpose of multiple use scenarios. First, the database system of the application uses a remote process calling mechanism, and uses a general communication protocol, calling mechanism and programming specification in the client to access the database running on the server. Secondly, the database system of the application realizes cross-platform support based on a mature remote process calling standard library, so that the client can run on multiple operating systems. In addition, the use of the remote process calling standard library also enables the client to be implemented in multiple programming languages, thereby greatly reducing the threshold of the database use. Finally, the access of the database system of the application follows the SQL structured query language design specification, so that the query of the database can be written in the structured query statement familiar to the developer, thereby enabling the user to obtain a similar experience of writing a traditional relational database query statement, and reducing the difficulty of using the database system.

[0101] Figure 4 is a flowchart of a data processing method provided by a fourth embodiment of the application, as Figure 4As shown, the data processing method provided by the embodiment of the application can be applied to the data processing method of the database system in any of the above embodiments, and includes the following steps.

[0102] S401, the processing module acquires to-be-stored data, the to-be-stored data including a data table;

[0103] Specifically, the processing module can acquire to-be-stored data, the to-be-stored data including a data table, and the data table including column data corresponding to each field. In the embodiment of the application, data represents a column-based data organization form, and the data table is stored in units of columns, which is just opposite to the row-based storage method adopted by a traditional relational database. The advantages of column-based storage include: (1) when performing a query, only column data related to a query statement needs to be processed, and the entire data row does not need to be accessed, thereby greatly reducing the number of disk accesses and improving data access efficiency; (2) since each column of data is of the same type, the data processing efficiency is high when performing a query on column data.

[0104] For example, the processing module can receive to-be-stored data sent remotely through the communication module. The processing module can also obtain to-be-stored data from a mobile storage device through a peripheral interface locally.

[0105] S402, the processing module selects a corresponding compression algorithm to segmentally compress each column of data according to the access frequency of each column of data of the data table; wherein the corresponding relationship between the access frequency and the compression algorithm is preset.

[0106] Specifically, for each column of data of the data table, the processing module acquires the access frequency of each column of data, then obtains a corresponding compression algorithm according to the access frequency of each column of data, and then segmentally compresses each column of data according to the compression algorithm to obtain compressed data of each column of data. The corresponding relationship between the access frequency and the compression algorithm is preset, and the compression algorithm is selected according to actual needs, which is not limited in the embodiment of the application.

[0107] S403, the processing module stores each column of segmentally compressed data to the storage module.

[0108] Specifically, the processing module stores each column of segmentally compressed data to the storage module for subsequent access and query.

[0109] The data processing method provided in this embodiment of the invention involves a processing module acquiring data to be stored, which includes a data table. The processing module selects a corresponding compression algorithm based on the access frequency of each column of data in the data table to perform segmented compression on each column. The processing module then stores the segmented compressed data into a storage module. Because it uses a column-based table data organization and segmented compression, it enables the storage of large amounts of data on a single electronic device, reducing the cost of big data storage. Furthermore, the column-based table data organization improves the access efficiency of high-dimensional data, making it suitable for applications with high real-time requirements.

[0110] Figure 5 This is a flowchart illustrating the data processing method provided in the fifth embodiment of the present invention, as shown below. Figure 5 As shown, based on the above embodiments, further, the step of selecting a corresponding compression algorithm to perform segmented compression of each column of data according to the access frequency of each column of data in the data table includes:

[0111] S501. Obtain the access frequency corresponding to each column of data in the data table;

[0112] Specifically, each column of data in the data table has a corresponding access frequency, and the processing module can obtain the access frequency corresponding to each column of data. The initial value of the access frequency is set according to actual needs, and this embodiment of the invention does not impose any limitations.

[0113] For example, each column of data corresponds to a field, and the access frequency of each column is stored in relation to that field. The access frequency corresponding to a field in a column is retrieved, and this retrieved frequency is used as the access frequency for that column.

[0114] For example, you can set the initial value of the access frequency for a column of data to 0, and increment the access count by 1 for each subsequent access to the column data. Calculate the quotient of the access count of the column data within a preset time period and the preset time period itself, and update the access frequency for the column data accordingly.

[0115] If, based on experience, the column data in the data table is accessed frequently, a large initial value can be set for the access frequency of the corresponding column data. This will allow the column data to be compressed using a compression algorithm with a relatively low compression ratio but shorter compression and decompression times during subsequent segmented compression, thereby improving the access efficiency of the column data.

[0116] S502. Obtain the corresponding compression algorithm based on the access frequency of each column of data in the data table;

[0117] Specifically, the processing module can obtain the compression algorithm corresponding to each column of data based on the access frequency of each column of data in the data table.

[0118] For example, a frequency threshold is set. If the access frequency corresponding to the column data is greater than the frequency threshold, the corresponding compression algorithm is the lz4 or zstd compression algorithm. If the access frequency corresponding to the column data is less than or equal to the frequency threshold, the corresponding compression algorithm is the zlib, bzip2 or lzm compression algorithm. The frequency threshold is set according to actual needs, and the embodiments of the present application are not limited.

[0119] S503, segmentally compressing each column data of the data table according to the compression algorithm corresponding to each column data of the data table.

[0120] Specifically, for each column data of the data table, the processing module can segmentally compress the column data according to the compression algorithm corresponding to the column data after obtaining the compression algorithm. Segmental compression means that the column data is divided into several segments for compression. Each segment of the column data can be compressed in parallel to improve the data compression efficiency.

[0121] It can be understood that for the data table stored by the storage module, the access frequency corresponding to the column data in the data table can be updated. If the compression algorithm obtained based on the updated access frequency corresponding to the column data changes, the column data can be re-segmentally compressed according to the re-obtained compression algorithm. For example, the access frequency corresponding to the column data is greater than the frequency threshold before updating, which indicates that the column data has a high access frequency. After updating, the access frequency corresponding to the column data is less than the frequency threshold, which indicates that the column data becomes data with a low access frequency. Therefore, the column data can be re-segmentally compressed according to the compression algorithm obtained based on the updated access frequency.

[0122] Figure 6 is a flowchart of a data processing method provided by the sixth embodiment of the present application, as shown in Figure 6 On the basis of the above embodiments, further, the segmentally compressing each column data of the data table according to the compression algorithm corresponding to each column data of the data table includes:

[0123] S601, obtaining the segmental data length corresponding to the column data according to the data length of the column data and the number of processor threads;

[0124] Specifically, the processing module can obtain the segmental data length corresponding to the column data according to the data length of the column data and the number of processor threads. The number of processor threads is known.

[0125] For example, the database system provided by the embodiment of the present application is implemented on a dual-server with 256 cores and 512 threads, and the number of processor threads is 512. If a column of data in a data table has 512,000 rows of data, the data length of the column of data is 512,000, and the corresponding segmented data length is 512,000 / 512=1000.

[0126] S602, if it is judged that the segmented data length corresponding to the column of data is less than the first preset value, the column of data is segmented based on the first preset value to obtain each segment of data corresponding to the column of data and the number of segments;

[0127] Specifically, the processing module compares the segmented data length corresponding to the column of data with the first preset value. If the segmented data length corresponding to the column of data is less than the first preset value, the column of data is segmented based on the first preset value, and each segment of data of the column of data and the number of segments can be obtained. The first preset value is set according to actual experience, and the embodiment of the present application does not limit it. The first preset value is set considering that using multi-thread parallel processing of data does not cause resource waste. When the segmented data length is less than the first preset value, the column of data is segmented according to the segmented data length, and multi-thread parallel processing is used, which will cause resource waste.

[0128] For example, the data length of the column of data is 512,000, and the first preset value is 2000. The column of data is segmented based on the first preset value 2000, and the number of segments that can be obtained is 512,000 / 2000=256. The column of data can be evenly divided into 256 segments. The first row to the 256th row of data of the column of data is the first segment of data, the 257th row to the 512th row of data is the second segment of data, the 513th row to the 768th row of data is the third segment of data, and so on. There are 256 segments of data.

[0129] S603, according to the compression algorithm corresponding to the column of data, a number of threads corresponding to the number of segments are used to parallel compress each segment of data corresponding to the column of data.

[0130] Specifically, since the number of segments is less than the number of processor threads, the processing module allocates a thread for each segment of data of the column of data, and parallel compresses each segment of data based on the compression algorithm corresponding to the column of data to improve the compression efficiency of the data.

[0131] Figure 7 is a flowchart of the data processing method provided by the seventh embodiment of the present application, as shown in Figure 7 According to the data table, each column of data is segmented and compressed according to the compression algorithm corresponding to each column of data.

[0132] S701, obtaining a segment data length corresponding to the column data according to a data length of the column data and a number of processor threads;

[0133] Specifically, the processing module can obtain the segment data length corresponding to the column data according to the data length of the column data and the number of processor threads. The number of processor threads is known.

[0134] S702, if it is judged that the segment data length corresponding to the column data is greater than or equal to a first preset value and less than a second preset value, segmenting the column data based on the segment data length to obtain each segment data corresponding to the column data;

[0135] Specifically, the processing module compares the segment data length corresponding to the column data with the first preset value and the second preset value respectively. If the segment data length corresponding to the column data is greater than or equal to the first preset value and less than the second preset value, the column data is segmented based on the segment data length corresponding to the column data, and each segment data of the column data can be obtained. The second preset value is set according to actual experience, and the embodiment of the present application is not limited. The second preset value is set considering that using multi-thread parallel processing of data does not cause system crash due to insufficient computer resources. When the segment data length is greater than or equal to the second preset value, the column data is segmented according to the segment data length, and multi-thread parallel processing is used, which is easy to cause system crash due to insufficient resources.

[0136] S703, compressing each segment data corresponding to the column data in parallel using a number of threads of the processor thread according to a compression algorithm corresponding to the column data.

[0137] Specifically, the processing module allocates a thread to each segment data of the column data, uses a number of threads of the processor thread, and performs parallel compression processing on each segment data based on the compression algorithm corresponding to the column data, so as to improve the compression efficiency of the data.

[0138] Figure 8 is a flowchart of a data processing method provided by the eighth embodiment of the present application, as shown in Figure 8 On the basis of the above embodiments, further, the segmenting and compressing each column data of the data table according to the compression algorithm corresponding to the column data includes:

[0139] S801, obtaining a segment data length corresponding to the column data according to a data length of the column data and a number of processor threads;

[0140] Specifically, the processing module can obtain the segment data length corresponding to the column data according to the data length of the column data and the number of processor threads.

[0141] S802, if it is judged that the segment data length corresponding to the column data is greater than or equal to the second preset value, the column data is segmented based on the second preset value to obtain each segment data corresponding to the column data.

[0142] Specifically, the processing module compares the segment data length corresponding to the column data with the second preset value. If the segment data length corresponding to the column data is greater than or equal to the second preset value, the column data is segmented based on the second preset value, and each segment data of the column data can be obtained.

[0143] S803, according to the compression algorithm corresponding to the column data, the thread parallelism of the number of processor threads is used to compress each segment data corresponding to the column data.

[0144] Specifically, the processing module allocates a thread for each segment data of the column data, uses the thread parallelism of the number of processor threads, and performs parallel compression processing on each segment data based on the compression algorithm corresponding to the column data, so as to improve the compression efficiency of the data.

[0145] Figure 9 is a flowchart of the data processing method provided by the ninth embodiment of the present application, as shown in Figure 9 On the basis of the above embodiments, further, the data processing method provided by the embodiments of the present application further comprises:

[0146] S901, if the processing module judges that the data type corresponding to the column data in the data table is a preset type, the column data is de-duplicated to obtain de-duplicated column data.

[0147] Specifically, the processing module obtains the data type corresponding to the column data in the data table. If the data type corresponding to the column data is a preset type, the processing module compares each row data included in the column data with each other, retains only one for the same data, removes the repeated data, and obtains the de-duplicated column data. The preset type is preset, which is a character type and other non-fixed-length data types. The present embodiment is not limited according to actual needs.

[0148] S902, the processing module performs hash on the de-duplicated column data according to the string length to obtain the hash value corresponding to each string data in the de-duplicated column data.

[0149] Specifically, each row of data of the deduplicated column data can be regarded as a string, and each row of data of the deduplicated column data is subjected to hash hashing according to the string length, so that a hash value corresponding to each string data in the deduplicated column data is obtained.

[0150] S903, the processing module groups each string data in the deduplicated column data according to the hash value corresponding to the string data, to obtain a plurality of string groups; wherein each string group includes string data having the same string length and corresponding to the same hash value.

[0151] Specifically, since the strings having the same length are equal in the hash value obtained after being subjected to hash hashing, the string data can be grouped according to the hash value corresponding to each string data in the deduplicated column data, and the string data having the same hash value is grouped into a group to form a string group, so that a plurality of string groups can be obtained.

[0152] S904, the processing module sorts the string data included in each string group to obtain each sorted string group.

[0153] Specifically, the processing module sorts the string data included in each string group to obtain each sorted string group. The sorting of the string data included in the string group is to arrange the string data in order, so as to improve the efficiency in subsequent query.

[0154] For example, the string data can be sorted according to a preset rule, and the preset rule is set according to actual needs, which is not limited in the embodiment of the application. For example, the preset rule is to sort the string data according to the dictionary order.

[0155] S905, the processing module generates an index table of the column data according to the index value corresponding to the deduplicated column data, the sorted string group and the hash value corresponding to each sorted string group; wherein the index value corresponding to the deduplicated column data is obtained in advance.

[0156] Specifically, the processing module generates an index table of the column data according to the index value corresponding to the deduplicated column data, the sorted string group and the hash value corresponding to each sorted string group. In the index table of the column data, each sorted string group has a unique corresponding hash value, and each string data of the deduplicated column data has a unique corresponding index value. Wherein the index value corresponding to the deduplicated column data is obtained in advance, and each string data of the deduplicated column data uniquely corresponds to an index value. The index value corresponding to the deduplicated column data is set according to actual needs, which is not limited in the embodiment of the application.

[0157] Figure 10 is a flowchart of a data processing method provided by the tenth embodiment of the application, asFigure 10 Further, based on the above embodiments, the data processing method provided by the embodiments of the present application further comprises:

[0158] S1001, the processing module replaces the corresponding column data according to the index value corresponding to the deduplicated column data, to obtain query column data corresponding to the column data;

[0159] Specifically, the processing module replaces each string data in the original column data corresponding to the deduplicated column data with an index value corresponding to each string data, to obtain query column data corresponding to the column data.

[0160] S1002, the processing module segments and compresses the query column data corresponding to the column data according to a compression algorithm corresponding to the column data, to obtain each segment of compressed data of the query column data corresponding to the column data.

[0161] Specifically, the query column data corresponding to the column data can be segmented and compressed according to the compression algorithm corresponding to the column data, to obtain each segment of compressed data of the query column data corresponding to the column data. The specific implementation process of this step is similar to that of step S503, and thus will not be described here.

[0162] Figure 11 is a flowchart of the data processing method provided by the eleventh embodiment of the present application, as Figure 11 The data processing method provided by the embodiments of the present application can be applied to the data processing method of the database system described in any of the above embodiments, and comprises:

[0163] S1101, the processing module obtains a query request from a query input queue, the query request comprising a data table identifier, at least one field, and a query keyword;

[0164] Specifically, the processing module can obtain a query request from a query input queue, the query request comprising a data table identifier, at least one field, and a query keyword. The data table identifier corresponds to a data table one-to-one, the field is a field included in the data table, each segment of the field corresponds to a column data, and the query keyword is set according to actual needs, which is not limited by the embodiments of the present application. The query input queue is obtained in advance.

[0165] S1102, the processing module queries and obtains each segment of compressed data corresponding to each field of the at least one field from a storage module according to the data table identifier and the at least one field, wherein each field corresponds to a column data of the data table;

[0166] Specifically, the processing module queries the corresponding data table from the data table stored in the storage module according to the data table identifier, and obtains the column data corresponding to each field from the queried data table according to each field in the at least one field, wherein the column data corresponding to each field is segmented compressed data.

[0167] S1103, decompressing each segment of compressed data of the column data corresponding to each field according to the compression algorithm corresponding to each field in the at least one field to obtain decompressed data corresponding to each field in the at least one field;

[0168] Specifically, the compression algorithm corresponding to the column data corresponding to each field is taken as the compression algorithm corresponding to each field. The processing module can decompress each segment of compressed data of the column data corresponding to each field according to the compression algorithm corresponding to each field in the at least one field, and can obtain the decompressed data corresponding to each field in the at least one field, i.e., the data of each segment of field corresponding column data before segmentation compression.

[0169] S1104, the processing module obtains a query result according to the query keyword and the decompressed data corresponding to each field in the at least one field.

[0170] Specifically, the processing module performs data query on the decompressed data corresponding to each field in the at least one field according to the query keyword to obtain a query result.

[0171] Each segment of compressed data can be assigned a thread corresponding to the decompressed data thereof, and the query keyword is queried to realize multi-thread parallel processing of the query task. In order to solve the synchronization problem of thread concurrent operation, the query task corresponding to the query request and the merging thread of the query result are connected through an output pipe, the pipe is designed as a queue, the input end of which is connected with each query thread, and the output end is connected with only the query result merging thread, which is the thread corresponding to each sub-query sub-task of the query task corresponding to the query request. The data in the pipe is protected by using thread synchronization mechanisms such as semaphores and condition numbers. Once there is a query result output from the query thread, the query result merging thread can immediately and automatically execute the merging task, and finally the results generated by multiple query threads are merged into the total query result.

[0172] Figure 12 is a flowchart of a data processing method provided by the twelfth embodiment of the present application, as shown in Figure 12 On the basis of the above embodiments, further, the decompressing each segment of compressed data of the column data corresponding to each field in the at least one field according to the compression algorithm corresponding to each field in the at least one field to obtain the decompressed data corresponding to each field in the at least one field comprises:

[0173] S1201, if it is judged that the number of segments of the column data corresponding to the field is greater than the number of processor threads, then the column data corresponding to the field is decompressed in parallel using the number of processor threads according to the compression algorithm corresponding to the column data corresponding to the field;

[0174] Specifically, the processing module compares the number of segments of the column data corresponding to the field with the number of processor threads. If the number of segments of the column data corresponding to the field is greater than the number of processor threads, then the processing module allocates the segment data to be decompressed to each of the threads of the number of processor threads, and performs parallel decompression processing on each segment of data according to the compression algorithm corresponding to the column data corresponding to the field. After an idle process appears, the idle process is allocated the segment data to be processed until the column data corresponding to the field is decompressed.

[0175] S1202, if it is judged that the number of segments of the column data corresponding to the field is less than or equal to the number of processor threads, then the column data corresponding to the field is decompressed in parallel using the number of segments of the column data corresponding to the field according to the compression algorithm corresponding to the column data corresponding to the field.

[0176] Specifically, the processing module compares the number of segments of the column data corresponding to the field with the number of processor threads. If the number of segments of the column data corresponding to the field is less than or equal to the number of processor threads, then the processing module allocates a thread to each segment of the column data corresponding to the field, and performs parallel decompression processing on each segment of data according to the compression algorithm corresponding to the column data corresponding to the field.

[0177] Figure 13 is a flowchart of a data processing method provided by the thirteenth embodiment of the present application, as shown in the above embodiments, further, the data processing method provided by the embodiment of the present application further comprises: Figure 13

[0178] S1301, if the processing module judges that the data type corresponding to the field is a preset type, then an index table of the column data corresponding to the field is obtained according to the field;

[0179] Specifically, the processing module obtains the data type corresponding to the column data corresponding to the field. If the data type corresponding to the column data is a preset type, then an index table of the column data corresponding to the field is obtained according to the field. The index table of the column data corresponding to the field is obtained in advance and stored corresponding to the field.

[0180] ​S1302, the processing module hashes the query keyword according to the string length to obtain a hash value corresponding to the query keyword.

[0181] Specifically, the query keyword is string data, and the processing module hashes the string length of the query keyword according to the string length of the query keyword to obtain a hash value corresponding to the query keyword.

[0182] S1303, the processing module obtains an index value corresponding to the query keyword according to the hash value corresponding to the query keyword and an index table of column data corresponding to the field.

[0183] Specifically, the processing module queries a corresponding sorting string group from the index table of column data corresponding to the field according to the hash value corresponding to the query keyword, and then uses a binary search algorithm to find the same string data as the query keyword in the sorting string group corresponding to the query keyword. If there is the same string data as the query keyword, the index value corresponding to the string data is taken as the index value corresponding to the query keyword.

[0184] It can be understood that if the index value corresponding to the query keyword cannot be found, it means that there is no data matching the query keyword in the column data corresponding to the field.

[0185] 1304, the processing module obtains a query result corresponding to the query keyword according to the index value corresponding to the query keyword and query column data corresponding to the column data corresponding to the field.

[0186] Specifically, the processing module performs data query in the query column data corresponding to the column data corresponding to the field according to the index value corresponding to the query keyword to obtain a query result corresponding to the query keyword. The query column data corresponding to the column data corresponding to the field is pre-stored.

[0187] When performing data query, the query keyword first obtains an index value according to an index table of column data, and then compares the index value with an index value in query column data, which converts comparison of strings into comparison of index values, greatly accelerating the query speed of strings and other variable length data.

[0188] On the basis of the above embodiments, further, before obtaining a query result corresponding to the query keyword according to the index value corresponding to the query keyword and query column data corresponding to the column data corresponding to the field, the method further comprises:

[0189] According to the field corresponding compression algorithm, each piece of compressed data of the query column data corresponding to the column data corresponding to the field is decompressed to obtain the query column data corresponding to the column data corresponding to the field.

[0190] Specifically, in order to save storage space, the query column data corresponding to the column data can be stored by segment compression. When the query column data corresponding to the column data is needed, it needs to be decompressed. The field corresponding compression algorithm of the processing module decompresses each piece of compressed data of the query column data corresponding to the column data corresponding to the field, and the query column data corresponding to the column data corresponding to the field can be obtained.

[0191] Figure 14 is a flowchart of the data processing method provided by the fourteenth embodiment of the application, as shown in Figure 14 On the basis of the above embodiments, further, the data processing method provided by the embodiments of the application further comprises:

[0192] S1401, the processing module acquires the query request sent by each client;

[0193] Specifically, each client can send a query request to the database system, and the processing module can receive the query request sent by each client through the communication module. The query request includes information such as data table identifier, field and query keyword, which is set according to actual needs, and the embodiments of the application are not limited.

[0194] S1402, the processing module puts each query request into a query input queue according to the receiving time sequence of each query request.

[0195] Specifically, the processing module can record the receiving time when receiving each query request. The processing module can put each query request into the query input queue according to the receiving time sequence of each query request according to the receiving time. Subsequently, the processing module will process the query request one by one in the way of first in first out.

[0196] Figure 15 is a schematic diagram of the entity structure of the electronic device provided by the fifteenth embodiment of the application, as shown in Figure 15As shown, the electronic device can include a processor 1501, a communications interface 1502, a memory 1503, and a communications bus 1504, wherein the processor 1501, the communications interface 1502, and the memory 1503 complete mutual communication through the communications bus 1504. The processor 1501 can invoke a logical instruction in the memory 1503 to execute the following method: obtaining to-be-stored data, the to-be-stored data including a data table; selecting a corresponding compression algorithm to segmentally compress each column of data according to an access frequency of each column of data of the data table; wherein a corresponding relationship between the access frequency and the compression algorithm is preset; and storing each column of segmentally compressed data to a storage module. Or

[0197] obtaining a query request from a query input queue, the query request including a data table identifier, at least one field, and a query keyword; querying the storage module to obtain, according to the data table identifier and the at least one field, segmentally compressed data corresponding to each field of the at least one field; wherein each field corresponds to a column of data of the data table; decompressing, according to a compression algorithm corresponding to each field of the at least one field, the segmentally compressed data of the column of data corresponding to each field to obtain decompressed data corresponding to each field of the at least one field; and obtaining a query result according to the query keyword and the decompressed data corresponding to each field of the at least one field.

[0198] In addition, the logical instruction in the memory 1503 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0199] The embodiment discloses a computer program product, the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the method provided by each method embodiment, for example, comprising: obtaining to be stored data, the to be stored data comprises a data table; according to the access frequency of each column data of the data table, selecting a corresponding compression algorithm to segment compression each column data; wherein, the correspondence between the access frequency and the compression algorithm is preset; each column data after segment compression is stored to a storage module. Or

[0200] obtaining a query request from a query input queue, the query request comprising a data table identifier, at least one field, and a query keyword; querying the at least one field corresponding to each segment compressed data from a storage module according to the data table identifier and the at least one field; wherein each field corresponds to a column data of the data table; decompressing each segment compressed data of the column data corresponding to each field according to the compression algorithm corresponding to each field in the at least one field, to obtain decompressed data corresponding to each field in the at least one field; obtaining a query result according to the query keyword and the decompressed data corresponding to each field in the at least one field.

[0201] The embodiment provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program enables the computer to execute the method provided by each method embodiment, for example, comprising: obtaining to be stored data, the to be stored data comprises a data table; according to the access frequency of each column data of the data table, selecting a corresponding compression algorithm to segment compression each column data; wherein, the correspondence between the access frequency and the compression algorithm is preset; each column data after segment compression is stored to a storage module. Or

[0202] obtaining a query request from a query input queue, the query request comprising a data table identifier, at least one field, and a query keyword; querying the at least one field corresponding to each segment compressed data from a storage module according to the data table identifier and the at least one field; wherein each field corresponds to a column data of the data table; decompressing each segment compressed data of the column data corresponding to each field according to the compression algorithm corresponding to each field in the at least one field, to obtain decompressed data corresponding to each field in the at least one field; obtaining a query result according to the query keyword and the decompressed data corresponding to each field in the at least one field.

[0203] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0204] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0205] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0207] In the description of the present specification, the description of the terms "one embodiment", "one specific embodiment", "some embodiments", "for example", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0208] The specific embodiments described above further illustrate the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A database system, characterized in that, It includes a storage module and a processing module, wherein: The processing module is connected to the storage module; The processing module is used to select the corresponding compression algorithm to perform segmented compression on each column of data according to the access frequency of each column of data in the data table; wherein, the correspondence between access frequency and compression algorithm is preset; The storage module is used to compress and save each column of data in the data table in segments; Specifically, the processing module is used to obtain the segment length of the column data based on the data length of the column data and the number of processor threads; if it is determined that the segment length of the column data is less than a first preset value, the column data is segmented based on the first preset value to obtain each segment of the column data and the number of segments; and each segment of the column data is compressed in parallel using the number of threads corresponding to the number of segments according to the compression algorithm corresponding to the column data.

2. The database system according to claim 1, characterized in that, The storage module includes a first-level storage unit, a second-level storage unit, and a third-level storage unit. The processing module is connected to the first-level storage unit, the second-level storage unit, and the third-level storage unit, respectively. The first-level storage unit is used to store data tables using a redundancy mechanism; The second-level storage unit is used to cache the data stored in the first-level storage unit; The third-level storage unit is used to cache the data stored in the second-level storage unit.

3. The database system according to claim 1 or 2, characterized in that, It also includes a communication module, which is connected to the processing module.

4. A data processing method based on the database system according to any one of claims 1 to 3, characterized in that, include: The processing module acquires data to be stored, which includes a data table. The processing module selects a corresponding compression algorithm to segment and compress each column of data according to the access frequency of each column of data in the data table; wherein, the correspondence between access frequency and compression algorithm is preset; The processing module stores the segmented and compressed data of each column into the storage module; The processing module obtains the segment length of the column data based on the data length of the column data and the number of processor threads. If it is determined that the segment length of the column data is less than a first preset value, the column data is segmented based on the first preset value to obtain each segment of the column data and the number of segments. The compression algorithm corresponding to the column data is used to compress each segment of the column data in parallel using the number of threads corresponding to the number of segments.

5. The data processing method according to claim 4, characterized in that, The step of selecting a corresponding compression algorithm to segment and compress each column of data based on its access frequency includes: Obtain the access frequency corresponding to each column of data in the data table; The corresponding compression algorithm is obtained based on the access frequency of each column of data in the data table; The data in each column of the data table is segmented and compressed according to the compression algorithm corresponding to each column of the data table.

6. The data processing method according to claim 5, characterized in that, The step of segmenting and compressing the data in each column of the data table according to the compression algorithm corresponding to each column of the data table includes: The segment length corresponding to the column data is obtained based on the data length of the column data and the number of processor threads; If it is determined that the length of the segmented data corresponding to the column data is greater than or equal to a first preset value and less than a second preset value, then the column data is segmented based on the segmented data length to obtain each segment of data corresponding to the column data; Based on the compression algorithm corresponding to the column data, the number of processor threads is used to compress each segment of data in parallel.

7. The data processing method according to claim 5, characterized in that, The step of segmenting and compressing the data in each column of the data table according to the compression algorithm corresponding to each column of the data table includes: The segment length corresponding to the column data is obtained based on the data length of the column data and the number of processor threads; If it is determined that the length of the segmented data corresponding to the column data is greater than or equal to the second preset value, then the column data is segmented based on the second preset value to obtain each segment of data corresponding to the column data; Based on the compression algorithm corresponding to the column data, the number of processor threads is used to compress each segment of data in parallel.

8. The data processing method according to claim 4, characterized in that, Also includes: If the processing module determines that the data type corresponding to the column data in the data table is a preset type, it will perform deduplication on the column data to obtain the deduplicated column data. The processing module performs hash hashing on the deduplicated column data according to the string length to obtain the hash value corresponding to each string data in the deduplicated column data. The processing module groups the data into multiple string groups based on the hash value corresponding to each string in the deduplicated column data; wherein each string group includes strings with the same string length and corresponding to the same hash value. The processing module sorts the string data included in each string group to obtain each sorted string group; The processing module generates an index table for the column data based on the index values ​​corresponding to the deduplicated column data, each sorted string group, and their respective hash values; wherein, the index values ​​corresponding to the deduplicated column data are obtained in advance.

9. The data processing method according to claim 8, characterized in that, Also includes: The processing module replaces the corresponding column data with the index value corresponding to the deduplicated column data to obtain the query column data corresponding to the column data. The processing module performs segmented compression on the query column data corresponding to the column data according to the compression algorithm corresponding to the column data, and obtains each segment of compressed data of the query column data corresponding to the column data.

10. A data processing method based on the database system according to any one of claims 1 to 3, characterized in that, include: The processing module obtains a query request from the query input queue. The query request includes a data table identifier, at least one field, and query keywords. The processing module retrieves compressed data corresponding to each field from the storage module based on the data table identifier and the at least one field; wherein each field corresponds to a column of data in the data table. The processing module decompresses each segment of compressed data of the column data corresponding to each field according to the compression algorithm corresponding to each field in the at least one field, and obtains the decompressed data corresponding to each field in the at least one field; The processing module obtains the query results based on the query keywords and the decompressed data corresponding to each of the at least one field; The step of decompressing the compressed data of each segment of column data corresponding to each field in the at least one field according to the compression algorithm corresponding to each field in the at least one field, to obtain the decompressed data corresponding to each field in the at least one field, includes: If it is determined that the number of segments of the column data corresponding to the field is greater than the number of processor threads, then the column data corresponding to the field is decompressed in parallel using the number of processor threads according to the compression algorithm corresponding to the column data of the field. If it is determined that the number of segments of the column data corresponding to the field is less than or equal to the number of processor threads, then the column data corresponding to the field is decompressed in parallel using the number of threads corresponding to the number of segments of the column data corresponding to the field, according to the compression algorithm corresponding to the column data corresponding to the field.

11. The data processing method according to claim 10, characterized in that, Also includes: If the processing module determines that the data type corresponding to the field is a preset type, it obtains an index table of the column data corresponding to the field based on the field. The query keywords are hashed according to their string length to obtain the hash value corresponding to the query keywords; The processing module obtains the index value corresponding to the query keyword based on the hash value corresponding to the query keyword and the index table of the column data corresponding to the field; The processing module obtains the query results corresponding to the query keywords based on the index value corresponding to the query keywords and the query column data corresponding to the column data of the field.

12. The data processing method according to claim 11, characterized in that, Before obtaining the query result corresponding to the query keyword based on the index value corresponding to the query keyword and the query column data corresponding to the column data of the field, the process also includes: The processing module decompresses each segment of compressed data of the query column data corresponding to the column data corresponding to the field according to the field-corresponding compression algorithm to obtain the query column data corresponding to the column data corresponding to the field.

13. The data processing method according to any one of claims 10 to 12, characterized in that, Also includes: The processing module obtains the query requests sent by each client; The processing module places each query request into a query input queue according to the order in which they are received.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 4 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 4 to 13.

Images