A constant encoding method, apparatus, and computing device cluster

Abstract

The application discloses a constant encoding method, comprising: acquiring a plurality of pages, and selecting a sample page from the plurality of pages; obtaining an encoding table of at least one constant column in the sample page according to a constant value of the at least one constant column; identifying constant values of corresponding columns of other pages except the sample page in the plurality of pages by using the encoding table of the at least one constant column respectively, and determining each constant column in the other pages; and performing constant encoding on data of each constant column in the other pages, and obtaining constant element information of each constant column in the other pages. In the application, when identifying constant values of the other pages, constant values recorded in the encoding table of each constant column of the sample page are used to identify constant values of data in corresponding columns of the other pages, and a hash algorithm is not needed to identify constant values, so that performance overhead of constant encoding on the other pages is reduced, and the speed of constant encoding on the plurality of pages is accelerated.

Description

Technical Field

[0001] This invention relates to the field of encoding technology, and in particular to a constant encoding method, apparatus, and computing device cluster. Background Technology

[0002] With the proliferation of data-intensive applications and the surge in data volume, the storage and transmission costs of databases are constantly rising. To improve the efficiency of data storage and transmission, database compression encoding has become a key data optimization technique. This technique reduces the amount of data stored and transmitted by decreasing redundant information. Existing compression encoding methods include dictionary encoding, prefix compression, variable-length integer encoding, bitmap compression, and character encoding, all of which effectively reduce data redundancy. However, in some extreme storage scenarios, such as when a column of data within a page is almost identical, existing encoding algorithms may not provide significant compression results. Therefore, further exploration and development of more efficient data compression techniques are needed to address these challenges. Summary of the Invention

[0003] To address the aforementioned problems, embodiments of this application provide a constant encoding method that reduces the performance overhead of constant encoding for pages and accelerates the speed of constant encoding for multiple pages. Furthermore, this application also provides a constant encoding apparatus and a computing device cluster corresponding to this constant encoding method.

[0004] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0005] In a first aspect, this application provides a constant encoding method, comprising: acquiring multiple pages and selecting a sample page from the multiple pages; obtaining an encoding table for at least one constant column based on the constant values ​​of at least one constant column in the sample page; each encoding table being used to record the column number and constant value of a constant column; a constant column being a column in which the ratio between the number of bytes occupied by the constant value and the total number of bytes in a column of data is greater than a set threshold; based on the column number and constant value recorded in the encoding table of at least one constant column, performing constant value identification on the corresponding columns in other pages to determine each constant column in other pages, wherein other pages are pages other than the sample page among the multiple pages; performing constant encoding on the data of each constant column in other pages to obtain constant metadata of each constant column in other pages, wherein each constant metadata is used to record the data of a constant column.

[0006] In this embodiment, after acquiring multiple pages each time, the method selects one page as the sampling page. The method performs constant encoding on the data in the constant columns of the sampling page, obtaining an encoding table for each constant column in the sampling page, including the column number, encoding method, and constant value. When identifying constant values ​​on other pages, the method can use the constant values ​​recorded in the encoding table of each constant column of the sampling page to identify the constant values ​​of the corresponding columns in other pages, eliminating the need for hash algorithms. This reduces the performance overhead of constant encoding on other pages and speeds up the constant encoding of multiple pages.

[0007] In one implementation, the constant metadata includes an encoding method bit, a constant value bit, and a flag bit. The encoding method bit records the encoding type of the current column, the constant value bit records the constant value, and the flag bit records "1" or "0". "1" indicates that all the data in the column corresponding to the constant metadata is constant value, and "0" indicates that there is an abnormal value in the column corresponding to the constant metadata.

[0008] In one implementation, when the flag bit is recorded as "0", the constant element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.

[0009] In this embodiment, the method can customize the format of constant metadata, allowing the constant metadata to include encoding mode bits, constant value bits, flag bits, abnormal value bitmap bits (optionally), and abnormal value bits (optionally), and allowing the encoding mode bits to occupy 1 byte and the flag bits to occupy 1 bit, thereby minimizing the total number of bytes occupied by the constant metadata and thus improving the compression rate of constant encoding.

[0010] In one implementation, after selecting a sample page from multiple pages, the method further includes: splitting the sample page into multiple columns of data and determining constant values ​​in the data of each column; determining whether the ratio between the number of bytes occupied by the constant values ​​in each column and the total number of bytes occupied by the data in the corresponding column is greater than a set threshold; and determining the constant column of the sample page when the ratio between the number of bytes occupied by the constant values ​​in the column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold.

[0011] In one implementation, the sampling page is a row-based page. The sampling page is split into multiple columns of data, specifically by stacking the values ​​at the same position in each tuple of the sampling page into columns to obtain multiple columns of data.

[0012] In this embodiment, for row-based pages, the method can stack the values ​​at the same position in multiple tuples in the sampled page into columns, so that row-based pages can also be encoded using constant encoding.

[0013] In one implementation, determining the constant value in the data of each column specifically includes: selecting a portion of the data from each column according to a preset ratio to obtain sample data for each column; and using a hash algorithm to identify the constant value in the sample data of each column to obtain the constant value in the data of each column.

[0014] In this embodiment, when performing constant encoding on the sample page, the method selects a predetermined proportion of sample data from each column and then uses a hash algorithm to identify the constant values ​​within that sample data. Typically, a column of data consists of hundreds, thousands, or even tens of thousands of values. Therefore, this method only identifies constant values ​​on a portion of the data. This avoids interfering with the accurate identification of constant columns and reduces the workload of the hash algorithm for constant value identification, thereby lowering the performance overhead of constant encoding on the sample page and accelerating the process.

[0015] In one implementation, based on the column number and constant value recorded in the encoding table of at least one constant column, constant value identification is performed on the corresponding columns in other pages to determine each constant column in other pages. Specifically, this includes: selecting data from the first page of the column with the same column number as the column number recorded in the first encoding table; at least one encoding table includes the first encoding table, and other pages include the first page; comparing the constant values ​​recorded in the first encoding table with the values ​​in the data of the selected columns in the first page to obtain the number of bytes occupied by the constant values ​​recorded in the first encoding table; determining whether the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected columns in the first page is greater than a set threshold; if the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected columns in the first page is greater than the set threshold, determining that the selected columns in the first page are constant columns.

[0016] In this embodiment, when the method performs constant value identification on other pages, it can use the constant values ​​recorded in the encoding table of each constant column of the sampled page to perform constant value identification on the data of the corresponding column in other pages. It does not require the use of a hash algorithm for constant value identification, thereby reducing the performance overhead of constant encoding on other pages and speeding up the constant encoding of multiple pages.

[0017] Secondly, this application provides a constant encoding device, comprising: a first processing module for acquiring multiple pages and selecting a sample page from the multiple pages; a second processing module for obtaining an encoding table for at least one constant column based on the constant values ​​of at least one constant column in the sample page; each encoding table for recording the column number and constant value of a constant column; a constant column is a column in which the ratio between the number of bytes occupied by the constant value and the total number of bytes in a column of data is greater than a set threshold; a third processing module for identifying constant values ​​in corresponding columns in other pages based on the column number and constant value recorded in the encoding table of at least one constant column, and determining each constant column in other pages, wherein other pages are pages other than the sample page among the multiple pages; and a fourth processing module for performing constant encoding on the data of each constant column in other pages to obtain constant metadata of each constant column in other pages, wherein each constant metadata is used to record the data of a constant column.

[0018] In one implementation, the constant metadata includes an encoding method bit, a constant value bit, and a flag bit. The encoding method bit records the encoding type of the current column, the constant value bit records the constant value, and the flag bit records "1" or "0". "1" indicates that all the data in the column corresponding to the constant metadata is constant value, and "0" indicates that there is an abnormal value in the column corresponding to the constant metadata.

[0019] In one implementation, when the flag bit is recorded as "0", the constant element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.

[0020] In one implementation, after selecting a sampling page from multiple pages, the first processing module further splits the sampling page into multiple columns of data and determines the constant value in the data of each column; determines whether the ratio between the number of bytes occupied by the constant value in each column and the total number of bytes occupied by the data in the corresponding column is greater than a set threshold; when the ratio between the number of bytes occupied by the constant value in the column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold, the constant column of the sampling page is determined.

[0021] In one implementation, when the sampling page is a row-based page, the first processing module is specifically used to stack the values ​​at the same position in each tuple of the sampling page into columns to obtain multiple columns of data.

[0022] In one implementation, the first processing module is specifically used to select a portion of the data from each column of data according to a preset ratio to obtain sample data for each column; and to use a hash algorithm to identify constant values ​​in the sample data of each column to obtain constant values ​​in the data of each column.

[0023] In one implementation, the third processing module is specifically configured to select data from the first page that has the same column number as the column number recorded in the first encoding table; at least one encoding table includes the first encoding table, and other pages include the first page; compare the constant values ​​recorded in the first encoding table with each value in the data of the selected column on the first page to obtain the number of bytes occupied by the constant values ​​recorded in the first encoding table; determine whether the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than a set threshold; if the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than the set threshold, determine that the column selected on the first page is a constant column.

[0024] Thirdly, embodiments of this application provide a computing device, including: at least one memory; and at least one processor, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the embodiments as described in the first aspect.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device, cause the computing device to perform various possible implementations of the first aspect.

[0026] Fifthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, cause the computing device to implement various possible implementations of the first aspect.

[0027] In a sixth aspect, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the various possible implementations of the first aspect.

[0028] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the various possible implementations of the first aspect.

[0029] Eighthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement various possible implementations of the first aspect. Attached Figure Description

[0030] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0031] Figure 1 This is a schematic diagram of the structure of a database storage system provided in an embodiment of this application;

[0032] Figure 2 This is a flowchart illustrating a constant encoding method provided in an embodiment of this application;

[0033] Figure 3(a) is a schematic diagram of the process of the row storage engine storing input data provided in the embodiment of this application;

[0034] Figure 3(b) is a schematic diagram of the process by which the column storage engine provided in this embodiment stores input data;

[0035] Figure 4(a) is a schematic diagram of the process of generating constant element information based on column data in the data processing system provided in the embodiment of this application;

[0036] Figure 4(b) is a schematic diagram of the process of generating a coding table based on columns in the data processing system provided in the embodiments of this application;

[0037] Figure 5(a) is a schematic diagram of the process of generating constant metadata for row-style pages provided in the embodiments of this application;

[0038] Figure 5(b) is a schematic diagram of the process of generating constant element information from the data of the columnar page provided in the embodiment of this application;

[0039] Figure 6 This is a schematic diagram illustrating a user's use of the data processing system as provided in the embodiments of this application;

[0040] Figure 7 This is a schematic diagram of the structure of a constant encoding device provided in the embodiments of this application;

[0041] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0042] Figure 9 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of another computing device cluster architecture provided in the embodiments of this application. Detailed Implementation

[0044] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0045] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0046] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0047] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0048] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0049] Before introducing the technical solution protected by this application, several technical terms involved in the technical solution protected by this application will be explained in advance, namely:

[0050] A data system employing a storage engine refers to using a storage engine to manage the storage, indexing, and retrieval of data. Storage engines can be categorized into row-based storage engines and column-based storage engines. Row-based storage engines store data row-by-row, meaning each row of data is stored contiguously. This is suitable for operations requiring frequent full-row reads, such as online transaction processing (OLTP) systems. Column-based storage engines store data column-by-column, meaning data from the same column is stored together. This is suitable for scenarios requiring extensive analysis and aggregation operations on specific columns, such as online analytical processing (OLAP) systems.

[0051] A page is the basic unit used for data storage, management, and transmission in a storage system. Its size and purpose depend on the specific application scenario and technical implementation. In a row-oriented storage system, data is stored row by row, with each page containing multiple rows of data, and each row containing all the values ​​in that row. In a column-oriented storage system, data is stored column by column, with each page containing one or more columns of data, rather than a single row of multiple columns.

[0052] Constant encoding is a database storage optimization technique specifically designed to handle columns containing a large number of duplicate values. This technique first examines each value in the column to identify those that appear frequently throughout the column. Next, it uses a hash algorithm combined with efficient pruning to quickly locate and identify these duplicate values. Pruning reduces the amount of data fed into the hash table and hash collisions, improving lookup efficiency. Then, constant encoding stores only the duplicate values ​​and the very few unique values ​​in the column, rather than storing every single value. Finally, constant encoding creates a mapping table that maps duplicate values ​​to smaller identifiers or indexes, thus using less space during storage and improving storage efficiency and query performance.

[0053] In the process of constant encoding, hash algorithms use hash functions to map values ​​in a column to a hash table. The key of the hash table is the hash value of the value, and the value is the corresponding value and its frequency. This technique helps to quickly identify which values ​​are constants—that is, values ​​that appear very frequently. However, hash algorithms consume significant resources when extracting constant values ​​due to the large amount of computation, storage, and input / output (I / O) operations involved, as well as the potential need for model retraining and adjustment. These factors collectively contribute to the increased resource consumption of hash algorithms.

[0054] A constant value is the value that appears most frequently in a column of data in a column-based page or in a column-based page where each tuple is stacked into a column.

[0055] Outliers are values ​​other than constant values ​​in a column of data in a columnar page or a column of data in a row-based page.

[0056] Constant metadata refers to descriptive information related to constant data in a database system. It describes the structure and meaning of the data, including but not limited to the specific numerical value, data type, and location of the constant. In database tables, constant metadata may also include the name of the constant value, storage engine, version, and storage space occupied. Constant metadata is crucial for database management systems because it enables the system to understand the meaning and organization of the data, ensuring that the data is processed and maintained correctly.

[0057] Next, the technical solution provided in this application will be introduced.

[0058] Generally, constant encoding requires checking the data in each column of every page for constant rules. If a column contains a large amount of duplicate data, this data can be identified as constant values, and only one copy is kept. This copy, along with the outlier data and the corresponding row number, forms constant metadata, which is then stored on disk. This method can significantly reduce the total number of bytes in a column, achieving data compression.

[0059] However, constant encoding schemes in related technologies incur certain overhead in compression performance. This is because all columns on all pages require the use of hash algorithms to search for constant values ​​and identify outliers, a process that introduces significant performance overhead. Therefore, although constant encoding can reduce data storage, its resource consumption during execution is also a factor that needs to be considered.

[0060] In view of this, embodiments of this application provide a constant encoding method. This method, after acquiring multiple pages each time, can select one page from the multiple pages as a sampling page. The method processes the data in the constant columns of the sampling page to obtain an encoding table corresponding to each constant column in the sampling page, including the column number, encoding method, and constant value. When identifying constant values ​​on other pages, this method can use the constant values ​​recorded in the encoding table of each constant column of the sampling page to identify the constant values ​​of the corresponding columns in other pages, without needing to use a hash algorithm for constant value identification. This reduces the performance overhead of constant encoding on other pages and speeds up the constant encoding of multiple pages.

[0061] This method, when performing constant encoding on a sample page, selects a predetermined proportion of sample data from each column and then uses a hash algorithm to identify the constant values ​​within that sample data. Typically, a column of data consists of hundreds, thousands, or even tens of thousands of values. Therefore, this method only identifies constant values ​​in a subset of the data. This avoids interfering with the accurate identification of constant columns and reduces the workload of the hash algorithm for constant value identification, thereby lowering the performance overhead of constant encoding on the sample page and accelerating the encoding process.

[0062] This method allows for customization of the format of constant metadata, including encoding mode bits, constant value bits, flag bits, exception value bitmap bits (optionally), and exception value bits (optionally). The encoding mode bits occupy 1 byte, and the flag bits occupy 1 bit, minimizing the total number of bytes occupied by constant metadata and thus improving the compression rate of constant encoding.

[0063] Figure 1 This is a schematic diagram of the structure of a database storage system provided in an embodiment of this application. Figure 1 As shown, the database storage system 100 may include a data processing system 110, a memory 120, and a database 130.

[0064] When the database storage system 100 receives input data, this data is first preprocessed by the data processing system 110. During the preprocessing stage, the data processing system 110 encodes the input data using either a row-based storage engine or a column-based storage engine, according to the encoding strategy of the table to be stored. After encoding, the data processing system 110 temporarily stores the encoded data in memory 120. Subsequently, the data processing system 110 stores this data in the database 130 in a format of 8192 bytes per page; this process is called "disk write".

[0065] When a user needs to access or manipulate this data, the data processing system 110 reads the data from the disk of the database 130 into the memory 120 and decodes it. Afterward, the data processing system 110 returns the decoded data to the upper-level scheduling engine for further processing.

[0066] In this embodiment of the application, when the data processing system 110 performs constant encoding, it can...

[0067] Multiple pages are retrieved each time, and one page is selected as the sampling page. The data processing system 110 performs constant encoding on the data of the constant columns in the sampling page to obtain constant metadata for each constant column. Based on this constant metadata, an encoding table is obtained for each constant column in the sampling page, including the column number, encoding method, and constant value. When the data processing system 110 performs constant value identification on other pages, it can use the constant values ​​recorded in the encoding table of each constant column in the sampling page to identify the constant values ​​of the corresponding columns in other pages, without needing to use a hash algorithm. This reduces the performance overhead of constant encoding on other pages and speeds up the constant encoding of multiple pages.

[0068] Figure 2 This is a flowchart illustrating a constant encoding method provided in an embodiment of this application. Figure 2As shown, this method can be executed by the aforementioned data processing system 110, and the specific implementation process is as follows:

[0069] Step S201: Select one page from N pages as the sampling page. N is a positive integer greater than 1.

[0070] Specifically, the data processing system 110 first creates a new table and defines a compression strategy for it. The compression strategy can be based on rows, columns, or pages. Then, after receiving input data, the data processing system 110 can write the input data into the new table. When the data processing system 110 writes the data in the new table to disk, it can first analyze the data in the table to determine which data is frequently accessed "hot data" and which data is accessed less frequently as "cold data." The data processing system 110 can encode the cold data according to the defined compression strategy and then write the encoded data to disk in the form of pages. After obtaining N pages, the data processing system 110 can randomly select one page from multiple pages as a sampling page and use the remaining N-1 pages as encoding pages.

[0071] The preset sampling frequency determines the size of N. That is, the higher the preset sampling frequency, the smaller N is; the lower the preset sampling frequency, the larger N is. The data processing system 110 can add a Grand Unified Configuration (GUC) parameter to control the sampling frequency, thereby optimizing the speed of constant rule encoding. Since processing sample pages is time-consuming and consumes significant performance, the data processing system 110 can increase the preset sampling frequency to reduce the ratio of sample pages to encoding pages, thereby increasing the speed of constant rule encoding and reducing performance overhead.

[0072] Pages can be categorized into row-based pages and column-based pages. Row-based pages refer to pages stored using a row-based storage engine. For example, as shown in Figure 3(a), when the row-based storage engine stores input data, it stores each value in the input data sequentially in a table, row by row. Data in the table structure is organized into multiple tuples. Each tuple includes multiple columns. When the data processing system 110 generates a page consisting of three tuples, it can select the data identified as "tuple 1", "tuple 2", and "tuple 9" from the table to form a row-based page.

[0073] A columnar page refers to a page stored using a column-oriented storage engine. For example, as shown in Figure 3(b), when storing input data, the column-oriented storage engine stores each value in the input data sequentially in a table, organized by column. Data in the table structure is organized together with multiple columns. When the data processing system 110 generates a page consisting of three columns, it can select data from the table labeled "col1", "col2", and "col9" to form a columnar page.

[0074] In this embodiment, the data processing system 110 can customize the format of constant metadata. The constant metadata includes an encoding type bit, a constant value bit, a flag bit, an outlier bitmap (optionally), and an outlier bit (optionally). The encoding type bit records the encoding type of the current column and typically occupies 1 byte. The constant value bit records the constant value (i.e., the value that appears most frequently in a column), and the number of bytes occupied is determined by the column's attributes. The flag bit records "1" or "0" and typically occupies 1 bit. "1" indicates that all non-NULL values ​​in the current column are constant values, and there are no subsequent outlier bitmaps or outlier bits. "0" indicates that not all non-NULL values ​​in the current column are constant values, and there are subsequent outlier bitmaps and outlier bits. The outlier bitmap records the row number corresponding to the outlier, and the number of bytes occupied is equal to the sum of the number of constant values ​​and outliers in the column. The outlier bit records outliers (i.e., values ​​other than constant values ​​in a column), and the number of bytes it occupies is determined by the column's attributes. This application uses custom constant metadata to minimize the total number of bytes occupied by constant metadata, thereby improving the compression ratio of constant encoding.

[0075] The data processing system 110 can pre-store an encoding list, which records the correspondence between various encoding algorithms and flag bits. For example, the flag bit for constant encoding is 1, the flag bit for equal value encoding is 2, and the flag bit for difference encoding is 3, etc. When recording the encoding type, the data processing system 110 can set a flag bit in the encoding method bit of the constant metadata to identify the encoding type. This ensures that the encoding method bit only occupies 1 byte, thereby effectively reducing the number of bytes required to store metadata.

[0076] For example, as shown in Figure 4(a), the data processing system 110 acquires three columns of data, namely “col 1”, “col 2” and “col 3”. Among them, “col 1” is a non-constant column, “col 2” and “col 3” are both constant columns, and “col 2” is a constant column with one outlier.

[0077] The constant metadata generated by the data processing system 110 based on the data in column "col 2" includes encoding mode bit, constant value bit, flag bit, outlier bit map bit, and outlier bit. The encoding mode bit records the column encoding type and occupies 1 byte. The constant value bit records the constant value "AAAAA" and occupies 5 bytes. The flag bit records "0" and occupies 1 bit. The outlier bit map records "1", "1", "1", "1", "1", "1", "0", "1", "1", and "1", a total of 10 bits, occupying N / 8 = 10 / 8 = 2 bytes. Here, "1" represents a constant value bit, and "0" represents an outlier bit. The outlier bit records the outlier value as "GGGGG", occupying the corresponding column attribute length (if the column attribute is a fixed-length field, it's the column attribute byte length; if the column attribute is a variable-length field, it's the actual constant value byte length).

[0078] The constant metadata generated by the data processing system 110 based on the data in column "col 3" includes an encoding method bit, a constant value bit, and a flag bit. The encoding method bit records the column encoding type and occupies 1 byte. The constant value bit records a constant value of "AAAAA" and occupies the corresponding column attribute length. The flag bit records "0" and occupies 1 bit.

[0079] Step S202: Based on the data of at least one constant column in the sampling page, obtain the constant element information and encoding table of at least one constant column in the sampling page.

[0080] In this embodiment of the application, during the process of identifying constant columns, the data processing system 110 can define a relationship between the number of bytes occupied by the constant value in each column and the total number of bytes, satisfying formula (1), and can consider that column as a constant column. Formula (1) is:

[0081] SUM(LEN(Ai|c)) ≥0.9 * SUM(LEN(Ai)) (1)

[0082] Where c represents the constant value in the column, Ai represents the value of a row in the column, LEN() represents the number of bytes occupied by the constant value in a row, and SUM() represents the total number of bytes occupied by all constant values ​​in the column.

[0083] Taking a sample page as an example (row-based page), after obtaining the sample page, the data processing system 110 can stack the values ​​at the same position in each tuple of the sample page into columns, thus obtaining multiple columns of data. For example, the sample page includes three tuples: "tuple 1", "tuple 2", and "tuple 3", each tuple containing three values. The data processing system 110 can stack the values ​​in the first column of "tuple 1", "tuple 2", and "tuple 3" into columns to obtain the first column of data; stack the values ​​in the second column of "tuple 1", "tuple 2", and "tuple 3" into columns to obtain the second column of data; and stack the values ​​in the third column of "tuple 1", "tuple 2", and "tuple 3" into columns to obtain the third column of data.

[0084] The data processing system 110 can extract a portion of sample data from each column of data according to a set ratio, thereby reducing the workload of the hash algorithm in identifying constant values. When searching for constant values, the data processing system 110 can use the hash function in the hash algorithm to identify constant values ​​in the sample data, so as to obtain the recurring constant values.

[0085] Typically, a data column consists of hundreds, thousands, or even tens of thousands of values. Therefore, the data processing system 110 extracts sample data according to a set ratio and performs constant value identification on the sample data. This not only avoids interfering with the accurate judgment of constant columns, but also reduces the workload of constant value identification using hash algorithms, thereby reducing the performance overhead of constant encoding on the sample pages and speeding up the constant encoding process.

[0086] After the data processing system 110 finds the constant values ​​from the sample data of each column, it can first calculate the number of bytes occupied by the values ​​in each row of each column and the number of bytes occupied by each constant value. Then, the data processing system 110 can calculate the total number of bytes occupied by the data in a column based on the number of bytes occupied by the values ​​in all rows of a column. The data processing system 110 can calculate the total number of bytes occupied by the constant values ​​in a column based on the number of bytes occupied by all constant values ​​in a column. Next, the data processing system 110 can obtain the proportion of constant values ​​based on the relationship between the number of bytes occupied by constant values ​​in each column and the total number of bytes. Finally, the data processing system 110 can use the above formula (1) to determine the relationship between the proportion of constant values ​​in each column and a set threshold. If the proportion of constant values ​​is less than the set threshold, the data processing system 110 considers the data in that column to be non-constant column data, and can either not encode the non-constant column data or encode it in other ways, and then store it in memory 120.

[0087] If the proportion of constant values ​​is greater than a set threshold, the data processing system 110 can consider the data in that column as constant column data. At this point, the data processing system 110 can further determine whether the proportion of constant values ​​is equal to 100%. If the proportion of constant values ​​is greater than the set threshold but less than 100%, the data processing system 110 can generate constant metadata based on the data in that column, including encoding mode bits, constant value bits, flag bits, outlier bitmap bits, and outlier bits, and then store this constant metadata in memory 120. If the proportion of constant values ​​is equal to 100%, the data processing system 110 can generate constant metadata based on the data in that column, including encoding mode bits, constant value bits, and flag bits, and then store this constant metadata in memory 120.

[0088] Meanwhile, when generating constant metadata, the data processing system 110 can also generate an encoding table for that column based on the data in that column. The data processing system 110 can store the encoding tables of each constant column of the sampling page in memory 120 to accelerate the encoding process of the remaining N-1 pages. The encoding table includes a sequence number bit, an encoding method bit, and a constant value bit. The sequence number bit records the sequence number of the current column in the sampling page, occupying 1 byte or 2 bytes. The encoding method bit records the encoding type of the current column. The constant value bit records the constant value. Optionally, the encoding table also includes a length bit for the constant value. The length bit for the constant value records the number of constant values.

[0089] For example, as shown in Figure 4(b), the data processing system 110 can acquire three columns of data, namely “col1”, “col2” and “col3”. Among them, “col1” is a non-constant column, “col2” and “col3” are both constant columns, and “col2” is a constant column with one outlier.

[0090] The encoding table generated by data processing system 110 based on the data in column "col 2" includes a sequence number bit, an encoding method bit, and a constant value bit. The sequence number bit records the column's sequence number, which is "2". The encoding method bit records the column's encoding type. The constant value bit records the constant value as "AAAAA".

[0091] The encoding table generated by data processing system 110 based on the data in column "col 3" includes a sequence number bit, an encoding method bit, and a constant value bit. The sequence number bit records the column number, which is "3". The encoding method bit records the column encoding type. The constant value bit records the constant value as "AAAAA".

[0092] For example, take the row-style page shown in Figure 3(a) as an example. As shown in Figure 5(a), the data processing system 110 determines based on formula (1) that the data in column “col 1” does not meet the requirements of a constant column, and can identify column “col1” as a non-constant column.

[0093] Data processing system 110 can find the constant value "AAAAA" in the data labeled "col 2", and since the proportion of constant values ​​is 100%, the data processing system 110 can identify the data labeled "col 2" as a constant column. At this time, the data processing system 110 can generate constant metadata based on the data labeled "col 2", which includes: encoding method bit, constant value bit (recorded as "AAAAA"), and flag bit (recorded as "1"), and generate an encoding table, which includes: sequence number bit (recorded as "2"), encoding method bit, and constant value bit (recorded as "AAAAA").

[0094] Data processing system 110 can find the constant value "BBBBBB" in the data labeled "col 3", and since the constant value accounts for 100%, the data labeled "col 3" can be identified as a constant column. At this time, data processing system 110 can generate constant metadata based on the data labeled "col 3", which includes: encoding method bit, constant value bit (recorded as "BBBBBB"), and flag bit (recorded as "1"), and generate an encoding table, which includes: sequence number bit (recorded as "3"), encoding method bit, and constant value bit (recorded as "BBBBBB").

[0095] Data processing system 110 can find the constant value "CCCC" in the data labeled "col 4", and since the proportion of constant values ​​is 100%, the data processing system 110 can identify the data labeled "col 4" as a constant column. At this time, the data processing system 110 can generate constant metadata based on the data labeled "col 4", which includes: encoding method bit, constant value bit (recorded as "CCCC"), and flag bit (recorded as "1"), and generate an encoding table, which includes: sequence number bit (recorded as "4"), encoding method bit, and constant value bit (recorded as "CCCC").

[0096] Taking a sample page as an example (using a columnar layout), after obtaining the sample page, the data processing system 110 can split it into multiple columns, obtaining data for each column. The data processing system 110 can then extract a portion of sample data from each column according to a set ratio. When searching for constant values, the data processing system 110 can use a hash function in a hash algorithm to locate and identify the sample data, thereby obtaining recurring constant values.

[0097] After the data processing system 110 finds the constant values ​​from the sample data of each column, it can first calculate the number of bytes occupied by the values ​​in each row of each column and the number of bytes occupied by each constant value. Then, the data processing system 110 can calculate the total number of bytes occupied by the data in a column based on the number of bytes occupied by the values ​​in all rows of a column. The data processing system 110 can calculate the total number of bytes occupied by the constant values ​​in a column based on the number of bytes occupied by all constant values ​​in a column. Next, the data processing system 110 can obtain the proportion of constant values ​​based on the relationship between the number of bytes occupied by constant values ​​in each column and the total number of bytes. Finally, the data processing system 110 can use the above formula (1) to determine the relationship between the proportion of constant values ​​in each column and a set threshold. If the proportion of constant values ​​is less than the set threshold, the data processing system 110 can consider the data in that column to be non-constant column data, and can either not encode the non-constant column data or encode it in other ways, and then store it in memory 120.

[0098] If the proportion of constant values ​​is greater than a set threshold, the data processing system 110 can consider the data in that column as constant column data. At this point, the data processing system 110 can further determine whether the proportion of constant values ​​is equal to 100%. If the proportion of constant values ​​is greater than the set threshold but less than 100%, the data processing system 110 can generate constant metadata based on the data in that column, including encoding mode bits, constant value bits, flag bits, outlier bitmap bits, and outlier bits, and then store this constant metadata in memory 120. If the proportion of constant values ​​is equal to 100%, the data processing system 110 can generate constant metadata based on the data in that column, including encoding mode bits, constant value bits, and flag bits, and then store this constant metadata in memory 120.

[0099] Meanwhile, when generating constant metadata, the data processing system 110 can also generate an encoding table for that column based on the data in that column. The data processing system 110 can store the encoding tables of each constant column of the sampled page in memory 120 to accelerate the encoding process of the remaining N-1 pages.

[0100] For example, consider a column-style page as shown in Figure 3(b) as an example. As shown in Figure 5(b), the data processing system 110 finds a constant value of "AAAAA" in the data labeled "col 1", and the proportion of constant values ​​is 100%, so the column labeled "col 1" can be identified as a constant column. At this time, the data processing system 110 can generate constant metadata based on the data labeled "col 1", which includes: encoding method bit, constant value bit (recorded as "AAAAA"), and flag bit (recorded as "1"), and generate an encoding table, which includes: sequence number bit (recorded as "1"), encoding method bit, and constant value bit (recorded as "AAAAA").

[0101] Data processing system 110 can find the constant value "aa" in the data labeled "col 2", and since the proportion of constant values ​​is 100%, the data processing system 110 can identify the data labeled "col 2" as a constant column. At this time, the data processing system 110 can generate constant metadata based on the data labeled "col 2", which includes: encoding method bit, constant value bit (recorded as "aa"), and flag bit (recorded as "1"), and generate an encoding table, which includes: sequence number bit (recorded as "2"), encoding method bit, and constant value bit (recorded as "aa").

[0102] The data processing system 110 determines, based on formula (1), that the data in column “col 9” does not meet the requirements of a constant column and can be identified as a non-constant column.

[0103] Step S203: The encoding table of at least one constant column of the sampling page is used to identify the constant values ​​of the corresponding columns of N-1 encoding pages, thereby determining each constant column in the encoding page.

[0104] Step S204: Perform constant encoding on the data of each constant column in each encoding page to obtain the constant element information of each constant column in the encoding page.

[0105] Specifically, after encoding the data in the constant columns of the sampling page, the data processing system 110 encodes the data in multiple encoding pages sequentially. The data processing system 110 can split the encoding page into multiple columns, obtaining data in multiple columns. The data processing system 110 can identify the constant values ​​of the corresponding columns in the encoding page based on the constant values ​​recorded in the encoding table of each constant column in the sampling page. Since the data in the sampling page and the encoding page both come from the same table, the constant values ​​in the same columns of the sampling page and the encoding page are highly likely to be the same. Therefore, when the data processing system 110 identifies the constant values ​​of each column of data in the encoding page, it uses the constant values ​​from the encoding table in the sampling page to identify the constant values ​​of the corresponding columns in the encoding page, ensuring data consistency and accuracy. For example, after extracting the encoding table of the first column of the sampling page, the data processing system 110 identifies the constant values ​​recorded in the encoding table against the data in the first column of the encoding page. Optionally, the data processing system 110 can identify the constant values ​​in each encoding table of the sampling page with the constant values ​​in each column of the encoding page, thereby increasing the number of constant values ​​in each column of the encoding page.

[0106] For example, after extracting the encoding table of the first column of the sampling page, the data processing system 110 compares the constant values ​​recorded in the encoding table with the values ​​in the data of the first column of the encoding page to obtain the value that is the same as the constant value recorded in the encoding table (hereinafter referred to as the "target value"). Then, the data processing system 110 can calculate the number of bytes occupied by the value in each row of the first column, and the number of bytes occupied by each target value. Next, the data processing system 110 can calculate the total number of bytes occupied by the data in the first column based on the number of bytes occupied by the values ​​in all rows of the first column. The data processing system 110 can calculate the total number of bytes occupied by the target value based on the number of bytes occupied by each target value. Finally, the data processing system 110 obtains the proportion of the target value based on the relationship between the number of bytes occupied by the target value in the first column and the total number of bytes, and uses the above formula (1) to determine the relationship between the proportion of the target value and the set threshold. If the proportion of the target value is less than the set threshold, the data processing system 110 considers the first column to be a non-constant column, and can not encode the data in the non-constant column or encode it in other ways, and then store it in the memory 120.

[0107] If the proportion of the target value is greater than a set threshold, the data processing system 110 can consider the first column as a constant column and the target value as a constant value. At this point, the data processing system 110 can further determine whether the proportion of the target value is equal to 100%. If the proportion of the target value is greater than the set threshold but less than 100%, the data processing system 110 can generate constant metadata based on the data in that column, including encoding mode bits, constant value bits, flag bits, outlier bitmap bits, and outlier bits, and then store this constant metadata in memory 120. If the proportion of the target value is equal to 100%, the data processing system 110 can generate constant metadata based on the data in that column, including encoding mode bits, constant value bits, and flag bits, and then store this constant metadata in memory 120.

[0108] In this embodiment, after completing the constant encoding of N pages of data, the data processing system 110 writes the non-constant column data and the constant metadata of the constant columns from the N pages of data stored in memory 120 to the database 130, thus achieving persistent storage of the N pages of data. Subsequently, the data processing system 110 selects N pages again from the table containing cold data and continues to perform the constant encoding operation. This process continues until all pages in the table containing cold data have undergone constant encoding.

[0109] When data processing system 110 reads data, it can read N (or other numbers) pages of data from database 130 and store them in memory 120. Data processing system 110 can retrieve constant metadata from memory 120 and perform decoding operations on the constant metadata to restore the data in the constant columns. After completing the decoding operation, data processing system 110 restores the data from the N pages into a table and then returns the table to the upper-level scheduling engine for further processing.

[0110] In this embodiment, after acquiring multiple pages each time, the data processing system 110 can select one page as a sampling page. The data processing system 110 performs constant encoding on the data in the constant columns of the sampling page, obtaining an encoding table corresponding to each constant column in the sampling page, including the column number, encoding method, and constant value. When the data processing system 110 performs constant value identification on other pages, it can use the constant values ​​recorded in the encoding table of each constant column of the sampling page to identify the constant values ​​of the corresponding columns in other pages, eliminating the need to use a hash algorithm for constant value identification. This reduces the performance overhead of constant encoding on other pages and speeds up the constant encoding of multiple pages.

[0111] The above is a description of the data processing system 110 provided in this application embodiment. It is understood that the data processing system 110 can be configured on a cloud management platform, for example, deployed on at least one virtual machine or container instance, so that the cloud management platform can provide data processing services. Of course, the data processing system 110 can also be configured on nodes other than the cloud management platform, for example, deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud management platform can provide pages related to public cloud services for users to remotely access public cloud services. In this embodiment, users can pre-purchase the data processing services provided by the data processing system 110 on the cloud management platform. For ease of understanding, the interaction between the user and the cloud management platform is described below.

[0112] like Figure 6As shown, the interaction between the user and the cloud management platform mainly includes: the user logs into the cloud management platform 600 through a web page on the client side, selects and purchases cloud services (i.e., data processing services) related to the data processing system 110 on the cloud management platform 600. After purchase, the user can generate the data processing system 110 on the cloud management platform 600 based on the functions provided by the data processing service. The cloud management platform 600 is primarily used to manage the infrastructure for running the data processing service. For example, the infrastructure for the data processing service may include multiple data centers located in different regions, each data center including multiple servers. The data centers can provide basic resources for the data processing service, such as computing resources and storage resources. Therefore, when purchasing and using the data processing service, the user mainly pays for the resources used. When using the data processing service, the user can input their needs for the data processing service through the configuration interface, application programming interface (API), or user interaction interface provided by the cloud management platform 600. The cloud management platform 600 can then generate a data processing service matching the user's (or other software / hardware, etc.) input requirements.

[0113] Alternatively, some modules in the data processing system 110 can be configured on the cloud side and others on the edge side, thereby realizing data processing services through edge-cloud collaboration. Furthermore, the data processing system 110 can also be entirely configured on the edge side, depending on the actual situation; no specific limitation is made here.

[0114] Based on the above description, this application provides a constant encoding device 700. For example... Figure 7 As shown, the device 700 includes:

[0115] The first processing module 710 is used to acquire multiple pages and select a sample page from the multiple pages; the second processing module 720 is used to obtain an encoding table for at least one constant column based on the constant value of at least one constant column in the sample page; each encoding table is used to record the column number and constant value of a constant column; a constant column is a column in which the ratio between the number of bytes occupied by the constant value and the total number of bytes in a column is greater than a set threshold; the third processing module 730 is used to identify the constant values ​​of the corresponding columns in other pages based on the column number and constant value recorded in the encoding table of at least one constant column, and determine each constant column in other pages, where other pages are pages other than the sample page among the multiple pages; the fourth processing module 740 is used to perform constant encoding on the data of each constant column in other pages to obtain constant metadata of each constant column in other pages, where each constant metadata is used to record the data of a constant column.

[0116] In one implementation, the constant metadata includes an encoding method bit, a constant value bit, and a flag bit. The encoding method bit records the encoding type of the current column, the constant value bit records the constant value, and the flag bit records "1" or "0". "1" indicates that all the data in the column corresponding to the constant metadata is constant value, and "0" indicates that there is an abnormal value in the column corresponding to the constant metadata.

[0117] In one implementation, when the flag bit is recorded as "0", the constant element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.

[0118] In one implementation, after selecting a sampling page from multiple pages, the first processing module 710 further splits the sampling page into multiple columns of data and determines the constant value in the data of each column; determines whether the ratio between the number of bytes occupied by the constant value in each column and the total number of bytes occupied by the data in the corresponding column is greater than a set threshold; when the ratio between the number of bytes occupied by the constant value in the column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold, the constant column of the sampling page is determined.

[0119] In one implementation, when the sampling page is a row-based page, the first processing module 710 is specifically used to stack the values ​​at the same position in each tuple of the sampling page into columns to obtain multiple columns of data.

[0120] In one implementation, the first processing module 710 is specifically used to select a portion of the data from each column of data according to a preset ratio to obtain sample data for each column; and to use a hash algorithm to identify constant values ​​in the sample data of each column to obtain constant values ​​in the data of each column.

[0121] In one embodiment, the third processing module 730 is specifically configured to select data from the first page that has the same column number as the column number recorded in the first encoding table; at least one encoding table includes the first encoding table, and other pages include the first page; compare the constant values ​​recorded in the first encoding table with each value in the data of the selected column on the first page to obtain the number of bytes occupied by the constant values ​​recorded in the first encoding table; determine whether the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than a set threshold; if the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than the set threshold, determine that the column selected on the first page is a constant column.

[0122] The first processing module 710, the second processing module 720, the third processing module 730, and the fourth processing module 740 can all be implemented in software or in hardware. For example, the implementation of the first processing module 710 will be described below. Similarly, the implementation of the second processing module 720, the third processing module 730, and the fourth processing module 740 can refer to the implementation of the first processing module 710.

[0123] As an example of a software functional module, the first processing module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first processing module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0124] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0125] As an example of a hardware functional module, the first processing module 710 may include at least one computing device, such as a server. Alternatively, the first processing module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0126] The multiple computing devices included in the first processing module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first processing module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first processing module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0127] It should be noted that, in other embodiments, the first processing module 710 can be used to perform, for example... Figure 2 In any step of the method shown, the second processing module 720 can be used to perform, as follows: Figure 2 In any step of the method shown, the third processing module 730 can be used to perform, as follows: Figure 2 In any step of the method shown, the fourth processing module 740 can be used to perform, as follows: Figure 2 In the method shown, any step implemented by the first processing module 710, the second processing module 720, the third processing module 730, and the fourth processing module 740 can be specified as needed. The steps implemented by the first processing module 710, the second processing module 720, the third processing module 730, and the fourth processing module 740 are respectively as follows: Figure 2 The different steps in the method shown achieve all the functions of device 700.

[0128] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 8As shown, the computing device 800 includes a bus 810, a processor 820, a memory 830, and a communication interface 840. The processor 820, memory 830, and communication interface 840 communicate with each other via the bus 810. The computing device 800 can be a server, computer, laptop, server rack, etc. It should be understood that this application does not limit the number of processors and memory in the computing device 800.

[0129] The bus 810 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 810 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 810 may include a path for transmitting information between various components of the computing device 800 (e.g., processor 820, memory 830, communication interface 840).

[0130] The processor 820 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0131] The memory 830 may include volatile memory, such as random access memory (RAM). The memory 830 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0132] The memory 830 stores executable program code, and the processor 820 executes this executable program code to implement the functions of the aforementioned multiple modules, such as the first processing module 710, the second processing module 720, the third processing module 730, and the fourth processing module 740, thereby achieving the following: Figure 2 The method shown. That is, the memory 830 stores the method for performing such... Figure 2 The instructions for the method shown.

[0133] Alternatively, the memory 830 stores executable code, and the processor 820 executes this executable code to implement the functions of the aforementioned modules, thereby achieving, for example... Figure 2 The method shown. That is, the memory 830 stores the method for performing such... Figure 2 The instructions for the method shown.

[0134] The communication interface 840 uses transceiver units such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0135] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0136] like Figure 9 As shown, the computing device cluster includes at least one computing device 800. The memory 830 of one or more computing devices 800 in the computing device cluster may store the same memory for performing tasks such as... Figure 2 The instructions for the method shown.

[0137] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store memory for performing tasks such as... Figure 2 The instructions of the method shown are partial instructions. In other words, a combination of one or more computing devices 800 can jointly execute instructions for performing tasks such as... Figure 2 The instructions for the method shown.

[0138] It should be noted that the memories 830 in different computing devices 800 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the first processing module 710, the second processing module 720, the third processing module 730, and the fourth processing module 740, respectively. That is, the instructions stored in the memories 830 of different computing devices 800 can implement the functions of one or more of the first processing module 710, the second processing module 720, the third processing module 730, and the fourth processing module 740.

[0139] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 One possible implementation is shown. For example... Figure 10As shown, computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 830 in computing device 800A stores instructions for executing some of the functions of the first processing module 710 and the second processing module 720. Meanwhile, the memory 830 in computing device 800B stores instructions for executing other parts of the functions of the third processing module 730 and the fourth processing module 740.

[0140] Figure 10 The connection method between the computing device clusters shown can be based on the provisions of this application, such as... Figure 2 The method shown requires a large amount of data storage, so it is considered to delegate the functions of another part of the modules in the first processing module 710, the second processing module 720, the third processing module 730 and the fourth processing module 740 to the computing device 800B.

[0141] It should be understood that Figure 10 The functions of the computing device 800A shown can also be performed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be performed by multiple computing devices 800.

[0142] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster. The difference is that the memory 830 in one or more computing devices 800 within this computing device cluster can store the same memory for performing tasks such as... Figure 2 The instructions for the method shown.

[0143] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store memory for performing tasks such as... Figure 2 The instructions of the method shown are partial instructions. In other words, a combination of one or more computing devices 800 can jointly execute instructions for performing tasks such as... Figure 2 The instructions for the method shown.

[0144] It should be noted that the memory 830 in different computing devices 800 within the computing device cluster can store different instructions for executing some functions of the computing device 800. That is, the instructions stored in the memory 830 of different computing devices 800 can implement the functions of one or more of the aforementioned first processing module 710, second processing module 720, third processing module 730, and fourth processing module 740.

[0145] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform actions such as... Figure 2 The method shown.

[0146] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform actions such as... Figure 2 The method shown.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A constant encoding method, characterized in that, include: Obtain multiple pages and select one sampling page from the multiple pages; Based on the constant values ​​of at least one constant column in the sampling page, an encoding table for the at least one constant column is obtained; each encoding table is used to record the column number and constant value of a constant column; the constant column is a column in which the ratio between the number of bytes occupied by the constant value and the total number of bytes is greater than a set threshold. Based on the column number and constant value recorded in the encoding table of the at least one constant column, constant value identification is performed on the corresponding columns in other pages to determine each constant column in the other pages, wherein the other pages are pages other than the sampling page among the plurality of pages; The data of each constant column in the other pages are constant encoded to obtain constant metadata of each constant column in the other pages. Each constant metadata is used to record the data of a constant column.

2. The method according to claim 1, characterized in that, The constant element information includes an encoding method bit, a constant value bit, and a flag bit. The encoding method bit records the encoding type of the current column, the constant value bit records the constant value, and the flag bit records "1" or "0". "1" indicates that all the data in the column corresponding to the constant element information is a constant value, and "0" indicates that there is an abnormal value in the column corresponding to the constant element information.

3. The method according to claim 1 or 2, characterized in that, When the flag bit is recorded as "0", the constant element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.

4. The method according to any one of claims 1-3, characterized in that, After selecting a sampling page from the plurality of pages, the method further includes: The sampled page is split into multiple columns of data, and the constant values ​​in each column of data are determined; Determine whether the ratio between the number of bytes occupied by constant values ​​in each column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold; When the ratio between the number of bytes occupied by constant values ​​in a column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold, the constant column of the sampling page is determined.

5. The method according to claim 4, characterized in that, The sampling page is a row-based page, and the process of splitting the sampling page into multiple columns of data specifically includes: The data of the multiple columns are obtained by stacking the values ​​at the same position in each tuple of the sampled page into columns.

6. The method according to claim 4 or 5, characterized in that, The determination of constant values ​​in each column of data specifically includes: According to a preset ratio, a portion of the data is selected from the data in each column to obtain the sample data for each column; A hash algorithm is used to identify constant values ​​in the sample data of each column, thereby obtaining the constant values ​​in the data of each column.

7. The method according to any one of claims 1-6, characterized in that, The step of identifying constant values ​​in corresponding columns on other pages based on the column number and constant value recorded in the encoding table of the at least one constant column, and determining each constant column on the other pages, specifically includes: Based on the column number recorded in the first encoding table, data from the column with the same column number is selected from the first page; the at least one encoding table includes the first encoding table, and the other pages include the first page; The constant values ​​recorded in the first encoding table are compared with the values ​​in the data of the selected column on the first page to obtain the number of bytes occupied by the constant values ​​recorded in the first encoding table. Determine whether the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than the set threshold; If the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column in the first page is greater than the set threshold, the selected column in the first page is determined to be a constant column.

8. A constant encoding device, characterized in that, include: The first processing module is used to acquire multiple pages and select one sampling page from the multiple pages; The second processing module is used to obtain an encoding table for at least one constant column based on the constant value of at least one constant column in the sampling page; each encoding table is used to record the column number and constant value of a constant column; the constant column is a column in which the ratio between the number of bytes occupied by the constant value and the total number of bytes in a column of data is greater than a set threshold; The third processing module is used to identify the constant values ​​of the corresponding columns in other pages based on the column number and constant value recorded in the encoding table of the at least one constant column, and to determine each constant column in the other pages, wherein the other pages are pages other than the sampling page among the plurality of pages; The fourth processing module is used to perform constant encoding on the data of each constant column in the other pages to obtain constant element information of each constant column in the other pages. Each constant element information is used to record the data of a constant column.

9. The apparatus according to claim 8, characterized in that, The constant element information includes an encoding method bit, a constant value bit, and a flag bit. The encoding method bit records the encoding type of the current column, the constant value bit records the constant value, and the flag bit records "1" or "0". "1" indicates that all the data in the column corresponding to the constant element information is a constant value, and "0" indicates that there is an abnormal value in the column corresponding to the constant element information.

10. The apparatus according to claim 8 or 9, characterized in that, When the flag bit is recorded as "0", the constant element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.

11. The apparatus according to any one of claims 8-11, characterized in that, The first processing module, after selecting a sampling page from the plurality of pages, is further configured to... The sampled page is split into multiple columns of data, and the constant values ​​in each column of data are determined; Determine whether the ratio between the number of bytes occupied by constant values ​​in each column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold; When the ratio between the number of bytes occupied by constant values ​​in a column and the total number of bytes occupied by the data in the corresponding column is greater than the set threshold, the constant column of the sampling page is determined.

12. The apparatus according to claim 11, characterized in that, The first processing module, when the sampling page is a row-style page, is specifically used for The data of the multiple columns are obtained by stacking the values ​​at the same position in each tuple of the sampled page into columns.

13. The apparatus according to claim 11 or 12, characterized in that, The first processing module is specifically used for According to a preset ratio, a portion of the data is selected from the data in each column to obtain the sample data for each column; A hash algorithm is used to identify constant values ​​in the sample data of each column, thereby obtaining the constant values ​​in the data of each column.

14. The apparatus according to any one of claims 8-13, characterized in that, The third processing module is specifically used for Based on the column number recorded in the first encoding table, data from the column with the same column number is selected from the first page; the at least one encoding table includes the first encoding table, and the other pages include the first page; The constant values ​​recorded in the first encoding table are compared with the values ​​in the data of the selected column on the first page to obtain the number of bytes occupied by the constant values ​​recorded in the first encoding table. Determine whether the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than the set threshold; If the ratio between the number of bytes occupied by the constant values ​​recorded in the first encoding table and the total number of bytes occupied by the data of the selected column on the first page is greater than the set threshold, the selected column on the first page is determined to be a constant column.

15. A computing device cluster, characterized in that, include: At least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-7.

17. A computer program product containing instructions, characterized in that, The computer program product stores instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1-7.

Images