An equivalence encoding method, apparatus, and computing device cluster
By using the equivalent encoding method, which utilizes the similarity of columns in row-based pages for encoding, the problem of low compression efficiency in existing row-based pages is solved, achieving higher data storage and transmission efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2025-01-14
- Publication Date
- 2026-07-14
AI Technical Summary
Existing compression encoding methods fail to effectively utilize the similarity between columns in row-based pages, resulting in limited improvements in data storage and transmission efficiency.
The equivalent encoding method is adopted. By selecting a reference column and comparing it with other columns, it is determined whether the byte ratio is greater than the threshold. Equivalent encoding is then performed to generate equivalent element information. Furthermore, existing encoding techniques are combined to encode the unencoded columns.
It improves the compression rate of line-based pages and enhances data storage and transmission efficiency.
Smart Images
Figure CN122394561A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of encoding technology, and in particular to an equivalent encoding method, apparatus, and computing device cluster. Background Technology
[0002] With the proliferation of data-intensive applications and the surge in data volume, the storage and transmission costs of databases are constantly rising. To improve the efficiency of data storage and transmission, database compression encoding has become a key data optimization technique. This technique reduces the amount of data stored and transmitted by decreasing redundant information in the data. Existing compression encoding methods include dictionary encoding, prefix compression, variable-length integer encoding, bitmap compression, and character encoding, all of which effectively reduce data redundancy. However, for row-oriented storage pages, data not only exhibits similarity between rows but also contains similar or identical information in many columns. Currently used encoding methods do not encode these row-oriented pages column-wise. Summary of the Invention
[0003] To address the aforementioned problems, embodiments of this application provide an equivalent encoding method that can encode row-based pages in a column-wise manner. Furthermore, this application also provides an equivalent encoding apparatus and a computing device cluster corresponding to this equivalent encoding method.
[0004] Therefore, the following technical solutions are adopted in the embodiments of this application:
[0005] In a first aspect, this application provides an equivalent encoding method, comprising: obtaining a target page; the target page is a row-based page, the target page includes a first reference column, the first reference column being data from any column in the target page; obtaining the byte ratio of each column in the first column set based on the values of each column in the first column set and the values of each column in the first reference column, the byte ratio being the relationship between the number of bytes occupied by the value in each column that is identical to the value in the same position in the first reference column and the total number of bytes occupied by the values in the first reference column, the first column set being columns in the target page other than the first reference column; when the byte ratio of the columns in the first column set is greater than a set threshold, performing equivalent encoding on the values of each column in the first column set to obtain equivalent element information of the columns in the first column set, the equivalent element information being used to record the values of each column.
[0006] In this implementation, after acquiring a row-based page each time, the method selects a column from the page as a reference column. The method compares each value in the other columns of the page with the corresponding value in the reference column, determining whether the ratio of the number of bytes occupied by the same value in the two columns to the total number of bytes is greater than a set threshold. If the ratio is greater than the threshold, the method performs equal-value encoding on the data in that column, thereby encoding the row-based page according to its columns.
[0007] In one implementation, the equivalent element information includes an encoding method bit, a sequence number bit, and a flag bit. The encoding method bit records the encoding type of the current column, the sequence number bit records the sequence number of the first reference column, and the flag bit records "1" or "0". "1" indicates that all values in the column corresponding to the equivalent element information are the same as the values at the corresponding positions in the first reference column, and "0" indicates that all values in the column corresponding to the equivalent element information are not completely the same as the values at the corresponding positions in the first reference column.
[0008] In one implementation, when the flag bit is recorded as "0", the equivalent element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.
[0009] In this embodiment, the method can customize the format of the equivalent metadata, allowing the equivalent metadata to include encoding mode bits, sequence number bits, flag bits, outlier bitmap bits (optionally), and outlier bits (optionally), with the encoding mode bits occupying 1 byte and the flag bits occupying 1 bit, thereby minimizing the total number of bytes occupied by the equivalent metadata and thus improving the compression rate of equivalent encoding.
[0010] In one embodiment, the method further includes: when the number of different columns is greater than 1, selecting a second reference column from multiple different columns, wherein the different columns are columns in the first column set whose byte proportion is less than or equal to a set threshold; obtaining the byte proportion of each column in the second column set based on the values of each column in the second column set and the values of each second reference column, wherein the second column set consists of columns other than the second reference column from multiple different columns; and when the byte proportion of a column in the second column set is greater than the set threshold, performing equivalent encoding on each value of a column in the second column set to obtain equivalent metadata of the column in the second column set.
[0011] In this implementation, if the number of different columns is greater than one, the method can select a new reference column from among the multiple different columns, and then compare each value in the remaining different columns with the corresponding value in the new reference column. Through this continuous selection and comparison process, the method can perform equivalent encoding on as many column data in the row-based page as possible, thereby improving the compression rate of the equivalent encoding.
[0012] In one implementation, the method further includes encoding each value of each column in the target page, excluding the columns with equivalence encoding.
[0013] In this implementation, after completing the equivalence encoding, the method continues to encode the data in the reference columns that have not yet been encoded, as well as the data in different columns. At this point, the encoding method can be existing encoding techniques such as dictionary encoding, prefix compression, variable-length integer encoding, bitmap compression, and character encoding. By comprehensively encoding all column data of the target page, this method can effectively improve the compression ratio of the target page.
[0014] Secondly, this application provides an equivalent encoding device, comprising: a first processing module for acquiring a target page; the target page is a row-based page, the target page includes a first reference column, the first reference column being data from any column in the target page; a second processing module for obtaining the byte ratio of each column in the first column set based on the values of each column in the first column set and the values of each column in the first reference column, the byte ratio being the relationship between the number of bytes occupied by the value in each column that is identical to the value in the same position in the first reference column and the total number of bytes occupied by the values in the first reference column, the first column set being columns in the target page excluding the first reference column; and a third processing module for performing equivalent encoding on each value in the column of the first column set when the byte ratio of the column in the first column set is greater than a set threshold, to obtain equivalent element information of the column in the first column set, the equivalent element information being used to record each value in the column.
[0015] In one implementation, the equivalent element information includes an encoding method bit, a sequence number bit, and a flag bit. The encoding method bit records the encoding type of the current column, the sequence number bit records the sequence number of the first reference column, and the flag bit records "1" or "0". "1" indicates that all values in the column corresponding to the equivalent element information are the same as the values at the corresponding positions in the first reference column, and "0" indicates that all values in the column corresponding to the equivalent element information are not completely the same as the values at the corresponding positions in the first reference column.
[0016] In one implementation, when the flag bit is recorded as "0", the equivalent element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.
[0017] In one implementation, the third processing module is further configured to: select a second reference column from multiple different columns when the number of different columns is greater than 1, wherein the different columns are columns in the first column set whose byte proportion is less than or equal to a set threshold; obtain the byte proportion of each column in the second column set based on the values of each column in the second column set and the values of each reference column, wherein the second column set consists of columns other than the second reference column from multiple different columns; and when the byte proportion of a column in the second column set is greater than the set threshold, perform equivalent encoding on each value of a column in the second column set to obtain equivalent metadata of the column in the second column set.
[0018] In one implementation, the third processing module is further configured to encode each value of each column in the target page, excluding the columns with equal-value encoding.
[0019] Thirdly, embodiments of this application provide a computing device, including: at least one memory; and at least one processor, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the embodiments as described in the first aspect.
[0020] Fourthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device, cause the computing device to perform various possible implementations of the first aspect.
[0021] Fifthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, cause the computing device to implement various possible implementations of the first aspect.
[0022] In a sixth aspect, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the various possible implementations of the first aspect.
[0023] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the various possible implementations of the first aspect.
[0024] Eighthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement various possible implementations of the first aspect. Attached Figure Description
[0025] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.
[0026] Figure 1 This is a schematic diagram of the structure of a database storage system provided in an embodiment of this application;
[0027] Figure 2 This is a schematic diagram illustrating the process of generating equivalent element information by encoding a row-style page in a column-based manner, as provided in this embodiment of the application.
[0028] Figure 3 This is a flowchart illustrating an equivalent encoding method provided in an embodiment of this application;
[0029] Figure 4 This is a schematic diagram illustrating the process of the row storage engine providing this embodiment storing input data;
[0030] Figure 5 A schematic diagram illustrating the process of generating equivalent element information for the row-style page provided in this application embodiment;
[0031] Figure 6 This is a schematic diagram illustrating a user's use of the data processing system as provided in the embodiments of this application;
[0032] Figure 7 This is a schematic diagram of the structure of an equivalent encoding device provided in the embodiments of this application;
[0033] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0034] Figure 9 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application;
[0035] Figure 10 This is a schematic diagram of another computing device cluster architecture provided in the embodiments of this application. Detailed Implementation
[0036] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0037] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.
[0038] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.
[0039] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0040] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.
[0041] Before introducing the technical solution protected by this application, several technical terms involved in the technical solution protected by this application will be explained in advance, namely:
[0042] A data system employing a storage engine refers to using a storage engine to manage the storage, indexing, and retrieval of data. Storage engines can be categorized into row-based storage engines and column-based storage engines. Row-based storage engines store data row-by-row, meaning each row of data is stored contiguously. This is suitable for operations requiring frequent full-row reads, such as online transaction processing (OLTP) systems. Column-based storage engines store data column-by-column, meaning data from the same column is stored together. This is suitable for scenarios requiring extensive analysis and aggregation operations on specific columns, such as online analytical processing (OLAP) systems.
[0043] A page is the basic unit used for data storage, management, and transmission in a storage system. Its size and purpose depend on the specific application scenario and technical implementation. In a row-oriented storage system, data is stored row by row, with each page containing multiple rows of data, and each row containing all column values for that row. In a column-oriented storage system, data is stored column by column, with each page containing one or more columns of row data, rather than a single row of multiple columns.
[0044] Next, the technical solution provided in this application will be introduced.
[0045] Generally, in row-based pages, the data in each column is usually related. Currently, compression techniques in related technologies first encode each row of data independently, and then further compress the encoded data using compression algorithms such as zstd and lz4. There are various compression encoding techniques used for row storage in databases, including dictionary encoding, prefix compression, variable-length integer encoding, bitmap compression, and character encoding, each optimized for specific data characteristics. For example, dictionary encoding is suitable for fields with many repeated values, while bitmap compression is suitable for Boolean or enumerated data types.
[0046] However, existing dictionary encoding techniques are generally not suitable for encoding entire columns of data in row-based pages; they are primarily used for encoding individual data items. They cannot effectively handle similar data and can only encode identical data items through index substitution. Variable-length integer encoding is mainly for integer data and also cannot encode entire columns of data. While bitmap encoding is effective, its advantage is limited to Boolean data; it is not suitable for other data types such as integers and strings. Therefore, compression techniques in related technologies cannot encode row-based pages column-by-column.
[0047] In view of this, embodiments of this application provide an equivalent encoding method. Each time a row-based page is acquired, a column can be selected from the page as a reference column. The method compares each value in other columns of the page with the corresponding value in the reference column, determining whether the byte ratio between the number of bytes occupied by the same value in the two columns and the total number of bytes is greater than a set threshold. If the byte ratio is greater than the set threshold, the method performs equivalent encoding on the data in that column, thereby enabling the row-based page to be encoded in a column-based manner.
[0048] Figure 1 This is a schematic diagram of the structure of a database storage system provided in an embodiment of this application. Figure 1 As shown, the database storage system 100 may include a data processing system 110, a memory 120, and a database 130.
[0049] When the database storage system 100 receives input data, this data is first preprocessed by the data processing system 110. During the preprocessing stage, the data processing system 110 encodes the input data using a row storage engine according to the encoding strategy of the table to be stored. After encoding, the data processing system 110 temporarily stores the encoded data in memory 120. Subsequently, the data processing system 110 stores this data in the database 130 in a format of 8192 bytes per page; this process is called "disk write".
[0050] When a user needs to access or manipulate this data, the data processing system 110 reads the data from the disk of the database 130 into the memory 120 and decodes it. Afterward, the data processing system 110 returns the decoded data to the upper-level scheduling engine for further processing.
[0051] In this embodiment, when the data processing system 110 performs equivalent encoding, after acquiring a row-style page each time, it can select a column from the page as a reference column. The data processing system 110 compares each value in the other columns of the page with the corresponding value in the reference column to determine whether the byte ratio between the number of bytes occupied by the same value in the two columns and the total number of bytes is greater than a set threshold. If the byte ratio is greater than the set threshold, the data processing system 110 performs equivalent encoding on the data in that column, thereby enabling the row-style page to be encoded in a column-based manner.
[0052] Equivalence encoding refers to encoding the data in the comparison column according to the data in the reference column to obtain equivalent metadata. In this embodiment, the format of the equivalent metadata can be customized. The equivalent metadata includes an encoding method (encode_type) bit, a sequence number bit, a flag (is_all_matched) bit, an outlier bitmap bit (optionally), and an outlier bit (optionally). The encoding method bit records the encoding type of the current column and generally occupies 1 byte. The sequence number bit records the sequence number of the reference column in the row format page and generally occupies 1 byte or 2 bytes. The flag bit records "1" or "0" and generally occupies 1 bit. "1" indicates that all non-NULL values in the current column are exactly the same as the values at the same position in the reference column, and there are no subsequent outlier bitmap bits or outlier bits. "0" indicates that all non-NULL values in the current column are not exactly the same as the values at the same position in the reference column, and there are subsequent outlier bitmap bits and outlier bits. The outlier bitmap bit records the row number corresponding to the outlier value and occupies the same number of bytes as the number of values in that column. The outlier field records abnormal values, and the number of bytes it occupies is determined by the column's attributes. An outlier is a value in this column that is different from the value at the same position in the reference column.
[0053] In this embodiment of the application, the data processing system 110 can customize the format of the equivalent element information, so that the equivalent element information includes the encoding method bit, the sequence number bit, the flag bit, the outlier bit map bit (optionally), and the outlier bit (optionally), and the encoding method bit occupies 1 byte and the flag bit occupies 1 bit, so as to minimize the total number of bytes occupied by the equivalent element information, thereby improving the compression rate of the equivalent encoding.
[0054] The data processing system 110 can store an encoding list, which records the correspondence between various encoding algorithms and flag bits. For example, the flag bit for equivalent encoding is 1, the flag bit for constant encoding is 2, and the flag bit for difference encoding is 3, etc. When recording the encoding type, the data processing system 110 can set a flag bit in the encoding method bit of the equivalent element information to identify the encoding type. This ensures that the encoding method bit only occupies 1 byte, thereby effectively reducing the number of bytes required to store the equivalent element information.
[0055] For example, such as Figure 2 As shown, the row-based page contains four rows of data, each containing four values. Assuming the data processing system 110 selects the data in the first column (col 1) as the reference column, then compared to the data in the first column, the data in the second column (col 2) are equal, the data in the third column (col 3) are similar, and the data in the fourth column (col 4) are different.
[0056] The data processing system 110 generates equivalent metadata for the second column of data based on the data in the first column. This metadata includes an encoding method bit, a sequence number bit, and a flag bit. The encoding method bit records the column encoding type and occupies 1 byte. The sequence number bit records the sequence number of the first column as "1" and occupies 1 byte or 2 bytes. The flag bit records "1" and occupies 1 bit.
[0057] The data processing system 110 generates equivalent metadata for the third column based on the data in the first column. This metadata includes encoding method bits, sequence number bits, flag bits, outlier bitmap bits, and outlier bit bits. The encoding method bit records the column encoding type and occupies 1 byte. The sequence number bit records the sequence number of the first column as "1", occupying 1 or 2 bytes. The flag bit records "0", occupying 1 bit. The outlier bitmap bits record "1", "1", "0", and "1", totaling 4 bits and occupying N / 8 = 4 / 8 = 1 byte. Here, "1" indicates that the value in a row of the third column is the same as the value at the same position in the first column, and "0" indicates that the value in a row of the third column is different from the value at the same position in the first column. The outlier bit records the outlier value as "ccc", occupying the corresponding column attribute length (if the column attribute is a fixed-length field, it's the column attribute byte length; if the column attribute is a variable-length field, it's the actual byte length of the value).
[0058] Figure 3 This is a flowchart illustrating an equivalence method provided in an embodiment of this application. Figure 3 As shown, this method can be executed by the aforementioned data processing system 110, and the specific implementation process is as follows:
[0059] Step S301: Obtain the target page and extract a column of data from the target page as a reference column.
[0060] Specifically, the data processing system 110 first creates a new table and defines a compression strategy for it. The compression strategy can be row-based compression. Then, after receiving input data, the data processing system 110 can write the input data into the new table. When the data processing system 110 writes the data in the new table to disk, it can first analyze the data in the table to determine which data is frequently accessed "hot data" and which data is accessed less frequently as "cold data." The data processing system 110 can encode the cold data according to the defined compression strategy and then write the encoded data to disk in the form of pages. During the data encoding process, the data processing system 110 continuously extracts multiple pages of data from the table containing cold data and encodes each page one by one.
[0061] The target page is a row-based page. A row-based page refers to a page stored using a row-based storage engine. For example, such as... Figure 4 As shown, when the row storage engine stores input data, it stores each value in the input data sequentially in a table, row by row. Data in the table structure is organized into multiple tuples. Each tuple includes four columns. When the data processing system 110 generates a page consisting of five tuples, it can select the data from the table identified as "tuple 1", "tuple 2", "tuple 3", "tuple 6", and "tuple 8" to form a row-based page.
[0062] Before encoding the row-based page, the data processing system 110 can stack the values at the same position in each tuple of the sampled page into columns to obtain multiple columns of data. For example, the target page includes three tuples: "tuple 1", "tuple 2", and "tuple 3", each tuple containing three values. The data processing system 110 can stack the values in the first column of "tuple 1", "tuple 2", and "tuple 3" into columns to obtain the first column of data; stack the values in the second column of "tuple 1", "tuple 2", and "tuple 3" into columns to obtain the second column of data; and stack the values in the third column of "tuple 1", "tuple 2", and "tuple 3" into columns to obtain the third column of data.
[0063] After obtaining data from multiple columns, the data processing system 110 can randomly select one column as a reference column. Optionally, the data processing system 110 can use the first column among the multiple columns as the reference column.
[0064] Step S302: Based on the values of each column in the other columns and the values of the first reference column, obtain the byte ratio of each column in the other columns.
[0065] Step S303: When the byte ratio of other columns is greater than a set threshold, perform equal-value encoding on each value of other columns to obtain the equal-value element information of other columns.
[0066] Specifically, when performing numerical comparison, the data processing system 110 compares each value in each column of other columns with the corresponding value in the reference column. After the comparison is complete, the data processing system 110 first calculates the number of bytes occupied by each identical value in each of the other columns, and the number of bytes occupied by each value in each of the other columns. Then, based on the number of bytes occupied by each identical value in each of the other columns, the data processing system 110 calculates the total number of bytes occupied by each identical value in each of the other columns. Next, based on the relationship between the number of bytes occupied by each value in each of the other columns and the total number of bytes, the data processing system 110 obtains the proportion of bytes occupied by identical values. Finally, the data processing system 110 determines the relationship between the proportion of bytes occupied by identical values in each of the other columns and a set threshold.
[0067] If this byte ratio is less than a set threshold, the data processing system 110 can consider that the data in the current comparison column differs significantly from the data in the reference column. In this case, the data processing system 110 can treat the currently compared column as a different column.
[0068] If the proportion of this byte is greater than a set threshold, the data processing system 110 can consider that the data in the current comparison column is relatively close to or completely identical to the data in the reference column. At this time, the data processing system 110 can further determine whether the proportion of this byte is 100%. If the proportion of this byte is greater than the set threshold but less than 100%, the data processing system 110 can treat the current comparison column as a similar column, and generate equivalent metadata based on the data in the current comparison column, including encoding method bits, sequence number bits, flag bits, outlier bitmap bits, and outlier bits, and then store the equivalent metadata in memory 120.
[0069] If the byte ratio is equal to 100%, the data processing system 110 can take the currently compared column as the equal column, and generate equivalent element information including encoding mode bit, sequence number bit and flag bit based on the data of the currently compared column, and then store the equivalent element information in memory 120.
[0070] For example, with Figure 4 The example shown is a line-style page. Figure 5 As shown, the first column in this row-style page is the reference column. The data processing system 110 compares the data in column "col 1" and column "col 2". If the proportion of bytes occupied by the same value in column "col 2" is greater than a set threshold and this proportion is 100%, then column "col 2" can be considered an equal column. At this time, the data processing system 110 can generate equivalent metadata based on the data in column "col 2", which includes: encoding method bit, sequence number bit (recorded as "1"), and flag bit (recorded as "1").
[0071] Data processing system 110 compares the data in columns "col 1" and "col 3". If the proportion of bytes occupied by the same value in column "col 3" is greater than a set threshold and this proportion is less than 100%, then column "col 3" can be considered a similar column. At this time, data processing system 110 can generate equivalent metadata based on the data in column "col 3", which includes: encoding method bit, sequence number bit (recorded as "1"), flag bit (recorded as "0"), outlier bitmap bit (recorded as "1", "1", "0", "1" and "1"), and outlier bit (recorded as "ccc").
[0072] The data processing system 110 compares the data in columns "col 1" and "col 4" and finds that the proportion of bytes occupied by the same value in column "col 3" is less than a set threshold, so column "col 4" can be considered as a different column.
[0073] After comparing the data in other columns, the data processing system 110 counts the number of different columns. If the number of different columns is greater than one, the data processing system 110 can select a new column from the multiple different columns as a new reference column, and compare each value in the remaining different columns with the corresponding value in the new reference column. The specific comparison process can be found in steps S202-S203 above. Through this continuous selection and comparison process, the data processing system 110 can perform equivalent encoding on as many column data in the row-based page as possible, thereby improving the compression rate of the equivalent encoding.
[0074] After completing the equivalence encoding, the data processing system 110 will continue to encode the data in the reference columns and the data in different columns that have not yet been encoded. At this point, the encoding method can be existing encoding techniques such as dictionary encoding, prefix compression, variable-length integer encoding, bitmap compression, and character encoding. By comprehensively encoding all column data of the target page, the data processing system 110 can effectively improve the compression ratio of the target page.
[0075] After encoding the target page, the data processing system 110 writes the encoded data of the target page stored in memory 120 to the database 130, thus achieving persistent storage of the set page data. Subsequently, the data processing system 110 selects page data again from the table containing cold data and continues to perform the equivalent encoding operation. This process continues until all pages containing cold data in the table have undergone equivalent encoding.
[0076] When reading data, the data processing system 110 can read a set number of page data from the database 130 and store them in memory 120. The data processing system 110 can first retrieve the encoded reference column data from memory 120, and then decode the encoded reference column data to restore the reference column data. After completing the data decoding operation of the reference column, the data processing system 110 retrieves the equivalent element information from memory 120, and then decodes the equivalent element information to restore the data of that column. After completing the decoding operation, the data processing system 110 restores the set number of page data into a table, and then returns the table to the upper-level scheduling engine for further operations.
[0077] In this embodiment, after acquiring a row-style page, the data processing system 110 can select a column from the page as a reference column. The data processing system 110 compares the values in the other columns of the page with the corresponding values in the reference column to determine whether the byte ratio between the number of bytes occupied by the same value in the two columns and the total number of bytes is greater than a set threshold. If the byte ratio is greater than the set threshold, the data processing system 110 performs equivalent encoding on the data in that column, thereby enabling the row-style page to be encoded in a column-based manner.
[0078] The above is a description of the data processing system 110 provided in this application embodiment. It is understood that the data processing system 110 can be configured on a cloud management platform, for example, deployed on at least one virtual machine or container instance, so that the cloud management platform can provide data processing services. Of course, the data processing system 110 can also be configured on nodes other than the cloud management platform, for example, deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud management platform can provide pages related to public cloud services for users to remotely access public cloud services. In this embodiment, users can pre-purchase the data processing services provided by the data processing system 110 on the cloud management platform. For ease of understanding, the interaction between the user and the cloud management platform is described below.
[0079] like Figure 6As shown, the interaction between the user and the cloud management platform mainly includes: the user logs into the cloud service platform 600 through a web page on the client, selects and purchases cloud services (i.e., data processing services) related to the data processing system 110 on the cloud service platform 600. After purchase, the user can generate the data processing system 110 on the cloud service platform 600 based on the functions provided by the data processing service. The cloud service platform 600 is primarily used to manage the infrastructure for running the data processing service. For example, the infrastructure for the data processing service may include multiple data centers located in different regions, each data center including multiple servers. The data centers can provide basic resources for the data processing service, such as computing resources and storage resources. Therefore, when purchasing and using the data processing service, the user mainly pays for the resources used. When using the data processing service, the user can input their needs for the data processing service through the configuration interface, application programming interface (API), or user interaction interface provided by the cloud service platform 600. The cloud service platform 600 can then generate a data processing service matching the user's (or other software / hardware, etc.) input requirements.
[0080] Alternatively, some modules in the data processing system 110 can be configured on the cloud side and others on the edge side, thereby realizing data processing services through edge-cloud collaboration. Furthermore, the data processing system 110 can also be entirely configured on the edge side, depending on the actual situation; no specific limitation is made here.
[0081] Based on the above description, this application provides a constant encoding device 700. For example... Figure 7 As shown, the device 700 includes:
[0082] The first processing module 710 is used to obtain the target page; the target page is a row-based page, and the target page includes a first reference column, which is the data of any column in the target page; the second processing module 720 is used to obtain the byte ratio of each column in the first column set according to the values of each column in the first column set and the values of each column in the first reference column. The byte ratio is the relationship between the number of bytes occupied by the value in each column that is the same as the value in the same position in the first reference column and the total number of bytes occupied by each value in the first reference column. The first column set is the columns in the target page other than the first reference column; the third processing module 730 is used to perform equivalent encoding on each value in the column of the first column set when the byte ratio of the column in the first column set is greater than a set threshold, to obtain the equivalent element information of the column in the first column set. The equivalent element information is used to record each value in the column.
[0083] In one implementation, the equivalent element information includes an encoding method bit, a sequence number bit, and a flag bit. The encoding method bit records the encoding type of the current column, the sequence number bit records the sequence number of the first reference column, and the flag bit records "1" or "0". "1" indicates that all values in the column corresponding to the equivalent element information are the same as the values at the corresponding positions in the first reference column, and "0" indicates that all values in the column corresponding to the equivalent element information are not completely the same as the values at the corresponding positions in the first reference column.
[0084] In one implementation, when the flag bit is recorded as "0", the equivalent element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.
[0085] In one embodiment, the third processing module 730 is further configured to: select a second reference column from multiple different columns when the number of different columns is greater than 1; the different columns are columns in the first column set whose byte proportion is less than or equal to a set threshold; obtain the byte proportion of each column in the second column set based on the values of each column in the second column set and the values of each reference column; the second column set consists of columns other than the second reference column among multiple different columns; and when the byte proportion of a column in the second column set is greater than the set threshold, perform equivalent encoding on each value of a column in the second column set to obtain equivalent element information of the column in the second column set.
[0086] In one implementation, the third processing module 730 is further configured to encode each value of each column in the target page, excluding the columns with equal value encoding.
[0087] The first processing module 710, the second processing module 720, and the third processing module 730 can all be implemented in software or in hardware. For example, the implementation of the first processing module 710 will be described below. Similarly, the implementation of the second processing module 720 and the third processing module 730 can refer to the implementation of the first processing module 710.
[0088] As an example of a software functional module, the first processing module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first processing module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0089] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0090] As an example of a hardware functional module, the first processing module 710 may include at least one computing device, such as a server. Alternatively, the first processing module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0091] The multiple computing devices included in the first processing module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first processing module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first processing module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0092] It should be noted that, in other embodiments, the first processing module 710 can be used to perform, for example... Figure 3 In any step of the method shown, the second processing module 720 can be used to perform, as follows: Figure 3 In any step of the method shown, the third processing module 730 can be used to perform, as follows: Figure 3 Any step in the method shown, implemented by the first processing module 710, the second processing module 720, and the third processing module 730, can be specified as needed. The steps implemented by the first processing module 710, the second processing module 720, and the third processing module 730 are respectively as follows: Figure 3 The different steps in the method shown achieve all the functions of device 700.
[0093] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 8 As shown, the computing device 800 includes a bus 810, a processor 820, a memory 830, and a communication interface 840. The processor 820, memory 830, and communication interface 840 communicate with each other via the bus 810. The computing device 800 can be a server, computer, laptop, server rack, etc. It should be understood that this application does not limit the number of processors and memory in the computing device 800.
[0094] The bus 810 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 810 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 810 may include a path for transmitting information between various components of the computing device 800 (e.g., processor 820, memory 830, communication interface 840).
[0095] The processor 820 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).
[0096] The memory 830 may include volatile memory, such as random access memory (RAM). The memory 830 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0097] The memory 830 stores executable program code, and the processor 820 executes this executable program code to implement the functions of the aforementioned multiple modules, such as the first processing module 710, the second processing module 720, and the third processing module 730, thereby achieving the following: Figure 3 The method shown. That is, the memory 830 stores the method for performing such... Figure 3 The instructions for the method shown.
[0098] Alternatively, the memory 830 stores executable code, and the processor 820 executes this executable code to implement the functions of the aforementioned modules, thereby achieving, for example... Figure 3 The method shown. That is, the memory 830 stores the method for performing such... Figure 3 The instructions for the method shown.
[0099] The communication interface 840 uses transceiver units such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.
[0100] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0101] like Figure 9As shown, the computing device cluster includes at least one computing device 800. The memory 830 of one or more computing devices 800 in the computing device cluster may store the same memory for performing tasks such as... Figure 3 The instructions for the method shown.
[0102] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store memory for performing tasks such as... Figure 3 The instructions of the method shown are partial instructions. In other words, a combination of one or more computing devices 800 can jointly execute instructions for performing tasks such as... Figure 3 The instructions for the method shown.
[0103] It should be noted that the memories 830 in different computing devices 800 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the first processing module 710, the second processing module 720, and the third processing module 730 described above. That is, the instructions stored in the memories 830 of different computing devices 800 can implement the functions of one or more of the first processing module 710, the second processing module 720, and the third processing module 730 described above.
[0104] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 One possible implementation is shown. For example... Figure 10 As shown, computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 830 in computing device 800A stores instructions for executing some of the functions of the first processing module 710 and the second processing module 720. Meanwhile, the memory 830 in computing device 800B stores instructions for executing another portion of the functions of the third processing module 730.
[0105] Figure 10 The connection method between the computing device clusters shown can be based on the provisions of this application, such as... Figure 3 The method shown requires a large amount of data storage, so it is considered to delegate the functions of another part of the modules in the first processing module 710, the second processing module 720 and the third processing module 730 to the computing device 800B.
[0106] It should be understood that Figure 10The functions of the computing device 800A shown can also be performed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be performed by multiple computing devices 800.
[0107] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster. The difference is that the memory 830 in one or more computing devices 800 within this computing device cluster can store the same memory for performing tasks such as... Figure 3 The instructions for the method shown.
[0108] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store memory for performing tasks such as... Figure 3 The instructions of the method shown are partial instructions. In other words, a combination of one or more computing devices 800 can jointly execute instructions for performing tasks such as... Figure 3 The instructions for the method shown.
[0109] It should be noted that the memory 830 in different computing devices 800 within the computing device cluster can store different instructions to execute some functions of the computing device 800. That is, the instructions stored in the memory 830 of different computing devices 800 can implement the functions of one or more of the first processing module 710, the second processing module 720, and the third processing module 730 mentioned above.
[0110] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform actions such as... Figure 3 The method shown.
[0111] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform actions such as... Figure 3 The method shown.
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. An equivalent encoding method, characterized in that, include: Obtain the target page; the target page is a row-based page, and the target page includes a first reference column, the first reference column being data from any column in the target page; Based on the values of each column in the first column set and the values of the first reference column, the byte ratio of each column in the first column set is obtained. The byte ratio is the relationship between the number of bytes occupied by the value in each column that is the same as the value in the same position in the first reference column and the total number of bytes occupied by each value in the first reference column. The first column set is the columns in the target page other than the first reference column. When the proportion of bytes in the first column set is greater than a set threshold, the values of each column in the first column set are encoden using equivalent values to obtain the equivalent element information of the column in the first column set. The equivalent element information is used to record each value of a column.
2. The method according to claim 1, characterized in that, The equivalent element information includes an encoding method bit, a sequence number bit, and a flag bit. The encoding method bit records the encoding type of the current column, the sequence number bit records the sequence number of the first reference column, and the flag bit records "1" or "0". "1" indicates that all values in the column corresponding to the equivalent element information are the same as the values at the corresponding positions in the first reference column, and "0" indicates that the values in the column corresponding to the equivalent element information are not completely the same as the values at the corresponding positions in the first reference column.
3. The method according to claim 2, characterized in that, When the flag bit is recorded as "0", the equivalent element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.
4. The method according to any one of claims 1-3, characterized in that, The method further includes: When the number of different columns is greater than 1, a second reference column is selected from multiple different columns, wherein the different columns are columns in the first column set whose byte proportion is less than or equal to the set threshold; Based on the values of each column in the second column set and the values of the second reference column, the byte ratio of each column in the second column set is obtained. The second column set consists of columns other than the second reference column among the plurality of different columns. When the proportion of bytes in the column of the second column set is greater than the set threshold, the values of each column in the second column set are encoden to obtain the equivalent element information of the column in the second column set.
5. The method according to any one of claims 1-4, characterized in that, The method further includes: Encode each value in each column of the target page, except for the columns with equal value encoding.
6. An equivalent encoding device, characterized in that, include: The first processing module is used to obtain the target page; The target page is a row-based page, and the target page includes a first reference column, which is data from any column in the target page. The second processing module is used to obtain the byte ratio of each column in the first column set based on the values of each column in the first column set and the values of the first reference column. The byte ratio is the relationship between the number of bytes occupied by the value in each column that is the same as the value in the same position in the first reference column and the total number of bytes occupied by the values in the first reference column. The first column set is the columns in the target page other than the first reference column. The third processing module is used to encode each value of the column in the first column set into equivalent values when the byte ratio of the column in the first column set is greater than a set threshold, so as to obtain the equivalent element information of the column in the first column set. The equivalent element information is used to record each value of a column.
7. The apparatus according to claim 6, characterized in that, The equivalent element information includes an encoding method bit, a sequence number bit, and a flag bit. The encoding method bit records the encoding type of the current column, the sequence number bit records the sequence number of the first reference column, and the flag bit records "1" or "0". "1" indicates that all values in the column corresponding to the equivalent element information are the same as the values at the corresponding positions in the first reference column, and "0" indicates that the values in the column corresponding to the equivalent element information are not completely the same as the values at the corresponding positions in the first reference column.
8. The apparatus according to claim 7, characterized in that, When the flag bit is recorded as "0", the equivalent element information also includes an outlier bitmap and an outlier bit. The outlier bitmap records the row number corresponding to the outlier, and the outlier bit records the outlier.
9. The apparatus according to any one of claims 6-8, characterized in that, The third processing module is also used for When the number of different columns is greater than 1, a second reference column is selected from multiple different columns, wherein the different columns are columns in the first column set whose byte proportion is less than or equal to the set threshold; Based on the values of each column in the second column set and the values of the second reference column, the byte ratio of each column in the second column set is obtained. The second column set consists of columns other than the second reference column among the plurality of different columns. When the proportion of bytes in the column of the second column set is greater than the set threshold, the values of each column in the second column set are encoden to obtain the equivalent element information of the column in the second column set.
10. The apparatus according to any one of claims 6-9, characterized in that, The third processing module is also used for Encode each value in each column of the target page, except for the columns with equal value encoding.
11. A computing device cluster, characterized in that, include: At least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-5.
12. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-5.
13. A computer program product containing instructions, characterized in that, The computer program product stores instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1-5.