A data encoding method and a method for comparing encoded data

Abstract

This specification provides a data encoding method and an encoded data comparison method, which determines the number of bytes of the payload in the data to be encoded; the payload is specified data that can characterize the size of the data to be encoded; an index byte is generated; the index byte includes at least: length data, which is used to characterize the number of bytes of the payload; the generated index byte is combined with the payload of the data to be encoded, and the combined data is used as the encoding result of the data to be encoded. Since the number of bytes in the payload is related to the size of a data, by increasing the index byte to record the number of bytes of the data payload, compression is achieved while ensuring that the compressed data can participate in sorting and can be compared with other compressed data.

Description

Technical Field

[0001] This specification relates to the field of computer application technology, and in particular to a data encoding method and a method for comparing encoded data. Background Technology

[0002] Computer devices typically store all data in fixed-length data types (such as int64 or int32) by default. This can lead to a waste of storage resources or data transmission bandwidth for smaller datasets.

[0003] To address the aforementioned issues, related technologies typically employ data compression by altering the data encoding method. However, the encoding methods in these technologies prevent comparisons and sorting of multiple data sets, causing significant inconvenience in scenarios requiring sorting or comparison. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data encoding method and a method for comparing encoded data.

[0005] According to a first aspect of one or more embodiments of this specification, a data encoding method is provided, comprising:

[0006] Determine the number of bytes of payload in the data to be encoded; the payload is specified data that can characterize the size of the data to be encoded.

[0007] Generate index bytes; the index bytes include at least: length data, the length data being used to characterize the number of bytes in the payload;

[0008] The generated index byte is combined with the payload of the data to be encoded, and the combined data is used as the encoding result of the data to be encoded.

[0009] According to a second aspect of one or more embodiments of this specification, a method for comparing encoded data is provided, comprising:

[0010] Two encoded data sets to be compared are obtained; the encoded data is obtained by encoding using the data encoding method described above.

[0011] The length of each encoded data is determined based on the index byte of each encoded data.

[0012] The two encoded data are compared based on their lengths.

[0013] According to a third aspect of the embodiments of this specification, a data encoding apparatus is provided, comprising:

[0014] A byte count determination module is used to determine the number of bytes of the payload in the data to be encoded; the payload is specified data that can characterize the size of the data to be encoded.

[0015] An index byte generation module is used to generate index bytes; the index bytes include at least: length data, which is used to characterize the number of bytes in the payload;

[0016] The combination module is used to combine the generated index byte with the payload of the data to be encoded, and use the combined data as the encoding result of the data to be encoded.

[0017] According to a fourth aspect of the embodiments of this specification, an encoded data comparison apparatus is provided, comprising:

[0018] The encoded data acquisition module is used to acquire two encoded data to be compared; the encoded data is obtained by encoding using the data encoding method described above.

[0019] The length data determination module is used to determine the length of each encoded data based on the index byte of each encoded data.

[0020] The comparison module is used to compare the size of two encoded data based on their length data.

[0021] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processor, implement the data encoding method or data decoding method described above.

[0022] According to a sixth aspect of the embodiments of this specification, a computer device is provided, the computer device comprising:

[0023] processor;

[0024] Memory used to store processor-executable instructions;

[0025] The processor implements the data encoding or data decoding method described above by running the executable instructions.

[0026] According to a sixth aspect of the embodiments of this specification, a computer program is provided, which, when executed by a processor, implements the above-described data encoding method or data decoding method.

[0027] According to a seventh aspect of the embodiments of this specification, a database system is provided in which data is encoded using the data encoding method described above; the database system sorts the data using the encoded data comparison method described above to obtain a sorting result.

[0028] When the database system receives a user's data query request, it responds to the user's data query request based on the sorting results.

[0029] This specification provides a data encoding method and an encoded data comparison method, which determines the number of bytes of the payload in the data to be encoded; the payload is specified data that can characterize the size of the data to be encoded; an index byte is generated; the index byte includes at least: length data, the length data being used to characterize the number of bytes of the payload; the generated index byte is combined with the payload of the data to be encoded, and the combined data is used as the encoding result of the data to be encoded.

[0030] Since the number of bytes in the payload is related to the size of a single data item, the number of bytes in the payload is recorded by adding index bytes. If two data items have different index byte sizes, their size can be determined by comparing only their index bytes. If the index byte sizes are the same, it means their payload portions are the same length, allowing for further comparison of their payload portions and thus their overall size. In this way, compression is achieved while ensuring that the compressed data can be sorted and compared with other compressed data.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0033] Figure 1 This is a schematic diagram illustrating the encoding range of VarInt according to an exemplary embodiment of this specification.

[0034] Figure 2 This is a flowchart illustrating a data encoding method according to an exemplary embodiment of this specification.

[0035] Figure 3 This is a schematic diagram illustrating a binary data storage method according to an exemplary embodiment of this specification.

[0036] Figure 4 This is a schematic diagram illustrating encoded data according to a specific embodiment of this specification.

[0037] Figure 5 This is a flowchart illustrating a data decoding method according to an exemplary embodiment of this specification.

[0038] Figure 6 This is a flowchart illustrating an coded data comparison method according to an exemplary embodiment of this specification.

[0039] Figure 7 This is a block diagram illustrating a data encoding apparatus according to an exemplary embodiment of this specification.

[0040] Figure 8 This is a block diagram illustrating an coded data comparison apparatus according to an exemplary embodiment of this specification.

[0041] Figure 9 This specification is a hardware structure diagram of a computer device containing a data encoding device or a data decoding device according to an exemplary embodiment. Detailed Implementation

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0043] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0044] Computer devices typically store data using fixed-length data types. For example, for integers, the data types can be int64 (8 bytes, 64 bits), int32 (4 bytes, 32 bits), or int16 (2 bytes, 16 bits). Since computers are designed to only compare or perform addition and subtraction operations on two data points of the same length, this storage method ensures efficient processing.

[0045] To save storage space and transmission bandwidth, data compression is commonly used in related technologies. For integers, compression is typically achieved using variable-length quantity (VLQ) encoding methods, such as VarInt and LEB128.

[0046] The VLQ approach works by recognizing that not all integer payloads occupy all the bytes of a given data type. For example, an integer of type int64 typically occupies 8 bytes, but when storing the integer 100, only one byte is needed to represent its payload, making the remaining seven bytes unnecessary. Furthermore, smaller integers generally appear more frequently in databases than larger ones; therefore, VLQ compression can save space occupied by integers.

[0047] It should be noted that VLQ is called an encoding method rather than a compression method because there are many compression methods, and achieving compression by changing the encoding method is just one type of compression method. The VLQ methods described in this specification and the methods in the embodiments of the following text (which can also be considered VLQ methods) achieve compression by changing the encoding of integers. This encoding method achieves overall compression by encoding a large amount of data with a payload of less than 8 bytes (taking int64 as an example). It may not be effective for individual integers. For example, an integer that already occupies 8 bytes (64 bits) may become larger using VLQ. It should also be noted that the encoding methods mentioned in this specification are all binary encoding methods.

[0048] The encoding method of VarInt in related technologies will be explained next. Compared with uncompressed methods, an important problem that VLQ and similar methods need to solve is that, since different integers occupy different numbers of bytes, the encoding method needs to be set so that the computer can know the number of bytes occupied by each integer.

[0049] To address the aforementioned issues, VarInt sets the first bit of each byte as a connection bit. If the connection bit is 1, it indicates that the byte containing the connection bit is not the last byte of the integer, and reading the integer still requires reading the next byte; if the connection bit is 0, it indicates that the byte containing the connection bit is the last byte of the integer, and reading the integer does not require reading the next byte. After setting the connection bit, for each integer, multiple bits corresponding to its payload are extracted and grouped into sets of seven bits, starting from the least significant bit (the last bit). The number of groups corresponding to the payload bits is determined (if the last most significant bit is insufficient for a group, zeros are added before the most significant bit). Then, based on the grouping, the connection bit is set to generate the compressed integer. The encoding range of VarInt is as follows: Figure 1 As shown.

[0050] For example, the decimal number 127, before compression, has the binary byte encoding of an int16 type as 0000000001111111, and after VarInt compression, the byte encoding is as follows: 0 1111111, the leftmost consecutive 0 bits. The decimal number 128, before compression, is encoded as 00000000 10000000 in binary (int16 type). After compression, the byte encoding is... 1 0000001 0 0000000, the two underlined bits (i.e., the leftmost bits of each of the two bytes) are consecutive bits.

[0051] Other compression methods for VLQ in related technologies are similar and will not be elaborated here.

[0052] The core idea of ​​the compression method described above is to remove bytes from the data that are useless in representing its size, and only retain the useful bytes. The example above uses integers as an example; it should be noted that the compression method using this idea can also be applied to other data that may include many bytes that are useless in representing its size.

[0053] Databases typically store large amounts of data, creating a need for data compression. Furthermore, to more easily retrieve a specific piece of data from the database, the data needs to be sorted first (the reason for this sorting is that without sorting, the user-input search criteria must be compared with each piece of data; sorting allows for a more efficient binary search). However, VLQ-encoded data in related technologies cannot be directly compared or sorted.

[0054] In the methods described above, the compressed data may not be of the same length, making direct comparison or sorting impossible using the encoding methods employed in related technologies. If comparison is required, the compressed data must be decompressed. Decompression consumes significant resources, and comparing or sorting after decompression will further increase resource consumption and reduce the efficiency of the comparison or sorting process.

[0055] To enable sorting or comparison of compressed data with minimal resource consumption, considering that for some types of data, the size of two data items is related to the number of bytes, the length of the data payload (i.e., the number of bytes included in the payload) can be stored in the header of each compressed (i.e., encoded, hereinafter the same) data (i.e., the position that the computer will read first when reading the data). In this way, when the number of bytes in the payloads of two data items are different, the size of the two data items can be compared by only comparing the length of the payload stored in the header; when the length of the payloads of two data items is the same, the size of the two data items can be further compared by reading the payload bytes.

[0056] In addition, when reading data, the length of the payload can be determined based on the length data recorded in the data header (hereinafter referred to as the index byte), thereby determining the position of the boundary between data.

[0057] It should be noted that while knowing the lengths of two data payloads allows for direct comparison, failing to record this information in the header and instead calculating the length of each payload only when sorting will consume significant processing resources and reduce sorting or comparison efficiency. Specifically, using the method provided in this manual only requires reading the index byte to compare or sort the two data items. However, without this method, reading all bytes of a single data item is necessary, which consumes computer resources and undoubtedly reduces processing efficiency.

[0058] This specification provides a data encoding method and an encoded data comparison method, which determines the number of bytes of the payload in the data to be encoded; the payload is specified data that can characterize the size of the data to be encoded; an index byte is generated; the index byte includes at least: length data, the length data being used to characterize the number of bytes of the payload; the generated index byte is combined with the payload of the data to be encoded, and the combined data is used as the encoding result of the data to be encoded.

[0059] Since the number of bytes in the payload is related to the size of a single data item, the number of bytes in the payload is recorded by adding index bytes. If two data items have different index byte sizes, their size can be determined by comparing only the two index bytes. If the index byte sizes are the same, it means their payload portions are the same length, and their sizes can be compared further. In this way, compression is achieved while ensuring that the compressed data can be sorted and compared with other compressed data.

[0060] The following section will describe one of the data encoding methods shown in this specification.

[0061] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a data encoding method according to an exemplary embodiment, comprising:

[0062] Step 201: Determine the number of bytes of the payload in the data to be encoded.

[0063] The payload is specified data that can characterize the size of the data to be encoded.

[0064] Specifically, in order for the encoded (i.e. compressed) data to be compared, it is first necessary to determine the length of the payload of each piece of data to be encoded, that is, to determine the number of bytes of the payload.

[0065] First, the execution subject of this method can be any computer device capable of storage and computation. It can also be performed by a system of multiple computers; this specification does not limit the execution subject of this method.

[0066] As mentioned above, the application scenarios for this method include situations where the database needs to save storage space and transmission bandwidth, and there is a need to compare and sort data in the same column. Therefore, the method described in this specification can be applied to databases. Of course, the method described in this specification can also be applied to other scenarios that require saving storage space and transmission bandwidth, and where there is a need to sort or compare compressed data. The above-mentioned application scenarios to databases do not represent a limitation of this specification.

[0067] Secondly, regarding the integers involved in this method, it should be noted that the data to be encoded must meet the following conditions in order for the encoded integers to be comparable and for the encoding to achieve the compression effect: First, the data to be encoded stored in a fixed-length data type must have a lot of leading zeros (or bytes that do not represent the actual size), otherwise the compression effect cannot be achieved; Second, the length of the payload of the data to be encoded stored in a fixed-length data type must be related to the size of the data, and the size of the two data can be determined by comparing the lengths of the payloads of the two data.

[0068] Data that meets the above conditions can be integers. Of course, it can also be other data that meets the above conditions; this specification does not limit the specific data type.

[0069] Next, we will explain the terms used in step 201.

[0070] The data to be encoded is the data to be encoded, that is, the data to be compressed using a fixed-length data type. The data to be encoded is the data stored in the computer device, and it can be any data type, such as int64 or other types of data. This specification does not limit the data type of the data to be encoded.

[0071] It should be noted that the bytes and bits involved in the encoding methods described in this specification are all bytes and bits that store data in binary form.

[0072] Payload bytes can be understood as the specified number of bytes required to represent the size of the data. For example, if an integer is 0082, then both 082 and 82 can be used as the payload of that data. In other words, the payload is the number of bytes that can represent the size of the data and does not exceed the space occupied by the data.

[0073] If the goal is to improve the compression ratio, then the payload can be the minimum number of bytes required to represent the size of the data.

[0074] When the payload is the minimum number of bytes required to represent the size of the data, for positive and unsigned data, the payload represents the bytes remaining after removing the leading zero bytes from the binary data to be encoded. These leading zero bytes represent bytes where all bits are leading zeros. It should be noted that in some cases, the data to be encoded does not include leading zero bytes; in such cases, the payload after removing the leading zero bytes is equivalent to all bytes of the data to be encoded.

[0075] For negative numbers, since computers store negative numbers in two's complement form, the negative number has its prefix removed (the 0 closest to the MSB is identified, and the 1s between the 0 and the MSB are the prefix).

[0076] Leading zeros are defined as follows: for a given data, starting from the most significant bit (MSB) and continuing until the first non-zero digit is encountered, the zeros between this non-zero digit and the MSB constitute the leading zeros. Leading zeros are also zeros that are not used to represent the size of the data, i.e., zeros before the payload. It should be noted that, as described above, the method in this specification is for binary data, and the leading zeros in step 201 are the leading zeros used to represent the binary data.

[0077] A leading zero byte is a byte in which all bits are stored with leading zeros. A byte consists of 8 bits, each of which stores a binary 0 or 1. In other words, a leading zero byte is the byte preceding the nearest byte containing a 1 (MSB).

[0078] The method to obtain the payload of the data to be encoded is to remove the prefix part of the data to be encoded (i.e., the leading zero byte or the negative prefix mentioned above) to obtain the payload.

[0079] Step 203: Generate index bytes.

[0080] The index byte includes at least: length data, which is used to characterize the number of bytes in the payload.

[0081] Specifically, to enable comparison of encoded bytes, index bytes are added for comparison and determining the boundaries between data. To achieve this, the index bytes include at least the length of the payload. This allows for size comparison and sorting (when two data items have different lengths) after reading the index bytes. Furthermore, reading the index bytes determines the length of the payload portion of the current data, thus identifying where the payload portion ends and distinguishing between different encoded data.

[0082] As mentioned above, the index byte (header byte) is the byte used for size comparison. Since the index byte records the length of the payload, and the length is determined based on the number of bytes in the payload, the longest data to be encoded generally does not exceed 8 bytes. One byte is sufficient to represent the decimal number 8, meaning the index byte is a single byte.

[0083] Regarding the order of index bytes and payload, if the position of each data index byte can be known in advance before reading the data (for example, the relevant information can be stored in the file header of the data storage file), then the index bytes in the encoded data can be before or after the payload.

[0084] If the position of the index byte for each data item is not pre-recorded, it is best to store the index byte before the payload. As mentioned earlier, the index byte stores the number of payload bytes. The length of the payload bytes needs to be determined by reading the index byte, thereby determining where the current byte ends.

[0085] The following section will explain the length data contained in the index byte.

[0086] As mentioned above, the length data is obtained based on the number of bytes in the payload. In other words, the length data can reflect the number of bytes in the payload. By comparing the index bytes of two data sets that have been encoded using the data encoding method provided in this specification, the number of bytes in the two data sets can be compared, thereby determining the size of the two data sets.

[0087] The following examples illustrate the specific format of length data. It should be noted that these examples do not represent a limitation on the length data used in this specification; any length data that can represent the number of bytes in the payload can be used as length data in this specification.

[0088] First, the length data can directly record the number of bytes in the payload. This allows for direct comparison of the length data when comparing the sizes of different encoded data.

[0089] When comparing data types, for some data types (such as integers), if the data is unsigned (e.g., unsigned integers), all data are positive, so a larger length indicates a larger data type. If the data is signed, and both data are positive, the process is the same as for unsigned data; if both data are negative, a larger length indicates a smaller data type; if one data is negative and the other positive, the positive number is always greater than the negative number.

[0090] In the above methods, for signed data, the sign bit can be added to the index byte, so that the size of the data can be easily determined by reading only the index byte; the sign bit can also be added to the first bit of the payload, so that the size comparison can be completed by reading only the index byte plus one bit of the payload.

[0091] When the sign bit is added to the index byte, in other words, the data to be encoded is signed data; the index byte includes at least: a sign bit and length data; the sign bit is used to indicate the positive or negative sign of the data to be encoded.

[0092] It should be noted that, in the above cases, the length data is not limited to the number of bytes of the directly recorded payload, but can also be in other forms, such as the following.

[0093] Secondly, length data can also be in other forms, such as the following:

[0094] When the data to be encoded is a positive number (including both unsigned and signed positive numbers), the length of the record is positively correlated with the number of bytes in the payload. For example, the length can be directly recorded as the number of bytes in the payload. For instance, if a data payload consists of 4 bytes, then the number 4 (00000100 in binary) can be directly recorded in the index byte.

[0095] When the data to be encoded is negative, the recorded length data and the number of bytes of the payload are negatively correlated. For example, if the length data is encoded in total of 4 bits, it can be the inverse of the number of bytes of the binary payload recording the length data. For example, if the payload is 4, the binary value is 0100 (assuming that the number of bytes of the payload is only encoded in 4 bits, these 4 bits can be located at any position of the index byte, and other positions can store 0 or other content. For details on other content stored in the index byte, please refer to the previous and later texts, which will not be explained here). After reversing the 01s of the above binary payload bytes (i.e., calculating the inverse), the data obtained is 1011, which can be used as the stored length data.

[0096] Thus, when comparing the size of two encoded data, if the two encoded data have the same sign (i.e., both are positive or both are negative), the length data can be directly compared, and the data with the larger length is larger. If the two encoded data have different signs (i.e., one is positive and the other is negative), the size of the two can be directly determined by reading the sign bit from the payload or other locations.

[0097] Furthermore, in addition to the length data as described above, to facilitate size comparison, the index byte can include not only the length data but also a sign bit. Moreover, if the sign bit is placed before the length data, and is 1 for positive numbers and 0 for negative numbers, then the two data points can be directly compared based on their length data. The magnitude of the length data of the two data points (regardless of whether they are positive or negative) reflects their relative size.

[0098] In other words, the index byte includes at least a sign bit and length data, with the sign bit preceding the length data. When the data to be encoded is positive, the sign bit is 1, and the length data is the number of bytes in the payload; when the data to be encoded is negative, the sign bit is 0, and the length data is the one's complement of the number of bytes in the binary payload.

[0099] For example, a positive number 8, a payload of 1 byte, and an index byte with a sign bit set to 1. Assume the sign bit is set in the 7th bit of the index byte (i.e., the MSB is the 7th bit, and the LSB is the 0th bit). Figure 3 As shown, Figure 3 The stored data is 10100000 (each box represents one bit, and all boxes combined represent one byte), and the length data is set in the 6th-3rd bits of the payload (it should be noted that the above positions of the index byte and length data are just examples; they can also be set in other positions, such as the sign bit in the 7th bit and the length data in the 0th-3rd bits). Therefore, the index byte of this integer after 8-bit encoding is 10001000. A positive number 600 has a payload of 2 bytes, and the sign bit of the index byte is set to 1. Following the above encoding method, the index byte is encoded as 10010000. By directly comparing the index bytes of the two numbers, it is easy to see that the index byte of 600 is larger than the index byte of 8-bit.

[0100] A negative number -8 still has a payload of 1 byte. According to the encoding method described above, the length data should be encoded as 1110, with the sign bit set to 0. Therefore, the index byte would be 01110000. Similarly, the index byte for -600 should be encoded as 01101000. It is clear that the index byte for -8 is larger than that for -600. Furthermore, the index bytes for all positive numbers are larger than those for negative numbers.

[0101] It should be noted that there are bits without meaning in the above examples. These bits are set to 0 in the examples. Of course, these bits can also be set to 1 or any fixed value. For example, bits 0-2 can be set to 101, etc. Of course, the above examples do not represent a limitation on this application. Bits without meaning can be set to any fixed value.

[0102] The forms of other index bytes will be explained later and will not be elaborated here.

[0103] Step 205: Combine the generated index byte with the payload of the data to be encoded, and use the combined data as the encoding result of the data to be encoded.

[0104] Specifically, by combining the index byte and the payload, the encoded data to be encoded is obtained.

[0105] Using the previous example, the index byte of the integer 8 is 10001000, and the payload is 00001000. Therefore, the encoded 8 is 10001000 00001000.

[0106] The following sections will explain other forms of the index byte.

[0107] First, as mentioned above, int64 only includes 8 bytes, and 4 bits can represent 8. Therefore, in addition to encoding the length data, the index byte can also encode the first few bits of the data to be encoded (i.e., the few bits closest to the most significant bit; the larger these values ​​are, the larger the data is). Thus, if two data items have the same index and length, there's no need to read the payload portion; the size of the two data items can be compared by comparing the first few bits of the data items in the index byte, further improving the efficiency of comparison or sorting. If the index bytes of the two data items are completely identical, then the payload portion of both data items is read to determine their size relationship.

[0108] In other words, the index byte includes at least length data and first payload data; the first payload data is the first N bits of the payload, where N is the number of bits remaining in the index byte after removing the length data.

[0109] It should be noted that, under the above scheme, the index byte can also store a sign bit, which allows for faster data comparison through the index byte.

[0110] It should also be noted that encoding the first payload data requires reading the payload and writing a portion of the payload data into the index byte, which may consume some encoding processing efficiency. To improve encoding efficiency, it is possible to choose not to write the first payload data into the index byte.

[0111] Furthermore, after the first payload data is written to the index byte, it may or may not be retained in the payload portion; this specification does not limit this.

[0112] Secondly, in order to save on the number of bytes occupied by the encoded data, if the number of bytes of the payload of the data to be encoded is less than 1, and the number of bits used to represent the data in that byte is less than N, this part of the data can be put into the index byte, and then the payload part is not retained, so as to save on the number of bytes of the encoded data.

[0113] In other words, determining the number of bytes of payload in the data to be encoded includes: if the number of bits of the binary data to be encoded after removing leading zeros is less than N, determining that the number of bytes of payload in the data to be encoded is 0; N is the number of bits remaining after removing the length data from the index byte. Generating the index byte includes: setting the index byte to include: length data and second payload data; the second payload data is used to characterize the data to be encoded after removing leading zeros. Combining the generated index byte with the payload of the data to be encoded, and using the combined data as the encoding result of the data to be encoded, includes: using the generated index byte as the encoding result of the data to be encoded.

[0114] Furthermore, the number of bits occupied by the length data can be set to a fixed value, such as 4 bits (as mentioned above, 4 bits can represent the number of bytes of the payload of all data). To reduce the number of bits occupied by the length data and leave more space in the header data to store other content, the number of bits occupied by the length data can also be determined based on the following method.

[0115] First, the number of bits occupied by the length data can be determined based on the following method:

[0116] Obtain the number of bytes of the payload of all data to be encoded; classify all the data to be encoded, with the same number of bytes of payload for data of the same class; determine the number of types of data to be encoded, and based on the determined number of types, determine the number of bits occupied by the length data.

[0117] For example, if the payload of all data has only three byte counts: 1, 2, and 8, then the length data can occupy only two bytes. For instance, 01 can represent a payload of 1 byte, 10 can represent a payload of 2 bytes, and 11 can represent a payload of 8 bytes. This approach is suitable for scenarios where the data to be encoded is relatively concentrated.

[0118] Secondly, another method for determining the number of bits occupied by length data is:

[0119] If all data to be encoded has the same data type, obtain the data types of all data to be encoded; determine the number of bytes occupied by the data type; and determine the number of bits occupied by the length data based on the determined number of bytes.

[0120] In other words, the maximum number of bytes that the payload of the data to be encoded can occupy is determined based on its data type, thus determining the number of bits required for the length data. For example, for int64 data, the payload can occupy a maximum of 8 bytes. Since 8 in binary is 1000, which requires 4 bits to represent, the length data requires 4 bits. For int32 data, the payload can occupy a maximum of 4 bytes. Since 4 in binary is 100, which requires 3 bits to represent, the length data requires 3 bits. For int16 data, the payload can occupy a maximum of 2 bytes. Since 2 in binary is 10, which requires 2 bits to represent, the length data requires 2 bits.

[0121] The following will describe a data encoding method shown in this specification through a specific embodiment. This example will use integers as an example to illustrate the encoding method.

[0122] The encoding format for integers includes an index byte and a payload.

[0123] The format of the index byte is:

[0124] The 7th or most significant bit: the sign bit, where 1 indicates a positive number and 0 indicates a negative number. For unsigned numbers, no sign bit is needed, and all subsequent bits can be shifted forward accordingly.

[0125] Bits 6-3: Length data. For positive numbers, the length data is the number of bytes in the payload. For negative numbers, the length data is the inverted version of the number of bytes in the binary payload.

[0126] Bits 2-0: Payload bits. This position stores the payload if the number of payload bits is less than 3 (and in this case, the encoded integer only includes the index byte). Otherwise, this position stores 0.

[0127] After obtaining the index byte, the index byte and the payload are concatenated (the method for obtaining the payload is described above and will not be repeated here) to obtain the encoded data.

[0128] It should be noted that in this embodiment, the index byte occupies a fixed 1 byte, while the payload can occupy 0-8 bytes, and the payload length varies with the stored integer.

[0129] The encoded integer includes the following parts: Figure 4 As shown.

[0130] In this embodiment, for the purpose of comparing numerical sizes, the index bytes are designed to be used for sorting integers: an integer with a larger index byte represents a larger integer. If the index bytes are equal, then the payload bytes must be compared.

[0131] Because the sign bit is the most significant bit (MSB), when comparing index byte values, positive numbers are always greater than negative numbers. Since longer positive numbers are always greater than shorter positive numbers, and longer negative numbers are always less than shorter negative numbers, length bit comparisons also follow this rule (i.e., the size of the index byte is directly positively correlated with the size of the integer).

[0132] By comparing with methods in related technologies, it can be seen that the VarInt method in related technologies uses "continuous bits" to indicate the length of the encoded integer. Since the continuous bits are at the beginning of each byte, if it is necessary to compare the size of two data, all bytes of the encoded integer must be read, otherwise the number of bytes included cannot be directly determined.

[0133] In this application, an index byte is used to represent the length of the effective payload of the encoded integer. It is only necessary to read an index byte or a byte plus a sign bit (if the integer to be encoded is a signed integer and the sign bit is not set in the index byte, the sign bit also needs to be read) to compare the size of two integers, which solves the problem that the size of integers compressed by the VLQ method in related technologies cannot be compared.

[0134] In addition, this specification also provides a data decoding method, such as Figure 5 As shown, the method includes the following steps:

[0135] Step 501: Determine the number of bytes of the payload of the data to be decoded from the index bytes of the data to be decoded.

[0136] The data to be decoded is obtained by encoding using the data encoding method described above.

[0137] As mentioned above, the length data in the index byte can be stored in various forms. In the above method, the number of bytes corresponding to the payload is determined based on the index byte according to the encoding method.

[0138] Step 503: Read the payload based on the determined number of bytes in the payload.

[0139] Once the number of bytes in the payload is determined, the payload can be obtained by reading that number of bytes backward from the index byte.

[0140] Step 505: Based on a preset data type, process the payload to obtain the decoding result corresponding to the data to be decoded.

[0141] Since the payload is bytes with the prefix removed, after obtaining the payload, it is necessary to pad the payload with a sufficient number of 0s or 1s and add a sign bit based on the data type (to determine how many prefixes need to be added) and the sign bit (if it is a positive number, the prefix is ​​0; if it is a negative number, the prefix is ​​1) to obtain the decoding result.

[0142] Furthermore, this specification also provides a seventh aspect according to embodiments of this specification, providing a database system in which data is encoded using the aforementioned data encoding method; the database system sorts the data using the aforementioned encoded data comparison method to obtain a sorting result.

[0143] When the database system receives a user's data query request, it responds to the user's data query request based on the sorting results.

[0144] Encoding the data using the methods described above allows for both good compression and comparability. Furthermore, as mentioned above, to improve retrieval efficiency, the database system can pre-sort the encoded data using these methods (generally sorting data within the same column). This allows the database system to retrieve data using more efficient methods based on the sorting results when it receives a user's data retrieval request.

[0145] like Figure 6 As shown, this specification also provides a method for comparing encoded data, including the following steps:

[0146] Step 601: Obtain two encoded data to be compared; the encoded data is obtained by encoding using the data encoding method described above.

[0147] This method can be applied to scenarios where coded data needs to be compared or sorted, such as when searching data in a database (as mentioned earlier, in which case the data in the database needs to be sorted).

[0148] Encoded data is data obtained by encoding using the methods described above, which includes the index byte and the payload portion of the data.

[0149] Step 603: Determine the length of each encoded data based on the index byte of each encoded data.

[0150] The encoded data includes an index byte, which contains length data. Therefore, the corresponding length data can be read from the index byte.

[0151] Step 605: Compare the two encoded data based on their lengths.

[0152] Length data is used to characterize the number of bytes in the payload portion of the encoded data. The number of bytes in the payload is related to the size of the encoded data. For positive numbers, the more bytes in the payload portion, the larger the data is. For negative numbers, the opposite is true. Therefore, data comparison can be performed based on length data.

[0153] If the index byte stores length data and a sign bit, and the sign bit precedes the length data; when the data to be encoded is positive, the sign bit is 1, and the length data is the number of bytes of the payload; when the data to be encoded is negative, the sign bit is 0, and the length data is the inverse of the number of bytes of the binary payload.

[0154] In the above case, the larger the encoded data, the larger the corresponding index byte will be. Therefore, the size of the two data can be determined directly based on the index byte.

[0155] In other words, capturing 605 involves: comparing the size of the index bytes of two encoded data, and determining the comparison result of the two encoded data based on the comparison result of the index bytes; wherein, the size of the index byte is positively correlated with the size of the encoded data.

[0156] like Figure 7 As shown, Figure 7 This is a block diagram illustrating a data encoding apparatus according to an exemplary embodiment of this specification, the apparatus comprising:

[0157] The byte count determination module 710 is used to determine the number of bytes of the payload in the data to be encoded; the payload is specified data that can characterize the size of the data to be encoded.

[0158] An index byte generation module 720 is used to generate index bytes; the index bytes include at least: length data, which is used to characterize the number of bytes in the payload;

[0159] The combination module 730 is used to combine the generated index byte with the payload of the data to be encoded, and use the combined data as the encoding result of the data to be encoded.

[0160] In one optional embodiment, the data to be encoded is signed data; the index byte includes at least: a sign bit and length data; the sign bit is used to indicate the positive or negative sign of the data to be encoded.

[0161] In one optional embodiment, the sign bit is ordered before the length data; when the data to be encoded is a positive number (i.e., greater than 0), the sign bit is 1, and the length data is the number of bytes of the payload; when the data to be encoded is a negative number (i.e., less than 0), the sign bit is 0, and the length data is the inverse of the number of bytes of the binary payload.

[0162] In one optional embodiment, the index byte includes at least length data and first payload data; the first payload data is the first N bits of the payload, where N is the number of bits remaining in the index byte after removing the length data.

[0163] In an optional embodiment, the byte count determination module 710 is used to determine that the number of bytes of the effective payload in the binary data to be encoded is 0 when the number of bits of the data after removing leading zeros is less than N; N is the number of bits remaining in the index byte after removing the length data; the index byte generation module 720 is used to set the index byte to include: length data and second payload data; the second payload data is used to characterize the data after removing leading zeros from the data to be encoded; and the combination module 730 is used to use the generated index byte as the encoding result of the data to be encoded.

[0164] In an optional embodiment, the apparatus further includes a first bit count determination module 750 (not shown in the figure), configured to, when all data to be encoded have the same data type, obtain the data type of all data to be encoded that needs to be encoded; determine the number of bytes occupied by the data type; and determine the number of bits occupied by the length data based on the determined number of bytes.

[0165] In an optional embodiment, the apparatus further includes a second bit count determination module 760 (not shown in the figure), used to obtain the number of bytes of the payload of all data to be encoded; classify all the data to be encoded, with the same number of bytes of payload for data of the same type; determine the number of types of data to be encoded, and determine the number of bits occupied by the length data based on the determined number of types.

[0166] like Figure 8 As shown, Figure 8 This specification illustrates an coded data comparison apparatus according to an exemplary embodiment, comprising:

[0167] The encoded data acquisition module 810 is used to acquire two encoded data to be compared; the encoded data is obtained by the data encoding method described above.

[0168] The length data determination module 820 is used to determine the length data of each encoded data based on the index byte of each encoded data.

[0169] The comparison module 830 is used to compare the size of two encoded data based on their length data.

[0170] In an optional embodiment, the index byte of the encoded data further includes a sign bit, which precedes the length data; when the data to be encoded is positive, the sign bit is 1, and the length data is the number of bytes of the payload; when the data to be encoded is negative, the sign bit is 0, and the length data is the inverse of the number of bytes of the binary payload.

[0171] In the above case, the comparison module 830 is used to compare the size of the index bytes of the two encoded data, and determine the comparison result of the two encoded data based on the comparison result of the index bytes; wherein, the size of the index byte is positively correlated with the size of the encoded data.

[0172] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0173] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0174] like Figure 9 As shown, Figure 9 This diagram illustrates a hardware structure of a computer device containing an embodiment of a data encoding or decoding device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are internally connected to each other via the bus 1050.

[0175] The processor 1010 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above-described methods by running executable instructions.

[0176] The memory 1020 for storing processor-executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020.

[0177] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0178] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0179] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0180] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0181] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described data encoding or data decoding method.

[0182] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0183] This specification also provides a computer program that, when executed by a processor, implements the above-described data encoding or data decoding method.

[0184] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0185] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A data encoding method, comprising: Determine the number of bytes of payload in the data to be encoded; The payload is specified data that can characterize the size of the data to be encoded, wherein the payload is the data to be encoded after removing the prefix portion, and the prefix portion includes leading zero bytes and / or negative numbers; Generate an index byte; the index byte includes at least: a sign bit and length data, the length data being used to characterize the number of bytes of the payload, the index byte being used to compare the size of the data to be encoded with the size of other data to be encoded, the sign bit being ordered before the length data; when the data to be encoded is greater than 0, the sign bit is 1, and the length data is the number of bytes of the payload; when the data to be encoded is less than 0, the sign bit is 0, and the length data is the inverse of the number of bytes of the binary payload; The generated index byte is combined with the payload of the data to be encoded, and the combined data is used as the encoding result of the data to be encoded.

2. The method according to claim 1, wherein the index byte includes at least length data and first payload data; the first payload data is the first N bits of the payload, where N is the number of bits remaining in the index byte after removing the length data.

3. The method according to claim 1, wherein determining the number of bytes of payload in the data to be encoded includes: If the number of bits of the binary data to be encoded after removing leading zeros is less than N, the number of bytes of the effective payload in the data to be encoded is determined to be 0. N is the number of bits remaining in the index byte after removing the length data; The generated index bytes include: The index byte is configured to include: length data and second payload data; the second payload data is used to characterize the data to be encoded after removing leading zeros. The step of combining the generated index byte with the payload of the data to be encoded, and using the combined data as the encoding result of the data to be encoded, includes: The generated index byte is used as the encoding result of the data to be encoded.

4. The method according to claim 1, wherein the method for determining the number of bits occupied by the length data includes: Given that all data to be encoded has the same data type, obtain the data type of all data to be encoded that needs to be encoded. Determine the number of bytes occupied by the data type; Based on the determined number of bytes, the number of bits occupied by the length data is determined.

5. The method according to claim 1, wherein the method for determining the number of bits occupied by the length data includes: Get the number of bytes in the payload of all the data to be encoded; All the acquired data to be encoded are classified, and the payload of data in the same category has the same number of bytes. The number of types of data to be encoded is determined, and based on the determined number of types, the number of bits occupied by the length data is determined.

6. A method for comparing encoded data, comprising: Obtain the two encoded data to be compared; The encoded data is obtained by encoding using the method described in any one of claims 1-5; The length of each encoded data is determined based on the index byte of each encoded data. The two encoded data are compared based on their lengths.

7. The method according to claim 6, wherein the index byte of the encoded data further includes a sign bit, the sign bit preceding the length data; when the data to be encoded is greater than 0, the sign bit is 1, and the length data is the number of bytes of the payload; when the data to be encoded is less than 0, the sign bit is 0, and the length data is the inverse of the number of bytes of the binary payload. The comparison of the two encoded data based on their lengths includes: The index bytes of two encoded data are compared to determine the comparison result of the two encoded data; the size of the index byte is positively correlated with the size of the encoded data.

8. A database system, wherein data in the database system is encoded by the method according to any one of claims 1-5; the database system sorts the data by the method according to claim 6 or 7 to obtain a sorting result; When the database system receives a user's data query request, it responds to the user's data query request based on the sorting results.

9. A computer device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

11. A computer program product, wherein a processor of the computer program product implements the method as described in any one of claims 1-7 when the processor runs.

Images