Bit width compression method and device, query method and device, equipment and storage medium

Abstract

The invention discloses a bit width compression method and device, a query method and device, equipment and a storage medium, which are applied to the technical field of data processing, and the method comprises the following steps: dynamically determining a storage bit width based on the value quantity of a tag; coding the label value of the current label into a binary code based on the storage bit width; and filling the binary codes into a buffer area according to a preset sequence to obtain a binary string, converting the binary string into integer coded data, and storing the integer coded data and the initial offset position corresponding to each binary code into a metadata table of a database. Compared with a current fixed bit width scheme which cannot be optimized according to actual distribution of label values, so that the utilization rate of a storage space is low, and the flexibility of coping with the change of the number of values is insufficient, the method can dynamically distribute the most suitable storage bit width for each label according to the actual number of values of each label, so that the storage efficiency is improved on the premise of ensuring the storage capacity. The compression ratio is maximized, the storage space is saved, and meanwhile, the method is compatible with efficient query of an existing database system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a bit-width compression method, query method, apparatus, device, and storage medium. Background Technology

[0002] In IoT scenarios, the number of possible values ​​for different tags varies significantly (e.g., "on / off status" has only 2 values, "device type" may have 10 values, and "signal strength" has 6 levels). If a fixed bit-width compression scheme is used (e.g., uniformly allocating 4 bits): for the "on / off status" (2 values) tag, only 1 bit is needed, and using 4 bits results in 3 bits (75%) of wasted space. For tags with more than 16 values, 4 bits cannot represent them, making the scheme unsuitable and inflexible. Therefore, the fixed bit-width scheme cannot be optimized according to the actual distribution of tag values, resulting in significant room for improvement in storage space utilization and insufficient flexibility in dealing with changes in the number of values.

[0003] It is evident that improving storage space utilization is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a bit-width compression method, query method, apparatus, device and storage medium, which solves the problem of wasted space in the prior art of bit-width storage.

[0005] To solve the above-mentioned technical problems, the present invention provides a bit-width compression method, comprising:

[0006] The storage width is dynamically determined based on the number of values ​​that can be taken from the label;

[0007] The tag value of the current tag is encoded into binary code based on the storage bit width;

[0008] The binary codes are filled into a buffer in a preset order to obtain a binary string, and the binary string is converted into integer encoded data. The integer encoded data and the starting offset position corresponding to each binary code are stored in the metadata table of the database.

[0009] Optionally, the storage bit width can be dynamically determined based on the number of possible values ​​for the label, including:

[0010] The storage width is obtained by dynamically predicting the number of values ​​based on the label using a storage width prediction model; wherein the storage width prediction model is a model obtained by training a machine learning model based on historical labels and their corresponding historical value counts.

[0011] Optionally, before dynamically predicting the storage width based on the number of values ​​for the label using a storage width prediction model, the method further includes:

[0012] Collect metadata sequences for each tag over a historical time period. The metadata sequences include the number of values, historical growth trends, and query access patterns. The historical growth trend refers to the growth pattern of the number of tag values. The query access patterns are parameters related to storage efficiency and query performance.

[0013] A training sample set is constructed based on the metadata sequence;

[0014] The machine learning model is trained using the training sample set to obtain the storage bit width prediction model.

[0015] Optionally, the binary codes are filled into a buffer in a preset order to obtain a binary string, and the binary string is converted into integer encoded data. The integer encoded data and the starting offset position corresponding to each binary code are stored in the metadata table of the database, including:

[0016] Determine the number of bits already occupied in the binary string in the buffer, and determine whether the number of bits already occupied has reached the set number of bits;

[0017] When the set number of bits is reached, the binary string is converted into the integer encoded data and stored in the tag value storage table.

[0018] Optionally, the storage bit width can be dynamically determined based on the number of possible values ​​for the label, including:

[0019] Based on the number of values, the storage width is determined using a storage bit width function; wherein, the storage bit width function is a function that characterizes the correspondence between the number of values ​​and the storage bit width;

[0020] The present invention also provides a query method, including:

[0021] The tags and tag values ​​for the query are determined based on the query conditions;

[0022] Based on the pre-stored metadata table, query the storage bit width of the tag and its starting offset position in the integer encoded data; wherein, the metadata table is a metadata table determined based on the bit width compression method described above;

[0023] Obtain the integer code corresponding to the tag value;

[0024] The mask is determined based on the starting offset position and the storage bit width;

[0025] The target value is determined based on the integer encoding and the starting offset position;

[0026] Determine whether the result of a bitwise AND operation between the target integer field and the mask is equal to the target value;

[0027] Use the target integer field that is equal to the target value as the query result.

[0028] The present invention also provides a bit-width compression device, comprising:

[0029] The storage bit width determination module is used to dynamically determine the storage bit width based on the number of values ​​that the tag can take;

[0030] The encoding module is used to encode the tag value of the current tag into binary code based on the storage bit width;

[0031] The storage module is used to fill the binary code into the buffer in a preset order to obtain a binary string, convert the binary string into integer encoded data, and store the integer encoded data and the starting offset position corresponding to each binary code into the metadata table of the database.

[0032] The present invention also provides a query device, comprising:

[0033] The tag and tag value determination module is used to determine the tags and tag values ​​of a query based on the query conditions.

[0034] The storage bit width and starting offset position determination module is used to query the storage bit width and starting offset position of the tag in the integer encoded data based on a pre-stored metadata table; wherein, the metadata table is a metadata table determined based on the bit width compression method described above.

[0035] An integer encoding determination module is used to obtain the integer encoding corresponding to the tag value;

[0036] The mask determination module is used to determine the mask based on the starting offset position and the storage bit width;

[0037] The target value determination module is used to determine the target value based on the integer encoding and the starting offset position;

[0038] The judgment module is used to determine whether the result of performing a bitwise AND operation between the target integer field and the mask is equal to the target value.

[0039] The query result determination module is used to select the target integer field that is equal to the target value as the query result.

[0040] The present invention also provides an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor for executing the computer program to implement the steps of the above method.

[0043] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method.

[0044] As can be seen, this invention dynamically determines the storage bit width based on the number of tag values; based on the storage bit width, the tag value of the current tag is encoded into binary code; the binary code is filled into a buffer in a preset order to obtain a binary string, and the binary string is converted into integer encoded data. The integer encoded data and the starting offset position corresponding to each binary code are stored in the metadata table of the database. The beneficial effects of this invention are: compared with the current fixed bit width scheme, which cannot be optimized according to the actual distribution of tag values, resulting in a large room for improvement in storage space utilization and insufficient flexibility in dealing with changes in the number of values, this invention can dynamically allocate the most suitable storage bit width according to the actual number of values ​​for each tag, thereby maximizing the compression ratio, saving storage space, supporting seamless expansion, and being compatible with the efficient query of existing database systems while ensuring storage capacity.

[0045] In addition, the present invention also provides a bit-width compression device, apparatus and storage medium, as well as a query method, apparatus, apparatus and storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart of a bit-width compression method provided in an embodiment of the present invention;

[0048] Figure 2 A flowchart of a query method provided in an embodiment of the present invention;

[0049] Figure 3 This is a system architecture diagram corresponding to a dynamic bit-width compression method provided in an embodiment of the present invention;

[0050] Figure 4 A flowchart illustrating a bit-width compression method provided in an embodiment of the present invention;

[0051] Figure 5 A flowchart illustrating a query method provided in an embodiment of the present invention;

[0052] Figure 6This is a schematic diagram of a bit-width compression device provided in an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of the structure of a query device provided in an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Some terms that appear in the description of the embodiments of this application are subject to the following interpretation:

[0057] A tag is an attribute or characteristic used to describe an object (such as an IoT device or a user). This attribute usually has a limited, predefined range of possible values.

[0058] Number of possible values: This is the total number of all possible, non-repeating values ​​for a given tag. It is a natural number, usually represented by "N".

[0059] Bit width refers to the number of binary bits required in computer storage to represent a specific value of a tag. It is a natural number, usually represented by "W".

[0060] Key-value pair storage: This is a common form of storing tag data, where each tag is a key and its value is stored separately.

[0061] Binary bitwise operations: These are basic arithmetic functions supported by modern databases (such as Oracle and MySQL), such as BITAND and BITSHR, which can be used to extract and judge specific bits in integer fields.

[0062] Data compression: To save storage space, common techniques include fixed-width compression (such as storing all tags in 4-bit format) and columnar compression (such as dictionary encoding and run-length encoding).

[0063] Tag value storage table: Used to store tag values. Each table can contain multiple fields, and each field contains the values ​​of several tags.

[0064] Metadata table: Used to record descriptive information about tags, including the bit width of the value, the name of the data table where the value is stored and the field name, and the offset of the value in the data table field.

[0065] Buffer: This is a temporary storage space with a preset length allocated in memory before being stored in the database table. The binary string of the tag value is appended from right to left according to the order in which the tag value is passed in, until the preset length is reached. Then, the binary value is converted into a decimal integer and stored in the field of the tag value storage table.

[0066] In IoT scenarios, the number of values ​​for different tags varies greatly (e.g., "Switch Status" has only 2 values, "Device Type" may have 10 values, and "Signal Strength" has 6 levels). If a fixed bit-width compression scheme (e.g., uniformly allocating 4 bits) is used: for the "Switch Status" tag (2 values), only 1 bit is needed, and using 4 bits results in 3 bits (75%) of wasted space. For the "Device Type" tag (10 values), although 4 bits (representing up to 16 values) can meet the storage requirements, it is not optimal. For tags with more than 16 values, 4 bits cannot represent them, the scheme is unsuitable, and the flexibility is poor. The fixed bit-width scheme cannot be optimized according to the actual distribution of tag values, resulting in significant room for improvement in storage space utilization and insufficient flexibility in dealing with changes in the number of values. This invention aims to provide a dynamic bit-width compression storage method to solve the shortcomings of the above-mentioned fixed bit-width scheme.

[0067] Please refer to Figure 1 , Figure 1 A flowchart illustrating a bit-width compression method provided in an embodiment of the present invention. The method may include:

[0068] S101, dynamically determine the storage bit width based on the number of values ​​that the tag can take.

[0069] Each step in this embodiment can be executed by a designated electronic device, which can be a server, a portable terminal, or other forms. This embodiment does not limit the specific method for dynamically determining the storage bit width based on the tag. For example, this embodiment can determine the storage bit width based on predicting the number of values ​​for the current tag using a prediction model; or it can determine the storage bit width based on a functional relationship between the number of values ​​and the storage bit width; or it can determine the storage bit width based on a mapping relationship between the number of values ​​and the storage bit width. For example, querying the current number of values ​​(N) for the tag, and allocating a corresponding storage bit width (W) based on the interval N falls into (e.g., 2 values ​​→ 1 bit, 3-4 values ​​→ 2 bits, 5-8 values ​​→ 3 bits, 9-16 values ​​→ 4 bits).

[0070] It should be further explained that, based on any of the above embodiments, the dynamic determination of storage width based on the number of tag values ​​can include: dynamically predicting the storage width based on the number of tag values ​​using a storage width prediction model; wherein, the storage width prediction model is a model trained on a machine learning model based on historical tags and their corresponding historical value counts. This embodiment can train the machine learning model based on historical tags and their corresponding historical value counts. This embodiment does not limit the specific machine learning model. For example, the machine learning model in this embodiment can be a neural network model; or the machine learning model in this embodiment can be a linear model. The main function of the storage width prediction model in this embodiment is to predict the optimal storage width based on learning the actual number of occurrences of historical tag values, thus avoiding storage waste. For example, if a tag has 10 possible values, but only 6 will actually occur, the AI ​​model will determine 6, thereby reducing the storage width from 4 to 3.

[0071] It should be further noted that, based on any of the above embodiments, before dynamically predicting the storage width using a storage width prediction model based on the number of tag values ​​to obtain the storage width, the following may also be included:

[0072] Step 1: Collect metadata sequences for each tag over a historical time period. The metadata in the metadata sequence includes the number of values, historical growth trends, and query access patterns. The historical growth trend refers to the growth pattern of the number of tag values, and the query access patterns are parameters related to storage efficiency and query performance.

[0073] Step 2: Construct a training sample set based on the metadata sequence;

[0074] Step 3: Train the machine learning model using the training sample set to obtain the storage bit width prediction model.

[0075] This embodiment provides a specific method for training a machine learning model. By considering not only the number of values ​​but also historical growth trends and query access patterns, the accuracy of model training is improved, thereby enabling optimization of space allocation at a finer granularity.

[0076] It should be further explained that, based on any of the above embodiments, the dynamic determination of the storage bit width based on the number of tag values ​​may include: determining the storage bit width using a storage bit width function based on the number of values; wherein, the storage bit width function is a function characterizing the correspondence between the number of values ​​and the storage bit width; the storage bit width function in this embodiment is... Where W represents the storage bit width and N represents the number of values ​​to be retrieved. Represents the floor function This represents a base-2 logarithm. This embodiment ensures that a given number of values ​​is represented using the minimum number of bits by employing a storage bit-width function.

[0077] S102, based on the storage bit width, encode the tag value of the current tag into binary code.

[0078] This embodiment encodes the tag value corresponding to the current tag. For example, if the current tag is "Device Type" (10-bit value), the specific tag value (such as "Air Conditioner") is encoded as a W-bit (e.g., 4-bit) binary number (such as "0010").

[0079] S103, fill the binary code into the buffer in a preset order to obtain a binary string, convert the binary string into integer encoded data, and store the integer encoded data and the starting offset position corresponding to each binary code into the metadata table of the database.

[0080] In this embodiment, the binary codes corresponding to each tag value are filled into a packing buffer in a preset order. When the buffer accumulates enough tags (e.g., 20) to reach 64 bits, or when all tags have been processed, the entire binary string is converted into a 64-bit integer and finally written into the NUMBER / BIGINT (numeric type) field of the database for storage. This embodiment does not limit the specific preset order; for example, the preset order in this embodiment can be chronological order. The starting offset position stored in the metadata table in this embodiment can be in binary form or decimal form. It should be noted that the overall steps are as follows: when a device type tag value is received, its storage bit width W=3 is first calculated according to the bit width calculation logic, and the tag value Value=101; it is first appended to the buffer. Assuming that the preset maximum width of the buffer is 64 bits, and other tags have already occupied 10 bits, then the device type tag will occupy bits 11-13 (from right to left), and the buffer value becomes 101********** after storage. When subsequent tag values ​​are added, and the buffer reaches 64 bits (or the width is insufficient to store the next tag value), the binary value in the buffer is converted to a decimal integer and stored in the Col-x field of the data storage table Dtable. The metadata table records: Tag ID = Device Type Tag, Storage Location = Dtable#Col-x, Bit Width W = 3, Offset P = 10.

[0081] It should be further explained that, based on any of the above embodiments, filling the binary code into the buffer in a preset order to obtain a binary string, converting the binary string into integer encoded data, and storing the integer encoded data and the starting offset position corresponding to each binary code in the metadata table of the database may include:

[0082] S1031, determine the number of bits already occupied in the binary string in the buffer, and determine whether the number of bits already occupied has reached the set number of bits;

[0083] S1032, when the set number of bits is reached, the binary string is converted into an integer encoded data and stored in the tag value storage table.

[0084] This embodiment does not limit the specific number of bits set. For example, the number of bits set in this embodiment can be 64 bits; or it can also be 128 bits; or it can also be a binary large object (BLOB) to accommodate more tags and adapt to different scenarios. It is understandable that a 64-bit integer is chosen as the packing unit because it is both efficient and large enough to accommodate a considerable number of tags (such as about 20), and it is also a data type that computers process efficiently.

[0085] This invention provides a bit-width compression method, which may include: S101, dynamically determining the storage bit width based on the number of tag values; S102, encoding the tag value of the current tag into binary code based on the storage bit width; S103, filling the binary code into a buffer in a preset order to obtain a binary string, converting the binary string into integer encoded data, and storing the integer encoded data and the starting offset position corresponding to each binary code in the metadata table of the database. Compared with the current fixed bit-width scheme, which cannot be optimized according to the actual distribution of tag values, resulting in significant room for improvement in storage space utilization and insufficient flexibility in dealing with changes in the number of values, this invention can dynamically allocate the most suitable storage bit width according to the actual number of values ​​for each tag, thereby maximizing the compression ratio, saving storage space, supporting seamless expansion, and being compatible with the efficient querying of existing database systems while ensuring storage capacity.

[0086] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A flowchart of a query method provided in an embodiment of the present invention may specifically include:

[0087] S201, determine the tags and tag values ​​for the query based on the query conditions.

[0088] The implementation of this invention requires a database that supports bitwise operations (such as Oracle, MySQL, PostgreSQL, etc.).

[0089] S202, based on the pre-stored metadata table, query the storage bit width of the tag and its starting offset position in the integer encoded data; wherein, the metadata table is a metadata table determined based on the bit width compression method described above.

[0090] S203, obtain the integer code corresponding to the tag value.

[0091] S204, determine the mask based on the starting offset position and storage bit width.

[0092] S205 determines the target value based on integer encoding and starting offset position.

[0093] S206, determine whether the result of the bitwise AND operation between the target integer field and the mask is equal to the target value.

[0094] S207, use the target integer field that is equal to the target value as the query result.

[0095] For ease of understanding, for example, when querying data with device type tag = 101, the steps are as follows: First, obtain the storage width W and storage offset P of the tag based on the metadata table; Second, calculate two values: mask M = (2^W - 1) << P, resulting in M ​​= 1110000000000, target value V = (Value) << P, resulting in V = 1010000000000; Third, extract the decimal integer D (target integer field) based on the storage location Dtable#Col-x of the record value in the metadata table; Fourth, perform a bitwise AND operation between D and mask M to obtain the result B; Fifth, compare whether B and V are equal. If they are equal, it means that a device of the target device class has been found.

[0096] When the query in the embodiment of the present invention starts, the system analyzes the query conditions to determine the tag to be queried (such as "device type") and the tag value (such as "air conditioner"). Subsequently, it queries the meta-data table pre-stored in the database to obtain two key parameters of the tag: its storage bit width (W = 3) and the starting offset position (P = 10) in the 64-bit field. At the same time, it obtains the binary code Value = 101 corresponding to the tag value "air conditioner". Then, it calculates the mask (Mask) required for the query: M = (2^W - 1) << P. This mask is used to precisely isolate the bits occupied by the target tag in the 64-bit field, and calculates the target value: V = (Value) << P. For example, if Value = 101 (binary), the tag value with W = 3 is stored at the offset P = 10, then M=(2^W - 1) << P, resulting in M = 1110000000000, and V = (Value) << P, resulting in V = 1010000000000. Finally, it constructs an SQL query statement, uses the BITAND function of the database to determine whether the result B after the bitwise AND operation between the target field (the field corresponding to the decimal number) and the mask is equal to the target value V, and returns the query result. The target field in this embodiment is obtained according to the records in the meta-data table. In addition to directly using bitwise operations in the WHERE (condition) clause of SQL (Structured Query Language) in this embodiment, complex bit operation logics can also be encapsulated by creating a database user-defined function (UDF) to provide a simpler query interface for the application layer (such as WHERE GET_TAG(tags_field, 'device type') = 'air conditioner').

[0097] The application scenarios of this embodiment can be any scenarios that require efficient processing of a large number of multi-value tags, such as the Internet of Things device management platform, user portrait system, real-time recommendation system, etc. And after verification, in typical scenarios (assuming 20 tags, with an average bit width of 3.2 bits), compared with the original key-value storage, it is expected to save about 95% of the storage space; compared with the fixed 4-bit storage scheme, it is expected to save another 20% of the space. The query efficiency is equivalent to the fixed bit width scheme and much higher than the string matching query.

[0098] To make the present invention easier to understand, please specifically refer to Figure 3 , Figure 3 which is the system architecture diagram corresponding to a dynamic bit width compression method provided by the embodiment of the present invention, Figure 3 The system mainly includes a dynamic bit width allocation engine, a binary packer, and an in-situ query engine. After the tag data is input, the allocation engine determines the optimal bit width for it, and the packer compresses multiple tag values into a binary string and stores it in the database. During query, the query engine converts the conditions into bitwise operations for retrieval.

[0099] With Figure 3 Please refer to the flowchart of a corresponding bit-width compression method. Figure 4 The process begins with receiving tag data. The system first queries the current number of values ​​(N) for the tag and allocates a corresponding storage width (W) based on the range N falls into (e.g., 2 values ​​→ 1 bit, 3-4 values ​​→ 2 bits, 5-8 values ​​→ 3 bits, 9-16 values ​​→ 4 bits). Then, the specific tag value (e.g., "air conditioner") is encoded as a W-bit binary number (e.g., "101"). This binary code is filled into a packing buffer in a preset order. When the buffer has enough tags (e.g., 20) to fill the 64 bits, or when all tags have been processed, the entire binary string is converted into a 64-bit integer and finally written to the NUMBER / BIGINT field of the database for storage. This embodiment can dynamically determine the mapping rules and methods of the storage width (W) based on the actual number of tag values ​​(N) (e.g., N∈[1,2] → W=1; N∈[3,4] → W=2 ...); when the number of values ​​for a tag exceeds the current storage width capacity, this embodiment can automatically upgrade its storage width to the next level.

[0100] and Figure 3 Please refer to the flowchart of one corresponding query method. Figure 5 When a query begins, the system parses the query conditions to determine the tag and value to be queried. Then, it queries the pre-stored metadata table to obtain the tag's storage width and starting offset position in the 64-bit field. Simultaneously, it obtains the binary code corresponding to the value "air conditioner". Next, it calculates the required mask for the query: M = (2^W - 1) << P. This mask is used to precisely isolate the bits occupied by the target tag in the 64-bit field. The target value is calculated: V = (Value) << P. Finally, an SQL query statement is constructed, and the database's BITAND function is used to determine whether the result of a bitwise AND operation between the target field and the mask equals the target value V, and the query result is returned. This embodiment can dynamically generate a query mask (M) and a target value (V) based on the tag's bit width and offset, and utilize native database bitwise operation functions (such as BITAND) to achieve efficient querying without decompression.

[0101] This invention, by allocating bit width "on demand" for each tag, reduces the space over-allocated to low-cardinality tags compared to fixed-bit-width schemes, while ensuring the storability of high-cardinality tags. Therefore, this invention significantly improves storage space utilization and offers excellent flexibility, adapting to an increase in the number of tag values. Furthermore, since this embodiment ultimately stores compressed data as a general integer type field and uses standard database bitwise operations for querying, no special storage format or database kernel modification is required. Therefore, this invention can be seamlessly integrated with existing relational databases, exhibiting excellent compatibility, low implementation cost, and achieving highly efficient millisecond-level queries.

[0102] The bit width compression device provided in the embodiments of the present invention will be described below. The bit width compression device described below can be referred to in correspondence with the bit width compression method described above.

[0103] Please refer to the details. Figure 6 , Figure 6 A schematic diagram of a bit-width compression device provided in an embodiment of the present invention may include:

[0104] The storage bit width determination module 100 is used to dynamically determine the storage bit width based on the number of values ​​that the tag can take;

[0105] Encoding module 200 is used to encode the tag value of the current tag into binary code based on the storage bit width;

[0106] The storage module 300 is used to fill the binary code into a buffer in a preset order to obtain a binary string, convert the binary string into integer encoded data, and store the integer encoded data and the starting offset position corresponding to each binary code into the metadata table of the database.

[0107] Furthermore, based on any of the above embodiments, the storage bit width determination module 100 may include:

[0108] The storage bit width prediction module is used to dynamically predict the storage bit width based on the number of values ​​of the label using a storage bit width prediction model; wherein, the storage bit width prediction model is a model obtained by training a machine learning model based on historical labels and their corresponding historical value counts.

[0109] Furthermore, based on any of the above embodiments, the bit-width compression device may further include:

[0110] The data acquisition module is used to collect metadata sequences for each tag over a historical time period. The metadata sequences include the number of values, historical growth trends, and query access patterns. The historical growth trend refers to the growth pattern of the number of tag values, and the query access patterns are parameters related to storage efficiency and query performance.

[0111] A training sample set construction module is used to construct a training sample set based on the metadata sequence;

[0112] The storage bit width prediction model training module is used to train the machine learning model using the training sample set to obtain the storage bit width prediction model.

[0113] Furthermore, based on any of the above embodiments, the storage module 300 may include:

[0114] The judgment unit is used to determine the number of bits already occupied in the binary string in the buffer, and to determine whether the number of bits already occupied has reached a set number of bits;

[0115] A storage unit is used to convert the binary string into the integer encoded data and store it in the tag value storage table when the set number of bits is reached.

[0116] Furthermore, based on any of the above embodiments, the storage bit width determination module 100 may include:

[0117] The storage bit width unit is determined based on a function, which is used to determine the storage bit width based on the number of values ​​using a storage bit width function; wherein, the storage bit width function is a function that characterizes the correspondence between the number of values ​​and the storage bit width;

[0118] It should be noted that the order of the modules and units in the aforementioned bit-width compression device can be changed without affecting the logic.

[0119] This invention provides a bit-width compression device, which may include: a storage bit-width determination module 100, used to dynamically determine the storage bit-width based on the number of tag values; an encoding module 200, used to encode the tag value of the current tag into binary code based on the storage bit-width; and a storage module 300, used to fill the binary code into a buffer in a preset order to obtain a binary string, convert the binary string into integer encoded data, and store the integer encoded data and the starting offset position corresponding to each binary code in the metadata table of the database. Compared with the current fixed bit-width scheme, which cannot be optimized according to the actual distribution of tag values, resulting in a large room for improvement in storage space utilization and insufficient flexibility in dealing with changes in the number of values, this invention can dynamically allocate the most suitable storage bit-width according to the actual number of values ​​of each tag, thereby maximizing the compression ratio, saving storage space, supporting seamless expansion, and being compatible with the efficient query of existing database systems while ensuring storage capacity.

[0120] The query device provided in the embodiments of the present invention will be described below. The query device described below and the query method described above can be referred to in correspondence.

[0121] Please refer to the details. Figure 7 , Figure 7 A schematic diagram of a query device provided in an embodiment of the present invention may include:

[0122] The tag and tag value determination module 400 is used to determine the tags and tag values ​​of a query based on the query conditions;

[0123] The storage bit width and starting offset position determination module 500 is used to query the storage bit width and starting offset position of the tag in integer encoded data based on a pre-stored metadata table; wherein, the metadata table is a metadata table determined based on the bit width compression method described above.

[0124] Integer encoding determination module 600 is used to obtain the integer encoding corresponding to the tag value;

[0125] The mask determination module 700 is used to determine the mask based on the starting offset position and the storage bit width;

[0126] The target value determination module 800 is used to determine the target value based on the integer encoding and the starting offset position;

[0127] The judgment module 900 is used to determine whether the result of performing a bitwise AND operation between the target integer field and the mask is equal to the target value.

[0128] The query result determination module 1000 is used to take the target integer field that is equal to the target value as the query result.

[0129] It should be noted that the order of the modules and units in the above query device can be changed without affecting the logic.

[0130] This invention provides a query device that may include: a tag and tag value determination module 400, used to determine the tag and tag value to be queried based on query conditions; a storage bit width and starting offset position determination module 500, used to query the storage bit width and starting offset position of the tag in integer encoded data based on a pre-stored metadata table; wherein the metadata table is a metadata table determined based on the aforementioned bit width compression method; an integer encoding determination module 600, used to obtain the integer encoding corresponding to the tag value; a mask determination module 700, used to determine a mask based on the starting offset position and the storage bit width; a target value determination module 800, used to determine a target value based on the integer encoding and the starting offset position; a judgment module 900, used to judge whether the result of a bitwise AND operation between the target integer field and the mask is equal to the target value; and a query result determination module 1000, used to take the target integer field that is equal to the target value as the query result. This invention can perform queries based on a created metadata table, improving query efficiency.

[0131] The following describes an electronic device provided by an embodiment of the present invention. The electronic device described below can be referred to in correspondence with the bit width compression method and query method described above.

[0132] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention may include:

[0133] Memory 10 is used to store computer programs;

[0134] Processor 20 is used to execute computer programs to implement the bit-width compression method and the polling method described above.

[0135] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0136] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0137] The storage width is dynamically determined based on the number of values ​​that can be taken from the label;

[0138] The tag value of the current tag is encoded into binary code based on the storage bit width;

[0139] The binary codes are filled into the buffer in a preset order to obtain a binary string, and the binary string is converted into integer encoded data. The integer encoded data and the starting offset position corresponding to each binary code are stored in the metadata table of the database.

[0140] Or a program that can perform the following functions:

[0141] The tags and tag values ​​for the query are determined based on the query conditions;

[0142] Based on the pre-stored metadata table, query the storage bit width of the tag and its starting offset position in the integer encoded data; wherein, the metadata table is the metadata table determined based on the bit width compression method described above;

[0143] Get the integer code corresponding to the tag value;

[0144] The mask is determined based on the starting offset position and the storage bit width;

[0145] The target value is determined based on integer encoding and starting offset position;

[0146] Determine whether the result of a bitwise AND operation between the target integer field and the mask is equal to the target value;

[0147] Use the target integer field that is equal to the target value as the query result.

[0148] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0149] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0150] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0151] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0152] Of course, it should be noted that, Figure 8 The structure shown does not constitute a limitation on the electronic device in the embodiments of the present invention. In practical applications, the electronic device may include more than Figure 8 More or fewer components as shown, or combinations of certain components.

[0153] The following describes the computer-readable storage medium provided in the embodiments of the present invention. The computer-readable storage medium described below can be referred to in correspondence with the bit-width compression method and query method described above.

[0154] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the bit-width compression method and the query method described above.

[0155] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0158] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0159] The present invention has provided a detailed description of a bit-width compression method, query method, apparatus, device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A bit-width compression method, characterized in that, include: The storage width is dynamically determined based on the number of values ​​that can be taken from the label; The tag value of the current tag is encoded into binary code based on the storage bit width; The binary codes are filled into a buffer in a preset order to obtain a binary string, and the binary string is converted into integer encoded data. The integer encoded data and the starting offset position corresponding to each binary code are stored in the metadata table of the database.

2. The bit-width compression method according to claim 1, characterized in that, The storage width is dynamically determined based on the number of possible values ​​for the tag, including: The storage width is obtained by dynamically predicting the number of values ​​based on the label using a storage width prediction model; wherein the storage width prediction model is a model obtained by training a machine learning model based on historical labels and their corresponding historical value counts.

3. The bit-width compression method according to claim 2, characterized in that, Before dynamically predicting the storage width using a storage width prediction model based on the number of values ​​based on the label, the method further includes: Collect metadata sequences for each tag over a historical time period. The metadata sequences include the number of values, historical growth trends, and query access patterns. The historical growth trend refers to the growth pattern of the number of tag values. The query access patterns are parameters related to storage efficiency and query performance. A training sample set is constructed based on the metadata sequence; The machine learning model is trained using the training sample set to obtain the storage bit width prediction model.

4. The bit-width compression method according to any one of claims 1 to 3, characterized in that, The binary codes are filled into a buffer in a preset order to obtain a binary string, and the binary string is converted into integer encoded data. The integer encoded data and the starting offset position corresponding to each binary code are stored in the metadata table of the database, including: Determine the number of bits already occupied in the binary string in the buffer, and determine whether the number of bits already occupied has reached the set number of bits; When the set number of bits is reached, the binary string is converted into the integer encoded data and stored in the tag value storage table.

5. The bit-width compression method according to claim 1, characterized in that, The storage width is dynamically determined based on the number of possible values ​​for the tag, including: Based on the number of values, the storage width is determined using a storage bit width function; wherein, the storage bit width function is a function that characterizes the correspondence between the number of values ​​and the storage bit width.

6. A query method, characterized in that, include: The tags and tag values ​​for the query are determined based on the query conditions; Based on a pre-stored metadata table, the storage bit width and starting offset position of the tag in the integer encoded data are queried; wherein, the metadata table is a metadata table determined based on the bit width compression method according to any one of claims 1 to 5; Obtain the integer code corresponding to the tag value; The mask is determined based on the starting offset position and the storage bit width; The target value is determined based on the integer encoding and the starting offset position; Determine whether the result of a bitwise AND operation between the target integer field and the mask is equal to the target value; Use the target integer field that is equal to the target value as the query result.

7. A bit-width compression device, characterized in that, include: The storage bit width determination module is used to dynamically determine the storage bit width based on the number of values ​​that the tag can take; The encoding module is used to encode the tag value of the current tag into binary code based on the storage bit width; The storage module is used to fill the binary code into the buffer in a preset order to obtain a binary string, convert the binary string into integer encoded data, and store the integer encoded data and the starting offset position corresponding to each binary code into the metadata table of the database.

8. A query device, characterized in that, include: The tag and tag value determination module is used to determine the tags and tag values ​​of a query based on the query conditions. The storage bit width and starting offset position determination module is used to query the storage bit width and starting offset position of the tag in the integer encoded data based on a pre-stored metadata table; wherein, the metadata table is a metadata table determined based on the bit width compression method described above. An integer encoding determination module is used to obtain the integer encoding corresponding to the tag value; The mask determination module is used to determine the mask based on the starting offset position and the storage bit width; The target value determination module is used to determine the target value based on the integer encoding and the starting offset position; The judgment module is used to determine whether the result of performing a bitwise AND operation between the target integer field and the mask is equal to the target value. The query result determination module is used to select the target integer field that is equal to the target value as the query result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the bit-width compression method as described in any one of claims 1 to 5, or the query method as described in claim 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the bit-width compression method as described in any one of claims 1 to 5, or the query method as described in claim 6.

Images