Data query method and device, electronic equipment and storage medium

Abstract

The invention discloses a data query method and device, electronic equipment and a storage medium, and relates to the technical field of computers.The data query method comprises the steps that a query key of to-be-queried data is constructed based on obtained storage features and data identifiers of the to-be-queried data; under the condition that the query key length of the query key is greater than a preset length threshold value, extracting a first prefix key from the query key according to the storage feature, and extracting a second prefix key from a target key according to the storage feature of the target key in the plurality of keys; a first prefix value of a first prefix key and a second prefix value of a second prefix key are respectively determined, a preset key value database is queried through the first prefix value and the second prefix value, and a key value matched with a query key is determined as a query result of data to be queried. And compared with a bitwise query mode, the comparison frequency can be remarkably reduced, and the data query efficiency is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data query method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of cloud computing and big data technology, there is a very high requirement for data access efficiency, query speed and storage cost. As a key technology supporting mass data storage and fast access, the performance optimization of the storage system has become one of the research hotspots.

[0003] However, when processing large-scale data sets, especially those with high-frequency access and a large number of deletion operations, the query performance of the storage system is limited. This is mainly because in the current query process, it is generally queried byte by byte. This query method will significantly increase unnecessary computing overhead, thereby affecting the data query efficiency of the entire storage system. SUMMARY

[0004] The present application provides a data query method and device, electronic equipment and storage medium to at least solve the problem of low data query efficiency caused by the bit-by-bit query method in the query process in the related art.

[0005] The present application provides a data query method, comprising: constructing a query key of the to-be-queried data based on the storage feature and the data identifier of the to-be-queried data obtained; in the case that the query key length of the query key is greater than a preset length threshold, extracting a first prefix key from the query key according to the storage feature, and extracting a second prefix key from a target key in a plurality of keys according to the storage feature of the target key; the target key is any one of the plurality of keys; determining a first prefix value of the first prefix key and a second prefix value of the second prefix key; based on the first prefix value and the second prefix value, querying in a preset key-value database, and determining the key value matched with the query key as the query result of the to-be-queried data; wherein the key-value database comprises the plurality of keys and the key values corresponding to the plurality of keys, respectively.

[0006] The application further provides a data query device, comprising: a construction module, configured to construct a query key of to-be-queried data based on an obtained storage feature and a data identifier of the to-be-queried data; an extraction module, configured to extract a first prefix key from the query key according to the storage feature in a case where a query key length of the query key is greater than a preset length threshold, and extract a second prefix key from a target key in a plurality of keys according to a storage feature of the target key; the target key is any one of the plurality of keys; a determination module, configured to determine a first prefix value of the first prefix key and a second prefix value of the second prefix key; a query module, configured to query in a preset key-value database based on the first prefix value and the second prefix value, and determine a key value matched with the query key as a query result of the to-be-queried data; wherein the key-value database comprises the plurality of keys and key values corresponding to the plurality of keys respectively.

[0007] The application further provides an electronic device, comprising: a memory, configured to store a computer program; and a processor, configured to execute the computer program to implement the steps of any one of the data query methods.

[0008] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the data query methods.

[0009] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any one of the data query methods.

[0010] According to the application, a query key of to-be-queried data is constructed based on an obtained storage feature and a data identifier of the to-be-queried data; in a case where a query key length of the query key is greater than a preset length threshold, a first prefix key is extracted from the query key according to the storage feature, and a second prefix key is extracted from a target key in a plurality of keys according to a storage feature of the target key; further, a first prefix value of the first prefix key and a second prefix value of the second prefix key are determined; based on the first prefix value and the second prefix value, a query is performed in a preset key-value database, and a key value matched with the query key is determined as a query result of the to-be-queried data; wherein the suffix key is determined by the data identifier of the to-be-queried data. That is, by extracting the first prefix key of the query key and the second prefix key of the target key, and respectively determining the first prefix value of the first prefix key and the second prefix value of the second prefix key, the query is performed based on the first prefix value and the second prefix value, which can significantly reduce the number of comparisons and greatly improve the data query efficiency, compared with the bit-by-bit query mode. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0012] Figure 1 The structural block diagram of a server device of a data query method provided by an embodiment of the present application is shown in FIG. 1.

[0013] Figure 2 The flow chart of a data query method provided by an embodiment of the present application is shown in FIG. 2.

[0014] Figure 3 The flow chart of a data query method provided by another embodiment of the present application is shown in FIG. 3.

[0015] Figure 4 The structural block diagram of a data query device provided by an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0017] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0018] In order for those skilled in the art to better understand the solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0019] The data query method embodiments provided in the embodiments of the present application can be executed in a server device or similar computing device. Taking the case of running on a server device, Figure 1 The hardware structural block diagram of a server device of a data query method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the server device can include one or more (CPU) 101, memories 102, buses 103, input devices 104, output devices 105, storage devices 106 and network interfaces 107. Figure 1 The hardware structural block diagram of a server device of a data query method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the server device can include one or more (CPU) 101, memories 102, buses 103, input devices 104, output devices 105, storage devices 106 and network interfaces 107. Figure 1The server device shown in FIG. 1 includes only one processor 102 (the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. The server device can further include a transmission device 106 for communication functions, and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown in FIG. 1 is only schematic, and does not limit the structure of the server device. For example, the server device can include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1. Figure 1 Figure 1 The structure shown in FIG. 1 is only schematic, and does not limit the structure of the server device. For example, the server device can include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0020] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the data query method in the embodiments of the present application. The processor 102 can execute various functional applications and data processing by running the computer programs stored in the memory 104, that is, implement the method described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, and the remote memory can be connected to the server device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0021] The transmission device 106 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication service provider of the server device. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0022] The embodiments of the present application provide a data query method applied to the server device. The method will be described in detail in combination with the execution flow of the data query method. As shown in FIG. 2, the method includes the following steps S202-S208: Figure 2

[0023] S202, constructing a query key of the to-be-queried data based on the obtained storage characteristics and data identifier of the to-be-queried data;

[0024] ​​The storage characteristics of data refer to the classification and organization of data in the storage system, which can include storage pool ID (identifier), namespace ID, object type identifier, etc. The storage characteristics can provide a basis for subsequent data retrieval. The determination of storage characteristics is often static and depends on the metadata information of data when it is stored in the system.

[0025] The data identifier is specific information used to uniquely identify data, such as object name, file name, snapshot ID, etc. These identifiers ensure that different data can be distinguished even within the same namespace.

[0026] Specifically, when constructing the query key of the data to be queried, the storage characteristics of the data to be queried can be identified from its metadata, such as storage pool ID and namespace ID, etc. Then, the data identifier of the data to be queried is determined, such as the object name. This is the unique identifier of the data, which is used to accurately locate the data item. The storage characteristics and data identifier are concatenated to form the query key.

[0027] In some embodiments, after concatenating the query key, the query key can be formatted to meet the storage and retrieval requirements of the storage system. The formatting process can include encoding the query key, ensuring that the length of the query key meets the database requirements, or adding separators to improve readability.

[0028] S204, in the case where the query key length of the query key is greater than a preset length threshold, a first prefix key is extracted from the query key according to the storage characteristics, and a second prefix key is extracted from the target key according to the storage characteristics of the target key in the plurality of keys; the target key is any one of the plurality of keys;

[0029] The preset length threshold is a key length value set in advance, which is used to distinguish whether the query key is too long to extract and use the prefix key. The selection of the preset length threshold depends on multiple factors, including the data type in the storage system, the data distribution characteristics, the query frequency, and the processing capacity of the storage system itself, etc. For example, 64 bits (8 bytes) can be set as the preset length threshold.

[0030] Specifically, when querying, it can be judged whether the query key length exceeds the preset length threshold. If the query key length is less than or equal to the preset length threshold (e.g. 64 bits), the default comparator can be used for key value comparison (i.e. bitwise comparison), without additional processing. But if the query key length exceeds the preset length threshold, the prefix key extraction operation needs to be performed next.

[0031] It can be understood that the first prefix key is extracted from the query key according to the storage feature when the length of the query key is greater than the preset length threshold. First, the field length of the storage feature in the query key is determined. Then, the prefix of the corresponding length is extracted from the query key as the first prefix key according to the field length of the storage feature. For example, if the total length of the storage pool ID and the namespace ID is 64 bits, 64 bits of data is extracted from the starting position of the query key. For multiple keys in the key-value database, the second prefix key needs to be extracted from any one of the keys (target key) so as to be compared with the first prefix key of the query key. This process is similar to extracting the first prefix key, except that the object of the operation is changed to the target key.

[0032] S206, determining a first prefix value of the first prefix key and a second prefix value of the second prefix key;

[0033] It can be understood that the first prefix key is a part of the key containing the storage feature extracted from the query key, which is used for quick filtering and positioning data. The second prefix key is extracted from the target key and has the same structure as the first prefix key, which is used for comparison with the first prefix key.

[0034] Specifically, the storage feature contained in the first prefix key is converted into a numerical representation. For example, if the first prefix key is 64 bits, the 64 bits can be converted into an integer, which is the first prefix value of the first prefix key. Similarly, the same length of the storage feature is extracted from the second prefix key and converted into an integer, which is the second prefix value.

[0035] In an exemplary embodiment, the above step S206 can be implemented by determining the first prefix value of the first prefix key and the second prefix value of the second prefix key, including: determining a first byte sequence of the first prefix key, and determining a second byte sequence of the second prefix key; performing numerical conversion on the first byte sequence to obtain the first prefix value, and performing numerical conversion on the second byte sequence to obtain the second prefix value.

[0036] The first byte sequence is the byte representation of the first prefix key. It is the actual content of the first prefix key in byte units. The second byte sequence is the byte representation of the second prefix key. It is the actual content of the second prefix key in byte units.

[0037] It can be understood that the conversion of the byte sequence to the numerical value is the process of converting the byte sequence (binary form) into a single numerical value (such as unsigned integer uint64_t). This conversion allows the use of the efficiency of integer comparison and avoids the overhead of byte-by-byte comparison.

[0038] It should be noted that the numerical conversion can be implemented in various ways. For example, the values of each byte of the byte sequence can be merged into an integer using a bit shift operation. Alternatively, the byte sequence can be parsed in hexadecimal representation and converted into an integer. By converting the first byte sequence into an integer, a first prefix value is obtained. In the same way as the first prefix value, the second byte sequence is also converted into a second prefix value by numerical conversion. In this way, the two prefix keys are represented as integers that can be directly compared numerically.

[0039] In one specific application, the first prefix key is DB_001_NS001 (first 64 bits) and the second prefix key is DB_002_NS002 (first 64 bits). First, they are converted into byte sequences. Then, using bit operations or numerical conversion functions, the two byte sequences are converted into integers of type uint64_t. For example, DB_001_NS001 can be converted into the numerical value 123456789, and DB_002_NS002 can be converted into 987654321.

[0040] In the above embodiment, by determining and converting the first prefix value and the second prefix value, the query process can be optimized, unnecessary bitwise comparison can be reduced, and the query efficiency can be improved.

[0041] S208, based on the first prefix value and the second prefix value, querying in a preset key-value database, determining the key value matching the query key as the query result of the data to be queried; wherein the key-value database comprises: a plurality of keys, and a plurality of key values corresponding to the plurality of keys respectively.

[0042] The key-value database is a data storage system, and its core data structure is a set of (key, value) pairs. The key (Key) is used to uniquely identify data, and the key value (Value) stores the actual data content. The key can be defined by the user and can contain various information, such as storage pool ID, namespace ID, object name, etc., while the key value contains data associated with the key.

[0043] Specifically, when searching for data in the key-value database, the first prefix value and the second prefix value can be compared. Since the first prefix value and the second prefix value are both integer values, such comparison is usually faster than byte-by-byte key value comparison. If the two prefix values do not match, the key can be immediately excluded without comparing the remaining part of the key, thereby greatly reducing unnecessary comparisons. Only when the first prefix value matches the second prefix value, further comparison of the remaining part of the key is needed to determine whether the key completely matches the query key. When the key completely matches the query key, the corresponding key value is the query result.

[0044] The steps S202-S208 described above, based on the obtained storage characteristics and data identifiers of the data to be queried, construct the query key of the data to be queried; in the case where the query key length of the query key is greater than the preset length threshold, the first prefix key is extracted from the query key according to the storage characteristics, and the second prefix key is extracted from the target key according to the storage characteristics of the target key in the plurality of keys; further, the first prefix value of the first prefix key and the second prefix value of the second prefix key are determined; based on the first prefix value and the second prefix value, the key value matching the query key is determined as the query result of the data to be queried in the preset key-value database; wherein the suffix key is determined by the data identifier of the data to be queried. That is, by extracting the first prefix key of the query key and the second prefix key of the target key, and respectively determining the first prefix value of the first prefix key and the second prefix value of the second prefix key, the query is expanded through the first prefix value and the second prefix value, which can significantly reduce the number of comparisons and greatly improve the data query efficiency, compared with the bit-by-bit query mode.

[0045] In some exemplary embodiments, according to the storage characteristics, the first prefix key is extracted from the query key, and according to the storage characteristics of the target key in the plurality of keys, the second prefix key is extracted from the target key, comprising: determining the field length of the storage characteristics; based on the storage characteristics at the starting position of the query key, the first prefix key of the field length is extracted, and based on the storage characteristics at the starting position of the target key, the second prefix key of the field length is extracted.

[0046] The storage characteristics include storage pool ID, namespace ID, etc. For example, the storage pool ID is 3 characters, and the namespace ID is 5 characters. Then, according to the starting position, an 8-bit field can be determined as the prefix of the storage characteristics.

[0047] It can be understood that in the query key, the starting position of the storage characteristics, i.e. the beginning part of the query key, is first identified. Based on the determined field length of the storage characteristics, the first prefix key of the corresponding length is extracted from the starting position of the query key. For example, if the field length of the storage characteristics is 8 bits, the first prefix key can be extracted from the beginning part of the query key. The target key refers to the key stored in the database, which is potentially matched with the query key. Using the same standard as the query key, the same length field is extracted from the starting position of the target key as the second prefix key.

[0048] In some embodiments, the field length of the storage feature is 64 bits (8 bytes), and the query key is DB_001_NS001_ObjectName001, and the first 8 bytes DB_001_NS001 is extracted from the start of the key as the first prefix key. For the target key DB_002_NS002_AnotherObjectName, the first 8 bytes DB_002_NS002 is also extracted from the start as the second prefix key.

[0049] In the above embodiments, the prefix keys are effectively extracted from the query key and the target key, and then the prefix keys are used to optimize the query process, improve the query efficiency, and reduce unnecessary comparisons.

[0050] In an exemplary embodiment, based on the first prefix value and the second prefix value, the key value matching the query key is determined as the query result of the to-be-queried data in the preset key-value database, including: comparing the first prefix value and the second prefix value to obtain a comparison result; in the case that the comparison result is that the first prefix value and the second prefix value are the same, querying the key-value database based on the identification suffix key of the query key, and determining the key value matching the query key as the query result of the to-be-queried data; wherein the identification suffix key is determined by the data identification of the to-be-queried data.

[0051] The identification suffix key refers to the remaining part of the query key other than the storage feature prefix, such as the object name or the data identification, etc. When the first prefix value and the second prefix value are equal, the identification suffix key is used to determine the exact match of the key.

[0052] It can be understood that when querying, the first prefix value (the value converted from the prefix part of the query key) and the second prefix value (the value converted from the prefix part of the target key in the key-value database) can be compared to quickly determine whether the target key is likely to match the query key. Since the first prefix value and the second prefix value are both integer values, standard integer comparison operations such as <, > or == can be used to determine their size relationship or whether they are equal.

[0053] When the first prefix value is equal to the second prefix value, it means that the target key and the query key are the same in the storage feature part. At this time, the suffix part of the key can be further compared to determine whether it is a complete match. If the first prefix value is not equal to the second prefix value, the target key can be immediately excluded because the different storage features mean that the key is not the target of the query. In the case of prefix value matching, the identification suffix key can be used to further query in the database. The identification suffix key is compared with the remaining part of the target key to find the key value pair that completely matches.

[0054] In the above embodiments, based on the first prefix value and the second prefix value, the numerical representation of the stored feature can be quickly compared, the possible matching items can be screened out, and the query result can be determined, thereby significantly improving the query performance of the key-value database.

[0055] In an exemplary embodiment, in the case where the first prefix value and the second prefix value are the same, the identification suffix key of the query key is used to query the preset key-value database, and the key value matching the query key is determined as the query result of the data to be queried, including: in the case where the first prefix value and the second prefix value are the same, the identification suffix key of the query key is compared with the identification suffix key of the target key bit by bit, and the bit-by-bit comparison result is determined; based on the bit-by-bit comparison result, the preset key-value database is queried, and the key value matching the query key is determined as the query result of the data to be queried.

[0056] It should be noted that the prefix value being equal indicates that the target key and the query key share the same context environment such as the storage pool ID and the namespace ID, thereby possibly pointing to the same batch of data sets, and thus preparing for accurate matching. In the case where the prefix values are equal, the identification suffix key of the query key and the identification suffix key of the target key are further used for bit-by-bit accurate comparison. The identification suffix key is the part of the key other than the prefix, and is usually used to distinguish different data objects with the same storage feature.

[0057] It can be understood that the bit-by-bit comparison means starting from the first byte of the suffix key and comparing one by one until the last byte, so as to ensure that each byte of the two is the same. The result of the bit-by-bit comparison can explicitly indicate whether there is a difference between the two suffix keys. If all the bytes are the same, it means that the keys are completely matched; if any byte is different, it means that the keys are not matched. Once the bit-by-bit comparison result of the identification suffix key shows that the two keys are completely matched, the key corresponding to the query key can be located in the key-value database, and the stored key value can be obtained.

[0058] In the above embodiments, in the case where the first prefix value and the second prefix value are the same, the identification suffix key of the query key and the identification suffix key of the target key are further used for bit-by-bit accurate comparison, the quick comparison of the prefix value effectively reduces the search range, and the subsequent bit-by-bit comparison of the identification suffix key ensures the accurate retrieval of data, thereby making the data query not only fast but also accurate.

[0059] In an example embodiment, based on the bit-by-bit comparison result, the query in the preset key-value database is performed, and the key value matching the query key is determined as the query result of the data to be queried, including: in the case that the bit-by-bit comparison result is that the identification suffix key of the query key and the identification suffix key of the target key are consistent, the key value of the target key is determined as the query result of the data to be queried; in the case that the bit-by-bit comparison result is that the identification suffix key of the query key and the identification suffix key of the target key are inconsistent, a new target key is determined from the plurality of keys, a second prefix key is extracted from the new target key according to the storage characteristics of the new target key, and the step of returning the first prefix value of the first prefix key and the second prefix value of the second prefix key is continued to be executed until the query result of the data to be queried is determined.

[0060] When the bit-by-bit comparison result of the identification suffix key shows complete consistency, it means that the target key matches the query key. At this time, the key value (i.e. the stored data) associated with the target key will be marked as the query result and then returned to the query initiator. If the bit-by-bit comparison result of the identification suffix key shows inconsistency, the next target key will be searched in the key-value database, and the process of extracting the second prefix key and performing bit-by-bit comparison will be repeated. The termination condition of the loop can be that all key values matching the query key are found or there is no key left to be compared in the key-value database, which is not limited in the embodiments of the present application.

[0061] In some embodiments, the query key is DB_001_NS001_ObjectName001, the first 64-bit prefix value matches, and the bit-by-bit comparison stage of the identification suffix key is entered: target key 1: DB_001_NS001_AnotherObjectName, the identification suffix key is inconsistent, and is skipped. Target key 2: DB_001_NS001_ObjectName001, the identification suffix key is consistent, and it is confirmed as the query result, and its key value is read from the key-value database.

[0062] In the above embodiments, the query mechanism based on the bit-by-bit comparison result can efficiently locate the data, thereby significantly improving the query performance.

[0063] In an example embodiment, the storage characteristics include the data storage type, the namespace, and the storage pool of the data to be queried, and the data identifier includes the object name of the data to be queried. Based on the obtained storage characteristics and data identifier of the data to be queried, the query key of the data to be queried is constructed, including: based on the data storage type, the namespace, and the storage pool, the first prefix key of the data to be queried is constructed, and based on the object name, the identification suffix key of the data to be queried is constructed; and the query key of the data to be queried is constructed according to the first prefix key and the identification suffix key.

[0064] The data storage type is the storage structure or category of the data. The namespace is used to isolate the logical space of different data sets, ensuring the independence and security of different business data. The storage pool is a physical or logical data storage area used to manage storage resources and data distribution. The data identifier, such as the object name, is used to uniquely identify a single data object in a data set with the same storage characteristics, so that even in the case of the same storage pool and namespace, different objects can be accurately distinguished and retrieved by their identifiers.

[0065] In some embodiments, the format of the first prefix key can be defined, typically including information of the data storage type, namespace, and storage pool. For example, the format can be set as <storage type> _ <storage pool ID> _ <namespace ID>. The storage type, storage pool ID, and namespace ID of the data to be queried are combined according to the predefined format to form the first prefix key. For example, for a data object with a storage type of RocksDB, a storage pool ID of 001, and a namespace ID of NS001, the first prefix key can be RocksDB_001_NS001.

[0066] In some embodiments, the unique identifier of the data to be queried, i.e., the object name, is extracted. The object name can be formatted to ensure that it meets the key structure requirements of the key-value database.

[0067] Specifically, the first prefix key and the identification suffix key are connected according to the predefined rules to form the complete query key. For example, the first prefix key RocksDB_001_NS001 is connected with the identification suffix key ObjectName001 to form the query key RocksDB_001_NS001_ObjectName001.

[0068] It can be understood that, in order to optimize the query efficiency, the length of the first prefix key can be set to a fixed value, such as 64 bits. This helps to quickly locate and compare the storage characteristics during the query, reducing unnecessary key-value comparisons. In actual construction of the query key, the storage characteristics and data identifier can be converted from their original form (such as a string) to a specific encoding format to meet the storage requirements.

[0069] In a specific application, let's query a data object stored in RocksDB. Its storage characteristics are: storage type RocksDB, storage pool ID 001, namespace ID NS001, and data identifier (object name) ObjectName001. Following the predefined format <storage type>_<storage pool ID>_<namespace ID>, the first prefix key is RocksDB_001_NS001. The identifier suffix key is ObjectName001. The first prefix key and the identifier suffix key are concatenated with an underscore to form the complete query key RocksDB_001_NS001_ObjectName001.

[0070] In the above embodiment, by combining storage characteristics and data identifiers to construct query keys, data can be efficiently and accurately located. The first prefix key is used to quickly identify the storage environment, and the identification suffix key is used to accurately match data objects. The combination of the two significantly improves database query efficiency and system response speed.

[0071] The embodiments described above are only part of the embodiments of the present application, not all of the embodiments. In order to better understand the above method, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application. Specifically:

[0072] With the explosive growth of data, traditional storage methods are no longer able to meet increasingly stringent requirements. In this context, software-defined distributed storage technology, with its highly scalable and adaptable system architecture, meets the real requirements of ever-increasing data volumes.

[0073] In distributed storage systems, object data is stored directly on block devices, while object metadata is stored in RocksDB (an embedded key-value storage system). However, with the continuous growth of data volumes and the increasing complexity of business scenarios, RocksDB faces challenges in query efficiency when processing certain types of data. This is particularly true for data that is frequently used or subject to a large number of deletions. This is because traditional comparators require a complete bit-by-bit comparison of key values. This results in an excessive number of comparisons when the data volume is large, severely impacting query performance and, consequently, the performance of the distributed storage system.

[0074] This application provides a data query method. Based on the data type recorded in the RocksDB distributed storage system, it fully utilizes the characteristics of the RocksDB comparator to design a fixed-length prefix as the RocksDB key to store data. A new key comparator is designed to first compare the fixed-length prefix and then compare the remaining suffix, thereby reducing the number of comparisons and improving the query efficiency of RocksDB.

[0075] Reference Figure 3 As shown in the flowchart of the data query method, comprising the following steps:

[0076] S301, determine whether the query key length is greater than 64 bits. In a distributed storage system, there are various data types, but there are some high-frequency use or specific pattern data, such as metadata, snapshot data, etc. These data usually have certain classification characteristics, such as belonging to a certain storage pool, namespace or having certain business attributes. According to the above data type analysis, the query key is constructed, wherein the query key format is PREFIX+storage pool ID+namespace+object name. PREFIX refers to a fixed format character sequence, which is used to classify data.

[0077] Because shorter query keys and longer query keys may have different characteristics when compared, targeted comparison methods can improve efficiency. If the query key length is less than 64 bits, direct bitwise comparison is performed, that is, using ByteWiseComparator (a comparator that compares two keys byte by byte in the storage system) to compare. When the query key length is small, enter S303, the overhead of byte-by-byte comparison is relatively small, and the default ByteWiseComparator can meet the demand, and will not have a significant impact on the overall performance. When the query key length is greater than or equal to 64 bits, enter S302;

[0078] S302, determine the first prefix key from the query key; wherein the fixed prefix part is PREFIX+storage pool ID+namespace, and the length of this field is set to 64 bits, ensuring that the first 64 bits of data are consistent;

[0079] S303, bitwise comparison;

[0080] S304, determine the second prefix key of the target key from the key-value database;

[0081] S305, determine the first prefix value of the first prefix key and the second prefix value of the second prefix key;

[0082] S306, determine whether the prefix values are equal. That is, compare the first prefix value of the first prefix key with the second prefix value of the second prefix key. Using the feature that the first 64 bits of the same type of information in the distributed storage system are the same, the prefix key can be converted into a uint64_t type value, so that the comparison times are reduced from 64 times to 1 time. If the first 64 bits are the same, delete the first 64 bits of the key value, and enter S303, the suffix key is compared using the ByteWiseComparator comparator;

[0083] S307, determine whether the suffix key is consistent to determine the query result.

[0084] The application discloses a data query method. In the query, a quick comparison is first performed on the first 64 bits, a large number of bit-by-bit comparison operations are reduced, the number of comparisons is reduced, calculation resources and time are saved, and the query process is more efficient. In processing large-scale data query, the optimization can significantly reduce the query response time, improve the query performance of the system, effectively reduce the load of the entire system, and improve the stability and scalability of the system.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platforms, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment.

[0086] Embodiments of the application also provide a parameter adjustment device for implementing the above embodiments and preferred embodiments, which have been described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the modules described in the following embodiments are preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0087] Figure 4 is a structural block diagram of a data query device according to an embodiment of the application, and the device comprises:

[0088] The construction module 402 is configured to construct a query key of the to-be-queried data based on the obtained storage feature and data identifier of the to-be-queried data.

[0089] The extraction module 404 is configured to extract a first prefix key from the query key according to the storage feature in a case where a query key length of the query key is greater than a preset length threshold, and extract a second prefix key from a target key in the plurality of keys according to a storage feature of the target key; the target key is any one of the plurality of keys.

[0090] The determination module 406 is configured to determine a first prefix value of the first prefix key and a second prefix value of the second prefix key.

[0091] The query module 408 is configured to query a preset key-value database based on the first prefix value and the second prefix value, and determine a key value matched with the query key as a query result of the to-be-queried data; wherein the key-value database comprises the plurality of keys and key values corresponding to the plurality of keys respectively.

[0092] By the above device, based on the obtained storage feature and data identifier of the to-be-queried data, the query key of the to-be-queried data is constructed; in the case that the query key length of the query key is greater than a preset length threshold, a first prefix key is extracted from the query key according to the storage feature, and a second prefix key is extracted from the target key according to the storage feature of the target key in the plurality of keys; further, a first prefix value of the first prefix key and a second prefix value of the second prefix key are determined; based on the first prefix value and the second prefix value, a query is performed in the preset key-value database, and a key value matched with the query key is determined as the query result of the to-be-queried data; wherein the identification suffix key is determined by the data identifier of the to-be-queried data. That is, by extracting the first prefix key of the query key and the second prefix key of the target key, and respectively determining the first prefix value of the first prefix key and the second prefix value of the second prefix key, the query is expanded by the first prefix value and the second prefix value, which can significantly reduce the number of comparisons and greatly improve the data query efficiency, compared with the bit-by-bit query mode.

[0093] In an example embodiment, the extraction module 404 is further configured to determine a field length of the storage feature; extract a first prefix key of the field length at a starting position of the query key based on the storage feature, and extract a second prefix key of the field length at a starting position of the target key based on the storage feature.

[0094] In an example embodiment, the determination module 406 is further configured to determine a first byte sequence of the first prefix key, and determine a second byte sequence of the second prefix key; perform numerical conversion on the first byte sequence to obtain the first prefix value, and perform numerical conversion on the second byte sequence to obtain the second prefix value.

[0095] In an example embodiment, the query module 408 is further configured to compare the first prefix value and the second prefix value to obtain a comparison result; in the case that the comparison result is that the first prefix value and the second prefix value are the same, perform a query in the key-value database based on an identification suffix key of the query key, and determine a key value matched with the query key as the query result of the to-be-queried data; wherein the identification suffix key is determined by the data identifier of the to-be-queried data.

[0096] In an example embodiment, the query module 408 is further configured to, in the case that the first prefix value and the second prefix value are the same, perform a bit-by-bit comparison between an identification suffix key of the query key and an identification suffix key of the target key based on the identification suffix key of the query key, determine a bit-by-bit comparison result, and perform a query in the preset key-value database based on the bit-by-bit comparison result, and determine a key value matched with the query key as the query result of the to-be-queried data.

[0097] In an example embodiment, the query module 408 is further configured to, in a case where the bit-by-bit comparison result is that the identification suffix key of the query key is consistent with the identification suffix key of the target key, determine the key value of the target key as the query result of the data to be queried; in a case where the bit-by-bit comparison result is that the identification suffix key of the query key is inconsistent with the identification suffix key of the target key, determine a new target key from the plurality of keys again, extract a second prefix key from the new target key according to the storage feature of the new target key, and continue to execute the step of determining the first prefix value of the first prefix key and the second prefix value of the second prefix key until the query result of the data to be queried is determined.

[0098] In an example embodiment, the storage feature includes a data storage type, a namespace, and a storage pool of the data to be queried, and the data identification includes an object name of the data to be queried; the construction module 402 is further configured to construct the first prefix key of the data to be queried based on the data storage type, the namespace, and the storage pool, and construct the identification suffix key of the data to be queried based on the object name; and construct the query key of the data to be queried according to the first prefix key and the identification suffix key.

[0099] The features of the embodiments of the data query apparatus can refer to the related descriptions of the embodiments of the data query method, which will not be repeated here.

[0100] The embodiments of the present application further provide an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the above-mentioned data query method embodiments.

[0101] The embodiments of the present application further provide a computer readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned data query method embodiments when executed.

[0102] In an example embodiment, the above-mentioned computer readable storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0103] The embodiments of the present application further provide a computer program product including a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned data query method embodiments.

[0104] The embodiment of the present application further provides another computer program product, comprising a nonvolatile computer readable storage medium, the nonvolatile computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the steps in any of the above data query method embodiments.

[0105] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the elements of the examples have been described in general terms in the above description. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in varying ways for each particular application, but such implementation should not be interpreted as a departure from the scope of the present application.

[0106] The above has introduced in detail a data query method provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in the present text, and the above description of the examples is only for helping to understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data query method, characterized in that: include: Constructing a query key for the data to be queried based on the acquired storage characteristics and data identifier of the data to be queried; If the query key length of the query key is greater than a preset length threshold, extracting a first prefix key from the query key according to the stored characteristics, and extracting a second prefix key from the target key according to the stored characteristics of a target key among the multiple keys; the target key is any one of the multiple keys; determining a first prefix value for the first prefix key and a second prefix value for the second prefix key; Based on the first prefix value and the second prefix value, a query is performed in a preset key-value database, and a key value matching the query key is determined as a query result of the data to be queried; wherein the key-value database includes: the multiple keys, and the key values ​​corresponding to the multiple keys respectively.

2. The data query method according to claim 1, characterized in that: The extracting a first prefix key from the query key according to the storage feature, and extracting a second prefix key from the target key according to the storage feature of the target key among the multiple keys, comprises: Determining the field length of the stored feature; A first prefix key of the field length is extracted based on the storage feature at a starting position of the query key, and a second prefix key of the field length is extracted based on the storage feature at a starting position of the target key.

3. The data query method according to claim 1, characterized in that: The determining a first prefix value of the first prefix key and a second prefix value of the second prefix key includes: determining a first byte sequence for the first prefix key, and determining a second byte sequence for the second prefix key; The first byte sequence is numerically converted to obtain the first prefix value, and the second byte sequence is numerically converted to obtain the second prefix value.

4. The data query method according to claim 1, characterized in that: The querying in a preset key-value database based on the first prefix value and the second prefix value, and determining a key value matching the query key as a query result of the data to be queried, includes: Comparing the first prefix value and the second prefix value to obtain a comparison result; When the comparison result is that the first prefix value and the second prefix value are the same, a query is made in the key-value database based on the identification suffix key of the query key, and the key value matching the query key is determined as the query result of the data to be queried; wherein, the identification suffix key is determined by the data identification of the data to be queried.

5. The data query method according to claim 4, characterized in that: When the first prefix value and the second prefix value are the same, searching a preset key-value database based on the identification suffix key of the query key, and determining a key value matching the query key as a query result of the data to be queried, includes: In the case where the first prefix value and the second prefix value are the same, performing a bitwise comparison between the identification suffix key of the query key and the identification suffix key of the target key to determine a bitwise comparison result; Based on the bitwise comparison result, a query is performed in a preset key-value database, and a key value matching the query key is determined as a query result of the data to be queried.

6. The data query method according to claim 5, characterized in that: The step of searching a preset key-value database based on the bitwise comparison result and determining a key value matching the query key as a query result of the data to be queried includes: If the bitwise comparison result shows that the identification suffix key of the query key and the identification suffix key of the target key are consistent, determining the key value of the target key as the query result of the data to be queried; When the bitwise comparison result is that the identification suffix key of the query key and the identification suffix key of the target key are inconsistent, a new target key is re-determined from the multiple keys, a second prefix key is extracted from the new target key according to the storage characteristics of the new target key, and the step of determining the first prefix value of the first prefix key and the second prefix value of the second prefix key is returned to continue execution until the query result of the data to be queried is determined.

7. The data query method according to claim 1, characterized in that: The storage characteristics include: the data storage type, namespace and storage pool of the data to be queried; the data identifier includes: the object name of the data to be queried; The step of constructing a query key for the data to be queried based on the acquired storage characteristics and data identifier of the data to be queried includes: Constructing a first prefix key of the data to be queried based on the data storage type, the namespace, and the storage pool, and constructing an identification suffix key of the data to be queried based on the object name; A query key for the data to be queried is constructed according to the first prefix key and the identification suffix key.

8. A data query device, characterized in that: include: A construction module, configured to construct a query key for the data to be queried based on the acquired storage characteristics and data identifier of the data to be queried; an extraction module, configured to extract a first prefix key from the query key according to the stored characteristics, and extract a second prefix key from the target key according to the stored characteristics of the target key among the multiple keys, if the query key length of the query key is greater than a preset length threshold; a determining module, configured to determine a first prefix value of the first prefix key and a second prefix value of the second prefix key; A query module is used to query in a preset key-value database based on the first prefix value and the second prefix value, and determine the key value matching the query key as the query result of the data to be queried; wherein, the key-value database includes: the multiple keys, and the key values ​​corresponding to the multiple keys respectively.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data query method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data query method according to any one of claims 1 to 7.

Images