Data query method and device

Abstract

The present specification provides a data query method and device, wherein the data query method includes: obtaining a data query request, wherein the data query request carries a partition key and query attribute information; determining a target partition corresponding to the partition key in a first database, and reading attribute information of a data block contained in the target partition; when the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information; determining index information according to the reading result, and reading target data corresponding to the data query request in the second database based on the index information.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data query method and device. Background Art

[0002] With the development of Internet technology, more and more information has begun to be digitized, and some online service platforms will provide users with log storage services in order to meet the user's backtracking requests, so that when the user traces back to the relevant data, the log storage service provided by the online service platform can be used to read the user's relevant data and feedback. In the prior art, the multi-tenant log database used by the online service platform uses a column-based data file format to store relevant data. Although the storage and retrieval of relevant data can be achieved, since the data is stored in a column-based data file format, it is impossible to provide an index structure and an effective retrieval method, resulting in the log database not only being unable to meet high write traffic, but also having the problem of low retrieval performance. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a data query method. This specification also relates to a data query device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data query method is provided, including:

[0005] Obtaining a data query request, wherein the data query request carries a partition key and query attribute information;

[0006] Determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition;

[0007] In the case that the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0008] Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

[0009] Optionally, before the step of obtaining the data query request is executed, the step further includes:

[0010] Extracting the data to be migrated whose storage time is greater than a preset time threshold in the first database, and determining the migration partition key corresponding to the data to be migrated;

[0011] compressing the data to be migrated into compressed data, and writing the compressed data into a partition corresponding to the migration partition key in the second database;

[0012] Determine the compression attribute information and compression index information corresponding to the compressed data according to the writing result, and establish a storage correspondence between the migration partition key and the compression attribute information and the compression index information;

[0013] The storage relationship table corresponding to the second database is updated based on the storage correspondence relationship.

[0014] Optionally, determining the compression index information corresponding to the compressed data according to the writing result includes:

[0015] Determining whether the compressed data meets the preset index creation condition;

[0016] If so, create a compression index corresponding to the compressed data according to the writing result, and determine the compression index information based on the compression index.

[0017] Optionally, creating a compression index corresponding to the compressed data according to the writing result includes:

[0018] Determining the data type of the compressed data;

[0019] In the case where the data type is a text type, creating an inverted index corresponding to the compressed data according to the writing result as the compressed index;

[0020] In the case where the data type is a numerical type, a tree index corresponding to the compressed data is created according to the writing result as the compressed index.

[0021] Optionally, before the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data block contained in the target partition is performed, the step further includes:

[0022] Detecting whether the query attribute information contains time information;

[0023] If not, the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data blocks included in the target partition is executed.

[0024] Optionally, if the result of the detection of whether the query attribute information contains time information is yes, the following steps are performed:

[0025] Determining whether the time information meets the preset time division condition;

[0026] If yes, determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data blocks included in the target partition;

[0027] The attribute information is matched with the query attribute information, and target data corresponding to the data query request is read in the data block according to the matching result.

[0028] Optionally, if the result of determining whether the time information meets the preset time division condition is no, executing the following steps:

[0029] Read the storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0030] Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

[0031] Optionally, when the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information before executing the step further includes:

[0032] Parsing the attribute information to obtain a first value interval, and parsing the query attribute information to obtain a second value interval;

[0033] determining whether the first numerical interval includes the second numerical interval;

[0034] If so, reading the target data corresponding to the data query request in the data block based on the query attribute information;

[0035] If not, the step of determining a second database having a data migration relationship with the first database and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information is executed.

[0036] Optionally, reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information includes:

[0037] Performing a hash operation on the partition key and the query attribute information to obtain a query hash value;

[0038] Determine, in the hash table corresponding to the second database, candidate partition keys and time interval information corresponding to the query hash value;

[0039] Determine a second target partition in the second database according to the candidate partition key and the time interval information, and read second attribute information of the candidate data block included in the second target partition;

[0040] In the case where the second attribute information matches the query attribute information, the storage relationship table corresponding to the second database is read according to the query attribute information.

[0041] Optionally, determining the index information according to the reading result includes:

[0042] When a query index is obtained according to the reading result, the query index is used as the index information;

[0043] Correspondingly, reading the target data corresponding to the data query request in the second database based on the index information includes:

[0044] The target data corresponding to the data query request is read in the second database based on the query index.

[0045] According to a second aspect of an embodiment of this specification, a data query device is provided, including:

[0046] An acquisition request module is configured to acquire a data query request, wherein the data query request carries a partition key and query attribute information;

[0047] an information reading module, configured to determine a target partition corresponding to the partition key in the first database, and read attribute information of a data block contained in the target partition;

[0048] a database determination module configured to, when the query attribute information does not match the attribute information, determine a second database having a data migration relationship with the first database, and read a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0049] The data reading module is configured to determine index information according to the reading result, and read the target data corresponding to the data query request in the second database based on the index information.

[0050] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0051] Memory and processor;

[0052] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0053] Obtaining a data query request, wherein the data query request carries a partition key and query attribute information;

[0054] Determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition;

[0055] In the case that the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0056] Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

[0057] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data query method are implemented.

[0058] The data query method provided in this specification, when obtaining the partition key and query attribute information, will determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition. When the query attribute information does not match the attribute information, the second database having a data migration relationship with the first database can be determined at this time, and the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information. The index information can be determined based on the reading, and finally the target data corresponding to the data query request is read in the second database based on the index information, so that the problem of large write traffic and inability to store data is solved by combining the first database and the second database, and different methods are used for data query for the first database and the second database, which effectively improves the data query efficiency, thereby realizing a rapid response to the data query request. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a data query method provided in an embodiment of this specification;

[0060] Figure 2 It is a schematic diagram of a cold data format in a data query method provided in an embodiment of this specification;

[0061] Figure 3 It is a processing flow chart of a data query method applied to a log audit scenario provided by an embodiment of this specification;

[0062] Figure 4It is a structural schematic diagram of a data query device provided in an embodiment of this specification;

[0063] Figure 5 It is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0064] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0065] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0066] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0067] First, the terms involved in one or more embodiments of this specification are explained.

[0068] Tenant: refers to the user who uses the log storage service (Log Storage) provided by the business service provider. The same Log Storage can be used by multiple tenants.

[0069] Hot data: refers to real-time data with a short writing time. The data structure of hot data is write-optimized.

[0070] Cold data: refers to data that takes a long time to write and is converted from hot data. The data structure of cold data is read-optimized.

[0071] In this specification, a data query method is provided. This specification also relates to a data query device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0072] In practical applications, in order to improve the computing performance and storage efficiency of Hadoop Hive, the ORC column storage data format is usually used for data storage. This format usually includes Postscript, File Footer, Stripe stripe data blocks and column statistics to achieve data storage. Although file-level retrieval and filtering can be achieved, due to the characteristics of the ORC column storage data format, it cannot provide a data structure optimized for writing real-time data. In addition, in the audit log scenario of the log storage service, ORC itself does not provide an index structure and retrieval method that can quickly retrieve data, which makes this method unable to be applied to the audit log scenario. Therefore, an effective solution is urgently needed to solve the above problems.

[0073] The data query method provided in this specification, when obtaining the partition key and query attribute information, will determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition. When the query attribute information does not match the attribute information, the second database having a data migration relationship with the first database can be determined at this time, and the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information. The index information can be determined based on the reading, and finally the target data corresponding to the data query request is read in the second database based on the index information, so that the problem of large write traffic and inability to store data is solved by combining the first database and the second database, and different methods are used for data query for the first database and the second database, which effectively improves the data query efficiency, thereby realizing a rapid response to the data query request.

[0074] Figure 1 A flow chart of a data query method provided according to an embodiment of this specification is shown, which specifically includes the following steps:

[0075] Step S102: obtaining a data query request, wherein the data query request carries a partition key and query attribute information.

[0076] In actual applications, due to the characteristics of online services, most online services will establish record logs to record business-related data and corresponding business processing procedures. As the amount of log data continues to increase, it is difficult for the service provider to store a large amount of log data. Therefore, in order to ensure that the service provider can retain a large amount of log data, the log storage service provides tenants with a log data storage service. By establishing a dedicated database to store log data for tenants, the service provider can trace back the log data at any time to meet the audit of log data.

[0077] Based on this, the data query request specifically refers to a request for querying log-related data in the audit log scenario, which is submitted by the tenant to the log storage service provider; accordingly, the partition key specifically refers to the ID corresponding to the tenant, which is used to determine the partition corresponding to the tenant in the database. It should be noted that in order to improve the utilization rate of the log database, the same log database can provide log data storage services to multiple tenants at the same time, and each tenant corresponds to a partition, and the key of each partition is the tenant's ID, and the tenant's ID is unique; accordingly, the query attribute information specifically refers to the attributes of the data itself, including but not limited to the maximum value of the data, number of rows, occupied space, type, storage time, generation time, etc.

[0078] Further, when the data query request is received, it indicates that the tenant needs to query the historical log data. Before that, in order to improve the query performance, different methods can be used to store data in different time periods, thereby ensuring that the log storage service provider can quickly respond to different query requests from different tenants. In this embodiment, the specific implementation method is as follows:

[0079] Extracting the data to be migrated whose storage time is greater than a preset time threshold in the first database, and determining the migration partition key corresponding to the data to be migrated;

[0080] compressing the data to be migrated into compressed data, and writing the compressed data into a partition corresponding to the migration partition key in the second database;

[0081] Determine the compression attribute information and compression index information corresponding to the compressed data according to the writing result, and establish a storage correspondence between the migration partition key and the compression attribute information and the compression index information;

[0082] The storage relationship table corresponding to the second database is updated based on the storage correspondence relationship.

[0083] Specifically, the preset time threshold refers to a condition for determining data migration. If the storage time in the first database exceeds the time threshold, it can be determined that the data needs to be converted from hot data to cold data. If the storage time in the first database does not exceed the time threshold, it can be determined that the current state of the data is hot data. Accordingly, the first database specifically refers to hot data whose storage write time does not exceed the preset time threshold, and the second database specifically refers to cold data whose storage write time exceeds the preset time threshold, and the cold data is obtained by converting the hot data that exceeds the preset time threshold.

[0084] It should be noted that since the first database and the second database are both log databases for storing log data, and the first database is a database for storing hot data, and the second database is a database for storing cold data, and the cold data is obtained by converting the hot data, the second database and the first database contain partitions corresponding to the same partition key, that is, the same partition key can determine the partition in the first database, and can also determine the partition in the second database.

[0085] In addition, since the data stored in the first database is hot data, and the conversion of hot data to cold data requires more computing resources, when the data is received, it will be written to the first database first. When the storage time in the first database exceeds the preset time threshold, the hot data in the first database will be converted to cold data and stored in the second database; further, in order to improve the query performance, the format of the data stored in the second database can be composed of three parts: meta information, index and data, see Figure 2 As shown, the meta information includes table meta information, column list and index list, etc. The index includes index meta information and index data, etc. In order to improve query performance, an inverted index can be established for text data, and a kd tree index can be established for numerical types. The data includes data statistics (maximum value, number of rows) and compressed original data, etc.

[0086] Among them, both the first database and the second database contain partitions corresponding to the partition keys, the compressed data specifically refers to the data obtained after the data to be migrated is compressed, and the space occupied by the compressed data is smaller than the space occupied by the data to be migrated; the compression attribute information specifically refers to the maximum value and number of rows of the data to be migrated, and the compression index information specifically refers to the information whether there is an index.

[0087] Based on this, the data storage time in the first database will be detected at set intervals. When data that needs to be migrated is detected, the data to be migrated whose storage time is greater than a preset time threshold will be extracted from the first database, and the migration partition key corresponding to the data to be migrated will be determined. The migration partition key specifically refers to the key corresponding to the partition storing the data to be migrated. Then, in order to improve the subsequent query performance, the data to be migrated can be compressed to obtain compressed data, and the compressed data can be written into the partition corresponding to the migration partition key in the second database. When the writing is completed, in order to facilitate subsequent data query, the compression attribute information and compression index information of the compressed data can be determined, and then a storage correspondence between the migration partition key and the compression attribute information and the compression index information is established. Finally, based on the storage correspondence, the storage relationship table corresponding to the second database can be updated. Among them, the storage relationship table of the second database records the correspondence between the partition key, data attribute information and index information. When retrieving data, the information that needs to be read can be determined by reading the table.

[0088] Furthermore, after the compressed data is written into the second database, an index may be created for the compressed data according to the writing result. In the process of creating the index, since the index-related data also needs to occupy storage space, considering the space utilization, different compressed data may choose whether to create an index according to the demand. In this embodiment, the specific implementation method is as follows:

[0089] Determining whether the compressed data meets the preset index creation condition;

[0090] If so, determine the data type of the compressed data; when the data type is a text type, create an inverted index corresponding to the compressed data according to the writing result as the compressed index; when the data type is a numerical type, create a tree index corresponding to the compressed data according to the writing result as the compressed index, and determine the compressed index information based on the compressed index.

[0091] If not, the information without index is used as the compressed index information.

[0092] Specifically, the index creation condition refers to a condition for determining whether it is necessary to create an index for index data. The index creation condition may be determining whether the space occupied by the compressed data is greater than a preset occupancy threshold, or determining whether the length of the compressed data is greater than a preset length threshold. Accordingly, since different types of data have different structures, different methods will be used to create indexes for different types of data when the index creation condition is met.

[0093] Based on this, since the data corresponding to the index also needs to occupy a certain amount of space, when deciding whether to create the index, it can be determined whether the created index is larger than the space occupied by the compressed data. If it is larger, it does not need to be created; if it is not larger, it needs to be created; further, after the compressed data is written into the second database, it will be determined whether the compressed data meets the preset index creation conditions. If so, the data type of the compressed data can be determined. When the data type is a text type, an inverted index corresponding to the compressed data can be created based on the writing result, and the inverted index can be used as the compressed index; when the data type is a numerical type, a tree index corresponding to the compressed data can be created based on the writing result as the compressed index; finally, the compressed index information can be determined based on the compressed index.

[0094] For example, the log data generated by tenant 1 at a certain moment is stored in the hot database for more than 1 hour, and it is determined that this part of the log data needs to be converted into cold data and stored in the cold database; based on this, it is determined that the migration partition key corresponding to the log data to be migrated is cid_1, and the partition corresponding to the migration partition key cid_1 is determined in the cold data. At this time, the preset compression algorithm is used to compress the log data to be migrated to obtain compressed log data, and then the compressed log data is written to the partition corresponding to the migration partition key cid_1; after the writing is completed, in order to perform subsequent log auditing, the compression attribute information of the compressed log data is determined to be {maximum value L max , minimum value L min , number of rows N, storage time T, creation time TS}, and at the same time determine the kd tree index established for the compressed log data, determine the compressed index information as having an index, and then establish a storage correspondence between the migration partition key cid_1 and the compressed attribute information and the compressed index information, and write the storage correspondence into the storage relationship table corresponding to the cold database, so that the corresponding log data can be quickly queried in the cold database during the subsequent log audit process.

[0095] Among them, the inverted index is used to store the mapping of the storage location of a word in a document or a group of documents under full-text search. It is the most commonly used data structure in document retrieval systems. Through the inverted index, you can quickly get a list of documents containing a word based on the word.

[0096] In summary, by periodically converting the write-optimized data structure (hot data) into the read-optimized data structure (cold data), the efficiency of querying the data in the second database is effectively improved, so that query requests from different tenants can be quickly responded to, thereby improving the tenant's user experience.

[0097] Step S104: determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data blocks included in the target partition.

[0098] Specifically, based on the data query request obtained above, further, the target partition will be determined in the first database according to the partition key carried in the data query request. The target partition specifically refers to the partition storing the log data of a certain tenant. After determining the target partition, in order to be able to provide feedback for the data query request, the data information of the data block contained in the target partition can be read at this time. The data block specifically refers to a data block composed of partial log data. The attribute information of the data block specifically refers to the maximum value, number of rows, occupied space, type, storage time, generation time, etc. corresponding to the data stored in the data block, for subsequent query processing operations.

[0099] Further, since different data query requests provide different query conditions, if the received data query request contains more information, the database storing the data to be queried can be accurately determined according to the data query request, and the query can be performed. If the received data query request contains less information, the database may not be accurately determined according to the information contained in the data query request, and the query can be performed in the manner of steps S102 to S108. In this embodiment, how to select the correct query method according to the data query request is specifically implemented as follows:

[0100] Detecting whether the query attribute information contains time information;

[0101] If not, the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data blocks included in the target partition is executed.

[0102] If so, determining again whether the time information meets the preset time division condition;

[0103] If it matches, determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition; match the attribute information with the query attribute information, and read the target data corresponding to the data query request in the data block according to the matching result.

[0104] If not matched, the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information; index information is determined according to the reading result, and the target data corresponding to the data query request is read in the second database based on the index information.

[0105] Specifically, the time information refers to the time-related information contained in the query attribute information, and the time node corresponding to the data to be queried can be determined through the time information; accordingly, the preset time division condition refers to the condition for determining whether the data to be queried is stored in the first database or the second database.

[0106] Based on this, after obtaining the data query request, it is possible to detect whether the query attribute information contains time information. If not, it means that the database to be queried cannot be accurately determined based on the data query request, and step S104 can be executed; if it is contained, it means that the database corresponding to the data to be queried can be preliminarily determined through the time information contained in the query attribute information.

[0107] When it is determined that the query attribute information includes the time information, it can be determined whether the time information included in the query attribute information meets the preset time division condition. If it does, it means that the target data corresponding to the data query request can be queried in the first database. At this time, the target partition can be determined in the first database according to the partition key included in the data query request, and then the attribute information of the data block included in the target partition is read, and the attribute information and the query attribute information are matched. According to the matching result, the target data block that meets the data query request can be determined to be stored, and finally the target data is read from the target data block and feedback is provided for the data query request.

[0108] If not, it means that the target data corresponding to the data query request can be queried in the second database, and since the second database is a database for storing cold data, the storage relationship table corresponding to the second database can be read according to the partition key and the query attribute information, and the index information corresponding to the data query request can be determined according to the reading result. Finally, based on the index information, the target data corresponding to the data query request can be read in the second database, and the target data can be fed back as a response to the data query request.

[0109] Using the above example, when the log data of tenant 1 needs to be audited, a data query request will be submitted. By parsing the query attribute information contained in the data query request, it is determined that the time information contained in the query attribute information is T1. At this time, the time information can be analyzed to determine which database the log data to be audited is stored in; based on this, it will be determined whether the time information T1 contained in the query attribute information meets the preset time division condition;

[0110] If it does, it means that the log data to be audited is stored in the hot database. At this time, the target partition can be determined in the hot database according to the partition key cid_1 contained in the data query request, and then the attribute information of the data block contained in the target partition is read to determine that the attribute information is (maximum value 10, minimum value 0, number of rows N1, time T2), and the query attribute information is (maximum value 5, minimum value 0, number of rows N2, time T1). By comparing the query attribute information in the data query request with the data information of the data block, it is determined that there is data corresponding to the data query request in the data block. At this time, the target log data that meets the query attribute information can be extracted from the data block according to the query attribute information, and feedback can be given to the data query request for subsequent audit processing.

[0111] If it does not match, it means that the log data to be audited is stored in the cold database. At this time, the storage relationship table corresponding to the cold database can be read according to the partition key cid_1 and the query attribute information (maximum value 5, minimum value 0, number of rows N2, time T1), and the index for reading the target log data is determined according to the reading result. Then, the target log data that meets the data query request can be obtained by loading and retrieving the index, and feedback is provided for the data query request for subsequent audit processing.

[0112] In summary, before performing data query, in order to improve query efficiency, the query method to be adopted can be accurately distinguished by time information, so that the target data can be quickly fed back for the data query request, thereby improving the query performance in the log audit scenario and improving the log audit efficiency.

[0113] Step S106, when the query attribute information does not match the attribute information, determine a second database having a data migration relationship with the first database, and read a storage relationship table corresponding to the second database according to the partition key and the query attribute information.

[0114] Specifically, based on the above determination of the attribute information of the data block, the query attribute information can be further matched with the attribute information. If the query attribute information matches the attribute information, it means that the target data corresponding to the data query request exists in the first database, and the target data corresponding to the data query request can be queried in the first database, that is, by filtering the PartitionId Map that records cid (partition key) and row_count (number of rows), the target partition corresponding to the partition key is obtained. After that, for the columns other than the partition key in the data query request, a preliminary filtering will be performed according to hot block statistics (including min and max values). If it cannot be completely filtered, it means that the target data that meets the data query request can be queried in the data block, and the data block can be scanned for deep filtering to obtain the target data corresponding to the data query request. If the query attribute information does not match the attribute information, it means that the data corresponding to the data query request does not exist in the first database. At this time, a second database having a data migration relationship with the first database can be determined to continue the query processing of the target data.

[0115] It should be noted that the data migration relationship specifically means that the data in the first database will be converted to the second database if the conditions are met. The specific implementation method can refer to the corresponding description content in the above embodiment, and this embodiment does not make any limitation here.

[0116] Based on this, after determining the second database, since the second database is a database for storing cold data, it is necessary to read the storage relationship table corresponding to the second database during query to determine the compressed data of the data block corresponding to the data query request, as well as the corresponding index information, so as to directly read the compressed data through the index and decompress it to obtain the target data corresponding to the data query request.

[0117] Furthermore, in the process of determining whether the query attribute information and the attribute information match, in order to improve the matching accuracy and realize rapid feedback of the target data for the data query request, the determination can be made according to the value of the data. In this embodiment, the specific implementation method is as follows:

[0118] Parsing the attribute information to obtain a first value interval, and parsing the query attribute information to obtain a second value interval;

[0119] determining whether the first numerical interval includes the second numerical interval;

[0120] If so, reading the target data corresponding to the data query request in the data block based on the query attribute information;

[0121] If not, execute step S106.

[0122] Specifically, the first numerical interval specifically refers to an interval consisting of the maximum and minimum values ​​of the data values ​​in the data block, and the second numerical interval specifically refers to the maximum and minimum values ​​corresponding to the data to be queried; based on this, the first numerical interval is obtained by parsing the attribute information, and the second numerical interval is obtained by parsing the query attribute information; then, it is determined whether the first numerical interval contains the second numerical interval. If not, it is determined that the attribute information and the query attribute information do not match. At this time, step S106 can be continued to determine the target data corresponding to the data query request in the second database; if included, it is determined that the attribute information and the query attribute information match. At this time, the target data corresponding to the data query request can be read in the data block based on the query attribute information, that is, the data identical to the query attribute information is obtained in the first database as the target data.

[0123] In summary, by querying the target data corresponding to the data query request, by comparing the first numerical interval and the second numerical interval, it is possible to accurately determine whether the target data corresponding to the data query request exists in the first database, thereby saving the method of traversing all the data contained in the database and saving query time.

[0124] Furthermore, in the process of reading the storage relationship table corresponding to the second database according to the partition key and the query attribute information, since the second database is a cold database, the second database has a higher query efficiency than the first database. In order to further improve the efficiency of querying data in the second database and reduce the space occupied by other data except the data itself, a hash table can be used to record the corresponding information. In this embodiment, the specific implementation method is as follows:

[0125] Performing a hash operation on the partition key and the query attribute information to obtain a query hash value;

[0126] Determine, in the hash table corresponding to the second database, the candidate partition key and time interval information corresponding to the query hash value;

[0127] Determine a second target partition in the second database according to the candidate partition key and the time interval information, and read second attribute information of the candidate data block included in the second target partition;

[0128] In the case where the second attribute information matches the query attribute information, the storage relationship table corresponding to the second database is read according to the query attribute information.

[0129] Specifically, when it is determined that the target data exists in the second database, a hash operation can be performed on the partition key and the query attribute information to obtain a query hash value, and then the candidate partition key and time interval information corresponding to the query hash value are determined in the hash table corresponding to the second database; then the second target partition storing the target data is determined in the second database according to the candidate partition key and the time interval information, and since the second target partition contains a large number of data blocks, the candidate data blocks can be determined in the second target partition according to the query attribute information and the time interval information, and at this time it is determined that the data is stored in the candidate data blocks, and then the second attribute information of the candidate data blocks is read, and the second attribute information is matched with the query attribute information. When the second attribute information matches the query attribute information, it means that the target data corresponding to the data query request exists in the second database, and then the storage relationship table corresponding to the second database can be further read according to the query attribute information to determine the relevant information of the stored target data for subsequent query processing operations.

[0130] In actual applications, during the data query in the second database, the query can be based on the cid and<min_ts,max_ts> Information is filtered to obtain one or more data blocks to be queried. Then, since each data block stores the statistics of each column (maximum value, minimum value, set of null values, etc.), for columns other than the partition key in the query conditions, further filtering can be performed according to the query attribute information in the data query request to determine whether the target data corresponding to the data query request can be queried in the second database. If there is a match between the attribute information and the query, subsequent query processing can be performed.

[0131] In summary, the relevant information in the second database is recorded by using a hash table, which not only reduces the space occupied by the relevant information, but also prevents the information from being tampered with, thereby improving the efficiency of data query in the second database.

[0132] Step S108, determining index information according to the reading result, and reading target data corresponding to the data query request in the second database based on the index information.

[0133] Specifically, after querying the storage relationship table corresponding to the second database according to the partition key and the query attribute information, the index information can be determined according to the reading result. The index information may be information containing an index or not containing an index. After determining the index information, the target data corresponding to the data query request can be read in the second database according to the index information.

[0134] Furthermore, in the process of querying the target data according to the index information, since there are two situations of the index information, different methods will be used to query the target data in different ways. In this embodiment, the specific implementation method is as follows:

[0135] When a query index is obtained according to the reading result, the query index is used as the index information; and the target data corresponding to the data query request is read in the second database based on the query index.

[0136] If no query index is obtained according to the query result, a data block matching the query attribute information may be selected from the second database, and then target data corresponding to the data query request may be selected from the data block for feedback.

[0137] The data query method provided in this specification, when obtaining the partition key and query attribute information, will determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition. When the query attribute information does not match the attribute information, the second database having a data migration relationship with the first database can be determined at this time, and the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information. The index information can be determined based on the reading, and finally the target data corresponding to the data query request is read in the second database based on the index information, so that the problem of large write traffic and inability to store data is solved by combining the first database and the second database, and different methods are used for data query for the first database and the second database, which effectively improves the data query efficiency, thereby realizing a rapid response to the data query request.

[0138] The following combination Figure 3 , taking the application of the data query method provided in this specification in the log audit scenario as an example, the data query method is further explained. Figure 3 A processing flow chart of a data query method applied to a log audit scenario provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0139] Step S302: extracting the data to be migrated from the first database whose storage time is greater than a preset time threshold, and determining the migration partition key corresponding to the data to be migrated.

[0140] Step S304: compress the data to be migrated into compressed data, and write the compressed data into the partition corresponding to the migration partition key in the second database.

[0141] Step S306: Determine the compression attribute information and compression index information corresponding to the compressed data according to the writing result, and establish a storage correspondence between the migration partition key and the compression attribute information and the compression index information.

[0142] Step S308: updating the storage relationship table corresponding to the second database based on the storage correspondence relationship.

[0143] Step S310: obtaining a data query request, wherein the data query request carries a partition key and query attribute information.

[0144] Step S312: determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data blocks included in the target partition.

[0145] Step S314: when the queried attribute information does not match the attribute information, determine a second database having a data migration relationship with the first database.

[0146] Step S316: perform a hash operation on the partition key and the query attribute information to obtain a query hash value.

[0147] Step S318: determine the candidate partition key and time interval information corresponding to the query hash value in the hash table corresponding to the second database.

[0148] Step S320: determine a second target partition in the second database according to the candidate partition key and the time interval information, and read second attribute information of the candidate data blocks included in the second target partition.

[0149] Step S322: when the second attribute information matches the query attribute information, read the storage relationship table corresponding to the second database according to the query attribute information.

[0150] Step S324: determine index information according to the reading result, and read target data corresponding to the data query request in the second database based on the index information.

[0151] The data query method provided in this specification, when obtaining the partition key and query attribute information, will determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition. When the query attribute information does not match the attribute information, the second database having a data migration relationship with the first database can be determined at this time, and the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information. The index information can be determined based on the reading, and finally the target data corresponding to the data query request is read in the second database based on the index information, so that the problem of large write traffic and inability to store data is solved by combining the first database and the second database, and different methods are used for data query for the first database and the second database, which effectively improves the data query efficiency, thereby realizing a rapid response to the data query request.

[0152] Corresponding to the above method embodiment, this specification also provides a data query device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a data query device provided by an embodiment of this specification. Figure 4 As shown, the device comprises:

[0153] The request acquisition module 402 is configured to acquire a data query request, wherein the data query request carries a partition key and query attribute information;

[0154] The information reading module 404 is configured to determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition;

[0155] A database determination module 406 is configured to determine a second database having a data migration relationship with the first database when the query attribute information does not match the attribute information, and read a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0156] The data reading module 408 is configured to determine index information according to the reading result, and read the target data corresponding to the data query request in the second database based on the index information.

[0157] In an optional embodiment, the data query device further includes:

[0158] The migration module is configured to extract the data to be migrated whose storage time is greater than a preset time threshold in the first database, and determine the migration partition key corresponding to the data to be migrated; compress the data to be migrated into compressed data, and write the compressed data into the partition corresponding to the migration partition key in the second database; determine the compression attribute information and compression index information corresponding to the compressed data according to the writing result, and establish a storage correspondence between the migration partition key and the compression attribute information and the compression index information; and update the storage relationship table corresponding to the second database based on the storage correspondence.

[0159] In an optional embodiment, the migration module is further configured to:

[0160] Determine whether the compressed data meets the preset index creation condition; if so, create a compressed index corresponding to the compressed data according to the writing result, and determine the compressed index information based on the compressed index.

[0161] In an optional embodiment, the migration module is further configured to:

[0162] Determine the data type of the compressed data; when the data type is a text type, create an inverted index corresponding to the compressed data according to the writing result as the compressed index; when the data type is a numerical type, create a tree index corresponding to the compressed data according to the writing result as the compressed index.

[0163] In an optional embodiment, the data query device further includes:

[0164] The detection module is configured to detect whether the query attribute information contains time information; if not, run the information reading module 404.

[0165] In an optional embodiment, if the judgment result of the detection is yes, the following modules are executed:

[0166] The first judgment module is configured to judge whether the time information meets the preset time division condition; if so, determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition; match the attribute information with the query attribute information, and read the target data corresponding to the data query request in the data block according to the matching result.

[0167] In an optional embodiment, the first determination module is further configured to:

[0168] The storage relationship table corresponding to the second database is read according to the partition key and the query attribute information; index information is determined according to the reading result, and target data corresponding to the data query request is read in the second database based on the index information.

[0169] In an optional embodiment, the data query device further includes:

[0170] a parsing module configured to parse the attribute information to obtain a first value interval, and to parse the query attribute information to obtain a second value interval;

[0171] A second determination module is configured to determine whether the first numerical interval contains the second numerical interval; if so, read the target data corresponding to the data query request in the data block based on the query attribute information;

[0172] If not, run the determination database module 406

[0173] In an optional embodiment, the database determination module 406 is further configured to:

[0174] Perform a hash operation on the partition key and the query attribute information to obtain a query hash value; determine the candidate partition key and time interval information corresponding to the query hash value in the hash table corresponding to the second database; determine the second target partition in the second database based on the candidate partition key and the time interval information, and read the second attribute information of the candidate data block contained in the second target partition; when the second attribute information matches the query attribute information, read the storage relationship table corresponding to the second database based on the query attribute information.

[0175] In an optional embodiment, the data reading module 408 is further configured to:

[0176] When a query index is obtained according to the reading result, the query index is used as the index information; and the target data corresponding to the data query request is read in the second database based on the query index.

[0177] The data query device provided in this embodiment, when obtaining the partition key and query attribute information, will determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition. When the query attribute information does not match the attribute information, the second database having a data migration relationship with the first database can be determined, and the storage relationship table corresponding to the second database is read according to the partition key and the query attribute information. The index information can be determined based on the reading, and finally the target data corresponding to the data query request is read in the second database based on the index information, so that the problem of large write traffic and inability to store data is solved by combining the first database and the second database, and different methods are used for data query for the first database and the second database, which effectively improves the data query efficiency, thereby realizing a rapid response to the data query request.

[0178] The above is a schematic scheme of a data query device of this embodiment. It should be noted that the technical scheme of the data query device and the technical scheme of the above data query method belong to the same concept, and the details not described in detail in the technical scheme of the data query device can be referred to the description of the technical scheme of the above data query method.

[0179] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0180] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0181] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0182] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0183] The processor 520 is used to execute the following computer executable instructions:

[0184] Obtaining a data query request, wherein the data query request carries a partition key and query attribute information;

[0185] Determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition;

[0186] In the case that the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0187] Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

[0188] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data query method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data query method.

[0189] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:

[0190] Obtaining a data query request, wherein the data query request carries a partition key and query attribute information;

[0191] Determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition;

[0192] In the case that the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information;

[0193] Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

[0194] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data query method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data query method.

[0195] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0196] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0197] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0198] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0199] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data query method, comprising: Obtain a data query request, wherein the data query request carries a partition key and query attribute information, wherein the partition key is used to determine a partition corresponding to the tenant; Determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition; In the case that the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information; Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

2. The data query method according to claim 1, before the step of obtaining the data query request is executed, further comprising: Extracting the data to be migrated whose storage time is greater than a preset time threshold in the first database, and determining the migration partition key corresponding to the data to be migrated; compressing the data to be migrated into compressed data, and writing the compressed data into a partition corresponding to the migration partition key in the second database; Determine the compression attribute information and compression index information corresponding to the compressed data according to the writing result, and establish a storage correspondence between the migration partition key and the compression attribute information and the compression index information; The storage relationship table corresponding to the second database is updated based on the storage correspondence relationship.

3. The data query method according to claim 2, wherein determining the compression index information corresponding to the compressed data according to the writing result comprises: Determining whether the compressed data meets the preset index creation condition; If so, create a compression index corresponding to the compressed data according to the writing result, and determine the compression index information based on the compression index.

4. The data query method according to claim 3, wherein the step of creating a compressed index corresponding to the compressed data according to the writing result comprises: Determining the data type of the compressed data; In the case where the data type is a text type, creating an inverted index corresponding to the compressed data according to the writing result as the compressed index; In the case where the data type is a numerical type, a tree index corresponding to the compressed data is created according to the writing result as the compressed index.

5. The data query method according to claim 1, before the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data block contained in the target partition is performed, it also includes: Detecting whether the query attribute information contains time information; If not, the step of determining the target partition corresponding to the partition key in the first database and reading the attribute information of the data blocks included in the target partition is executed.

6. The data query method according to claim 5, if the result of the determination of whether the query attribute information contains time information is yes, executing the following steps: Determining whether the time information meets the preset time division condition; If yes, determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data blocks included in the target partition; The attribute information is matched with the query attribute information, and target data corresponding to the data query request is read in the data block according to the matching result.

7. The data query method according to claim 6, if the result of judging whether the time information meets the preset time division condition is no, executing the following steps: Read the storage relationship table corresponding to the second database according to the partition key and the query attribute information; Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

8. The data query method according to claim 1, wherein when the query attribute information does not match the attribute information, before the step of determining a second database having a data migration relationship with the first database and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information is performed, the method further comprises: Parsing the attribute information to obtain a first value interval, and parsing the query attribute information to obtain a second value interval; determining whether the first numerical interval includes the second numerical interval; If so, reading the target data corresponding to the data query request in the data block based on the query attribute information; If not, the step of determining a second database having a data migration relationship with the first database and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information is executed.

9. The data query method according to claim 1, wherein the step of reading the storage relationship table corresponding to the second database according to the partition key and the query attribute information comprises: Performing a hash operation on the partition key and the query attribute information to obtain a query hash value; Determine, in the hash table corresponding to the second database, candidate partition keys and time interval information corresponding to the query hash value; Determine a second target partition in the second database according to the candidate partition key and the time interval information, and read second attribute information of the candidate data block included in the second target partition; In the case where the second attribute information matches the query attribute information, the storage relationship table corresponding to the second database is read according to the query attribute information.

10. The data query method according to claim 9, wherein determining index information according to the reading result comprises: When a query index is obtained according to the reading result, the query index is used as the index information; Correspondingly, reading the target data corresponding to the data query request in the second database based on the index information includes: The target data corresponding to the data query request is read in the second database based on the query index.

11. A data query device, comprising: An acquisition request module is configured to acquire a data query request, wherein the data query request carries a partition key and query attribute information, wherein the partition key is used to determine a partition of a corresponding tenant; an information reading module, configured to determine a target partition corresponding to the partition key in the first database, and read attribute information of a data block contained in the target partition; a database determination module configured to, when the query attribute information does not match the attribute information, determine a second database having a data migration relationship with the first database, and read a storage relationship table corresponding to the second database according to the partition key and the query attribute information; The data reading module is configured to determine index information according to the reading result, and read the target data corresponding to the data query request in the second database based on the index information.

12. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the following method: Obtain a data query request, wherein the data query request carries a partition key and query attribute information, wherein the partition key is used to determine a partition corresponding to the tenant; Determine the target partition corresponding to the partition key in the first database, and read the attribute information of the data block contained in the target partition; In the case that the query attribute information does not match the attribute information, determining a second database having a data migration relationship with the first database, and reading a storage relationship table corresponding to the second database according to the partition key and the query attribute information; Index information is determined according to the reading result, and target data corresponding to the data query request is read from the second database based on the index information.

13. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the data query method according to any one of claims 1 to 10.

Images