Data query processing method, device and storage medium

Abstract

The application provides a data query processing method and device and a storage medium, and relates to the technical field of data analysis. The method comprises the following steps: acquiring a service level agreement (SLA) delay; partitioning a storage space according to the size of a storage partition corresponding to the delay to obtain a plurality of storage partitions, wherein different delays correspond to different storage partition sizes, the delay is positively correlated with the size of the storage partition; and establishing a query index for each storage partition in a sequence according to query information of each storage partition, wherein the plurality of storage partitions are kept in an ordered arrangement through the query index, and the data in a same storage partition is kept in a disordered arrangement. According to the method, the data in a same partition is stored in a disordered manner, the data is conveniently loaded in batches, the order is kept between different partitions, the query index is optimized according to the query information, the batch data is quickly analyzed, the data access efficiency is improved, and the timeliness requirement of a user for batch data analysis is met.

Description

Technical Field

[0001] This application relates to data analysis technology, and more particularly to a data query processing method, device and storage medium. Background Technology

[0002] Online Analytical Processing (OLAP) studies the problem of extracting information from massive amounts of data, enabling analysts to quickly, consistently, and interactively observe information from various perspectives in order to achieve a deeper understanding of the information.

[0003] To support efficient query analysis, current mainstream OLAP engines require the creation of complete indexes for data before providing analysis services. Among these, non-volatile memory (NVM) leverages its superior characteristics—non-volatile and read / write performance approaching that of dynamic random access memory (DRAM)—to push the performance limits of storage systems. This allows for the migration of indexes from traditional SSD / disk storage media to NVM, enabling the construction of NVM indexes and improving data loading speed.

[0004] However, in the current big data scenario, building an NVM index for massive amounts of data still requires a long waiting time, which cannot meet the demand for timeliness in data analysis. Summary of the Invention

[0005] This application provides a data query processing method, device, and storage medium to solve the problem that in the current big data scenario, building an NVM index for massive amounts of data still requires a long waiting time, which cannot meet the demand for timeliness of data analysis.

[0006] Firstly, this application provides a data query and processing method, including:

[0007] The delay in obtaining the Service Level Agreement (SLA);

[0008] Based on the size of the storage partition corresponding to the latency, the storage space is partitioned to obtain multiple storage partitions, wherein different latencies correspond to different storage partition sizes, and the latency is positively correlated with the size of the storage partition;

[0009] Based on the query information of each storage partition, a query index is built for each storage partition in chronological order. Multiple storage partitions are kept in an ordered manner through the query index, while the data within the same storage partition is arranged in an unordered manner.

[0010] In one possible design, after partitioning the storage space according to the size of the storage partition corresponding to the delay to obtain multiple storage partitions, the method further includes:

[0011] Retrieve the data to be stored;

[0012] The data to be stored is written to the cache partition, and after the cache partition is full, the data in the cache partition is sent to the write index queue.

[0013] The data in the write index queue is written to the storage partition sequentially.

[0014] In one possible design, writing the data to be stored into a cache partition, and sending the data from the cache partition to a write index queue after the cache partition is full, includes:

[0015] The data is written to the hot write cache partition, and after the hot write cache partition is full, the data in the hot write cache partition is sent to the hot write cache queue.

[0016] The data in the hot write cache queue is written to the cold write cache partition. After the cold write cache partition is full, the data in the cold write cache partition is sent to the write index queue.

[0017] In one possible design, the number of cold write cache partitions is equal to the number of storage partitions.

[0018] In one possible design, the step of building a query index for each storage partition in chronological order based on the query information for each storage partition includes:

[0019] Obtain the query count based on the query information for each storage partition;

[0020] Write the storage partition into the queue of indexes to be built;

[0021] The priority of the storage partition is obtained based on the query count, and the query count is positively correlated with the priority of the storage partition;

[0022] Query indexes are created for the storage partitions in the queue of indexes to be created based on the priority of the storage partitions.

[0023] In one possible design, writing the storage partition into the queue of indexes to be built includes:

[0024] When a query request is received, the query count is updated to obtain the updated query count;

[0025] Based on the updated query count, a corresponding random number is generated, and the value of the random number is between 0 and 1;

[0026] The index probability corresponding to the updated query count is obtained based on the online decision algorithm;

[0027] Determine whether the index probability is less than the random number. If not, write the storage partition into the queue of indexes to be built.

[0028] In one possible design, obtaining the index probability corresponding to the updated query count based on an online decision algorithm includes:

[0029] Based on the ratio of the cost of building a query index to the benefit of building a query index per query, and the cost coefficient, the index probability corresponding to the next query count is obtained.

[0030] Secondly, this application provides a data query and processing device, comprising:

[0031] The acquisition module is used to acquire the latency of the Service Level Agreement (SLA).

[0032] The first processing module is used to partition the storage space according to the size of the storage partition corresponding to the delay, thereby obtaining multiple storage partitions, wherein different delays correspond to different storage partition sizes, and the delay is positively correlated with the size of the storage partition;

[0033] The second processing module is used to build a query index for each storage partition according to the query information of each storage partition in a sequential order. The multiple storage partitions are kept in an ordered manner through the query index, while the data within the same storage partition is arranged in an unordered manner.

[0034] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0035] The memory stores computer-executed instructions;

[0036] The processor executes computer execution instructions stored in the memory to implement the data query processing method.

[0037] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a data query processing method.

[0038] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements a data query processing method.

[0039] The data query processing method, device, and storage medium provided in this application obtain the latency of the Service Level Agreement (SLA); partition the storage space according to the size of the storage partition corresponding to the latency to obtain multiple storage partitions, wherein different latencies correspond to different storage partition sizes, and the latency is positively correlated with the size of the storage partition; based on the query information of each storage partition, a query index is built for each storage partition in chronological order, wherein multiple storage partitions are kept in an ordered arrangement through the query index, while data within the same storage partition is arranged in an unordered manner, making data storage within the same partition unordered, facilitating fast batch loading of data, and maintaining order between different partitions, and optimizing the query index according to the query information, thereby realizing rapid analysis of batch data and improving data access efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating the data query processing method provided in the embodiments of this application. Figure 1 ;

[0042] Figure 2 A flowchart illustrating the data query processing method provided in the embodiments of this application. Figure 2 ;

[0043] Figure 3 A flowchart illustrating the data query processing method provided in the embodiments of this application. Figure 3 ;

[0044] Figure 4 This is a schematic diagram of the structure of the data query processing device provided in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. The collection, storage, use, processing, transmission, provision, and disclosure of information such as financial data or user data involved in the technical solutions of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] First, the relevant concepts or terms involved in this application will be explained:

[0048] Online Analytical Processing (OLAP) is a software technology that enables analysts to quickly, consistently, and interactively observe information from various perspectives to achieve a deeper understanding of the data.

[0049] Dynamic Random Access Memory (DRAM) is a type of semiconductor memory that primarily works by using the amount of charge stored in a capacitor to represent whether a binary bit is 1 or 0.

[0050] Solid State Drive (SSD): refers to a hard drive made of solid-state electronic storage chip arrays, also known as a solid-state drive.

[0051] Non-volatile memory (NVM) refers to a new type of storage medium that is non-volatile, accessed by bytes, has high storage density, low power consumption, and read / write performance close to that of DRAM.

[0052] A Service Level Agreement (SLA) is a mutually agreed-upon agreement or contract between a service provider and its customer regarding the quality, standard, and performance of the service.

[0053] Prefix Tree: Also known as a dictionary tree, trie tree, word search tree, or key tree, it is a multi-branch tree structure indexing method.

[0054] Adaptive Radix Tree (ARTree): This refers to a basic tree that provides indexing methods for intermediate and leaf nodes of various sizes. Different nodes store different numbers of keys and have different indexing methods. Currently, the main types are nodes that store 4, 16, 48, and 256 keys.

[0055] Existing storage media with query indexing capabilities typically require data to be ordered during storage to establish a complete index analysis service and ensure the normal operation of the indexing function. However, when processing massive amounts of data, this leads to excessively long data loading times, failing to meet users' timeliness requirements for data analysis. The data query processing method provided in this application determines the storage partition size based on the data's service level agreement (SLA), then stores the data in an unordered manner across multiple partitions, and finally builds an ordered index on these partitions, thereby improving storage and analysis efficiency and aiming to solve the aforementioned technical problems of existing technologies.

[0056] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0057] Figure 1 Flowchart of the data query and processing method provided in this application Figure 1 .like Figure 1 As shown, the method includes:

[0058] S101, Delay in obtaining the Service Level Agreement (SLA);

[0059] Specifically, larger partitions require fewer partition index entries to maintain, allowing for faster, unordered data storage and loading, resulting in higher data loading performance. However, queries require scanning all unnecessary data within the partition, which takes longer; therefore, larger partitions lead to poorer query performance. For data to be loaded, a uniform partitioning strategy is typically used. A higher Data Service Level Agreement (SLA) latency indicates a larger memory footprint. Therefore, based on uniform partitioning, the actual partition size must be determined according to the SLA latency. Thus, the first step is to obtain the data's SLA latency.

[0060] S102. The storage space is partitioned according to the size of the storage partition corresponding to the delay to obtain multiple storage partitions, wherein different delays correspond to different storage partition sizes, and the delay is positively correlated with the size of the storage partition.

[0061] Specifically, after obtaining the SLA latency of the data, a proportional scaling method is used to determine the partition ratio based on the latency ratio. That is, the higher the data latency, the larger the allocated storage partition, thus balancing the contradiction between the number of partitions and their respective partition ratios. Since the actual stored data consists of key values, each key value occupies a different amount of data. If only the number of key values ​​is considered, it will lead to uneven memory allocation; conversely, if only the memory size is considered, it may disrupt the integrity of boundary key values. Therefore, each partition needs to have a defined partition range. After determining the partition size, the range of each partition is obtained through data sampling. For example, if the maximum latency of the SLA for 10GB of data is 100Mbps, then if 1000 key values ​​are sampled, each partition will contain 10 key values.

[0062] S103. Based on the query information of each storage partition, a query index is established for each storage partition in chronological order. The multiple storage partitions are kept in an ordered manner through the query index, while the data within the same storage partition is arranged in an unordered manner.

[0063] Specifically, during data loading, although data is directly written to the partition, reducing the cost of data sorting and splitting, the write granularity of the NVM storage medium is 256 bytes, which can cause write amplification. Therefore, the data writing process to the partition adopts a strategy based on multi-level caching and cascading updates. This involves sequentially writing data through three types of threads: read / parse, write cache, and write index, to avoid cache misses caused by large amounts of data writing. After storing data in different partitions, since the data is stored in an unordered, discrete state, a significant amount of traversal time is consumed during the search process, leading to frequent cache misses during queries. Therefore, without building query indexes within partitions, query indexes need to be built for each partition. Based on the query information, existing prefix trees or adaptive radix trees are used to build inter-partition indexes. Since the query index is determined based on the query information of each partition, the query frequency is determined by the query count; the higher the query frequency of a partition, the higher its query priority.

[0064] The method provided in this embodiment obtains the latency of the Service Level Agreement (SLA); partitions the storage space according to the size of the storage partition corresponding to the latency, resulting in multiple storage partitions, wherein different latencies correspond to different storage partition sizes, and the latency is positively correlated with the size of the storage partition; and establishes a query index for each storage partition in chronological order based on the query information of each storage partition. The multiple storage partitions are kept in an ordered manner through the query index, while the data within the same storage partition is arranged in an unordered manner, facilitating rapid batch loading of data. Furthermore, the ordered nature of the data across different partitions, combined with the optimization of the query index based on the query information, enables rapid analysis of batch data and improves data access efficiency.

[0065] The data query processing method of this application will be described in detail below with reference to a specific embodiment.

[0066] Figure 2 Flowchart of the data query and processing method provided in this application Figure 2 .like Figure 2 As shown, the method includes:

[0067] S201, Delay in obtaining the Service Level Agreement (SLA);

[0068] Specifically, larger partitions require fewer partition index entries to maintain, allowing for faster, unordered data storage and loading, resulting in higher data loading performance. However, queries require scanning all unnecessary data within the partition, which takes longer; therefore, larger partitions lead to poorer query performance. For data to be loaded, a uniform partitioning strategy is typically used. A higher Data Service Level Agreement (SLA) latency indicates a larger memory footprint. Therefore, based on uniform partitioning, the actual partition size must be determined according to the SLA latency. Thus, the first step is to obtain the data's SLA latency.

[0069] S202. The storage space is partitioned according to the size of the storage partition corresponding to the delay to obtain multiple storage partitions, wherein different delays correspond to different storage partition sizes, and the delay is positively correlated with the size of the storage partition.

[0070] Specifically, after obtaining the SLA latency of the data, a proportional scaling method is used to determine the partition ratio based on the latency ratio. That is, the higher the data latency, the larger the allocated storage partition, thus balancing the contradiction between the number of partitions and their respective partition ratios. Since the actual stored data consists of key values, each key value occupies a different amount of data. If only the number of key values ​​is considered, it will lead to uneven memory allocation; conversely, if only the memory size is considered, it may disrupt the integrity of boundary key values. Therefore, each partition needs to have a defined partition range. After determining the partition size, the range of each partition is obtained through data sampling. For example, if the maximum latency of the SLA for 10GB of data is 100Mbps, then if 1000 key values ​​are sampled, each partition will contain 10 key values.

[0071] S203. Obtain the data to be stored;

[0072] Specifically, after partitioning the storage medium, the data to be stored needs to be written. The data to be stored first needs to be read and parsed. For example, the string in the original data file is converted into the identifier int. This step requires starting a read / parse thread. Each read / parse thread maintains a hot write cache to temporarily store the data to be written. The hot write cache is usually smaller than the CPU cache (such as 4M or 8M) so that the hot write cache can reside in the CPU cache and avoid frequent cache misses.

[0073] S204. Write the data into the hot write cache partition, and after the hot write cache partition is full, send the data in the hot write cache partition to the hot write cache queue.

[0074] Specifically, memory is divided into cold write cache and hot write cache. The hot write cache, as the first-level cache, is also partitioned. When a partition of the hot write cache is full, the hot write cache is set to an immutable state, waiting for the data of the partition that is full to be sent to the cold write cache. Since the write speed and the sending speed are not the same, the sending speed is faster than the write speed. When the data is in batches, too much data cannot be written to the cold write cache partition at the same time. Therefore, the data of the full hot write cache partition will first be sent to the hot write cache queue, and then sent to the cold write cache in the order of the queue.

[0075] S205. Write the data in the hot write cache queue to the cold write cache partition, and after the cold write cache partition is full, send the data in the cold write cache partition to the write index queue.

[0076] Specifically, each write cache thread maintains a fixed-size cold write cache. The cold write cache retrieves data from the hot write cache queue sequentially and writes it into the cold write cache partition. When the cold write cache partition is full, the same principle applies as the hot write cache partition: the data in the full cold write cache partition is first sent to the write index queue.

[0077] S206. Write the data in the write index queue to the storage partition in sequence; the number of cold write cache partitions is equal to the number of storage partitions.

[0078] Specifically, the write index thread retrieves data from the write index queue and writes it to the storage partition of the storage medium. To improve the orderliness of the partitioned data, the number of storage partitions corresponds one-to-one with the number of cold write cache partitions, and there is also a correspondence between cold write cache partitions and hot write cache partitions. Furthermore, the number of threads for the three types of operations—read / parsing, write caching, and write indexing—is roughly equal, and the total number of threads is roughly equal to the number of CPU cores, ensuring the stable and orderly execution of the multi-level caching process and avoiding cache misses.

[0079] S207. Based on the query information of each storage partition, a query index is established for each storage partition in chronological order. The multiple storage partitions are kept in an ordered manner through the query index, while the data within the same storage partition is arranged in an unordered manner.

[0080] Specifically, after the data is written to the storage partition, the analysis and query operations begin. Since data access is usually limited, creating an ordered index for all data would waste storage and computing resources, and would also render multi-level caching and cascading updates meaningless. Therefore, ordered indexes are only created for some frequently accessed hot data partitions, i.e., sorted according to access frequency. The higher the access frequency, the higher the priority. This allows for a secondary local sorting of the partition data, making the partition data gradually ordered during use. This does not consume computing resources or loading time during loading, and it also allows for further optimization of partition data analysis after all data has been loaded.

[0081] The method provided in this embodiment obtains the latency of the Service Level Agreement (SLA); partitions the storage space according to the size of the storage partition corresponding to the latency, resulting in multiple storage partitions, where different latencies correspond to different storage partition sizes, and the latency is positively correlated with the size of the storage partition; obtains the data to be stored; writes the data to a hot write cache partition, and after the hot write cache partition is full, sends the data in the hot write cache partition to a hot write cache queue; writes the data in the hot write cache queue to a cold write cache partition, and after the cold write cache partition is full, sends the data in the cold write cache partition to a write index queue; writes the data in the write index queue to the storage partitions sequentially; the number of cold write cache partitions is equal to the number of storage partitions; and establishes a query index for each storage partition according to the query information of each storage partition, wherein multiple storage partitions are kept in an ordered manner through the query index, while the data within the same storage partition is not ordered, thereby realizing multi-level caching and cascading updates of data, avoiding write amplification problems caused by direct writing during data storage, and avoiding frequent cache misses during querying due to the relatively discrete memory distribution.

[0082] Figure 3 Flowchart of the data query and processing method provided in this application Figure 3 .like Figure 3 As shown, based on the above embodiments, this embodiment provides a detailed explanation of the implementation method of building a query index for each storage partition according to the query information of each storage partition in a sequential order.

[0083] S301. Obtain the query count based on the query information for each storage partition;

[0084] Specifically, the query information for each storage partition includes a query count that corresponds one-to-one with the query content, and each storage partition maintains a query count l.

[0085] S302. When a query request is received, the query count is updated to obtain the updated query count;

[0086] Specifically, when a query request is received, the query count is updated first, that is, the query count changes from l to l+1.

[0087] S303. Based on the updated query count, generate a corresponding random number, wherein the value of the random number is between 0 and 1;

[0088] Specifically, assign a random number r between 0 and 1 to l+1.

[0089] S304. Based on the ratio of the cost of building a query index to the benefit of building a query index per query, and the cost coefficient, obtain the index probability corresponding to the next query count.

[0090] Specifically, the formula for calculating the index probability corresponding to the query count is:

[0091]

[0092] In the formula, probability p i represents the probability that the i-th query will trigger the creation of an index, and b represents the ratio of the cost of building the query index to the benefit of building the query index per query.

[0093] Furthermore, the method for calculating the ratio of the cost of building a query index to the benefit of building a query index per query is as follows:

[0094]

[0095] In the formula, c is the cost coefficient, which can be determined based on the actual index building and query execution time, k is the key length, and n is the partition size. For an unordered partition, the query cost of common point queries, range queries, quantile queries, etc., is usually O(n). If a prefix tree or ARTree index is built for the partition, the query cost can be reduced to O(k), so the time complexity of building the index is O(kn).

[0096] S305. Determine whether the probability of the index is less than the random number. If yes, do nothing. If no, execute S306.

[0097] S306. Write the storage partition into the index queue to be built;

[0098] Specifically, if p l+1 If the value is greater than or equal to r, then the partition is added to the queue of indexes to be built after the query is completed; otherwise, the partition is not optimized and sorted.

[0099] S307. Obtain the priority of the storage partition based on the query count, wherein the query count is positively correlated with the priority of the storage partition;

[0100] Specifically, the higher the query count, the higher the priority of the storage partition to be written into the queue of indexes to be built.

[0101] S308. Create a query index for the storage partition in the queue of indexes to be created according to the priority of the storage partition.

[0102] Specifically, the storage partitions in the queue to be indexed are used to create query indexes in descending order of priority, so that partitions with higher priority are retrieved first during the query process, thereby improving query retrieval efficiency.

[0103] The method provided in this embodiment obtains a query count based on the query information of each storage partition; when a query request is received, the query count is updated to obtain an updated query count; a corresponding random number is generated based on the updated query count, the random number being between 0 and 1; the index probability corresponding to the next query count is obtained based on the ratio of the cost of building the query index to the benefit of a single query for building the query index, and the cost coefficient; it is determined whether the index probability is less than the random number; if so, no action is taken; otherwise, the storage partition is written into the queue of indexes to be built; the storage partition is obtained based on the query count. The priority of the storage partition is positively correlated with the priority of the storage partition. A method for building query indexes on the storage partitions in the queue of indexes to be built based on the priority of the storage partitions achieves locally ordered arrangement of data within each partition. The index sequence of the data within each partition is optimized in real time according to the query frequency, so that the orderliness of the partition data gradually increases with the number of queries after loading, avoiding the consumption of computing resources during loading and gradually improving the efficiency of querying low-level partition data after loading. Furthermore, when the query stabilizes, the partition index optimization is also completed. The query performance of this method is comparable to that of existing full index methods.

[0104] In this embodiment of the invention, electronic devices or main control devices can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0105] Figure 4 A schematic diagram of the data query processing device provided in this application. Figure 4 As shown, the device 40 includes:

[0106] Module 401 is used to obtain the latency of the Service Level Agreement (SLA).

[0107] The first processing module 402 is used to partition the storage space according to the size of the storage partition corresponding to the delay, thereby obtaining multiple storage partitions, wherein different delays correspond to different storage partition sizes, and the delay is positively correlated with the size of the storage partition;

[0108] Specifically, the first processing module 402 is used to: after partitioning the storage space according to the size of the storage partition corresponding to the delay to obtain multiple storage partitions, the method further includes:

[0109] Retrieve the data to be stored;

[0110] The data to be stored is written to the cache partition, and after the cache partition is full, the data in the cache partition is sent to the write index queue.

[0111] The data in the write index queue is written to the storage partition sequentially.

[0112] Furthermore, the step of writing the data to be stored into the cache partition, and sending the data of the cache partition to the write index queue after the cache partition is full, includes:

[0113] The data is written to the hot write cache partition, and after the hot write cache partition is full, the data in the hot write cache partition is sent to the hot write cache queue.

[0114] The data in the hot write cache queue is written to the cold write cache partition. After the cold write cache partition is full, the data in the cold write cache partition is sent to the write index queue.

[0115] Furthermore, the number of cold write cache partitions is equal to the number of storage partitions.

[0116] The second processing module 403 is used to build a query index for each storage partition according to the query information of each storage partition in a sequential order. The multiple storage partitions are kept in an ordered arrangement through the query index, while the data within the same storage partition is arranged in an unordered manner.

[0117] Specifically, the second processing module 403 is used for: establishing a query index for each storage partition according to the query information of each storage partition in a sequential manner, including:

[0118] Obtain the query count based on the query information for each storage partition;

[0119] Write the storage partition into the queue of indexes to be built;

[0120] The priority of the storage partition is obtained based on the query count, and the query count is positively correlated with the priority of the storage partition;

[0121] Query indexes are created for the storage partitions in the queue of indexes to be created based on the priority of the storage partitions.

[0122] Furthermore, writing the storage partition into the queue of indexes to be built includes:

[0123] When a query request is received, the query count is updated to obtain the updated query count;

[0124] Based on the updated query count, a corresponding random number is generated, and the value of the random number is between 0 and 1;

[0125] The index probability corresponding to the updated query count is obtained based on the online decision algorithm;

[0126] Determine whether the index probability is less than the random number. If not, write the storage partition into the queue of indexes to be built.

[0127] Furthermore, obtaining the index probability corresponding to the updated query count based on the online decision algorithm includes:

[0128] Based on the ratio of the cost of building a query index to the benefit of building a query index per query, and the cost coefficient, the index probability corresponding to the next query count is obtained.

[0129] The data query processing device provided in this embodiment can execute the data query processing method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0130] In the specific implementation of the aforementioned data query processing device, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to perform the aforementioned data query processing method.

[0131] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in this application. For example... Figure 5 As shown, the electronic device 50 includes at least one processor 501 and a memory 502. The electronic device 50 also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0132] In the specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, causing at least one processor 501 to execute the data query processing method executed on the electronic device side as described above.

[0133] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0134] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0135] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0136] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0137] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0138] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described data query processing method.

[0139] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0140] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0141] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0142] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a data query processing method.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data query processing method, characterized in that, include: The delay in obtaining the Service Level Agreement (SLA); Based on the size of the storage partition corresponding to the latency, the storage space is partitioned to obtain multiple storage partitions, wherein different latencies correspond to different storage partition sizes, and the latency is positively correlated with the size of the storage partition; Obtain the query count based on the query information for each storage partition; Write the storage partition into the queue of indexes to be built; The priority of the storage partition is obtained based on the query count, and the query count is positively correlated with the priority of the storage partition; A query index is created for each storage partition in the queue of indexes to be created based on the priority of the storage partition; wherein, multiple storage partitions are kept in an ordered manner through the query index, and the data within the same storage partition is arranged in an unordered manner.

2. The method according to claim 1, characterized in that, After partitioning the storage space according to the size of the storage partition corresponding to the delay to obtain multiple storage partitions, the method further includes: Retrieve the data to be stored; The data to be stored is written to the cache partition, and after the cache partition is full, the data in the cache partition is sent to the write index queue. The data in the write index queue is written to the storage partition sequentially.

3. The method according to claim 2, characterized in that, The step of writing the data to be stored into the cache partition, and sending the data of the cache partition to the write index queue after the cache partition is full, includes: The data is written to the hot write cache partition, and after the hot write cache partition is full, the data in the hot write cache partition is sent to the hot write cache queue. The data in the hot write cache queue is written to the cold write cache partition. After the cold write cache partition is full, the data in the cold write cache partition is sent to the write index queue.

4. The method according to claim 3, characterized in that, The number of cold write cache partitions is equal to the number of storage partitions.

5. The method according to claim 1, characterized in that, The step of writing the storage partition into the queue of indexes to be built includes: When a query request is received, the query count is updated to obtain the updated query count; Based on the updated query count, a corresponding random number is generated, and the value of the random number is between 0 and 1; The index probability corresponding to the updated query count is obtained based on the online decision algorithm; Determine whether the index probability is less than the random number. If not, write the storage partition into the queue of indexes to be built.

6. The method according to claim 5, characterized in that, The step of obtaining the index probability corresponding to the updated query count based on the online decision algorithm includes: Based on the ratio of the cost of building a query index to the benefit of building a query index per query, and the cost coefficient, the index probability corresponding to the next query count is obtained.

7. A data query and processing device, characterized in that, include: The acquisition module is used to acquire the latency of the Service Level Agreement (SLA). The first processing module is used to partition the storage space according to the size of the storage partition corresponding to the delay, thereby obtaining multiple storage partitions, wherein different delays correspond to different storage partition sizes, and the delay is positively correlated with the size of the storage partition; The second processing module is used to obtain the query count based on the query information of each storage partition; write the storage partition into the index queue to be built; obtain the priority of the storage partition based on the query count, wherein the query count is positively correlated with the priority of the storage partition; and build a query index for the storage partition in the index queue to be built based on the priority of the storage partition; wherein multiple storage partitions are kept in an ordered arrangement through the query index, and the data within the same storage partition is arranged in an unordered manner.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images