Index recommendation method and system for cloud data analysis system

Abstract

The invention discloses an index recommendation method and system for a cloud data analysis system, and belongs to the technical field of index recommendation, and the method comprises the steps: for each current load, obtaining the feature representation of each query task on each data block, and fusing the feature representation as the input feature of the corresponding load; performing vectorization representation on the text information of each load, the input characteristics of each load and the current storage state information, and then inputting the text information, the input characteristics and the current storage state information into the trained index candidate evaluation model to obtain scores of all index candidates; the index candidate represents a scheme for creating an index for one column or multiple columns of one data block; the index candidate evaluation model is a neural network model based on an attention mechanism; screening out the index candidate with the highest score and then creating an index for the corresponding data block; updating the storage state information; if the storage overhead constraint is met and an index capable of being created exists, carrying out the next cycle; otherwise, ending the circulation. According to the method, scarce memory resources can be fully utilized to accelerate the index scanning effect.

Description

Technical Field

[0001] This invention belongs to the field of index recommendation technology, and more specifically, relates to an index recommendation method and system for cloud data analysis systems. Background Technology

[0002] In modern cloud computing environments, the data analytics field widely adopts a design strategy that separates storage and computing architectures. This architecture physically or logically decouples computing resources (e.g., compute clusters) from storage resources, allowing both to scale independently to adapt to changing resource demands. Furthermore, to optimize data transfer efficiency and reduce the frequency of input / output (I / O) operations, data blocks—typically containing tens of thousands to millions of data records from a single dataset—are defined as the basic unit for accessing remote storage resources, and also the smallest unit for data filtering and processing. This design not only increases data throughput but also improves overall system performance by reducing the number of I / O operations.

[0003] Indexes, as an important technique for improving data scanning efficiency in databases, are widely used in traditional database scenarios. However, in cloud data analytics scenarios with massive data volumes, secondary indexes are difficult to fully utilize because they consume too much scarce memory, thus hindering the system's performance bottlenecks.

[0004] Index recommendation technology, as an important means to improve memory utilization, can alleviate the problem of excessive index overhead to some extent. Academic research on this topic can be broadly divided into two categories. One category consists of heuristic-based index recommendation algorithms, such as Drop, Relaxation, and Extend. The core strategies of these algorithms can be further divided into two types: one starts from an empty set of indexes and continuously adds new candidate indexes that can improve query execution efficiency until the constrained storage space is full; the other starts from the entire set of indexes and continuously reduces the indexes that have the least impact on improving query execution efficiency until the storage overhead constraint is met. These heuristic index recommendation algorithms can perceive complex influences such as interactions between indexes, thus making better index selection decisions, but they are prone to high computational overhead, making them unsuitable for scenarios with excessively large scales (e.g., involving many attribute columns). The other category consists of reinforcement learning-based index recommendation algorithms, such as Swirl and DRLinda. These indexes can achieve efficient solutions for better index selection decisions at the cost of higher training overhead.

[0005] The aforementioned works all recommend index combinations using the dataset as the smallest granularity, wasting opportunities for index tuning at the data block level. Therefore, they struggle to fully utilize scarce memory resources to accelerate index scanning. To address this issue, some research has attempted index tuning at the data block level. For example, Alibaba attempts to build a single-column index for each data block; Slalom, on the other hand, can dynamically select the single-column index to build based on workload.

[0006] Even with the aforementioned techniques for indexing and recommending at the data block level, existing work struggles to meet the demands of effectively utilizing scarce memory resources in cloud computing scenarios. This is because such work can only support the construction of single-column indexes, not multi-column indexes, and it also cannot effectively handle issues such as interactions between indexes that affect the performance of index recommendation algorithms. Summary of the Invention

[0007] To address the shortcomings and improvement needs of existing technologies, this invention provides an index recommendation method and system for cloud data analysis systems. Its purpose is to establish appropriate single-column or multi-column indexes for data blocks with different distributions based on the differences in data distribution between data blocks, so as to make full use of scarce memory resources and accelerate the index scanning effect.

[0008] To achieve the above objectives, according to one aspect of the present invention, an index recommendation method for cloud data analytics systems is provided, comprising:

[0009] (S1) For each current load, obtain the feature representation of each query task on each data block and fuse them as the input features of the corresponding load.

[0010] (S2) The text information of each load, the input features of each load, and the current storage status information of the cloud data analysis system are vectorized to obtain the current system status feature vector.

[0011] (S3) Input the current system state feature vector into the trained index candidate evaluation model to obtain the scores of all index candidates; an index candidate represents a scheme for creating an index on one or more columns of a data block; the index candidate evaluation model is a neural network model based on an attention mechanism;

[0012] (S4) Filter out the index candidates with the highest scores, and create an index for one or more columns of the corresponding data block according to the index candidate;

[0013] (S5) Update storage status information; if the preset storage overhead constraint is met and there is an index that can be created, then proceed to (S1) to perform the next loop; otherwise, end the loop.

[0014] Furthermore, the feature representation w of query task q on data block b.b This includes the proportion of data accessed by query task q in data block b to the total data volume of the data block, as well as the range of data accessed by query task q in data block b.

[0015] Furthermore, the feature representation w of query task q on data block b. b The methods of obtaining it include:

[0016] Obtain the bitmap h of data block b respectively b And the bitmap h of query task q q Bitmap h b In the bitmap, each bit corresponds to a data range, and the bit corresponding to the data range containing the value of each data record in data block b is set to 1, while the remaining bits are set to 0; bitmap h q In the bitmap h, each bit corresponds to a data range, and the bit corresponding to the data range involved in the predicate of query task q is set to 1, while the remaining bits are set to 0; b and bitmap h q The lengths are equal, and the data ranges corresponding to the same bits are the same;

[0017] Bitmap h b and bitmap h q Performing a bitwise AND operation yields a union bitmap h that records the data range within data block b accessed by the queried task q. u ;

[0018] Calculate the joint bitmap h u The number of 1s in the middle and the bitmap h b The ratio p of the number of 1s in data block b is used to obtain the proportion of the amount of data accessed by the query task q in data block b to the total amount of data in the data block.

[0019] The ratio p and the joint bitmap h u By combining these features, we obtain the feature representation w of query task q on data block b. b .

[0020] Furthermore, attention-based neural network models include attention layers, linear layers, and output layers;

[0021] The attention layer uses the vectorized payload text information as the query and the vectorized payload input features as the key and value. It evaluates the relevance between each key and the query, uses it as the weight of the corresponding value, and performs a weighted summation of each value.

[0022] The linear layer takes the input from the attention layer and the vectorized storage state information as input;

[0023] The output layer is used to perform SoftMax processing on the output of the linear layer to obtain the scores of each index candidate.

[0024] According to another aspect of the present invention, a computer program product is provided, comprising a computer program; when the computer program is executed by a processor, it implements the above-described index recommendation method for cloud data analysis systems provided by the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the above-described index recommendation method for cloud data analysis systems provided by the present invention.

[0026] According to another aspect of the present invention, an index recommendation system for cloud data analysis systems is provided, comprising:

[0027] A computer-readable storage medium for storing computer programs;

[0028] And a processor for reading a computer program stored in a computer-readable storage medium and executing the index recommendation method for cloud data analysis systems provided by the present invention.

[0029] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0030] (1) This invention fuses the feature representations of each query task in the load on each data block as the input features of the corresponding load, and takes all possible index recommendations (including single-column indexes or index combinations) for each data block as index candidates. It uses a neural network model based on attention mechanism to score each index candidate based on load input features, load text information and system storage state information, and finally obtains the most suitable index candidate. This can fully consider the differences in data distribution between different data blocks, and use the neural network model to learn the interaction between indexes. While realizing data block-level index recommendation, it can establish suitable single-column indexes or multi-column indexes for data blocks with different distributions, thereby making full use of scarce memory resources to accelerate the index scanning effect.

[0031] (2) In the preferred embodiment of the present invention, the feature representation of the query task on the data block specifically includes the proportion of the amount of data accessed by the query task q in data block b to the total amount of data in the data block, and the data range of the data accessed by the query task in the data block. While fully considering the data distribution in the data block, it can fully consider the degree of fit between the load and the data block, thereby improving the feature expression capability of the load input feature and improving the accuracy of subsequent index candidate evaluation.

[0032] (3) In this invention, feature representations are obtained according to data blocks, which brings a large number of input features and affects the training efficiency of the model. This invention uses a neural network model based on attention mechanism to score each index candidate, which enables the model to focus its attention on the data blocks that are more suitable for creating indexes under the current workload, thereby improving the convergence efficiency and effect of the model. In its preferred scheme, the neural network model uses the feature representations loaded on the data blocks as both keys and values, and uses the text information of the load as a query. It learns the relevance between the query and the key, uses it as the weight of the corresponding value, performs a weighted summation operation on the value and uses it as the result output of the attention layer, and uses the result of the attention layer and the remaining results as the input of the linear layer. The output of the linear layer is processed by SoftMax to obtain the creation probability of each index candidate. The model structure is simple and can further improve the convergence speed of the model without affecting the index candidate scoring results. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of an index recommendation method for cloud data analysis systems provided in an embodiment of the present invention;

[0034] Figure 2 This is an example of a block load feature representation of a simplified histogram provided in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the index candidate evaluation model provided in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the attention layer in the index candidate evaluation model provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0038] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0039] To address the technical limitations of existing block-level index recommendation methods, which only support single-column index construction and cannot effectively handle interactions between indexes, the overall concept is as follows: Considering the differences in data distribution among different data blocks, when representing the load as features, the feature representations of each query task in the load on each data block are considered separately. All possible index recommendation schemes (including single-column and multi-column indexes) for each data block are treated as candidate indexes. An attention-based neural network model is used to score each candidate index, efficiently processing the large amount of feature data generated by the block feature representations and learning the interactions between indexes, thereby selecting appropriate index recommendation schemes for data blocks with different data distributions.

[0040] Before explaining the technical solution of the present invention in detail, the relevant technical terms are briefly introduced as follows:

[0041] A data block is a collection of tens of thousands to millions of data records from a single dataset.

[0042] Load: A collection of one or more query tasks.

[0043] Index: A data structure commonly used in databases to retrieve the values ​​of one or more columns.

[0044] Index recommendation: An algorithm commonly used in databases to select one or more columns to create an index in order to improve query efficiency.

[0045] Storage-compute separation: a new data architecture design paradigm, divided from top to bottom into a data analysis layer, a computation layer, and a storage layer. The computation and storage layers are decoupled, each being an independent distributed service. Its design aims to address three needs: data can be flexibly shared with different business units for data analysis; computation and storage can be independently scaled; and computation and storage resources are isolated, while also providing storage-compute performance equivalent to an integrated storage-compute architecture.

[0046] I / O: Input and output of data.

[0047] The following is an example.

[0048] Example 1:

[0049] An index recommendation method for cloud data analytics systems, such as Figure 1 As shown, it includes:

[0050] (S1) For each current load, obtain the feature representation of each query task on each data block and fuse them as the input features of the corresponding load.

[0051] (S2) The text information of each load, the input features of each load, and the current storage status information of the cloud data analysis system are vectorized to obtain the current system status feature vector.

[0052] (S3) Input the current system state feature vector into the trained index candidate evaluation model to obtain the scores of all index candidates; an index candidate represents a scheme for creating an index on one or more columns of a data block; the index candidate evaluation model is a neural network model based on an attention mechanism;

[0053] (S4) Filter out the index candidates with the highest scores, and create an index for one or more columns of the corresponding data block according to the index candidate;

[0054] (S5) Update storage status information; if the preset storage overhead constraint is met and there is an index that can be created, then proceed to (S1) to perform the next loop; otherwise, end the loop.

[0055] This embodiment fuses the feature representations of each query task in the load across each data block as the input features of the corresponding load, and takes all possible index recommendations (including single-column indexes or index combinations) for each data block as index candidates. It uses an attention-based neural network model to score each index candidate based on the load input features, load text information, and system storage state information, and finally obtains the most suitable index candidate. This fully considers the differences in data distribution between different data blocks and uses the neural network model to learn the interaction between indexes. While realizing data block-level index recommendation, it establishes appropriate single-column or multi-column indexes for data blocks with different distributions, thereby making full use of scarce memory resources to accelerate the index scanning effect.

[0056] The following provides a further explanation of the specific implementation methods for each step.

[0057] This embodiment, through step (S1), fuses the feature representations of each query task in the load across each data block as the input features for the corresponding load, thus fully considering the differences in data distribution between different data blocks. As a preferred implementation, in this embodiment, for any query task q, the feature representation w on any data block b... b This includes the proportion of data accessed by query task q in data block b to the total data volume of the data block, as well as the range of data accessed by query task q in data block b. This fully considers the data distribution in the data block and the degree of fit between the load and the data block, thereby improving the feature representation capability of the load input features and improving the accuracy of subsequent index candidate evaluation.

[0058] Specifically, this embodiment proposes a block load feature representation method based on simplified histograms to obtain the feature representation of a query task on a certain data block. Based on the proposed block load feature representation method based on simplified histograms, the feature representation w of query task q on data block b is obtained. b The methods of obtaining it include:

[0059] Obtain the bitmap h of data block b respectively b And the bitmap h of query task q q Bitmap h b In the bitmap, each bit corresponds to a data range, and the bit corresponding to the data range containing the value of each data record in data block b is set to 1, while the remaining bits are set to 0; bitmap h q In the bitmap h, each bit corresponds to a data range, and the bit corresponding to the data range involved in the predicate of query task q is set to 1, while the remaining bits are set to 0; b and bitmap h q The lengths are equal, and the data ranges corresponding to the same bits are the same;

[0060] Bitmap h b and bitmap h q Performing a bitwise AND operation yields a union bitmap h that records the data range within data block b accessed by the queried task q. u ;

[0061] Calculate the joint bitmap h u The number of 1s in the middle and the bitmap h b The ratio p of the number of 1s in data block b is used to obtain the proportion of the amount of data accessed by the query task q in data block b to the total amount of data in the data block.

[0062] The ratio p and the joint bitmap h u By combining these features, we obtain the feature representation w of query task q on data block b. b .

[0063] by Figure 2 For example, this includes a query task and three data blocks. Based on a preset partitioning scheme, the numerical distribution range of wages is divided into seven data ranges, and a bitmap is created for the query task. Figure 2 The query bitmap is used to create bitmaps for each data block. Figure 2 The block bitmap in the file;

[0064] Perform a bitwise AND operation between the created query bitmap and each block bitmap to obtain a Union Bitmap that represents the data range of the data accessed by the query task in each data block.

[0065] Calculate the ratio of the number of 1s in each joint bitmap to the number of 1s in the corresponding block bitmap. Use this ratio to estimate the proportion of the data accessed by the query task to the total data volume of each data block. Then integrate this ratio with the corresponding joint bitmap to obtain the feature representation of the query task on each data block.

[0066] Based on the feature representation obtained by the above method, it is possible to perceive the data distribution in different data blocks. Therefore, the final feature representation obtained is a block-aware load representation.

[0067] In this embodiment, after obtaining the feature representations of each query task in the workload on each data block using the above method, the obtained feature representations are concatenated as the input features of the workload. While this method fully considers the differences in data distribution of different data blocks, it will bring a large number of input features, which will affect the training efficiency of the model. This invention uses a neural network model based on the attention mechanism to score each index candidate, which enables the model to focus its attention on the data block that is more suitable for creating an index under the current workload, thereby improving the convergence efficiency and effect of the model.

[0068] As a preferred implementation method, such as Figure 3 As shown, in this embodiment, the established index candidate evaluation network specifically includes: an attention layer, a linear layer, and an output layer;

[0069] like Figure 4 As shown, the attention layer uses the vectorized payload text information as the query and the vectorized payload input features as the key and value. It evaluates the relevance between each key and the query (e.g., using dot product (DP)) as the weight of the corresponding value and performs a weighted summation on each value.

[0070] The linear layer takes the input of the attention layer and the vectorized storage state information as input; the storage state information includes the created index combinations, available storage overhead, and metadata information of data blocks such as the maximum and minimum values ​​of index columns and the number of data rows.

[0071] The output layer is used to perform SoftMax processing on the output of the linear layer to obtain the scores of each index candidate.

[0072] The model established in this embodiment can further improve the convergence speed of the model without affecting the index candidate scoring results.

[0073] In this embodiment, based on the index candidate scores output by the model, the index candidate with the highest score is selected, and a single-column or multi-column index is created for the corresponding data block. After that, the storage status information of the system will change. If the storage overhead constraint has not been exceeded, that is, there is still free memory space available for creating the index, the storage status information will be updated, and a suitable index recommendation scheme will be determined again through the above steps (S1) to (S4).

[0074] In summary, this embodiment provides an index recommendation method and system for cloud data analysis systems. It proposes a block load feature representation method using simplified histograms to obtain the input features of each load, enabling the model to learn the differences in data distribution between data blocks. At the same time, it uses a neural network model based on an attention mechanism to efficiently process the large amount of feature data brought by the block feature representation, thereby enabling the selection of appropriate index recommendation schemes (including single-column indexes and multi-column indexes) for data blocks with different data distributions.

[0075] Example 2:

[0076] A computer program product includes a computer program; when the computer program is executed by a processor, it implements the index recommendation method for cloud data analysis systems provided in Embodiment 1 above.

[0077] Example 3:

[0078] A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the index recommendation method for a cloud data analysis system provided in Embodiment 1 above.

[0079] Example 4:

[0080] An indexing and recommendation system for cloud data analytics systems includes:

[0081] A computer-readable storage medium for storing computer programs;

[0082] And a processor for reading a computer program stored in a computer-readable storage medium and executing the index recommendation method for a cloud data analysis system provided in Embodiment 1 above.

[0083] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An index recommendation method for cloud data analysis systems, characterized in that, include: (S1) For each current load, obtain the feature representation of each query task on each data block and fuse them as the input features of the corresponding load. (S2) The text information of each load, the input features of each load, and the current storage status information of the cloud data analysis system are vectorized to obtain the current system status feature vector. (S3) Input the current system state feature vector into the trained index candidate evaluation model to obtain the scores of all index candidates; the index candidate represents a scheme for creating an index on one or more columns of a data block; the index candidate evaluation model is a neural network model based on an attention mechanism; (S4) Filter out the index candidates with the highest scores, and create an index for one or more columns of the corresponding data block according to the index candidate; (S5) Update storage status information; if the preset storage overhead constraint is met and there is an index that can be created, then proceed to (S1) to perform the next loop; otherwise, end the loop.

2. The index recommendation method for cloud data analysis systems as described in claim 1, characterized in that, Query task q's feature representation w on data block b b This includes the proportion of data accessed by query task q in data block b to the total data volume of the data block, as well as the range of data accessed by query task q in data block b.

3. The index recommendation method for cloud data analysis systems as described in claim 2, characterized in that, Query task q's feature representation w on data block b b The methods of obtaining it include: Obtain the bitmap h of data block b respectively b And the bitmap h of query task q q Bitmap h b In the bitmap, each bit corresponds to a data range, and the bit corresponding to the data range containing the value of each data record in data block b is set to 1, while the remaining bits are set to 0; bitmap h q In the bitmap h, each bit corresponds to a data range, and the bit corresponding to the data range involved in the predicate of query task q is set to 1, while the remaining bits are set to 0; b and bitmap h q The lengths are equal, and the data ranges corresponding to the same bits are the same; Bitmap h b and bitmap h q Performing a bitwise AND operation yields a union bitmap h that records the data range within data block b accessed by the queried task q. u ; Calculate the joint bitmap h u The number of 1s in the middle and the bitmap h b The ratio p of the number of 1s in data block b is used to obtain the proportion of the amount of data accessed by the query task q in data block b to the total amount of data in the data block. The ratio p and the joint bitmap h u By combining these features, we obtain the feature representation w of query task q on data block b. b .

4. The index recommendation method for cloud data analysis systems as described in any one of claims 1 to 3, characterized in that, The attention-based neural network model includes an attention layer, a linear layer, and an output layer; The attention layer uses the vectorized payload text information as the query and the vectorized payload input features as the key and value. It evaluates the relevance between each key and the query, uses it as the weight of the corresponding value, and performs a weighted summation of each value. The linear layer takes the input of the attention layer and the vectorized storage state information as input; The output layer is used to perform SoftMax processing on the output of the linear layer to obtain the scores of each index candidate.

5. A computer program product, characterized in that, It includes a computer program; when the computer program is executed by a processor, it implements the index recommendation method for cloud data analysis systems as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The system includes a stored computer program; when executed by a processor, the computer program controls the device containing the computer-readable storage medium to perform the index recommendation method for a cloud data analysis system as described in any one of claims 1 to 4.

7. An indexing and recommendation system for cloud data analysis systems, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in the computer-readable storage medium and executing the index recommendation method for a cloud data analysis system as described in any one of claims 1 to 4.

Images