A machine learning-based distributed query system and its optimization method

Abstract

This invention relates to a machine learning-based distributed query system and its optimization method. The clustering module analyzes newly generated data in the system to obtain clustering results. The data sharding storage module stores the received shard key ranges into corresponding data shards, ensuring that similar data is allocated to the same or a few data shards. The regression module works with the data caching module to actively cache data, predicting future data usage and proactively caching high-usage data on high-speed read devices. The data caching module sorts the predicted future access rates of the data, obtains the shard keys corresponding to high-usage data, reads the corresponding data from the data sharding storage module, and passes this data to the data caching module, which then stores the received data on the high-speed read device. This addresses the problem of loose connections between data stored in a distributed manner on disks, making queries more difficult.

Description

Technical Field

[0001] This invention relates to the field of distributed querying, specifically a distributed query system based on machine learning and its optimization method. Background Technology

[0002] In recent years, research and development of distributed query systems have been on the rise both domestically and internationally. Due to the rapid increase in the volume and complexity of system data, distributed database systems have become the mainstream technology for data storage and processing. Internationally, representative distributed database technologies include Google's Bigtable, Amazon's DynamoDB, and Facebook's Cassandra. Domestically, distributed database systems such as Alibaba's OceanBase, PolarDB, and PingCAP's TiDB are also continuously developing and improving. Furthermore, with the continuous maturation of cloud computing technology, cloud databases have gradually become a new development trend. This trend is driven by the rapid growth of data and the increasing demand for real-time processing of large-scale data. To meet these demands, researchers have been developing new technologies to improve the performance, scalability, and reliability of distributed query systems. For example, new algorithms and data structures have been proposed to reduce communication costs and increase the parallelism of distributed queries. New technologies for load balancing and fault tolerance have also been developed to ensure the robustness and reliability of distributed query systems.

[0003] Furthermore, the trend of integrating machine learning techniques into distributed query systems is becoming increasingly apparent, and it is currently one of the hottest topics in the field of artificial intelligence. With the rapid growth of data volume and the continuous expansion of machine learning application scenarios, distributed machine learning technology has become a necessary solution with broad development prospects. For example, machine learning can be applied to the field of distributed data processing, such as distributed feature engineering and distributed machine learning model training. These tasks often require data partitioning, allocation, and aggregation, which often involve complex query plan optimization and resource allocation. Therefore, machine learning methods can provide efficient solutions for these tasks. Simultaneously, machine learning can analyze data during system operation, automatically identify bottlenecks and performance limitations, and provide corresponding optimization suggestions. These optimization suggestions can help developers quickly locate system problems and improve system reliability and stability. Summary of the Invention

[0004] This invention provides a machine learning-based distributed query system and its optimization method to solve two problems: one is that data is stored in a distributed manner on disks with loose connections between them, making queries more difficult; the other is the problem of uneven data access load, where some data is accessed for most of the time during system operation.

[0005] Compared with existing technologies, the present invention has the following advantages:

[0006] This invention uses a clustering algorithm to group data with similar features into the same data shard, which can improve query efficiency after data sharding. This is because the principle of spatial locality states that spatially proximate data are more likely to be accessed simultaneously in time, meaning that data with similar features will be accessed together during querying. Therefore, grouping data with similar features into the same data shard can avoid cross-shard queries, thus improving query efficiency.

[0007] This invention uses a linear regression algorithm to predict data usage and caches frequently accessed data on high-speed read devices, which can significantly shorten data access time and thus improve data access efficiency. This technology can reduce the number of times data is read from disks or remote storage devices, reduce system access latency and response time, thereby accelerating the data reading and processing process.

[0008] This invention reduces the number of times data is read from remote storage devices by pre-caching frequently accessed data, thereby reducing network transmissions and I / O operations and lowering system load. This allows the system to respond to user requests more quickly and improves system throughput.

[0009] After the regression module predicts the usage rate of all data, this invention sorts the usage rates and directly obtains the data with high usage rates. This avoids passing all data and its predicted usage rates to the data caching module, thereby reducing the burden of data transmission and improving caching speed.

[0010] This invention caches frequently accessed data predicted by a linear regression algorithm on a high-speed read device, thus avoiding system crashes and failures caused by a large number of data read and write operations. By concentrating these operations on the high-speed read device, the system's operating pressure is reduced. This improves system stability and reliability, and reduces system maintenance costs and risks.

[0011] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0012] A machine learning-based distributed query system, comprising a clustering module, a data sharding and storage module, a regression module, and a data caching module;

[0013] The clustering module is used to analyze newly generated data in the system and obtain clustering results;

[0014] The data sharding storage module is used to store the received sharding key range into the corresponding data shards, ensuring that similar data are allocated to the same or several data shards;

[0015] The regression module is used in conjunction with the data caching module to perform active data caching, predict the future usage rate of data, and actively cache data with high usage rate to high-speed reading devices.

[0016] The data caching module is used to sort the predicted future access rates of the data, obtain the shard keys corresponding to the data with high usage rates, then read the corresponding data from the data shard storage module, and pass this data to the data caching module. The data caching module stores the received data on the high-speed reading device.

[0017] Furthermore, the clustering module includes a data collection module, a feature engineering module, a model training module, and a model application module;

[0018] The data collection module is used to collect data about distributed queries;

[0019] The feature engineering module is used to select valid features and process and transform them;

[0020] The model training module is used to train a clustering model using the collected data;

[0021] The model application module is used to optimize queries in a distributed query system using a trained model.

[0022] Furthermore, the data sharding storage module includes disk blocks and a data backup module;

[0023] The disk block contains several shard keys. The shard keys in the disk block store the clustering results generated by the clustering module and are stored in shards according to data with similar data characteristics, in order to distinguish the shard to which the data belongs.

[0024] The data backup module is used to ensure high data availability, and new data will be backed up to different data shards in the data backup module.

[0025] Furthermore, the regression module includes a data processing module, a data receiving module, a usage rate prediction module, and a usage rate ranking module;

[0026] The data receiving module is used to receive data from the data fragmentation storage module;

[0027] The data processing module is used to process the data received by the data receiving module;

[0028] The utilization rate prediction module is used to predict the utilization rate of the processed data and send the result to the utilization rate sorting module;

[0029] The usage rate sorting module is used to cache data with high access rates on high-speed reading devices, which can avoid system crashes and failures caused by a large number of data read and write operations. With a large number of read and write operations concentrated on high-speed reading devices, the system's operating pressure is reduced.

[0030] An optimization method for a machine learning-based distributed query system, the optimization method using the machine learning-based distributed query system described above, the optimization method comprising the following steps:

[0031] Step 1: Use the clustering module to collect user data and pass information involving new data shard key values ​​to the data shard storage module;

[0032] Step 2: The data sharding and storage module provides the clustering module with the data required for training the clustering model;

[0033] Step 3: The regression module uses the data in the data sharding storage module to train a linear regression model, predicts the future usage rate of the data, and reads the data with high usage rate from the data sharding storage module.

[0034] Step 4: Store the data with high usage frequency from Step 3 into the data cache module;

[0035] Step 5: When querying data through the distributed query system, determine whether the data is frequently used. If the data is frequently used, retrieve it through the data caching module; if the data is infrequently used, retrieve it through the data sharding storage module.

[0036] Furthermore, the training of the clustering model specifically involves,

[0037] First, data is collected from the distributed system to prepare for feature engineering. In the feature engineering phase, features of the data are extracted. In the model training phase, the features of the data are analyzed, and K points are randomly selected from the data as cluster centers. The data in the dataset are assigned to the clusters containing the nearest cluster centers. The dataset consists of system data and user data from the distributed system. Then, the average value of all data points in a cluster is calculated to determine new cluster centers. This process is iterated until the cluster centers no longer change or the maximum number of iterations is reached, at which point the model training is complete.

[0038] Furthermore, the data distribution of the data sharding storage module is specifically as follows:

[0039] The clustering module analyzes newly generated data in the system to obtain clustering results. It then passes the clustering results and the range of shard keys corresponding to data shards with similar characteristics to the new data to the data shard storage module. The module stores the received shard key range into the corresponding data shards, ensuring that similar data is allocated to the same or a few data shards. Based on the range of shard keys, the module determines the location of the data shards and stores the new data there. The new data is also backed up to different data shards, and the data in the backup shards should all contain data with similar characteristics to the backup. Upon receiving a read / write request from the system, the module routes the request to the specific data shard, uses a hash function to calculate the shard key corresponding to the read / write request, obtains the data shard containing the target data, and finally returns the data to the user.

[0040] Furthermore, the predicted future usage rate of the data specifically refers to...

[0041] Model definition: In a linear regression model, the model can be represented as: y = θ0 + θ1x. Where θ0 and θ1 are the model parameters, which need to be obtained through training.

[0042] Determine the loss function: The loss function is mainly used to calculate the difference between the predicted value and the label value. Here, the mean squared error loss function is used.

[0043] Solving for parameters: The model parameters are calculated using the gradient descent method. In each iteration, θ0 and θ1 are updated along the direction of gradient descent until the loss function converges.

[0044] Prediction: After the model is trained, it is used to predict the future usage of the data, and the data with high usage is actively cached on high-speed reading devices.

[0045] Furthermore, step 4 specifically involves querying cached data.

[0046] Data reception: Receives the high-usage data predicted by the regression module and prepares to store the data in the high-speed reading device;

[0047] Determine the caching device: Select a suitable high-speed read device based on the system's budget and actual situation;

[0048] Establish query logic: When a user initiates a query request, first look for the data in the cache device. If the data is found, return it directly. If the target data is not in the cache, look for the data in the underlying data shard.

[0049] Update cached data: When data changes, the data in the cache needs to be updated in a timely manner to maintain the accuracy of the cached data and its consistency in the system. Attached Figure Description

[0050] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0051] Figure 1 A schematic diagram of the structure created by this invention;

[0052] Figure 2 The flowchart of the clustering module created for this invention.

[0053] Figure 3 The structural diagram of the data fragmentation storage module created for this invention is shown.

[0054] Figure 4 The flowchart of the regression module execution created for this invention.

[0055] Figure 5 A flowchart illustrating the data caching process created for this invention.

[0056] Figure 6 A flowchart illustrating the query cache data flow created for this invention.

[0057] Figure 7 The system structure diagram of the query function created for this invention. Detailed Implementation

[0058] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0059] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0060] A machine learning-based distributed query system, comprising a clustering module, a data sharding and storage module, a regression module, and a data caching module;

[0061] The clustering module is used to analyze newly generated data in the system and obtain clustering results;

[0062] The data sharding storage module is used to store the received sharding key range into the corresponding data shards, ensuring that similar data are allocated to the same or several data shards;

[0063] The regression module is used in conjunction with the data caching module to perform active data caching, predict the future usage rate of data, and actively cache data with high usage rate to high-speed reading devices.

[0064] The data caching module is used to sort the predicted future access rates of the data, obtain the shard keys corresponding to the data with high usage rates, then read the corresponding data from the data shard storage module, and pass this data to the data caching module. The data caching module stores the received data on the high-speed reading device.

[0065] Furthermore, the clustering module includes a data collection module, a feature engineering module, a model training module, and a model application module;

[0066] The data collection module is used to collect data about distributed queries, such as query operation time, network latency, and data size.

[0067] The feature engineering module is used to select effective features and process and transform them to facilitate model training.

[0068] The model training module is used to train a clustering model using the collected data;

[0069] The model application module is used to optimize queries in a distributed query system using a trained model.

[0070] Furthermore, the data sharding storage module includes disk blocks and a data backup module;

[0071] The disk block contains several shard keys. The shard keys in the disk block store the clustering results generated by the clustering module and are stored in shards according to data with similar data characteristics, in order to distinguish the shard to which the data belongs.

[0072] The data backup module is used to ensure high data availability. For new data, it will also back it up to different data shards in the data backup module. The data shards where the backup is located should contain data that are all similar to the backup characteristics.

[0073] Define sharding keys: For data of the same type, their keys are consecutive, which allows us to distinguish which shard the data belongs to;

[0074] Data allocation: Data is stored in various shards based on the value of the shard key;

[0075] Creating replicas: To ensure high data availability, replicas are distributed across different shards;

[0076] Data read / write: Based on the read / write request, locate the specific data shard and return the data to the user.

[0077] Furthermore, the regression module includes a data processing module, a data receiving module, a usage rate prediction module, and a usage rate ranking module;

[0078] The data receiving module is used to receive data from the data fragmentation storage module;

[0079] The data processing module is used to process the data received by the data receiving module;

[0080] The utilization rate prediction module is used to predict the utilization rate of the processed data and send the result to the utilization rate sorting module;

[0081] The usage rate sorting module is used to cache data with high access rates on high-speed reading devices, which can avoid system crashes and failures caused by a large number of data read and write operations. With a large number of read and write operations concentrated on high-speed reading devices, the system's operating pressure is reduced.

[0082] An optimization method for a machine learning-based distributed query system, the optimization method using the machine learning-based distributed query system described above, the optimization method comprising the following steps:

[0083] Step 1: Use the clustering module to collect user data and pass information involving new data shard key values ​​to the data shard storage module;

[0084] Step 2: The data sharding and storage module provides the clustering module with the data required for training the clustering model;

[0085] Step 3: The regression module uses the data in the data sharding storage module to train a linear regression model, predicts the future usage rate of the data, and reads the data with high usage rate from the data sharding storage module.

[0086] Step 4: Store the data with high usage frequency from Step 3 into the data cache module;

[0087] Step 5: When querying data through the distributed query system, determine whether the data is frequently used. If the data is frequently used, retrieve it through the data caching module; if the data is infrequently used, retrieve it through the data sharding storage module.

[0088] Furthermore, the training of the clustering model specifically involves,

[0089] First, data is collected from the distributed system to prepare for feature engineering. In the feature engineering phase, features of the data are extracted, such as data size and query time. In the model training phase, the features of the data are analyzed, and K points are randomly selected from the data as cluster centers. The data in the dataset are assigned to the clusters containing the nearest cluster centers. The dataset consists of system data and user data from the distributed system. Then, the average value of all data points in a cluster is calculated to determine new cluster centers. This process is iterated until the cluster centers no longer change or the preset maximum number of iterations is reached, at which point the model training is complete.

[0090] Since clustering algorithms are unsupervised, there is no concept of predicted values ​​and label values. For newly emerging data in a distributed system, the model can proactively classify it into data shards with similar features. The selection of the number of cluster centers is crucial. Too few cluster centers may lead to some data being misclassified, while too many may result in uneven data distribution, offering no advantage over traditional sharding methods. Therefore, the initial range for cluster centers is defined as: 5 <= k <= 10.

[0091] Furthermore, the data distribution of the data sharding storage module is specifically as follows:

[0092] The clustering module analyzes newly generated data in the system to obtain clustering results. It then passes the clustering results and the range of shard keys corresponding to data shards with similar characteristics to the new data to the data shard storage module. The module stores the received shard key range into the corresponding data shards, ensuring that similar data is allocated to the same or a few data shards. The module determines the location of the data shard based on the range of shard keys and stores the new data in the shard. To ensure high availability, new data is backed up to different data shards, and the data in the backup shard should contain only data with similar characteristics to the backup. Furthermore, to maintain data consistency in the system, multi-version control technology is used to process the data and ensure its stable and correct storage. When a read / write request is received from the system, it routes the request to the specific data shard, calculates the shard key corresponding to the read / write request using a hash function, obtains the data shard containing the target data, and finally returns the data to the user.

[0093] Furthermore, the predicted future usage rate of the data specifically refers to...

[0094] Model definition: In a linear regression model, the model can be represented as: y = θ0 + θ1x. Where θ0 and θ1 are the model parameters, which need to be obtained through training.

[0095] Determine the loss function: The loss function is mainly used to calculate the difference between the predicted value and the label value. Here, the mean squared error loss function is used.

[0096] Solving for parameters: The model parameters are calculated using the gradient descent method. In each iteration, θ0 and θ1 are updated along the direction of gradient descent until the loss function converges.

[0097] Prediction: After the model is trained, it is used to predict the future usage of the data, and the data with high usage is actively cached on high-speed reading devices.

[0098] The regression model first acquires data from data shards. This data has already been processed by the clustering module, with data exhibiting similar characteristics distributed within the same or a few data shards. This improves access speed compared to before processing, whether the regression module reads the data or a query request from the system. After acquiring the data shards, they are used as the training set for subsequent training of the linear regression model. Before training the model, the loss function needs to be determined. Here, the mean squared error loss function is used, which is the mean of the sum of squares of the differences between the predicted values ​​and the label values ​​of all samples. During the model training phase, gradient descent is mainly used to determine the two parameters θ0 and θ1 in the linear regression model. For the data in the system, future usage rates can be predicted using the linear regression model. Data with high access rates is proactively cached in high-speed reading devices, with the specific caching work handled by the data caching module.

[0099] Furthermore, step 4 specifically involves querying cached data.

[0100] Data reception: Receives the high-usage data predicted by the regression module and prepares to store the data in the high-speed reading device;

[0101] Determine the caching device: Select a suitable high-speed read device based on the system's budget and actual situation;

[0102] Establish query logic: When a user initiates a query request, first look for the data in the cache device. If the data is found, return it directly. If the target data is not in the cache, look for the data in the underlying data shard.

[0103] Update cached data: When data changes, the data in the cache needs to be updated in a timely manner to maintain the accuracy of the cached data and its consistency in the system.

[0104] The data caching module stores the data that needs to be cached in the cache device, while the regression module adjusts the model parameters in real time according to system changes, predicts future data access rates, and updates the data in the cache device. During queries, data not in the cache device is not directly cached from the underlying data shards to the high-speed read device; whether to cache the data is determined by the linear regression model of the regression module, avoiding wasted cache space. When the cache device is full, infrequently used data needs to be deleted, and new data with high usage needs to be placed in the cache device. When the data in the data shard storage module changes, the data in the cache also needs to be updated in real time.

[0105] The purpose of this invention is to address the aforementioned problems by providing two machine learning algorithms that implement proactive data sharding and data caching in a distributed query system. For data with similar characteristics, they are stored in the same data shard whenever possible. Furthermore, by predicting future data usage using regression algorithms, frequently used data can be proactively cached on high-speed read devices, thereby improving data access efficiency.

[0106] like Figure 1 As shown, the optimized architecture of the distributed query system consists of four main modules: a clustering module, a data sharding and storage module, a regression module, and a data caching module. The clustering module and the data sharding and storage module exchange data. The clustering module sends information related to the new data sharding key to the data sharding and storage module, helping the data sharding and storage module to partition the new data into data shards with similar characteristics. The data sharding and storage module provides the clustering module with the data it needs for training, helping it train a clustering model that conforms to the system characteristics. The regression module and the data caching module also exchange data with the data sharding and storage module. The regression module treats the data from the data sharding and storage module as its own dataset to train a linear regression model, predict the future usage rate of the data, and retrieves data with high usage rates from the data sharding and storage module. The data caching module receives information from the sharding and storage modules to cache the data. The specific structure of these four modules will be described in detail below.

[0107] The clustering module is further divided into a data collection module, a feature engineering module, a model training module, and a model application module. Generally, training data from the underlying storage system in the data sharding storage module is first transmitted to the clustering module through system coordination. The clustering module uses a clustering algorithm to train the received data, dividing it into several clusters with high similarity within the same cluster and low similarity between different clusters. Then, the trained clustering algorithm can be used to determine which data shard to store new data in, maximizing the similarity of data within the same data shard.

[0108] Upon receiving the data from the data sharding module, the first step is to determine which clustering algorithm to use. Here, the K-means algorithm is used. Then, the number of clusters is determined based on actual needs and the characteristics of the data in the system. Next, a similarity metric is selected based on the data characteristics. This is the core of clustering, determining which cluster the data will ultimately be assigned to. Here, Euclidean distance is chosen as the similarity metric. Euclidean distance refers to the distance between two points in n-dimensional space, and the calculation formula is: After determining the similarity metric, k data points are randomly selected as cluster centers. Here, k is the number of clusters divided based on data characteristics during the algorithm preparation phase, a hyperparameter. The distance between each data point and these cluster centers is then calculated, assigning each data point to a different cluster. Each cluster contains different data points; the average of these data points is used to recalculate the cluster centers, resulting in new cluster centers. This process of cluster division is repeated. Clustering is an iterative process that stops when the cluster distribution no longer changes, the intra-cluster distance is less than a certain threshold, or a pre-set number of iterations is reached, yielding the final clustering result. When a user generates new data, the model application module uses the trained clustering algorithm to determine the sharding key of the new data. Then, a custom hash function is used to hash the sharding key, ultimately locating it in the storage location. This ensures that sharding keys corresponding to data with similar characteristics are within a continuous range, avoiding cross-shard queries based on the principle of spatial locality. The detailed execution flow of the clustering module is as follows: Figure 2 As shown.

[0109] The clustering module may encounter some problems during use. If too few initial cluster centers are selected during the algorithm preparation phase, misclassification of data may occur. If too many initial cluster centers are selected, uneven data distribution may result, with some clusters containing too little data. The solution is to specify the initial number of cluster centers during the algorithm preparation phase based on the characteristics of the system data. Here, the range for the number of cluster centers is specified as: 5 <= k <= 10.

[0110] Closely connected to the clustering module is the data sharding module. In this module, based on the sharding key determined by the clustering module, data within the same cluster are considered to have similar characteristics, thus placing them within the same data range. After receiving the determined sharding key, the newly generated data blocks are divided, and a custom hash function is used to hash the sharding key. Different data are stored on different data shards. MySQL is used as the system's storage engine. Finally, the data shards are stored on the underlying storage device. Different data shards also exchange data, primarily to transmit synchronization information to maintain data consistency. Simultaneously, the data sharding module provides training data to the clustering and regression modules. The data provided to the clustering module can be directly used for feature extraction and training, while the data provided to the regression module needs to be labeled; here, the label represents the data usage rate. Because data with similar characteristics are stored in the same data shard, the order of the data can be appropriately shuffled when used as a training set to avoid negatively impacting model learning.

[0111] Due to the multi-user nature of distributed systems, data is not always available at any given time. To ensure high availability, the data sharding storage module provides a data backup mechanism. Data is stored in one or more data shards, ensuring data availability even when some shards fail or are occupied. First, a backup strategy needs to be determined. Since frequently accessed data is often selected, it should be backed up. A regression module can be used to predict data access rates, determining which data needs to be backed up. After backup, the availability and correctness of the backup data must be verified to ensure it can be used when other data is unavailable. When there are multiple copies of data in the storage device, inconsistency issues arise. The data sharding storage module uses Multi-Version Concurrency Control (MVCC) to achieve data consistency. When multiple transactions access the same data, consistency and isolation are ensured. MVCC maintains multiple versions of data on each data row, each with a timestamp or transaction ID, thus achieving data version isolation during concurrent access. In MVCC, each transaction reads a consistent view from when it starts up, i.e., the state of the data in the database at startup, unaffected by other transactions. When a transaction commits, the system checks whether it can be successfully committed, including whether the data read by the transaction has been modified by other transactions, and handles this accordingly to ensure data consistency.

[0112] The data sharding storage module and the clustering module complement each other. Data sharding storage relies on the clustering module to determine the range of sharding keys, while the training of the clustering algorithm relies on the data sharding storage module to provide training data. A detailed structure of the data sharding storage module is as follows... Figure 3 As shown.

[0113] The regression module is mainly divided into four parts: a data receiving module, a data processing module, a usage prediction module, and a usage ranking module. First, the regression module receives data from the data sharding and storage module as its training set, similar to the clustering module, except that the data now includes a usage rate label. After receiving the data from the underlying data sharding and storage module, the data receiving module performs preprocessing and feature extraction. Appropriate features are selected for the regression algorithm to predict usage rate; here, data source and data attributes, such as image resolution and creation time, are mainly used as training criteria. A linear regression model is trained using the preprocessed data. After training, the model's accuracy is verified using a validation set. If the model's loss meets the requirements, the obtained regression model can be used to predict the future access rate of each data point. The goal is to use an algorithm to predict future data access rates and then proactively cache high-access-rate data on high-speed read devices. Once the data usage rate is predicted, the usage rate sorting module within the regression module sorts the data in descending order of usage rate. Considering the massive data volume of distributed databases, an external sorting algorithm is chosen. The data to be sorted is loaded into memory in batches for sorting, and the final sorted result is the data's shard key value, facilitating the retrieval of high-access-rate data. Depending on the system budget, only a portion of the high-access-rate data is cached. The specific execution flow of the algorithm in the regression module is as follows... Figure 4 As shown.

[0114] The regression module exchanges data not only with the data sharding storage module but also with the caching module. After the usage ranking module in the regression module obtains the data usage ranking results, it retrieves the high-usage data from the storage device in the data sharding storage module and passes it to the caching module. Locating high-usage data requires a specific method; here, a hash function is used to calculate the data's shard key and locate its storage position. The custom hash function used here is the same as the hash function used in the clustering module; a suitable hash function can be used depending on the specific characteristics of the system. After hashing the data's shard key, the specific storage location of the data can be obtained. When the caching module obtains the data that needs to be cached, it stores it in the high-speed cache device.

[0115] The caching module consists of a cache control center and cache devices. The cache control center is responsible for scheduling data passed from the regression module, while the cache devices store the data. When the cache control center receives data, it first allocates a cache area in the cache device to store the new data. When the cache device is full, low-usage data needs to be deleted, and new high-usage data needs to be added. The low-usage data information can be obtained from the usage sorting module in the regression module, allowing for correct deletion and saving cache space. When the data in the storage devices of the data sharding storage module changes, the data in the cache also needs to be updated in real time. When the underlying data changes, if the data is in the cache, the cache data is updated synchronously. When a query request is generated in the distributed query system, it first requests data from the data cache module. If the corresponding data is in the cache module, the result can be returned directly, saving query time. If the requested data is not in the cache, the query is performed on the storage devices in the data sharding storage module. During the query, the shard key of the target data in the query statement needs to be parsed. For precise queries, a custom hash function can be used to locate the target data. Since the data stored in data shards has similar characteristics and their shard keys fall within a continuous range, a shard key range attribute can be added to the data shards. For range queries from the system, the shard key range can be used to locate the shard containing the data, thus returning the final result. The process for caching data and its query is as follows... Figure 5 , Figure 6 As shown.

[0116] After a query request is generated in the system, a series of processes are required. First, the SQL statement in the request needs to be parsed. After parsing, the SQL statement needs to be validated; invalid SQL statements will not be executed. Valid SQL statements undergo logical and physical optimization to speed up execution. Logical optimization mainly optimizes based on some pre-defined logical rules to form an optimized logical execution plan. Different logical units in the logical plan can form multiple physical units, meaning one logical execution plan can generate different physical execution plans. Physical optimization estimates the cost of the physical query plan and selects the plan with the lowest execution cost. Estimating the execution cost requires statistical information provided by the plan executor. After optimization, the query plan can be executed. If the data is in the cache, it can be queried directly from the cache. If the target data is not in the cache, the query request will be passed to the underlying query engine, where it will be queried in the underlying data shards. Ultimately, the query request will be executed in a distributed manner, returning the final result. The system architecture for the query function is as follows: Figure 7 As shown.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed query system based on machine learning, characterized in that, The distributed query system includes a clustering module, a data sharding and storage module, a regression module, and a data caching module; The clustering module is used to analyze newly generated data in the system and obtain clustering results; The data sharding storage module is used to store data into the corresponding data shards according to the received sharding key range, and similar data are allocated to the same or several data shards; The regression module is used in conjunction with the data caching module to perform active data caching, predict the future usage rate of data, and actively cache data with high usage rate to high-speed reading devices. The data caching module is used to sort the predicted future access rates of the data, obtain the sharding keys corresponding to the data with high usage rates, then read the corresponding data from the data sharding storage module, and pass this data to the data caching module. The data caching module stores the received data on the high-speed reading device. The clustering module includes a data collection module, a feature engineering module, a model training module, and a model application module; The data collection module is used to collect data about distributed queries; The feature engineering module is used to select valid features and process and transform them; The model training module is used to train a clustering model using the collected data; The model application module is used to optimize queries in a distributed query system using a trained model.

2. The distributed query system based on machine learning according to claim 1, characterized in that, The data sharding storage module includes disk blocks and a data backup module; The disk block contains several shard keys. The shard keys in the disk block store the clustering results generated by the clustering module and are stored in shards according to data with similar data characteristics, in order to distinguish the shard to which the data belongs. The data backup module is used to ensure high data availability, and new data will be backed up to different data shards in the data backup module.

3. The distributed query system based on machine learning according to claim 1, characterized in that, The regression module includes a data processing module, a data receiving module, a usage rate prediction module, and a usage rate ranking module; The data receiving module is used to receive data from the data fragmentation storage module; The data processing module is used to process the data received by the data receiving module; The utilization rate prediction module is used to predict the utilization rate of the processed data and send the result to the utilization rate sorting module; The usage rate sorting module is used to cache data with high access rates on high-speed reading devices, which can avoid system crashes and failures caused by a large number of data read and write operations. With a large number of read and write operations concentrated on high-speed reading devices, the system's operating pressure is reduced.

4. An optimization method for a distributed query system based on machine learning, characterized in that, The optimization method uses a machine learning-based distributed query system as described in any one of claims 1-3, and the optimization method includes the following steps: Step 1: Use the clustering module to collect user data and pass information involving new data shard key values ​​to the data shard storage module; Step 2: The data sharding and storage module provides the clustering module with the data required for training the clustering model; Step 3: The regression module uses the data in the data sharding storage module to train a linear regression model, predicts the future usage rate of the data, and reads the data with high usage rate from the data sharding storage module. Step 4: Store the data with high usage frequency from Step 3 into the data cache module; Step 5: When querying data through the distributed query system, determine whether the data is frequently used. If the data is frequently used, retrieve it through the data caching module; if the data is infrequently used, retrieve it through the data sharding storage module. The data distribution of the data sharding storage module is specifically as follows: The clustering module analyzes newly generated data in the system to obtain clustering results. It then passes the clustering results and the range of shard keys corresponding to data shards with similar characteristics to the new data to the data shard storage module. The module stores the received shard key range into the corresponding data shards, with similar data allocated to the same or several data shards. The module determines the location of the data shard based on the range of the shard keys and stores the new data in the shard. The new data is also backed up to different data shards, with the backup shard containing data with similar characteristics to the backup. Upon receiving a read / write request from the system, the module routes the request to the specific data shard, calculates the shard key corresponding to the read / write request using a hash function, obtains the data shard containing the target data, and finally returns the data to the user.

5. The optimization method for a machine learning-based distributed query system according to claim 4, characterized in that, The training of the clustering model specifically involves, First, data from the distributed system is collected to prepare for feature engineering; then, the features of the data are extracted during the feature engineering phase. The model training phase involves analyzing the characteristics of the data, randomly selecting K points from the data as cluster centers, and assigning the data in the dataset to the cluster containing the nearest cluster center. Here, the dataset consists of system data and user data from the distributed system. Then, the average value of all data points in a cluster is calculated to determine the new cluster center. This process is iterated until the cluster centers no longer change or the maximum number of iterations is reached, at which point the model training is complete.

6. The optimization method for a machine learning-based distributed query system according to claim 4, characterized in that, The predicted future usage rate of the data is specifically as follows: Model definition: In a linear regression model, the model can be represented as: ,in , These are the parameters of the model, which need to be obtained through training; Determine the loss function: The loss function is mainly used to calculate the difference between the predicted value and the label value. Here, the mean squared error loss function is used. Solving for parameters: The model parameters are calculated using gradient descent. In each iteration, the parameters are updated along the direction of gradient descent. , , until the loss function converges; Prediction: After the model is trained, it is used to predict the future usage of the data, and the data with high usage is actively cached on high-speed reading devices.

7. The optimization method for a machine learning-based distributed query system according to claim 4, characterized in that, Step 4 specifically involves querying cached data; Data reception: Receives the high-usage data predicted by the regression module and prepares to store the data in the high-speed reading device; Determine the caching device: Select a suitable high-speed read device based on the system's budget and actual situation; Establish query logic: When a user initiates a query request, first look for the data in the cache device. If the data is found, return it directly. If the target data is not in the cache, look for the data in the underlying data shard. Update cached data: When data changes, the data in the cache needs to be updated in a timely manner to maintain the accuracy of the cached data and its consistency in the system.

Images