Machine learning powered query processing tier of database and optimising query processing tier of database
The machine learning-powered query processing tier addresses inefficiencies in handling large data volumes by loading shards into RAM, inferring schema, and optimizing query plans, resulting in improved performance and resource balancing.
Patent Information
- Application Number
- PCT/IB2023/057564
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-07-26
- Publication Date
- 2025-01-30
AI Technical Summary
Existing database query processing systems face inefficiencies in handling large volumes of data from sources like IoT devices, log files, and web content, particularly in terms of query performance and data schema inference.
Implementing a machine learning-powered query processing tier that loads data shards into physical RAM across multiple query processing nodes, using adaptive data sampling to infer schema and optimize query plans, while aligning Query Processing Tier code with CPU architecture and leveraging SIMD instructions and AVX for improved performance.
This approach enables efficient parallel processing of large datasets without data transformation, improves query performance by predicting execution times and optimizing plans, and balances system resources for enhanced overall database performance.
Description
[0001] Machine Learning Powered Query Processing Tier of Database and Optimising Query Processing Tier of Database
[0002] Massive amounts of data like loT (Internet of Things) data, log files, web content, etc. which are stored in files or object stores can be loaded in parallel as different shards into the physical RAM (Random Access Memory) of different query processing nodes to perform query processing without any data transformation to persist the data in a traditional relational database. Shard size depends upon the instance type or class of the query processing nodes, and we can use adaptive data sampling coupled with machine learning which uses a few data samples from a large data set to infer the schema for the data or tables being loaded from files like Comma-Separated Values, etc (Note the above is not applicable for purely unstructured data). In another illustrative embodiment the query processing nodes might be the instances themselves where the shards are or where data of files or object stores reside. In the above scenario we don’t require loading of data and thus gives better query performance. Further the database engine tier uses machine learning to predict query execution time in addition to statistics to improve and better the query plans with every execution. The above is extremely useful and powerful where there are overlapping query constructs. Also, the Query Processing Tier code is aligned to the CPU (Central Processing Unit) architecture to balance different system resources like Compute, IO, Memory bandwidth and cache lookups between different query processing cores or threads. Further many query processing primitives like bit vector copy and column decompression leverage SIMD (Single Instruction Multiple Data) instructions. Query constructs like bloom filters are tuned to the size of the cache size of the query processing node or system. Further AVX (Advanced Vector Extensions) instructions are used for bloom filter lookups and hash computations as part of query processing for better performance.
Claims
Following are the claims for this invention1. Massive amounts of data like loT (Internet of Things) data, log files, web content, etc. which are stored in files or object stores can be loaded in parallel as different shards into the physical RAM (Random Access Memory) of different query processing nodes to perform query processing without any data transformation to persist the data in a traditional relational database.
2. Shard size depends upon the instance type or class of the query processing nodes, and we can use adaptive data sampling coupled with machine learning which uses a few data samples from a large data set to infer the schema for the data or tables being loaded from files like Comma-Separated Values, etc (Note the above is not applicable for purely unstructured data).
3. In another illustrative embodiment the query processing nodes might be the instances themselves where the shards are or where data of files or object stores reside. In the above scenario we don’t require loading of data and thus gives better query performance.
4. Further the database engine tier uses machine learning to predict query execution time in addition to statistics to improve and better the query plans with every execution. The above is extremely useful and powerful where there are overlapping query constructs.
5. Also, the Query Processing Tier code is aligned to the CPU (Central Processing Unit) architecture to balance different system resources like Compute, IO, Memory bandwidth and cache lookups between different query processing cores or threads.
6. Further many query processing primitives like bit vector copy and column decompression leverage SIMD (Single Instruction Multiple Data) instructions.
7. Query constructs like bloom filters are tuned to the size of the cache size of the query processing node or system.
8. Further AVX (Advanced Vector Extensions) instructions are used for bloom filter lookups and hash computations as part of query processing for better performance.