A compressed sensing streaming SQL query method, system, device and medium for a heterogeneous hybrid computing architecture

Abstract

This invention relates to a compressed-aware streaming SQL query method, system, device, and medium for heterogeneous hybrid computing architectures. The method includes: dynamically selecting the optimal compression encoding scheme for the input raw streaming data using an adaptive compression selection strategy, and processing the data in real time to obtain compressed data in the corresponding compressed encoding format; redesigning the basic SQL operators involved in the query analysis operation to obtain compressed-aware operators; decomposing the received SQL query statement into a series of compressed-aware operators, and dynamically making the optimal query execution decision based on the computational characteristics of each compressed-aware operator and the heterogeneous hardware status; and directly applying the compressed-aware operators to the compressed data to obtain the query results. This invention can be widely applied in the fields of database systems and data processing technology.

Description

Technical Field

[0001] This invention belongs to the field of database systems and data processing technology, and relates to an adaptive query and execution optimization technology that can directly execute operators on compressed data formats. In particular, it relates to a compression-aware streaming SQL query method, system, device and medium for heterogeneous hybrid computing architectures (including CPU, GPU and edge computing nodes, etc.). Background Technology

[0002] With the rapid development of data-intensive applications such as the Internet of Things (IoT), fintech, and intelligent monitoring, massive amounts of high-speed streaming data are continuously generated in cloud and edge environments. This poses unprecedented challenges to the processing capabilities of real-time query and analysis systems, especially in achieving an effective balance between performance, energy efficiency, and cost. Traditional streaming database systems typically follow a "decompress first, query later" processing paradigm when dealing with such data. This means that even if the data is stored or transmitted in compressed form, it must be completely decompressed before query execution. This process not only introduces significant data movement overhead and redundant decoding computations but also significantly increases memory bandwidth pressure, leading to limited system throughput and increased query latency, making it difficult to meet the stringent requirements of modern applications for high-concurrency, low-latency online analysis.

[0003] On the other hand, to enhance processing power, heterogeneous computing architectures (such as CPUs, GPUs, and emerging edge computing units working together) have become the mainstream platform for high-performance stream processing. However, existing systems on such platforms are often designed in a "separate" manner. The CPU side is typically responsible for control-intensive tasks such as query parsing, task management, and scheduling, while the GPU side focuses on executing computationally intensive operator acceleration. This separation of responsibilities results in the system lacking a unified data semantic layer and task scheduling mechanism throughout the entire process. Specifically, the compression features of data cannot be effectively perceived and utilized in various stages such as syntax analysis, query optimization, execution plan generation, and underlying operator implementation. As a result, there is a lack of deep collaboration between computation and storage. The storage and transmission advantages brought by data compression are greatly reduced during the computation stage due to architectural barriers, and new overhead may even be generated due to data format conversion between devices.

[0004] The academic and industrial communities have recognized the aforementioned problems and proposed the concept of "compression-domain computation," aiming to bypass the decompression step and directly perform query operations on compressed data. For example, in columnar storage databases, research has implemented aggregation and filtering operators based on specific encodings such as RLE encoding or bitpacking. However, these methods have significant limitations: First, they are usually limited to one or a few fixed compression algorithms, lacking a general operator framework compatible with multiple lightweight encoding formats; second, they support a limited range of operation types, mostly focusing on simple aggregation and selection, with insufficient or even complete support for complex operators such as joins, grouping, and sorting, or in complex query scenarios involving multiple operator combinations; finally, these solutions are often designed for homogeneous computing environments, failing to fully consider how to dynamically and efficiently schedule these compression-aware operators on heterogeneous devices such as CPUs and GPUs.

[0005] Furthermore, the selection strategies for compression algorithms in existing systems are often static or semi-static, failing to adaptively optimize based on the dynamic characteristics of streaming data, real-time fluctuations in network bandwidth, and the different computing characteristics of heterogeneous hardware. A compression algorithm that performs well under a certain data distribution and hardware configuration may quickly become ineffective after the scenario changes, resulting in low system resource utilization and drastic fluctuations in overall performance. Summary of the Invention

[0006] To address the problems of computation-storage separation, static selection of compression algorithms, and high data decompression overhead in existing streaming database systems under compressed storage and heterogeneous execution environments, the present invention aims to provide a compression-aware streaming SQL query method, system, device, and medium for heterogeneous hybrid computing architectures. This method can directly execute complete streaming SQL queries on diverse compression formats and deeply integrates the characteristics of heterogeneous computing architectures to achieve adaptive compression strategy selection and multi-level fine-grained task scheduling. This significantly improves the overall throughput, energy efficiency, and resource utilization of the system while ensuring the semantic correctness of queries.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a compressed-aware streaming SQL query method for heterogeneous hybrid computing architectures, comprising: An adaptive compression selection strategy is adopted to dynamically select the optimal compression encoding scheme for the input raw stream data, and the compressed data of the corresponding compression encoding format is obtained in real time. The basic SQL operators involved in query analysis operations were redesigned to obtain compressed sensing operators; The received SQL query statement is decomposed into a series of compressed sensing operators, and the optimal query execution decision is dynamically made based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status. The compressed sensing operators are then directly applied to the compressed data to obtain the query results.

[0008] Furthermore, the step of employing an adaptive compression selection strategy to dynamically select the optimal compression encoding scheme for the input raw stream data and processing it in real time to obtain compressed data in the corresponding compression encoding format includes: Multi-dimensional statistical features are extracted from each column of the input raw streaming data and constructed as a feature vector; The feature vector is input into a lightweight neural network model trained offline, and the optimal compression encoding scheme is dynamically selected for the input raw streaming data. The original streaming data is compressed using the optimal compression coding scheme to obtain compressed data.

[0009] Furthermore, the multi-dimensional statistical feature extraction for each column of the input raw stream data includes the extraction of: Basic statistical characteristics, including minimum, maximum, and mean; Data patterns include data repetition, data variability, number of distinct values, data order, and data disorder. And real-time network conditions.

[0010] Furthermore, the input of the lightweight neural network model is the feature vector of the original streaming data. The input feature vector is trained and inferred according to a comprehensive scoring formula, and the output is a nine-dimensional scoring vector, which corresponds to the nine lightweight compression algorithms integrated in the system. The lightweight compression algorithms include EG encoding, ED encoding, NS encoding, NSV encoding, RLE encoding, BD encoding, DICT encoding, BM encoding, and PLWAH encoding. The comprehensive scoring formula is as follows:

[0011] in, It is an algorithm For data columns The rating, It's the compression ratio. and These are compression and decompression times, respectively. It's network bandwidth. It is an adjustable parameter used to balance the weight of compression and decompression overhead.

[0012] Furthermore, the basic SQL operators involved in the query analysis operation are redesigned to obtain compressed sensing operators, including: When selecting an operation operator, for compressed data using EG, ED, and NS encoding formats, direct equality and range comparisons are performed; for compressed data using RLE encoding format, the numerical parts are directly compared and the run length is used to maintain the count, avoiding data expansion; for compressed data using BD encoding format, predicate pushdown is achieved by adjusting the base value and increment. The projection operator can directly forward the required attributes on compressed data in nine lightweight compression coding formats. The join operation operator directly compares compressed data using EG, ED, and NS encoding formats; compares numerical parts and maintains length using run length for compressed data using RLE encoding format; performs additional re-encoding steps to align data with different bases for compressed data using BD encoding format; and supports direct join when sharing the same dictionary for data using DICT and BM encoding formats, otherwise dictionary mapping or re-encoding is required. The grouping operator processes key-value pairs directly on the bit width of compressed data using EG, ED, NS, and BD encoding formats; for compressed data using RLE encoding format, it groups the numerical part and aggregates the counts on the run length; for compressed data using DICT and BM encoding formats, since a one-to-one mapping between the original values ​​and encoded values ​​is maintained, the equivalent grouping operation is performed directly on the compressed data. The sorting operator performs direct comparison and sorting on compressed data using EG, ED, and NS encoding formats; on compressed data using RLE encoding format, it sorts by numerical part and preserves run length for expansion during output; on compressed data using BD encoding format, it directly compares within the compressed domain after adjusting the increment based on the baseline value; on compressed data using DICT and BM encoding formats, it sorts directly if dictionary order is preserved, otherwise it requires lightweight remapping before sorting. The aggregation operators directly perform summation, counting, and maximum / minimum value calculations on compressed data using EG, ED, NS, RLE, and BD encoding formats. For RLE encoded compressed data, the run length is considered during calculation, while for BD encoded compressed data, the result is obtained by accumulating the increment and adjusting it according to the baseline value. Compressed data using DICT and BM encoding only directly supports counting and grouping aggregation; numerical aggregation usually requires decoding.

[0013] Furthermore, the process of decomposing the received SQL query statement into a series of compressed sensing operators, and dynamically making the optimal query execution decision based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status, and directly applying the compressed sensing operators to the compressed data to obtain the query result, includes: The system architecture level is selected by using a predefined comprehensive weighted scoring model to choose the most suitable execution platform for the current SQL query statement. The received SQL query statement is decomposed into a series of compressed sensing operators, and a greedy queue scheduling strategy is used to allocate resources to each compressed sensing operator on a determined execution platform according to the computational characteristics of each operator. The compressed sensing operator is directly applied to the compressed data to obtain the query results.

[0014] Furthermore, the process of decomposing the received SQL query statement into a series of compressed sensing operators and allocating resources for each compressed sensing operator on a determined execution platform using a greedy queue scheduling strategy based on the computational characteristics of each operator includes: The SQL query statement is decomposed into multiple compressed sensing operators, and device preferences are marked for each compressed sensing operator based on its computational characteristics and implementation method. Task queues are maintained for CPU and GPU respectively, and a greedy queue scheduling strategy is adopted to allocate resources to each compressed sensing operator on the execution platform based on the device preference.

[0015] Secondly, the present invention provides a compressed-aware streaming SQL query system for a heterogeneous hybrid computing architecture, comprising: The data acquisition unit is configured to dynamically select the optimal compression encoding scheme for the input raw stream data using an adaptive compression selection strategy, and process it in real time to obtain compressed data in the corresponding compression encoding format. The query processor is configured to redesign the underlying SQL operators involved in query analysis operations to obtain compressed sensing operators; The hierarchical multi-level scheduler is configured to decompose the received SQL query statement into a series of compressed sensing operators, and dynamically make the optimal query execution decision based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status, and apply the compressed sensing operators to the compressed data to obtain the query results.

[0016] Thirdly, the present invention provides a computer-readable storage medium for storing one or more programs, said one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any method.

[0017] Fourthly, the present invention provides a computing device comprising: one or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing any method.

[0018] The present invention has the following advantages due to the adoption of the above technical solutions: 1. This invention significantly reduces data transmission and decoding overhead by directly executing compression sensing operators on compressed data without requiring full decompression; 2. This invention utilizes a neural network-driven compression algorithm selector to dynamically select the optimal compression algorithm based on data statistical characteristics and network conditions, thereby achieving adaptive fusion of compression and computation. 3. This invention adopts a hierarchical multi-level scheduling mechanism to achieve dynamic task migration and load balancing among CPU, GPU, and edge devices, giving full play to the advantages of heterogeneous architecture; 4. This invention supports multiple lightweight compression algorithms and scalable encoding formats, has strong compatibility, and is easy to deploy; 5. This invention maintains significant performance advantages in both high-bandwidth and low-bandwidth environments, and greatly improves overall throughput and energy efficiency.

[0019] Therefore, this invention can be widely applied in the fields of database systems and data processing technology. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 This is a schematic diagram of the compressed sensing streaming SQL query system structure for heterogeneous hybrid computing architecture provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the query decomposition and operator allocation technology provided in the embodiments of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0022] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0023] Streaming data applications are now widely used in scenarios such as financial risk control, industrial IoT, real-time monitoring, and smart manufacturing. These scenarios share the common characteristics of continuous, high-speed, and massive data generation, requiring systems to perform low-latency real-time queries and analyses to support immediate decision-making. However, traditional streaming database systems face significant "computation and storage separation" problems when dealing with such demands, particularly in compressed data management and heterogeneous execution environments. Specifically, on the one hand, data compression mechanisms introduced to effectively save valuable network bandwidth and storage space create a processing bottleneck in traditional architectures: the system must completely decompress the data before executing queries. This process not only consumes a large number of CPU cycles but also introduces high data migration overhead, causing the transmission and storage advantages brought by compression to be partially offset by the additional latency during the query phase. On the other hand, modern computing platforms generally adopt a collaborative mode of heterogeneous computing units such as CPUs, GPUs, and even dedicated edge devices, but these units lack a unified data semantic interface that can understand compressed formats. This disconnect means that data compression information cannot be perceived or utilized in critical stages such as query optimization and execution plan generation. Compression and computation become two independent, even conflicting, processes, failing to create a synergistic effect.

[0024] After in-depth analysis, the technical root of the above problems lies in the lack of a unified expression and transmission mechanism for "compression semantics" throughout the data lifecycle. Specifically, this manifests as three "breaks": (1) At the data acquisition end, compression decisions are disconnected from data characteristics and query targets, making it impossible to provide the optimal encoding format for subsequent calculations; (2) At the execution engine layer, operator logic is tightly coupled with the compression encoding format, lacking a universal computing interface that can understand multiple compression formats; (3) At the resource management layer, the task scheduler cannot perceive the changes in data volume and differences in computational characteristics brought about by compression, making it difficult for it to make the optimal allocation decision on heterogeneous hardware.

[0025] To systematically address the aforementioned problems, some embodiments of this invention propose a compressed-sensory streaming SQL query method, system, device, and medium for heterogeneous hybrid computing architectures. Based on the core concept of "compressed-sensory computing," an end-to-end compressed collaborative processing pipeline is constructed, encompassing data acquisition, compression encoding, network transmission, and query execution. Its overall architecture and core methods mainly include the following three interrelated aspects: First, a feature extraction and adaptive compression selection module deployed at the data acquisition end performs real-time and intelligent processing on the input raw streaming data. This feature extraction and adaptive compression selection module can automatically analyze the statistical characteristics and patterns of the data, and, combined with real-time network bandwidth conditions, dynamically select the optimal one from a variety of lightweight compression algorithms (such as RLE, BD, DICT, etc.) through a pre-trained neural network model, thereby minimizing compression and transmission latency while ensuring a high compression ratio.

[0026] Secondly, on the query processing side, an innovative compression domain operator execution mechanism has been implemented. By redesigning core SQL operators such as filtering, projection, aggregation, and grouping, it has been given the ability to directly process compressed encoded data. This means that the query engine can complete most of the computational tasks on compressed data without restoring the data to its original format, fundamentally avoiding the performance loss caused by full decompression.

[0027] Finally, to efficiently execute the compressed sensing queries described above in complex heterogeneous hardware environments, a hierarchical multi-level scheduling method is introduced. This method allocates suitable computing platforms (such as discrete GPUs, integrated architectures, or edge nodes) to tasks at the macro-level query layer, and dynamically allocates tasks to CPUs or GPUs for execution at the micro-level operator layer based on operator characteristics and real-time device load, thereby achieving deep collaboration and real-time optimization of heterogeneous resources.

[0028] In summary, the technical solution of this invention significantly reduces the overhead of decompression and data transmission by deeply integrating compression sensing capabilities throughout the entire data processing chain, thereby effectively improving system throughput and energy efficiency ratio. It is particularly suitable for real-time analysis environments of high-concurrency, low-latency compressed streaming data.

[0029] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0030] Example 1 like Figure 1As shown, this invention provides a compressed-aware streaming SQL query method for heterogeneous hybrid computing architectures, which includes the following steps: 1) An adaptive compression selection strategy is adopted to dynamically select the optimal compression encoding scheme for the input raw stream data, and the compressed data of the corresponding compression encoding format is obtained in real time. 2) The basic SQL operators involved in query analysis operations are redesigned to obtain compressed sensing operators; 3) The received SQL query statement is decomposed into a series of compressed sensing operators, and the optimal query execution decision is dynamically made based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status. The compressed sensing operators are then directly applied to the compressed data to obtain the query results.

[0031] Furthermore, in step 1) above, the adaptive compression selection strategy is implemented based on multi-dimensional feature extraction and a lightweight neural network, and is used to perform real-time analysis and compression strategy decision-making on the input raw streaming data.

[0032] Specifically, it includes the following steps: 1.1) Perform real-time, low-overhead multi-dimensional statistical feature extraction on each column of the input raw streaming data and construct a feature vector; 1.2) Input the feature vector into a lightweight neural network model (e.g., MLP) trained offline, and dynamically select the optimal compression coding scheme for the input raw streaming data; 1.3) The original stream data is compressed using the optimal compression coding scheme to obtain compressed data.

[0033] Furthermore, in step 1.1) above, multi-angle feature extraction is performed on each column of the input raw stream data in order to automatically determine the optimal compression scheme based on the statistical characteristics of the raw stream data.

[0034] Specifically, this embodiment extracts thirteen categories of multi-dimensional statistical features, covering: basic statistical features, including minimum, maximum, and mean values; data patterns, including data repeatability (e.g., average run length), data variability (e.g., average increment, variance), number of distinct values, data orderliness (e.g., Kendall's τ, Spearman's ρ, absolute Kendall's τ), and data disorder (e.g., Shannon entropy, Gini impurity); and real-time network conditions (e.g., network bandwidth). Through these statistical features, the system can comprehensively characterize the data compression potential and algorithm adaptability.

[0035] Furthermore, in step 1.2 above, the lightweight neural network model established in this embodiment can learn the complex nonlinear relationship between data features, network conditions and the performance of different compression algorithms, and output a score vector for nine built-in lightweight compression algorithms. Based on this score vector, the optimal compression coding scheme is dynamically selected, realizing the leap from "static configuration" to "data-driven".

[0036] Specifically, the lightweight neural network model has a three-layer structure, containing 256, 128, and 64 neurons respectively. It uses the ReLU activation function, and its output is a nine-dimensional score vector, which corresponds to nine lightweight compression algorithms integrated in the system, including EG (Elias Gamma) encoding, ED (Elias Delta) encoding, NS (Null Suppression with Fixed Length) encoding, NSV (Null Suppression with Variable Length) encoding, RLE (RunLength Encoding) encoding, BD (Base Delta) encoding, DICT (Dictionary) encoding, BM (Bitmap) encoding, PLWAH (Position list word aligned hybrid) encoding, etc. (This invention only introduces this as an example, but is not limited to it).

[0037] The lightweight neural network model is trained and inferred based on a comprehensive scoring formula that uniformly considers the impact of compression ratio, compression time, decompression time, and network bandwidth, thereby achieving an optimal trade-off in end-to-end performance. The comprehensive scoring formula is defined as follows:

[0038] in, It is an algorithm For data columns The rating, It's the compression ratio. and These are compression and decompression times, respectively. It's network bandwidth. It is an adjustable parameter used to balance the weight of compression and decompression overhead. The innovation of this formula lies in unifying compression efficiency, computational overhead, and transmission latency into a single metric, ensuring optimal end-to-end performance from data acquisition to query result return.

[0039] Ultimately, the compression algorithm with the highest overall score was selected to compress the corresponding data column. This design enables the system to dynamically adapt to the time-varying characteristics of streaming data and the volatility of network conditions, significantly reducing processing latency caused by mismatch in fixed compression strategies while maintaining a high compression ratio.

[0040] Furthermore, in step 2) above, in order to realize the native computing capability for compressed data, this embodiment constructs a complete compressed domain query execution model. By redesigning the traditional SQL operator, it enables it to perceive and process multiple compressed encoding formats.

[0041] In this embodiment, the six basic SQL operators in query analysis and processing have been redesigned, including selection, projection, join, grouping, sorting and aggregation. The equivalent operation semantics of each basic SQL operator in various compressed encoding formats have been defined to form compression-aware operators, enabling them to be calculated in the compressed domain and avoiding the additional overhead caused by decompression.

[0042] Specifically, the compressed sensing operator in this embodiment includes: When selecting an operator, since EG encoding, ED encoding, and NS encoding preserve the original numerical values ​​and data layout, equality and range comparisons can be performed directly on the compressed data. For compressed data using RLE encoding, data expansion can be avoided by directly comparing the numerical parts and maintaining the count using the run length. For compressed data using BD encoding, predicate pushdown is achieved by adjusting the base value and increment.

[0043] The projection operation operator supports direct forwarding of the required attributes for compressed data in the nine lightweight compression encoding formats built into the system. The join operation operator supports direct joins for compressed data using EG, ED, and NS encoding formats because the values ​​can be directly compared. For compressed data using RLE encoding format, it is achieved by comparing the numerical parts and maintaining the length using run length. For compressed data using BD encoding format, an additional re-encoding step is required to align data with different bases. For compressed data using DICT and BM encoding formats, direct joins are supported when the same dictionary is shared; otherwise, dictionary mapping or re-encoding is required.

[0044] The grouping operator can directly process key-value pairs on the bit width of compressed data using EG, ED, NS, and BD encoding formats; on compressed data using RLE encoding format, it can group the data using the numerical part and aggregate the counts on the run length; on compressed data using DICT and BM encoding formats, since the compressed data maintains a one-to-one mapping between the original values ​​and the encoded values, equivalent grouping operations can be performed directly on the encoded data.

[0045] For compressed data using EG, ED, and NS encoding formats, the sorting operator can directly identify the encoding format and compare and sort without decompression. For compressed data using RLE encoding format, sorting is performed by numerical part, and run length is preserved for expansion during output. For compressed data using BD encoding format, the incremental value is adjusted by the baseline value and then directly compared within the compressed domain. For compressed data using DICT and BM encoding formats, sorting can be performed directly if dictionary order is preserved; otherwise, lightweight remapping is required.

[0046] The aggregation operators directly support summation, counting, and maximum / minimum value calculations on compressed data using EG, ED, NS, RLE, and BD encoding formats. However, for RLE-encoded compressed data, the run length needs to be considered during calculation. For BD-encoded compressed data, the result is obtained by accumulating the increment and adjusting it according to the baseline value. For DICT and BM-encoded compressed data, only counting and group aggregation are directly supported; numerical aggregations (such as summation) typically require decoding.

[0047] In this embodiment, the applicable boundaries of each compression-aware operator are clearly defined: for compression formats such as NSV and PLWAH that do not support direct operation, the system will fall back to the path of querying after decompression, but can still benefit from the compression gain during the transmission stage. Through this mechanism, the data is kept compressed throughout the entire query process, which not only eliminates the computational overhead caused by decompression, but also greatly reduces the number of memory accesses and the amount of data movement within the CPU / GPU, thereby releasing greater memory bandwidth and computational potential at the underlying level, which is especially suitable for data-intensive stream processing scenarios.

[0048] Furthermore, in step 3) above, in order to achieve efficient utilization of heterogeneous hardware resources, this invention proposes a hierarchical multi-level scheduling strategy for heterogeneous hybrid architectures, which is used to perform fine-grained resource allocation and execution optimization for compressed stream SQL query tasks on heterogeneous hybrid computing platforms that include discrete GPUs, integrated CPU-GPUs and edge GPUs, so as to efficiently execute compressed query tasks.

[0049] Specifically, this hierarchical multi-level scheduling method separates the query layer from the operator layer, and includes the following steps: 3.1) Select the most suitable execution platform for the current SQL query statement by using a predefined comprehensive weighted scoring model at the architecture level; The comprehensive weighted scoring model takes into account the indicators of different architecture platforms in three dimensions: computing power, energy consumption and hardware cost. It also allows users to flexibly specify optimization objectives (such as peak performance, energy efficiency priority or cost control) by configuring the weight factors corresponding to the three indicators. 3.2) The received SQL query statement is decomposed into a series of basic compressed sensing operators, and a greedy queue scheduling strategy is used to allocate resources to each compressed sensing operator on a determined execution platform according to the computational characteristics of each operator.

[0050] Furthermore, step 3.2 above includes the following steps: 3.2.1) Decompose the SQL query statement into multiple basic compressed sensing operators, and label each compressed sensing operator with device preference (CPU preference or GPU preference) according to its computational characteristics and implementation method. 3.2.2) Maintain task queues for CPU and GPU respectively, and allocate resources to each compressed sensing operator on the execution platform according to the preset allocation rules and the device preferences of each compressed sensing operator; 3.2.3) Apply the compressed sensing operator to the compressed data and return the query results.

[0051] Specifically, such as Figure 1 As shown, when the system receives an SQL query, the decomposition module breaks it down into three basic compressed sensing operators: "projection," "mean (aggregation)," and "grouping." Then, the system assigns a device preference (CPU preference or GPU preference) to each compressed sensing operator based on its computational characteristics. For example, the "projection" and "grouping" operators, due to their involvement in parallel processing of large-scale data, are labeled as GPU-preferred; while the "mean" operator, which may involve complex logic control and data distribution, is labeled as CPU-preferred in certain scenarios.

[0052] A lightweight greedy scheduler begins operation, maintaining separate task queues for the CPU and GPU, and adhering to an efficient allocation rule: compressed sensing operators are preferentially allocated to their preferred devices; if the preferred device is busy, they are dynamically allocated to an idle device to avoid blocking. This scheduling mechanism ensures that compressed sensing operators are promptly allocated to available computing resources, maintaining overall system progress even when executing on non-preferred devices. The compressed sensing operators ultimately execute on their respective CPU or GPU devices. The CPU, with its complex control logic units and cache architecture, handles logic-intensive tasks, while the GPU utilizes its massively parallel core array and high-speed cache to efficiently handle data-parallel tasks through data transfer and memory management interfaces.

[0053] The hierarchical, multi-level scheduling strategy provided in this embodiment enables the system to select the optimal hardware platform based on global objectives at a macro level, and to flexibly allocate tasks to CPUs or GPUs based on operator characteristics and real-time load conditions at a micro level. This achieves coordinated optimization of load balancing and resource utilization, ultimately resulting in significant improvements in several key indicators such as throughput, cost-effectiveness, and energy efficiency.

[0054] The three core aspects of this invention together constitute a closed-loop compressed sensing stream processing system. First, the data acquisition unit assigns the optimal compression format to the data through adaptive compression selection, laying the foundation for subsequent compressed domain computation. Second, the query processor's compressed domain operators ensure that the computation process does not destroy this format, maintaining the data's 'compressed state'. Finally, the hierarchical multi-level scheduler dynamically makes the optimal execution decision based on the data's current compression format, operator characteristics, and heterogeneous hardware status. These three stages are closely linked, allowing the benefits of compression to accumulate continuously throughout the end-to-end process, ultimately achieving a qualitative leap in system performance.

[0055] Example 2 The above-described embodiment 1 provides a compressed-sensing streaming SQL query method for heterogeneous hybrid computing architectures. Correspondingly, this embodiment provides a compressed-sensing streaming SQL query system for heterogeneous hybrid computing architectures. The system provided in this embodiment can implement the compressed-sensing streaming SQL query method for heterogeneous hybrid computing architectures of embodiment 1. This system can be implemented through software, hardware, or a combination of both. For example, the system may include integrated or separate functional modules or units to execute the corresponding steps in the methods of embodiment 1. Since the system in this embodiment is basically similar to the method embodiment, the description process in this embodiment is relatively simple. Relevant details can be found in the description of parts of embodiment 1. The system embodiment provided in this embodiment is merely illustrative.

[0056] like Figure 2 As shown, this embodiment provides a compressed sensing streaming SQL query system for heterogeneous hybrid computing architectures. This system is data flow oriented and seamlessly integrates the adaptive compression, compressed domain computing and hierarchical scheduling technologies in the previous embodiments, realizing end-to-end, full-link compressed sensing processing from data ingestion to query results.

[0057] Specifically, the system includes: The data acquisition unit is configured to dynamically select the optimal compression encoding scheme for the input raw stream data using an adaptive compression selection strategy, and process it in real time to obtain compressed data in the corresponding compression encoding format. The query processor is configured to redesign the underlying SQL operators involved in query analysis operations to obtain compressed sensing operators; The hierarchical multi-level scheduler is configured to decompose the received SQL query statement into a series of compressed sensing operators, and dynamically make the optimal query execution decision based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status, and apply the compressed sensing operators to the compressed data to obtain the query results.

[0058] Furthermore, the data acquisition unit includes a feature extraction module and an adaptive compression selection module. The feature extraction module is configured to perform real-time, low-overhead, multi-dimensional statistical feature extraction on each column of the input raw streaming data. The extracted features cover basic statistical information, data patterns, and real-time network bandwidth, and are constructed into feature vectors. The adaptive compression selection module includes a model prediction module and a compressor module. The model prediction module is configured to input the feature vectors into a pre-trained lightweight neural network model (e.g., MLP) to dynamically select the optimal compression encoding scheme for the input raw streaming data. The compressor module is configured to compress the raw streaming data using the optimal compression encoding scheme and directly transmit the compressed data to the query processor.

[0059] Furthermore, the hierarchical multi-level scheduler separates the query layer and the operator layer, including: The query layer is configured to select the most suitable execution platform for the current SQL query statement by using a predefined comprehensive weighted scoring model at the architecture level. The comprehensive weighted scoring model takes into account the indicators of different architecture platforms in three dimensions: computing power, energy consumption and hardware cost. It also allows users to flexibly specify optimization objectives (such as peak performance, energy efficiency priority or cost control) by configuring the weight factors corresponding to the three indicators. The operator layer is configured to decompose the received SQL query statement into a series of basic compressed sensing operators, and to use a greedy queue scheduling strategy to allocate resources to each compressed sensing operator on a determined execution platform according to the computational characteristics of each operator.

[0060] The data flow and processing procedure in this system are as follows: The raw streaming data first enters the data acquisition unit, where the feature extraction module extracts statistical features. Subsequently, the model prediction module invokes a lightweight neural network model to dynamically select the optimal solution from nine lightweight compression algorithms based on the statistical features. The compressor module immediately applies the selected compression algorithm to compress the raw streaming data, effectively balancing compression ratio and compression / decompression latency. This process transforms the raw data into a "compressed sensing data stream" with an optimal compression format, laying the foundation for space and computational efficiency for all subsequent processing.

[0061] The compressed data stream is transmitted directly to the query processor over the network. Because the data has been compressed, the entire transmission from the data source to the computing engine is compression-aware, minimizing network bandwidth usage.

[0062] Upon receiving the compressed data stream, the query processor does not decompress it but instead executes the query directly on it. The system decomposes the user-submitted SQL query into a series of compression-aware operators and uses these operators to directly process the compressed data. This means that the "compressed state" of the data is maintained throughout the entire computation process, completely avoiding the huge overhead of full decompression.

[0063] To efficiently execute compressed-sensing query plans, at the query layer, an optimal heterogeneous platform (such as discrete GPUs, integrated CPU-GPU, or edge GPUs) is selected for the current query task based on metrics such as computing power, energy consumption, and cost. At the operator layer, the scheduler further decomposes the query plan into multiple compressed-sensing operators and assigns them device preferences (CPU preference / GPU preference) based on their computational characteristics (such as parallelism and logical complexity). A dynamic greedy scheduler then allocates these operators to suitable CPU or GPU devices for execution, achieving fine-grained load balancing.

[0064] The final query results may be returned directly in compressed form for later use, or decompressed and returned to the user as needed.

[0065] As can be seen, the entire data stream remains compressed until it enters the query processing stage, thereby significantly reducing decompression and data migration overhead and providing a unified compression semantic basis for subsequent multi-level scheduling and heterogeneous execution.

[0066] Example 3 This embodiment provides a processing device corresponding to the compressed sensing streaming SQL query method for heterogeneous hybrid computing architecture provided in Embodiment 1. The processing device can be a client-side processing device, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Embodiment 1.

[0067] The processing device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to enable communication between them. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the compressed sensing streaming SQL query method for heterogeneous hybrid computing architectures provided in Embodiment 1.

[0068] Preferably, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0069] Preferably, the processor can be any type of general-purpose processor such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation herein.

[0070] Example 4 The compressed sensing streaming SQL query method for heterogeneous hybrid computing architecture described in Embodiment 1 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the compressed sensing streaming SQL query method for heterogeneous hybrid computing architecture described in Embodiment 1 are loaded.

[0071] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A compressed-aware streaming SQL query method for heterogeneous hybrid computing architectures, characterized in that, include: An adaptive compression selection strategy is adopted to dynamically select the optimal compression encoding scheme for the input raw stream data, and the compressed data of the corresponding compression encoding format is obtained in real time. The basic SQL operators involved in query analysis operations were redesigned to obtain compressed sensing operators; The received SQL query statement is decomposed into a series of compressed sensing operators, and the optimal query execution decision is dynamically made based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status. The compressed sensing operators are then directly applied to the compressed data to obtain the query results.

2. The compressed-aware streaming SQL query method for heterogeneous hybrid computing architecture as described in claim 1, characterized in that, The process of dynamically selecting the optimal compression encoding scheme for the input raw stream data using an adaptive compression selection strategy and processing it in real time to obtain compressed data in the corresponding compression encoding format includes: Multi-dimensional statistical features are extracted from each column of the input raw streaming data and constructed as a feature vector; The feature vector is input into a lightweight neural network model trained offline, and the optimal compression encoding scheme is dynamically selected for the input raw streaming data. The original streaming data is compressed using the optimal compression coding scheme to obtain compressed data.

3. The compressed-aware streaming SQL query method for heterogeneous hybrid computing architecture as described in claim 2, characterized in that, The process of extracting multi-dimensional statistical features from each column of the input raw stream data includes: Basic statistical characteristics, including minimum, maximum, and mean; Data patterns include data repetition, data variability, number of distinct values, data order, and data disorder. And real-time network conditions.

4. The compressed-aware streaming SQL query method for heterogeneous hybrid computing architecture as described in claim 2, characterized in that, The input of the lightweight neural network model is the feature vector of the raw streaming data. The input feature vector is trained and inferred according to a comprehensive scoring formula. The output is a nine-dimensional scoring vector, which corresponds to the nine lightweight compression algorithms integrated in the system. The lightweight compression algorithms include EG encoding, ED encoding, NS encoding, NSV encoding, RLE encoding, BD encoding, DICT encoding, BM encoding, and PLWAH encoding. The comprehensive scoring formula is as follows: in, It is an algorithm For data columns The rating, It's the compression ratio. and These are compression and decompression times, respectively. It's network bandwidth. It is an adjustable parameter used to balance the weight of compression and decompression overhead.

5. The compressed-aware streaming SQL query method for heterogeneous hybrid computing architecture as described in claim 4, characterized in that, The redesign of the basic SQL operators involved in query analysis operations to obtain compressed sensing operators includes: When selecting an operation operator, for compressed data using EG, ED, and NS encoding formats, direct equality and range comparisons are performed; for compressed data using RLE encoding format, the numerical parts are directly compared and the run length is used to maintain the count, avoiding data expansion; for compressed data using BD encoding format, predicate pushdown is achieved by adjusting the base value and increment. The projection operator can directly forward the required attributes on compressed data in nine lightweight compression coding formats. The join operation operator directly compares compressed data using EG, ED, and NS encoding formats; compares numerical parts and maintains length using run length for compressed data using RLE encoding format; performs additional re-encoding steps to align data with different bases for compressed data using BD encoding format; and supports direct join when sharing the same dictionary for data using DICT and BM encoding formats, otherwise dictionary mapping or re-encoding is required. The grouping operator processes key-value pairs directly on the bit width of compressed data using EG, ED, NS, and BD encoding formats; for compressed data using RLE encoding format, it groups the numerical part and aggregates the counts on the run length; for compressed data using DICT and BM encoding formats, since a one-to-one mapping between the original values ​​and encoded values ​​is maintained, the equivalent grouping operation is performed directly on the compressed data. The sorting operator performs direct comparison and sorting on compressed data using EG, ED, and NS encoding formats; on compressed data using RLE encoding format, it sorts by numerical part and preserves run length for expansion during output; on compressed data using BD encoding format, it directly compares within the compressed domain after adjusting the increment based on the baseline value; on compressed data using DICT and BM encoding formats, it sorts directly if dictionary order is preserved, otherwise it requires lightweight remapping before sorting. The aggregation operators directly perform summation, counting, and maximum / minimum value calculations on compressed data using EG, ED, NS, RLE, and BD encoding formats. For RLE encoded compressed data, the run length is considered during calculation, while for BD encoded compressed data, the result is obtained by accumulating the increment and adjusting it according to the baseline value. Compressed data using DICT and BM encoding only directly supports counting and grouping aggregation; numerical aggregation usually requires decoding.

6. The compressed-aware streaming SQL query method for heterogeneous hybrid computing architecture as described in claim 1, characterized in that, The process involves decomposing the received SQL query into a series of compressed sensing operators, dynamically making the optimal query execution decision based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status, and directly applying the compressed sensing operators to the compressed data to obtain the query results, including: The system architecture level is selected by using a predefined comprehensive weighted scoring model to choose the most suitable execution platform for the current SQL query statement. The received SQL query statement is decomposed into a series of compressed sensing operators, and a greedy queue scheduling strategy is used to allocate resources to each compressed sensing operator on a determined execution platform according to the computational characteristics of each operator. The compressed sensing operator is directly applied to the compressed data to obtain the query results.

7. The compressed-aware streaming SQL query method for heterogeneous hybrid computing architecture as described in claim 6, characterized in that, The process of decomposing the received SQL query statement into a series of compressed sensing operators and allocating resources for each compressed sensing operator on a determined execution platform using a greedy queue scheduling strategy based on the computational characteristics of each operator includes: The SQL query statement is decomposed into multiple compressed sensing operators, and device preferences are marked for each compressed sensing operator based on its computational characteristics and implementation method. Task queues are maintained for CPU and GPU respectively, and a greedy queue scheduling strategy is adopted to allocate resources to each compressed sensing operator on the execution platform based on the device preference.

8. A compressed-aware streaming SQL query system for heterogeneous hybrid computing architectures, characterized in that, include: The data acquisition unit is configured to dynamically select the optimal compression encoding scheme for the input raw stream data using an adaptive compression selection strategy, and process it in real time to obtain compressed data in the corresponding compression encoding format. The query processor is configured to redesign the underlying SQL operators involved in query analysis operations to obtain compressed sensing operators; The hierarchical multi-level scheduler is configured to decompose the received SQL query statement into a series of compressed sensing operators, and dynamically make the optimal query execution decision based on the computational characteristics of each compressed sensing operator and the heterogeneous hardware status, and apply the compressed sensing operators to the compressed data to obtain the query results.

9. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 7.

10. A computing device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 7.

Images