A stream processing method and system based on adaptive non-decompression direct calculation

Abstract

The present invention relates to a stream processing method and system based on adaptive non-decompression direct calculation, which determines the stream data to be processed and the application to be executed based on the stream data to be processed; selects the compression algorithm with the best performance of the stream data to be processed in real time, and adaptively compresses the stream data to be processed on the client to obtain compressed data; based on the available server resource queue maintained on the client, selects the most suitable server for the application to be executed, and transmits the compressed data and the application to be executed to the corresponding server; the corresponding server performs the operation processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain the stream processing result. The present invention can achieve obvious performance acceleration on the existing data set compared with the static method. Therefore, the present invention can be widely used in the field of big data processing technology.

Description

Technical Field

[0001] The invention relates to a stream processing method and system based on adaptive non-decompression direct calculation, belonging to the technical field of big data processing. Background Art

[0002] With the development of big data, stream processing technology has become more and more common in the current big data era. For example, in 2021, the number of connected IoT devices worldwide increased by 9%, with 12.3 billion active endpoints. Stream processing can analyze and query a large number of continuously arriving data stream tuples, such as sensor data and financial transactions. Its main features are low latency and real-time requirements. However, as the scale of data streams continues to increase, stream processing systems are facing pressure from the amount of data.

[0003] Processing such a large amount of streaming data faces dual challenges in terms of space and time. On the one hand, a large amount of streaming data puts tremendous pressure on the transmission bandwidth. If there is a significant delay in network transmission, the real-time performance of streaming processing cannot be guaranteed. On the other hand, streaming processing systems usually store data temporarily in memory. If the data arrives at a high rate, it will also put tremendous pressure on the server's memory usage. In particular, data compression, as a common method to reduce data size, is widely used in the field of big data to save time and space overhead. However, due to the real-time changes in streaming data, a single compression algorithm is difficult to meet the needs of streaming processing.

[0004] There is a large amount of literature on stream processing systems, but few studies focus on comprehensive compression methods in stream processing, especially compressed stream processing based on structured query language (SQL). Supporting SQL in stream processing helps users process data in a user-friendly way, and direct SQL processing on compressed data helps further improve system performance. Among the existing stream engines, TerseCades ensures the performance of compressed stream processing by optimizing the hardware architecture, but it only uses simple integer compression methods and floating-point compression methods, without diverse compression methods and adaptive performance models to provide adaptive processing. SABER and FineStream perform SQL stream processing on heterogeneous CPU-GPU architectures, but do not consider stream compression.

[0005] In traditional database processing, there are more diverse compression algorithm support, such as NS, NSV, DICT and BITMAP. In Succinct, the compressed representation of the input data can be directly queried efficiently. However, the characteristics of stream data processing are very different from those of traditional databases. The traditional database processing method is to perform overall operations after scanning all the data and select the compression algorithm according to the overall characteristics of the data. In stream processing, unbounded data streams arrive in real time. The dynamic characteristics of stream data are mainly reflected in the following three aspects. First, the attributes of stream data can change at any time, including value range, degree of repetition, etc. As the attributes of stream data change, the compression ratios that can be achieved by different compression algorithms are also different. Secondly, affected by factors such as data generation speed and network delay, the transmission rate of stream data may change at any time, thus affecting the waiting time of the system. Third, it is impossible to predict the frequency of data attribute changes. Since the characteristics of data streams are unpredictable, a fixed solution cannot be predetermined to effectively handle them. It needs to be handled by dynamic compression methods. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a stream processing method and system based on adaptive non-decompression direct calculation, which can adaptively compress stream data according to the performance model and perform direct processing in a compressed state to improve the performance of the stream processing system.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a stream processing method based on adaptive non-decompression direct calculation, comprising the following steps:

[0009] Determine stream data to be processed and an application to be executed based on the stream data to be processed;

[0010] Select the compression algorithm with the best performance for the stream data to be processed in real time, and perform adaptive compression on the stream data to be processed on the client to obtain compressed data;

[0011] Based on the available server resource queue maintained on the client, the most suitable server for the application to be executed is selected, and the compressed data and the application to be executed are transmitted to the corresponding server;

[0012] The corresponding server performs running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain a stream processing result.

[0013] Furthermore, the method of selecting in real time the compression algorithm with the best performance for the stream data to be processed, and adaptively compressing the stream data to be processed on the client to obtain compressed data includes:

[0014] Every time a batch is processed, a preset number of stream data tuples are pre-read and their data characteristics are counted;

[0015] According to the statistically obtained data characteristics, the total processing time brought by various preset compression algorithms is calculated, and the compression algorithm with the shortest total processing time is selected as the optimal compression algorithm for processing batch stream data in this section;

[0016] Based on the determined optimal compression algorithm, the stream data of the processing batch of this section is adaptively compressed, and the compression method and the number of bytes occupied by the stream data of the processing batch of this section are recorded at the same time as the final compressed data.

[0017] Furthermore, the total processing time brought by using various preset compression algorithms is calculated based on the statistically obtained data characteristics, and the compression algorithm with the shortest total processing time is selected as the optimal compression algorithm for processing batch stream data in this section, including:

[0018] Identify various compression algorithms that can compress stream data, including EG encoding, ED encoding, NS encoding, NSV encoding, BD encoding, RLE encoding, dictionary encoding, and bitmap;

[0019] Based on the statistical data characteristics, calculate the total processing time brought by various compression algorithms;

[0020] Based on the total processing time of different compression algorithms, the compression algorithm with the shortest total processing time is selected as the optimal compression method for processing batch stream data in this section.

[0021] Furthermore, the calculation formula of the total processing time is:

[0022] t=t compress +t trans +t query

[0023] Among them, t compress Indicates the compression time, t trans Indicates the transmission time, t query Indicates the query calculation time;

[0024] The compression time t compress The calculation formula is:

[0025]

[0026] Among them, a indicates whether the compression algorithm is greedy or lazy; t wait Indicates the time consumed waiting to read the entire data batch; Indicates the number of instructions used to access memory during compression; Indicates the number of instructions used for calculation during compression; N clientIndicates the processing performance of the client;

[0027] The transmission time t trans The calculation formula is:

[0028]

[0029] Among them, Size T Indicates the number of bytes occupied by each tuple; r indicates the compression ratio of the selected compression algorithm τ; Size B Indicates the number of tuples in each batch; latency indicates the transmission delay of the network;

[0030] The query calculation time t query The calculation formula is:

[0031]

[0032] in, Indicates the time consumed by the instructions used for calculation during the query process; Indicates the time consumed by the instructions used to read and write memory during this process.

[0033] Further, based on the available server resource queue maintained on the client, selecting the most suitable server for the application to be executed, and transmitting the compressed data and the application to be executed to the corresponding server, includes:

[0034] Based on the available server resource queues maintained on the client, select the first server with sufficient remaining available resources to perform the current task.

[0035] When the server receives the current task, it sends the current available resource information to all clients, updates the server resource queue s on the client for resource allocation of the new task, and sends the updated available resource information when returning the result to the client after completing the task.

[0036] Furthermore, the first server whose remaining available resources are sufficient to execute the current task satisfies the following conditions:

[0037] ①mem i ≥mem k ,c i ≥c k ,g i ≥g k ;

[0038] ② Meet mem j ≥mem k ,c j ≥c k ,gj ≥g k ;

[0039] Among them, mem i 、mem j and mem k Respectively represent the server s i Corresponding free memory, server s j The corresponding free memory and the memory required by the current task; c i 、c j and c k Respectively represent the server s i Corresponding CPU idle rate, server s j The corresponding CPU idle rate and the idle rate required by the current task; g i , g j and g k Respectively represent the server s i Corresponding CPU idle rate, server s j The corresponding CPU idle rate and the idle rate required by the current task.

[0040] Further, the corresponding server performs running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain a stream processing result, including:

[0041] For a given input data stream, assume that it has n data attributes <col 1 ,col 2 ,…,col n > and m sequences of operations to be processed <op 1 ,op 2 ,…,op m >

[0042] Traverse the operation sequence, for each operator op i , if the operation has better performance on the GPU, the operation is assigned to the GPU, otherwise it is assigned to the CPU;

[0043] The set of compression algorithms used by the corresponding server based on the one-to-one correspondence with the n data attributes sent by the client <τ 1 ,τ 2 ,…,τ n > and the number of bytes occupied by the compressed data <bytes 1 ,bytes 2 ,…,bytes n >, update the calculation function, and process the corresponding operator based on the updated calculation function.

[0044] In a second aspect, the present invention provides a stream processing system based on adaptive non-decompression direct calculation, comprising:

[0045] An input module, used to determine stream data to be processed and an application to be executed based on the stream data to be processed;

[0046] An adaptive compression module is used to select in real time the compression algorithm with the best performance for the stream data to be processed, and adaptively compress the stream data to be processed on the client to obtain compressed data;

[0047] A load balancing module is used to select the most suitable server for the application to be executed based on the available server resource queue maintained on the client, and transmit the compressed data and the application to be executed to the corresponding server;

[0048] The operation module is used for the corresponding server to perform running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain the stream processing result.

[0049] In a third aspect, the present invention provides a processing device, which includes at least a processor and a memory, wherein a computer program is stored in the memory, and when the processor runs the computer program, the steps of the stream processing method based on adaptive non-decompression direct calculation are executed.

[0050] In a fourth aspect, the present invention provides a computer storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the steps of the stream processing method based on adaptive non-decompression direct calculation.

[0051] The present invention adopts the above technical solution, which has the following advantages:

[0052] 1. The present invention proposes a multi-step performance model including compression, transmission, and query calculation, taking into account factors such as machine performance and network conditions, to estimate the performance of the stream processing system.

[0053] 2. The present invention integrates multiple compression algorithms in the system for high-performance processing of data with different characteristics.

[0054] 3. The present invention combines the performance model to adaptively compress the stream data, automatically switches the compression algorithm, adapts to the characteristics of the stream data, and improves the stream processing performance.

[0055] 4. The present invention performs load balancing among servers through a load balancing module, thereby improving the utilization rate of the servers.

[0056] 5. The present invention can directly calculate on compressed data to avoid the delay of decompression.

[0057] Therefore, the present invention can be widely applied in the field of big data processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings:

[0059] Figure 1 is a flow chart of a stream processing method based on fine-grained adaptive compression provided by an embodiment of the present invention;

[0060] Figure 2 This is the compression module processing flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0062] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0063] In some embodiments of the present invention, a method for stream processing based on adaptive non-decompression direct calculation is provided. First, a compression algorithm selector based on a performance model is used to select the best compression algorithm for the stream data to be processed in real time for compression. Then, the compressed data and the application to be executed are transmitted to the most suitable server for application processing to obtain the stream processing result. The present invention can adaptively compress the stream data according to the performance model, and perform direct processing in a compressed state, thereby improving the performance of the stream processing system.

[0064] Correspondingly, in some other embodiments of the present invention, a stream processing system, device and medium based on adaptive non-decompression direct calculation are provided.

[0065] Example 1

[0066] like Figure 1As shown, this embodiment provides a stream processing method based on adaptive non-decompression direct calculation, which includes the following steps:

[0067] 1) Determine the stream data to be processed and the application to be executed based on the stream data to be processed, that is, the SQL statement;

[0068] 2) Select the compression algorithm with the best performance for the stream data to be processed in real time, and adaptively compress the stream data to be processed on the client to obtain compressed data;

[0069] 3) Based on the available server resource queue maintained on the client, the most suitable server for the application to be executed is selected, and the compressed data and the application to be executed are transmitted to the corresponding server;

[0070] 4) The corresponding server performs running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain the stream processing result.

[0071] Preferably, in the above step 2), in a large data stream scenario, data is transmitted in a stream manner, and all the information to be processed cannot be known in advance. As the input data stream changes dynamically, its real-time processing method needs to be adaptively changed. In the present invention, a performance model for a compressed stream processing system is designed, and an adaptive selection of a compression algorithm is performed based on the performance model. By statistics on the specified characteristics of the input stream data, the compression ratio, the number of instructions to be executed, the memory and time overhead brought about by the use of various compression algorithms can be estimated, and then the system time required for processing using the corresponding compression algorithm can be calculated, and finally the compression method that consumes the least time can be selected for compression.

[0072] Specifically, Figure 2 As shown, the following steps are included:

[0073] 2.1) For every processing batch, pre-read a preset number of processing batches of stream data tuples and count their data characteristics.

[0074] The size of the processing batch is set by the user according to actual needs. In this embodiment, the size of the processing batch is set to 100, and the preset number of batches can be selected as 5 processing batches.

[0075] This embodiment performs statistics on the data characteristics of the stream data tuples, including the average data repetition, the maximum and minimum values, the variation range, the number of bytes occupied by leading zeros, etc. This embodiment uses the statistical results of the data characteristics of the stream data of these 5 processing batches as a prediction of the stream data characteristics of the next 100 processing batches.

[0076] 2.2) Based on the statistical data characteristics, calculate the total processing time brought by various preset compression algorithms, and select the compression algorithm with the shortest total processing time as the optimal compression algorithm for processing batch stream data in this section.

[0077] 2.3) Based on the optimal compression algorithm determined in step 2.2), the stream data of this processing batch is adaptively compressed, and the compression method and the number of bytes occupied by the stream data of this processing batch are recorded as the final compressed data.

[0078] Preferably, the above step 2.2) comprises the following steps:

[0079] 2.2.1) Determine the various compression algorithms that can be used to compress streaming data.

[0080] Specifically, in the compression module of the present invention, eight lightweight compression algorithms are integrated, each with different characteristics, for adaptive stream compression processing. The eight lightweight compression algorithms are described as follows:

[0081] ①Elias Gamma encoding (EG)

[0082] EG encoding uses the number of leading zeros to represent the binary validity of the data. Given a positive integer x, let That is, L represents the number of bits in the binary representation of x. The EG encoding form of x is L leading zeros plus its binary representation, which is represented by 2*L+1 bits in total.

[0083] ②Elias Delta encoding (ED)

[0084] ED coding is a variant of EG coding, which encodes L obtained in EG coding again. Given a positive integer x, let The ED encoding of X is N leading zeros, plus the binary representation of L+1, plus the last L bits of the binary representation of x, for a total of 2*N+L+1 bits. In comparison, EG encoding is more suitable for compressing small integers, while ED encoding is more suitable for compressing large integers.

[0085] ③Null suppression with fixed length(NS)

[0086] NS encoding achieves compression by removing the leading zeros from the binary representation of the elements. Because data is often represented in programs using aligned data types, there are many redundant leading zeros, so leading zeros can be removed to save space. The encoding result of this method has a fixed length, so for a group of data to be compressed, they are all aligned according to the longest number of digits after removing the leading zeros.

[0087] ④Null suppression with variable length(NSV)

[0088] NSV encoding is similar to NS encoding, and also achieves compression by removing leading zeros. The difference is that the number of bits in the compression result of NS is the same, and there will be redundant leading zeros for alignment, while NSV completely removes leading zeros, but requires an additional flag bit to record the number of bits of the compressed data. In comparison, NS encoding is more suitable for compressing a group of data with similar binary representation lengths, and NSV encoding has a significantly better compression effect than NS for data with uncertain binary encoding lengths.

[0089] ⑤Base-Delta encoding (BD)

[0090] BD encoding selects a base value in a set of data to be compressed and records it, and then for all elements, the difference between the element and the base value is used to represent it. Therefore, if the change between elements is much smaller than the element itself, it can occupy fewer bits.

[0091] ⑥Run length encoding (RLE)

[0092] RLE encoding is a traditional encoding method that uses key-value pairs to represent elements and the length of their consecutive appearances. For example, the string "AAAABBBBBCCC" can be encoded as [A,4][B,5][C,3], so it has a good compression effect on frequently repeated data.

[0093] ⑦Dictionary (DICT)

[0094] Dictionary encoding maps a large range of data to a small range of dictionary one by one. For example, in the string "ABABC", "A" can be encoded as "00", "B" as "01", and "C" as "10". Dictionary encoding works well for data with a large range of variation but a small number of values.

[0095] ⑧Bitmap (BITMAP)

[0096] The bitmap uses a binary string to complete the mapping of data. Each bit in the binary string corresponds to an original data. For example, for the string "AABCCD", each element in it can be mapped to a 4-bit binary string, "A" corresponds to "0001", "B" corresponds to "0010", "C" corresponds to "0100", and "D" corresponds to "1000", that is, A corresponds to the lowest bit in the binary string, and D corresponds to the highest bit. Because the more values ​​the data has, the longer the binary string will be, so the bitmap is only suitable for data with very few values.

[0097] Each compression algorithm has its own advantages and disadvantages, and they are suitable for data streams with different characteristics. For example, Elias Gamma encoding and Elias Delta encoding are respectively suitable for small values ​​and large values, run-length encoding is suitable for data with a large number of repetitions, and the compression effect of NS and NSV depends on the redundant leading zeros in the elements. The present invention divides these eight lightweight compression algorithms into two categories: eager and lazy. Among them, Elias Gamma encoding, Elias Delta encoding, NS encoding and NSV encoding belong to greedy compression algorithms, and the other four belong to lazy compression algorithms. The greedy compression algorithm compresses immediately when the input partial tuple arrives. In contrast, the lazy compression algorithm waits for the entire batch of data to be processed to enter before compressing. The advantage of the greedy compression algorithm is that each tuple can be processed immediately without waiting, while the advantage of the lazy algorithm is that a large amount of similar redundant data can be better utilized to obtain a higher compression ratio.

[0098] This embodiment uses a data set from a smart grid as an analysis case, which contains more than 4.055 billion energy consumption measurement data from 2,125 plugs distributed in 40 residences measured within a month. The data set contains seven attributes, including timestamp, energy consumption value, plug and house ID, etc. For real-time generated energy consumption data, the characteristics of stream data are constantly changing. The peaks and troughs of electricity consumption, as well as the different electricity consumption habits of different households, will cause the data stream to change continuously. Since the changing trend of data cannot be predicted, the processing method used in stream processing needs to be real-time and dynamic, and be able to automatically adapt to the changes in the characteristics of the data stream.

[0099] For example, when a family generates a large amount of electricity consumption data in a very short period of time, it will be shown in the data stream as a continuously repeated house ID and a constantly changing plug ID and energy consumption value. At this time, for a small range of timestamp changes in a short period of time, Base-Delta encoding may be used, for a continuously repeated house ID, Run-length encoding (RLE) may be used, and for a changing plug ID, dictionary encoding (DICT) or removal of redundant leading zeros (NS) and other compression methods may be used. When many families generate electricity consumption data at the same time, the house ID may change frequently, while the timestamp remains unchanged. At this time, the timestamp may be compressed by run-length encoding, and the house ID may be compressed by dictionary encoding. In this way, the first segment of data may be compressed by a combination of "Base-Delta, RLE, DICT", and the second segment of data may be compressed by a combination of "RLE, DICT, NS". As the characteristics of the stream data change, the compression method for the data stream in the present invention is constantly changing, and the processing method with the best performance is independently selected for each column attribute in the data stream.

[0100] 2.2.2) Based on the statistical data characteristics, calculate the total processing time brought by using various compression algorithms.

[0101] In order to guide the system to automatically select a suitable compression algorithm at runtime, this embodiment proposes a performance model. The difficulty in proposing a compression stream processing cost model lies mainly in the complexity of the processing process and scenario. In the processing scenario of the present invention, the present invention takes into account machine performance, network conditions and other extensive factors, and solves the above difficulties through a multi-step model. Considering that the processing of the present invention mainly includes three stages: compression, transmission and query calculation, the present invention establishes a system performance model of compression stream processing in these three stages, and expresses the total processing time t of the compression stream processing system as:

[0102] t=t compress +t trans +t query (1)

[0103] Among them, t compress Indicates the compression time, t trans represents the transmission time, t query Indicates the query calculation time.

[0104] Afterwards, the processing time of these three stages is calculated separately.

[0105] ① Compression time t compress

[0106] For a selected compression algorithm τ, the present invention uses To represent the number of instructions used for memory access during the compression algorithm processing, use Indicates the number of instructions used for computing operations. Since compression is performed on the client, the client's processing performance needs to be taken into account, and N is used client For computationally intensive programs, the client processing performance is represented by the CPU frequency; for memory-intensive programs, the client processing performance is represented by the memory read and write speed. At the same time, the total number of instructions required for different compression algorithms to operate can be calculated in advance, so that the performance of the compression algorithm can be combined with the client machine performance to obtain the time overhead of the compression stage. At the same time, for greedy compression algorithms, compression will be performed immediately as the data arrives, while lazy compression algorithms need to wait for a whole batch of data to be processed, so the present invention uses α·t wait represents the waiting time of the compression algorithm τ, where α is defined as:

[0107]

[0108] In summary, the compression time t compress The calculation formula is:

[0109]

[0110] Among them, α indicates whether the compression algorithm is greedy or lazy; t wait Indicates the time consumed waiting to read the entire data batch; Indicates the number of instructions used to access memory during compression; Indicates the number of instructions used for calculation during compression; N client Indicates the processing performance of the client.

[0111] ②Transmission time t trans

[0112]

[0113] Among them, Size T Indicates the number of bytes occupied by each tuple, which is determined by the format of the input data; r represents the compression ratio of the selected compression algorithm τ (number of bytes before compression / number of bytes after compression), that is, the size of the compressed tuple is Size T / r;Size B Indicates the number of tuples in each batch. Thus, for a compressed batch, the number of bytes to be transmitted is Size. T Size B / r, that is, the amount of data that needs to be transmitted depends on the compression ratio of the compression algorithm used. The higher the compression ratio of the algorithm, the more advantages it has in transmission performance; latency refers to the transmission delay of the network.

[0114] During the transmission process, the network status needs to be taken into account. When the network bandwidth is sufficient, that is, the number of bytes transmitted per second is less than the maximum number of bytes that the network can transmit per second, and there is no need to consider the delay of data queuing, t trans Formula (4) is used for calculation; if the network bandwidth is fully occupied, that is, the number of bytes transmitted per second is greater than the maximum number of bytes that the network can transmit per second, and a large amount of data needs to queue, t trans The calculation of is calculated by the following formula (5):

[0115]

[0116] ③Query calculation time

[0117] The query calculation is performed on the server side through the calculation function. The compression process mainly affects the efficiency of the calculation function in memory reading and writing, and has no effect on the efficiency of the calculation instruction. The reading and writing processing of the present invention uses bytes as the minimum unit, so the time consumed for memory reading and writing is proportional to the number of bytes occupied by the data in the memory. Indicates the time consumed by the instructions used for calculation during the query process. Indicates the time consumed by the instructions for reading and writing memory in the process. Since direct calculation can be performed without decompression, the reading and writing speed and calculation speed will be faster during calculation. Therefore, reducing the data size can speed up the calculation process. Different compression algorithms have different compression ratios, which will reflect different efficiencies in the calculation process. Specifically, the calculation formula for query calculation time is:

[0118]

[0119] in, Indicates the time consumed by the instructions used for calculation during the query process; It represents the time consumed by the instructions for memory reading and writing in this process; r represents the compression ratio of the selected compression algorithm τ (number of bytes before compression / number of bytes after compression).

[0120] 2.2.3) Based on the total processing time of different compression algorithms, the compression algorithm with the shortest total processing time is selected as the optimal compression method for processing batch stream data in this section.

[0121] Preferably, in the above step 3), based on the available server resource queue maintained on the client, the method of selecting the most suitable server for the application to be executed includes:

[0122] 3.1) Based on the available server resource queues s maintained on the client, select the first server whose remaining available resources are sufficient to execute the current task to be responsible for the current task.

[0123] On each client, an available server resource queue s is maintained, which contains n servers 1 ,s 2 ,…,s n >, each server s i There are corresponding free memory mem i , cpu idle rate c i And gpu idle rate g i The server resource queue s is an ordered queue, which is sorted in ascending order according to the resource availability. The sorting rules are:

[0124] For two servers i ,s j , if mem i <mem j , then s i j , if mem i >mem j , then s i >s j Otherwise, if c i <c j , then s i j , if c i >c j , then s i >s j Otherwise, if g i ≤g j , then s i j , then s i >s j .

[0125] When the client executes the load balancing module, it searches the server resource queue s and selects the first server with sufficient remaining available resources to perform the current task. Assume that the resources required to perform the current task are mem k ,c k ,g k , that is, find the server s i ,satisfy:

[0126] ①mem i ≥mem k ,c i ≥c k ,g i ≥g k .

[0127] ② Meet mem j ≥mem k ,c​​​​j ≥c k ,g j ≥g k .

[0128] The purpose of this is to select the smallest available server each time and make the best use of server resources.

[0129] 3.2) After receiving the current task, the server sends the current available resource information to all clients and updates the server resource queue s on the client for resource allocation of the new task. At the same time, after completing the task and returning the result to the client, the server will also send the updated available resource information.

[0130] Preferably, in the above step 4), when the corresponding server processes the SQL statement to be executed, it first further subdivides the SQL statement into different operators, which are the smallest data processing units. Specifically, the operators included in this embodiment support common relational operators, including projection, selection, aggregation, grouping, join, etc. In the context of stream processing, the data stream consists of a sequence of tuples, and a tuple is a record representing an event including elements such as timestamps, quantities, and measurements. Tuples constitute a batch. Batch processing represents a processing block containing a certain number of tuples. In batch processing, the present invention uses columns to represent elements of different tuples in the same field. Finally, the present invention uses a sliding window to process data. The window is composed of a finite sequence of tuples. The size of the window is set by the user and represents the size of the data unit to be processed.

[0131] In the computation module, stream processing is performed by query statements composed of these operators with a given sliding window size. After a certain time interval, the compression module uses the system performance model to predict the processing delay and update the compression algorithm used. The computation module is also updated accordingly, switching the corresponding computation function according to the selected new compression algorithm. Using these optimized query functions, the compression values ​​can be directly compared and calculated, allowing the present invention to perform calculations directly on the compressed data.

[0132] In addition, since different operators have different device preferences, for example, operations such as projection, grouping, and selection perform better on the GPU, while aggregation and connection perform better on the CPU. Therefore, in the computing module, the tasks to be executed will be assigned to the CPU or GPU in advance based on the SQL statements to be executed to achieve higher system performance.

[0133] Based on the above analysis, the processing flow of the processor includes the following steps:

[0134] 4.1) For a given input data stream, assume that it has n data attributes <col 1 ,col2 ,…,col n > and m sequences of operations to be processed <op 1 ,op 2 ,…,op m >

[0135] 4.2) Traverse the operation sequence, for each operator op i If the operation has better performance on the GPU, the operation is assigned to the GPU, otherwise it is assigned to the CPU.

[0136] 4.3) The set of compression algorithms used by the corresponding server based on the one-to-one correspondence with the n data attributes sent by the client <τ 1 ,τ 2 ,…,τ n > and the number of bytes occupied by the compressed data <bytes 1 ,bytes 2 ,…,bytes n >, update the calculation function, and process the corresponding operator based on the updated calculation function.

[0137] The calculation function built into the calculation module is used by default to process uncompressed data streams, while for the processing of compressed data streams, the corresponding calculation function needs to be updated. Due to the characteristics of the stream data, the present invention adopts a dynamic update processing method, so each time it is updated, the calculation function used will also be updated. Each time the processing method is dynamically updated, the client will send the server a set of compression algorithms corresponding to the n data attributes <τ 1 ,τ 2 ,…,τ n > and the number of bytes occupied by the compressed data <bytes 1 ,bytes 2 ,…,bytes n >. For an operator op i , whose default calculation function processes the data attributes of each column 1 ,q 2 ,…,q n >, the updated calculation function is expressed as <(q 1 ,τ 1 ,bytes 1 ),(q 2 ,τ 2 ,bytes 2 ),…,(q n ,τ n ,bytes n )>. Among them, the processing function q i ​The update method is determined by the corresponding compression algorithm τ i Determine, combined with the number of bytes after compression i To determine the memory space required for reading, writing and calculating data.

[0138] For byte-aligned compression algorithms, such as Base-Delta encoding, NS encoding, etc., the processing function q i The update method is relatively simple. For example, the original query is in the data attribute col i The corresponding operation (q i ,τ i ,bytes i ) You only need to change the data unit of memory access and operation to bytes i For other non-aligned compression algorithms, the processing function q i The update needs to be based on the compression algorithm τ i The principle and specific operation of the processing function are updated, such as q i is the grouping operation, τ i When it is RLE encoding, because the encoded result is the symbol value and the number of times the symbol is repeated in the original data, the corresponding (q i ,τ i ,bytes i ) can merge and summarize the encoded data.

[0139] Example 2

[0140] This embodiment is tested on several common data sets, including smart grids, which is a measurement of energy consumption of smart grids, cluster monitoring, which is monitoring data from Google computing clusters, and linear road benchmark, which is the monitoring and trajectory prediction of vehicle positions. On these three data sets, the present invention can achieve throughput improvements of 4.80 times, 2.38 times, and 2.55 times, and latency reductions of 66.0%, 58.0%, and 60.8%, respectively.

[0141] This embodiment uses dynamic workloads on these data sets and compares the performance with the existing best performance processing methods. Taking the smart grid data set as an example, under a network bandwidth of 10Mbps, a 1.70-fold speedup ratio can be achieved compared with the best performance static processing method, and under a network bandwidth of 100Mbps, a 1.21-fold speedup ratio can be achieved.

[0142] Example 3

[0143] The above-mentioned embodiment 1 provides a stream processing method based on adaptive non-decompression direct calculation. Correspondingly, this embodiment provides a stream processing system based on adaptive non-decompression direct calculation. The system provided by this embodiment can implement the stream processing method based on adaptive non-decompression direct calculation of embodiment 1, and the system can be implemented by software, hardware, or a combination of software and hardware. For example, the system may include integrated or separate functional modules or functional units to execute the corresponding steps in each method of embodiment 1. Since the system of this embodiment is basically similar to the method embodiment, the process described in this embodiment is relatively simple, and the relevant parts can refer to the partial description of embodiment 1. The embodiment of the system provided in this embodiment is only illustrative.

[0144] The stream processing system based on adaptive non-decompression direct calculation provided in this embodiment includes:

[0145] An input module, used to determine the stream data to be processed and the application to be executed based on the stream data to be processed, that is, an SQL statement;

[0146] An adaptive compression module is used to select in real time the compression algorithm with the best performance for the stream data to be processed, and adaptively compress the stream data to be processed on the client to obtain compressed data;

[0147] A load balancing module is used to select the most suitable server for the application to be executed based on the available server resource queue maintained on the client, and transmit the compressed data and the application to be executed to the corresponding server;

[0148] The operation module is used for the corresponding server to perform running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain the stream processing result.

[0149] Preferably, the adaptive compression module includes a compression algorithm selector and a compression module, wherein the compression algorithm selector is used to select the compression algorithm with the best performance for the processed stream data in real time based on the performance model; the compression module is used to compress the processed stream data using the optimal compression algorithm to obtain compressed data.

[0150] Example 4

[0151] This embodiment provides a processing device corresponding to the stream processing method based on adaptive non-decompression direct calculation provided in this embodiment 1. The processing device can be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of embodiment 1.

[0152] The processing device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete mutual communication. The memory stores a computer program that can be run on the processor, and the processor executes the stream processing method based on adaptive non-decompression direct calculation provided in this embodiment 1 when running the computer program.

[0153] In some embodiments, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0154] In other embodiments, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0155] Example 5

[0156] The stream processing method based on adaptive non-decompression direct calculation of this embodiment 1 can be specifically implemented as a computer program product, which may include a computer-readable storage medium carrying computer-readable program instructions for executing the stream processing method based on fine-grained adaptive compression described in this embodiment 1.

[0157] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A stream processing method based on adaptive non-decompression direct calculation, characterized in that: The following steps are involved: Determine stream data to be processed and an application to be executed based on the stream data to be processed; Select the compression algorithm with the best performance for the stream data to be processed in real time, and perform adaptive compression on the stream data to be processed on the client to obtain compressed data; Based on the available server resource queue maintained on the client, the most suitable server for the application to be executed is selected, and the compressed data and the application to be executed are transmitted to the corresponding server; The corresponding server performs running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain the stream processing result; The method of selecting in real time the compression algorithm with the best performance for the stream data to be processed, and adaptively compressing the stream data to be processed on the client to obtain compressed data includes: Every time a batch is processed, a preset number of stream data tuples are pre-read and their data characteristics are counted; According to the statistically obtained data characteristics, the total processing time brought by various preset compression algorithms is calculated, and the compression algorithm with the shortest total processing time is selected as the optimal compression algorithm for processing batch stream data in this section; Based on the determined optimal compression algorithm, the stream data of the processing batch of this section is adaptively compressed, and the compression method and the number of bytes occupied by the stream data of the processing batch of this section are recorded as the final compressed data; The method of calculating the total processing time brought by various preset compression algorithms based on the statistically obtained data characteristics and selecting the compression algorithm with the shortest total processing time as the optimal compression algorithm for processing batch stream data in this section includes: Identify various compression algorithms that can compress stream data, including EG encoding, ED encoding, NS encoding, NSV encoding, BD encoding, RLE encoding, dictionary encoding, and bitmap; Based on the statistical data characteristics, calculate the total processing time brought by various compression algorithms; The calculation formula of the total processing time is: t=t compress +t trans +t query Among them, t compress Indicates the compression time, t trans represents the transmission time, t query Indicates the query calculation time; The compression time t compress The calculation formula is: Among them, α indicates whether the compression algorithm is greedy or lazy; t wait Indicates the time consumed waiting to read the entire data batch; Indicates the number of instructions used to access memory during compression; Indicates the number of instructions used for calculation during compression; N client Indicates the processing performance of the client; The transmission time t trans The calculation formula is: Among them, Size T Indicates the number of bytes occupied by each tuple; r indicates the compression ratio of the selected compression algorithm τ; Size B Indicates the number of tuples in each batch; latency indicates the transmission delay of the network; The query calculation time t query The calculation formula is: in, Indicates the time consumed by the instructions used for calculation during the query process; Indicates the time consumed by the instructions used to read and write memory during the process; Based on the total processing time of different compression algorithms, the compression algorithm with the shortest total processing time is selected as the optimal compression method for processing batch stream data in this section; The method of selecting the most suitable server for the application to be executed based on the available server resource queue maintained on the client, and transmitting the compressed data and the application to be executed to the corresponding server includes: Based on the available server resource queues maintained on the client, select the first server with sufficient remaining available resources to perform the current task. When the server receives the current task, it sends the current available resource information to all clients, updates the server resource queue s on the client to provide resource allocation for the new task, and sends the updated available resource information when returning the result to the client after completing the task; The first server whose remaining available resources are sufficient to execute the current task satisfies the following conditions: ①mem i ≥mem k ,c i ≥c k ,g i ≥g k ; ② Meet mem j ≥mem k , c j ≥c k , g j ≥g k ; Among them, mem i 、mem j and mem k Respectively represent the server s i Corresponding free memory, server s j The corresponding free memory and the memory required by the current task; c i 、c j and c k Respectively represent the server s i Corresponding CPU idle rate, server s j The corresponding CPU idle rate and the idle rate required by the current task; g i , g j and g k Respectively represent the server s i Corresponding CPU idle rate, server s j The corresponding CPU idle rate and the idle rate required by the current task; The corresponding server performs running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain a stream processing result, including: For a given input data stream, assume that it has n data attributes <col1,col2,…,col n > and m sequences of operations to be processed <op1,op2,…,op m > Traverse the operation sequence, for each operator op i , if the operation has better performance on the GPU, the operation is assigned to the GPU, otherwise it is assigned to the CPU; The set of compression algorithms used by the corresponding server based on the one-to-one correspondence of the n data attributes sent by the client <τ1, τ2, ..., τ n > and the number of bytes occupied by the compressed data <bytes1,bytes2,…,bytes n >, update the calculation function, and process the corresponding operator based on the updated calculation function.

2. A stream processing system based on adaptive non-decompression direct calculation, characterized in that: include: An input module, used to determine stream data to be processed and an application to be executed based on the stream data to be processed; An adaptive compression module is used to select in real time the compression algorithm with the best performance for the stream data to be processed, and adaptively compress the stream data to be processed on the client to obtain compressed data; A load balancing module is used to select the most suitable server for the application to be executed based on the available server resource queue maintained on the client, and transmit the compressed data and the application to be executed to the corresponding server; The operation module is used for the corresponding server to perform the operation processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain the stream processing result; The method of selecting the compression algorithm with the best performance for the stream data to be processed in real time and adaptively compressing the stream data to be processed on the client to obtain compressed data includes: Every time a batch is processed, a preset number of stream data tuples are pre-read and their data characteristics are counted; According to the statistically obtained data characteristics, the total processing time brought by various preset compression algorithms is calculated, and the compression algorithm with the shortest total processing time is selected as the optimal compression algorithm for processing batch stream data in this section; Based on the determined optimal compression algorithm, the stream data of the processing batch of this section is adaptively compressed, and the compression method and the number of bytes occupied by the stream data of the processing batch of this section are recorded as the final compressed data; The method of calculating the total processing time brought by various preset compression algorithms based on the statistically obtained data characteristics and selecting the compression algorithm with the shortest total processing time as the optimal compression algorithm for processing batch stream data in this section includes: Identify various compression algorithms that can compress stream data, including EG encoding, ED encoding, NS encoding, NSV encoding, BD encoding, RLE encoding, dictionary encoding, and bitmap; Based on the statistical data characteristics, calculate the total processing time brought by various compression algorithms; The calculation formula of the total processing time is: t=t compress +t trans +t query Among them, t compress Indicates the compression time, t trans represents the transmission time, t query Indicates the query calculation time; The compression time t compress The calculation formula is: Among them, α indicates whether the compression algorithm is greedy or lazy; t wait Indicates the time consumed waiting to read the entire data batch; Indicates the number of instructions used to access memory during compression; Indicates the number of instructions used for calculation during compression; N client Indicates the processing performance of the client; The transmission time t trans The calculation formula is: Among them, Size T Indicates the number of bytes occupied by each tuple; r indicates the compression ratio of the selected compression algorithm τ; Size B Indicates the number of tuples in each batch; latency indicates the transmission delay of the network; The query calculation time t query The calculation formula is: in, Indicates the time consumed by the instructions used for calculation during the query process; Indicates the time consumed by the instructions used to read and write memory during the process; Based on the total processing time of different compression algorithms, the compression algorithm with the shortest total processing time is selected as the optimal compression method for processing batch stream data in this section; The method of selecting the most suitable server for the application to be executed based on the available server resource queue maintained on the client, and transmitting the compressed data and the application to be executed to the corresponding server includes: Based on the available server resource queues maintained on the client, select the first server with sufficient remaining available resources to perform the current task. When the server receives the current task, it sends the current available resource information to all clients, updates the server resource queue s on the client to provide resource allocation for the new task, and sends the updated available resource information when returning the result to the client after completing the task; The first server whose remaining available resources are sufficient to execute the current task satisfies the following conditions: ①mem i ≥mem k ,c i ≥c k ,g i ≥g k ; ② Meet mem j ≥mem k , c j ≥c k , g j ≥g k ; Among them, mem i 、mem j and mem k Respectively represent the server s i Corresponding free memory, server s j The corresponding free memory and the memory required by the current task; c i 、c j and c k Respectively represent the server s i Corresponding CPU idle rate, server s j The corresponding CPU idle rate and the idle rate required by the current task; g i , g j and g k Respectively represent the server s i Corresponding CPU idle rate, server s j The corresponding CPU idle rate and the idle rate required by the current task; The corresponding server performs running processing of the application to be executed on the compressed data according to the adopted compression algorithm to obtain a stream processing result, including: For a given input data stream, assume that it has n data attributes <col1,col2,…,col n > and m sequences of operations to be processed <op1,op2,…,op m > Traverse the operation sequence, for each operator op i , if the operation has better performance on the GPU, the operation is assigned to the GPU, otherwise it is assigned to the CPU; The set of compression algorithms used by the corresponding server based on the one-to-one correspondence of the n data attributes sent by the client <τ1, τ2, ..., τ n > and the number of bytes occupied by the compressed data <bytes1,bytes2,…,bytes n >, update the calculation function, and process the corresponding operator based on the updated calculation function.

3. A processing device, the processing device comprising at least a processor and a memory, the memory storing a computer program, characterized in that: When the processor runs the computer program, the processor executes the steps of the stream processing method based on adaptive non-decompression direct calculation according to claim 1.

4. A computer storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the steps of the stream processing method based on adaptive non-decompression direct calculation according to claim 1.

Images