A batch-flow integrated data processing system and method

Abstract

This application provides a batch-stream integrated data processing system and method, relating to the field of data processing technology. The system includes: a memory abstraction module for managing serialized memory data according to a data computation mode; a serialization and deserialization module for converting between data objects and memory data; an SQL code generation module for generating vectorized SQL execution code based on the Pax storage structure from the memory data to be processed according to the data computation mode; and an operator module for executing the vectorized SQL execution code using the Pax storage structure-based data processing mode to process the memory data. This application effectively balances both stream computing and batch processing modes, retaining the low latency performance of stream processing while providing higher batch processing throughput, and optimizing the efficiency of window computation, thus improving the computational performance for specific window statistical scenarios.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a batch-stream integrated data processing system, method, electronic device, and computer program product. Background Technology

[0002] Flink employs row-based data processing for low latency, so its batch processing implementation is also row-based. The problem with row-based processing is that it is not friendly to CPU caches. CPU caches are prone to cache misses, causing the CPU to idle while waiting to load data from memory. This prevents the full utilization of the SIMD (Single Instruction Multiple Data) feature of modern CPUs, where one instruction can only process one piece of data. In contrast, column-based storage can fully utilize the SIMD feature of modern CPUs to accelerate computation.

[0003] Existing Flink implementations have the following problems: 1) It is difficult to balance batch and streaming performance. Columnar execution engines can greatly improve the performance of statistical computation, but once there are a large number of branch jump instructions, the computation efficiency will drop sharply, making it difficult to guarantee low latency. Using row execution engines, it is difficult to take advantage of the CPU's SIMD feature to improve computation efficiency. 2) Poor performance in specific window statistical scenarios in stream computing. In stream computing, the amount of data processed depends on the length of the window. With a large data scale, it may be impossible to complete the computation within the specified time, resulting in continuous data backlog and a continuous increase in data processing latency. Summary of the Invention

[0004] This application provides a batch-stream integrated data processing system, method, electronic device, and computer program product to solve the problems of difficulty in balancing batch and stream performance and poor performance in specific window statistical scenarios in the prior art.

[0005] In a first aspect, embodiments of this application provide a batch-stream integrated data processing system, comprising:

[0006] The memory abstraction module is used to manage the memory of serialized memory data within the smallest unit of Flink memory allocation, according to the current data computation mode; wherein, the data computation mode includes stream computing mode and batch processing mode, and the memory data adopts the Pax storage structure;

[0007] The serialization and deserialization modules are used to convert between data objects and in-memory data;

[0008] The SQL code generation module is used to generate vectorized SQL execution code based on the Pax storage structure from the memory data to be processed according to the current data calculation mode.

[0009] The operator module is used to execute the vectorized SQL execution code in a data processing mode based on the Pax storage structure to process the memory data to be processed.

[0010] In one embodiment, the operator module includes multiple operator units, each of which is used to: acquire an input data stream, extract data records from the data pages of the data stream based on the vectorized SQL execution code, process multiple data records in batches in a loop, write them to a new data page, and then output a data stream in units of data pages.

[0011] In one embodiment, the memory abstraction module is further configured to: select appropriate data page storage parameters to manage the memory data according to the current data calculation mode; wherein, the data page storage parameters include the number of data records, the maximum waiting time, and the default maximum memory space of the data page.

[0012] In one embodiment, the memory abstraction module is further configured to:

[0013] In stream computing mode, the number of data records in the memory data is adaptively adjusted in real time according to the current data backlog. Under the constraint of the default maximum memory space of the data page, memory management is performed on the memory data based on the maximum waiting time and the adjusted number of data records.

[0014] In batch processing mode, memory management is performed on the memory data based on the default maximum memory space of the data page.

[0015] In one embodiment, the memory abstraction module includes a data insertion management unit, a data update management unit, and a data deletion management unit, wherein:

[0016] The data insertion management unit is used to generate a new data page and allocate sub-data pages to the new data page according to the length of the data attribute value when inserting data, and then copy the attributes of the data record to be inserted to different sub-data pages respectively.

[0017] The data update management unit is used to, when updating data, if it is determined that the available space of the sub-data page is exceeded based on the offset of the attribute value to be updated in different sub-data pages, then apply for space from the adjacent sub-data page of the sub-data page; if it is not possible to apply for space from the adjacent sub-data page, then a new data page is applied for, and the data record to be updated is moved to the newly applied data page, while the updated data record in the old data page is marked as deleted.

[0018] The data deletion management unit is used to mark the data to be deleted in a preset bitmap of the data page when data is deleted. When the number of marked data records to be deleted exceeds a preset threshold, the marked data records to be deleted are deleted and a new data page is generated for data storage.

[0019] In one embodiment, the serialization and deserialization module includes a first serialization and deserialization unit and a second serialization and deserialization unit, wherein:

[0020] The first serialization and deserialization unit is used to serialize the received event data object and write it into the corresponding sub-data page according to the Pax storage structure in stream computing mode; it is also used to read the corresponding column value from the Pax storage structure and reassemble it into an event data object; wherein, in stream computing mode, each sub-data page stores a sequence of event data objects.

[0021] The second serialization and deserialization unit is used to serialize multiple received event data objects by column in batch processing mode and write them into the corresponding sub-data pages according to the Pax storage structure; it is also used to read the corresponding column values ​​from the Pax storage structure and reassemble them into event data objects; wherein, in batch processing mode, each sub-data page stores a sequence of multiple event data objects.

[0022] In one embodiment, the SQL code generation module is specifically used to: request the resources required for computation after initialization according to the parameters of the SQL operator; generate temporary variables required during the computation process, extract data from the Pax storage structure for the columns involved in the computation, traverse the computation to generate the corresponding data structure variables using a loop computation method; write the computation result back to the memory data of the Pax storage structure, and release the computation resources requested for this computation.

[0023] Secondly, embodiments of this application provide a batch-stream integrated data processing method, including:

[0024] Within the smallest unit of Flink memory allocation, the memory abstraction module manages the serialized memory data according to the current data computation mode. The data computation mode includes stream processing mode and batch processing mode, and the memory data uses the Pax storage structure.

[0025] Use the serialization and deserialization modules to convert between data objects and in-memory data;

[0026] The SQL code generation module generates vectorized SQL execution code based on the Pax storage structure from the memory data to be processed according to the current data calculation mode.

[0027] The operator module employs a data processing mode based on the Pax storage structure to execute the vectorized SQL execution code, thereby processing the memory data to be processed.

[0028] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the batch-stream integrated data processing method described in the second aspect.

[0029] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the batch-stream integrated data processing method described in the second aspect.

[0030] This application provides a batch-stream integrated data processing system, method, electronic device, and computer program product. The system adopts a row-column hybrid Pax storage structure, supports columnar computing, and fully utilizes the SIMD optimization of modern CPUs. This effectively balances both stream computing and batch processing modes, retains the low latency performance of stream processing, provides higher batch processing throughput, and optimizes the efficiency of window computing, thereby improving the computing performance of specific window statistics scenarios. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is one of the structural schematic diagrams of the batch-stream integrated data processing system provided in the embodiments of this application;

[0033] Figure 2 This is a second schematic diagram of the integrated batch and stream data processing system provided in the embodiments of this application;

[0034] Figure 3 This is a comparative diagram of NSM row storage, Pax hybrid storage, and Pax caching provided in the embodiments of this application;

[0035] Figure 4 This is a schematic diagram of the Pax row-column hybrid storage structure provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of the improved serialization / deserialization structure provided in the embodiments of this application;

[0037] Figure 6This is a comparative diagram of the data layout in Flink MemorySegment provided in the embodiments of this application;

[0038] Figure 7 This is a comparative diagram of Flink data processing modes provided in the embodiments of this application;

[0039] Figure 8 This is a flowchart illustrating the batch-stream integrated data processing method provided in the embodiments of this application;

[0040] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] Reference Figure 1 This application provides a batch-stream integrated data processing system, comprising:

[0043] Memory abstraction module 1 is used to manage the memory of serialized memory data within the smallest unit of Flink memory allocation, according to the current data computation mode; wherein, the data computation mode includes stream computing mode and batch processing mode, and the memory data adopts Pax storage structure;

[0044] Serialization and deserialization module 2 is used to convert between data objects and memory data;

[0045] SQL code generation module 3 is used to generate vectorized SQL execution code based on Pax storage structure from the memory data to be processed according to the current data calculation mode.

[0046] Operator module 4 is used to execute the vector-based SQL execution code in a data processing mode based on Pax storage structure to process the memory data to be processed.

[0047] It should be noted that the embodiments of this application are based on Flink and are based on the Pax row-column hybrid memory storage structure. Flink is transformed into a vectorized execution computing engine, that is, the batch-stream integrated data processing system provided in the embodiments of this application mainly includes a memory abstraction module 1, a serialization and deserialization module 2, an SQL code generation module 3, and an operator module 4.

[0048] The batch-stream integrated data processing system provided in this application adopts a row-column hybrid Pax storage structure, supports columnar computation, and fully utilizes the SIMD optimization of modern CPUs. This effectively balances both stream computing and batch processing modes, retains the low latency performance of stream processing, provides higher batch processing throughput, and optimizes the efficiency of window computation, thereby improving the computational performance of specific window statistical scenarios.

[0049] In one embodiment, the operator module 4 includes multiple operator units, each of which is used to: acquire an input data stream, extract data records from the data pages of the data stream based on the vectorized SQL execution code, process multiple data records in batches in a loop, write them into a new data page, and then output a data stream in units of data pages.

[0050] In this embodiment, the operator module 4 has multiple operator units. The operator units adopt a data processing mode based on the Pax storage structure. Compared with the prior art, the processing mode is improved from processing by strip to processing by data page, which can take into account both batch and stream computing modes and further improve the computing performance of the data processing system.

[0051] In one embodiment, the memory abstraction module 1 is further configured to: select appropriate data page storage parameters to manage the memory data according to the current data calculation mode; wherein, the data page storage parameters include the number of data records, the maximum waiting time, and the default maximum memory space of the data page.

[0052] In this embodiment, when managing memory data, parameters such as the number of data records, maximum waiting time, and default maximum memory space per data page are included. The system adaptively adjusts these storage strategy parameters according to different data computation modes (stream computation mode and batch processing mode). Therefore, this embodiment can accommodate both batch and stream computation modes, further improving the computational performance of the data processing system.

[0053] In one embodiment, the memory abstraction module 1 is further configured to:

[0054] In stream computing mode, the number of data records in the memory data is adaptively adjusted in real time according to the current data backlog. Under the constraint of the default maximum memory space of the data page, memory management is performed on the memory data based on the maximum waiting time and the adjusted number of data records.

[0055] In batch processing mode, memory management is performed on the memory data based on the default maximum memory space of the data page.

[0056] In this embodiment, in stream computing mode, to improve the throughput of stream computing, the number of data entries stored in a data page can be appropriately increased. In batch processing mode, the focus is on overall throughput, minimizing the overall execution time of a single job. There's no need to consider the latency of processing each individual data entry; the goal is to maximize data throughput. The default maximum memory space of the data page is used as the actual memory size of the data page, and data is continuously written to the data page until it is full. In batch processing mode, the maximum wait time parameter is ineffective; Pax data page generation only stops after data processing is complete and a message indicating that data input is finished is received.

[0057] The batch-stream integrated data processing system provided in this application adaptively adjusts preset parameters for data storage and retrieval according to the different focuses of batch and stream computing modes, thereby maintaining low latency and high throughput in stream computing mode and improving batch processing performance in batch processing mode.

[0058] In one embodiment, the memory abstraction module 1 includes a data insertion management unit, a data update management unit, and a data deletion management unit, wherein:

[0059] The data insertion management unit is used to generate a new data page and allocate sub-data pages to the new data page according to the length of the data attribute value when inserting data, and then copy the attributes of the data record to be inserted to different sub-data pages respectively.

[0060] The data update management unit is used to, when updating data, if it is determined that the available space of the sub-data page is exceeded based on the offset of the attribute value to be updated in different sub-data pages, then apply for space from the adjacent sub-data page of the sub-data page; if it is not possible to apply for space from the adjacent sub-data page, then a new data page is applied for, and the data record to be updated is moved to the newly applied data page, while the updated data record in the old data page is marked as deleted.

[0061] The data deletion management unit is used to mark the data to be deleted in a preset bitmap of the data page when data is deleted. When the number of marked data records to be deleted exceeds a preset threshold, the marked data records to be deleted are deleted and a new data page is generated for data storage.

[0062] In this embodiment, the improved Flink memory data is managed by the data insertion management unit, data update management unit and data deletion management unit, thereby supporting the Pax row-column hybrid memory structure. Thus, this embodiment can take into account both batch and stream computing modes, further improving the computing performance of the data processing system.

[0063] In one embodiment, the serialization and deserialization module 2 includes a first serialization and deserialization unit and a second serialization and deserialization unit, wherein:

[0064] The first serialization and deserialization unit is used to serialize the received event data object and write it into the corresponding sub-data page according to the Pax storage structure in stream computing mode; it is also used to read the corresponding column value from the Pax storage structure and reassemble it into an event data object; wherein, in stream computing mode, each sub-data page stores a sequence of event data objects.

[0065] The second serialization and deserialization unit is used to serialize multiple received event data objects by column in batch processing mode and write them into the corresponding sub-data pages according to the Pax storage structure; it is also used to read the corresponding column values ​​from the Pax storage structure and reassemble them into event data objects; wherein, in batch processing mode, each sub-data page stores a sequence of multiple event data objects.

[0066] In this embodiment, the first serialization and deserialization unit is suitable for the streaming computing model, where events are processed as soon as they arrive at the computing node, without waiting for data as in batch processing mode, where a batch of data is accumulated in a Pax data page before processing, resulting in lower data processing latency. The second serialization and deserialization unit is suitable for the batch processing computing model, in which: (1) data can be processed in batches, with a single CPU instruction processing multiple data entries, the specific number of data entries processed depending on the CPU instruction set; (2) data can be transmitted in batches over the network, resulting in better data throughput compared to the streaming mode.

[0067] In one embodiment, the SQL code generation module 3 is specifically used to: apply for the resources required for calculation after initialization according to the parameters of the SQL operator; generate temporary variables required during the calculation process, extract data from the Pax storage structure for the columns involved in the calculation, traverse the calculation to generate the corresponding data structure variables in a loop calculation manner; write the calculation result back to the memory data of the Pax storage structure, and release the resources required for calculation applied for in this calculation.

[0068] In this embodiment, the SQL code generation module is improved to generate code for the Pax row-column hybrid memory structure. During the calculation process, the CPU caches a data page, and there is no need to load data from memory during the calculation process, which can greatly improve the computation efficiency of Flink.

[0069] Based on the above solution, and to facilitate a better understanding of the batch-stream integrated data processing system provided in this application's embodiments, the following detailed explanation is provided:

[0070] It's important to note that in the field of big data computing, there are two main computing modes: batch processing and stream processing. Batch processing is geared towards T+1 scenarios, enabling OLAP (Online Analytical Processing) offline analysis, with throughput and overall computational efficiency as its core focuses. Stream processing, on the other hand, is geared towards T+0 scenarios, enabling real-time business processing and incremental statistical analysis, with low latency and high throughput as its core focuses. These two modes have slightly different priorities, leading to differences in the implementation of their computing engines.

[0071] Internally, big data computing engines employ two main implementations: row-based and column-based. NSM, commonly known as row-based storage, is widely used for storing OLTP data, such as in MySQL. NSM stores data records sequentially across disk pages, with the offset of each record stored at the end of each page for quick location. This format is highly efficient when retrieving only one row or iterating through all records. However, if a query only needs a single column (e.g., `select name from R where age < 40`), each iteration of the `age` column may incur cache misses due to the inclusion of irrelevant data. Row-based engines typically use the Volcano computing model, found in Flink and Spark (the default execution mode). In stream computing scenarios, row-based execution is generally preferred, processing events one at a time to minimize end-to-end latency.

[0072] Columnar Storage Model (DSM) is the underlying data storage method used by almost all OLAP systems. Unlike Columnar Storage Model (NSM), DSM stores data according to different attributes in different data pages. For queries that only require a single attribute, the data page is read directly and processed, which is very efficient and CPU-friendly. However, if a query involves multiple different attributes, multiple I / O operations may be needed to combine the final data records. Furthermore, for writes, DSM can cause more random I / O because it writes data corresponding to different records to different data pages. Columnar storage typically uses column-based vectorized execution engines, such as ClickHouse, Spark (Arrow execution mode), and DirusDB. To further improve data analysis efficiency, each batch processes a single data set (e.g., 100,000 records).

[0073] Understandably, column-based implementations are throughput-friendly, fully utilizing CPU cache and SIMD features. However, they don't adequately support the row-by-row processing mode of stream computing. Therefore, existing Flink computing engines use row-based implementations to ensure low latency in data processing. The problem with row-based implementations is their poor CPU cache friendliness; CPU caches are prone to cache misses, causing the CPU to idle while waiting to load data from memory. This prevents full utilization of the SIMD (Single Instruction Multiple Data) features of modern CPUs, where one instruction can only process one data item. Column-based implementations, on the other hand, can fully utilize the SIMD features of modern CPUs, accelerating computation.

[0074] Existing Flink computing engines suffer from difficulties in balancing batch and streaming performance, as well as poor performance in specific window statistics scenarios. This application proposes an integrated batch and streaming data processing system by combining the advantages of columnar computation in batch processing with the low-latency requirements of event-based processing in streaming scenarios. This system optimizes streaming processing while improving batch processing efficiency, ensuring full utilization of CPU cache and SIMD features in batch processing, while also guaranteeing low latency in streaming processing.

[0075] It is understood that the embodiments of this application draw on the idea of ​​Pax row-column hybrid storage in the database field, improve Flink's in-memory data format, solve the above-mentioned problems of difficulty in balancing batch and stream performance and poor performance in statistical scenarios in Window computing mode of stream computing, and improve the efficiency of real-time large-scale feature data computing.

[0076] like Figure 2 As shown, this application embodiment is based on Flink and uses the Pax row-column hybrid memory storage structure to transform Flink into a vectorized execution computing engine. The main improvements are to three core Flink modules: the SQL code generation module, the serialization / deserialization module, and the memory abstraction module. The API layer of Flink is not modified to maintain compatibility with existing Flink applications, and the data processing mode (operator system) of Flink is also improved accordingly.

[0077] It's important to note that the DataStream API and SQL&Table API are two APIs within Flink, relying on a common runtime environment. When reading data, the data from Kafka or HDFS is directly converted into a Pax row-column hybrid storage structure using a serializer / deserializer and stored in memory.

[0078] 1. Pax-based hybrid row-column memory storage structure:

[0079] The Pax row-column hybrid memory storage structure is the basis for this improvement and optimization of Flink. Only by implementing this module can the CPU's SIMD parallel computing capabilities be fully utilized, and the differences between batch and stream execution modes can be taken into account.

[0080] (1) Detailed Explanation of PAX Memory Structure

[0081] Row-based NSM and column-based DSM each have their own advantages and disadvantages. Pax aims to combine the advantages of both while avoiding their disadvantages.

[0082] PAX stands for Partition Attributes Across. It uses a sub-data page approach within a data page to split and store event attributes in different sub-data pages.

[0083] Figure 3 The left side is a row-based NSM storage structure, with four consecutive events in sequence in the data page. The middle is a Pax row-column hybrid storage structure, with each attribute stored consecutively. The right side is a CPU cache in Pax storage mode, where a column of data is loaded into the CPU cache. When performing calculations on the column, data can be directly retrieved from the CPU's L1, L2, and L3 caches, improving computational efficiency.

[0084] If an event object has n properties, PAX will divide the data page into n mini pages, and then place the first property on the first mini page, the second on the second mini page, and so on.

[0085] At the beginning of each data page, the offset of each sub-data page is stored. For fixed-length attribute data, fixed-length sub-data pages (F-minipages) are used for storage, while for variable-length attribute data, variable-length sub-data pages (V-minipages) are used. For fixed-length sub-data pages, a bit vector is used to store null values. For variable-length sub-data pages, the offset of each value within the sub-data page is stored.

[0086] As you can see, the PAX format is actually a compromise between NSM and DSM. When reading based on a specific column, it allows for convenient sequential scanning within sub-data pages, making full use of the CPU cache. And when accessing multiple properties to assemble the final object, it only requires reading the relevant data between sub-data pages within the same data page.

[0087] (2) Improvements based on Pax to adapt to the difference in the number of data rows per data page in Flink's streaming and batch computing modes:

[0088] In stream processing mode, Flink processes data one item at a time, while in batch processing mode, Flink processes data batch by batch. Based on the Pax memory structure, in this embodiment, Pax is configured for both stream and batch processing modes to ensure compatibility with both batch and stream execution.

[0089] To adapt to Flink's scenarios, this embodiment makes adaptive settings and improvements to Pax. In Flink, the data page of Pax mainly includes two parameters: the number of data records and the maximum waiting time. The system adaptively adjusts these parameters according to batch and streaming computing modes, and also allows manual intervention in setting the data page parameters.

[0090] The number of data records, denoted as N, refers to how many data records can be stored in a data page (N is greater than or equal to 1, meaning at least 1 data record must be stored). The size of the records that can be stored in a data page is limited by the data page space. It is generally recommended that the maximum default memory space of the data page = Min(2M, CPU L2 cache size / (2 * number of CPU cores)).

[0091] Understandably, if a data page is too large, it cannot fit into the CPU's cache; if it is too small, it is difficult to utilize the CPU's SIMD features.

[0092] The actual memory size of a data page = (N * average record length) * (1 + P) + other overhead, which includes the data page header, BitMap, Offset pointer, etc. Here, P is the inflation factor, which defaults to 5%–10% to avoid the actual record length exceeding the average record length, which would prevent the data page from accommodating the data and lead to frequent requests for new data pages, thus impacting performance.

[0093] During execution, the size of the data page storage is dynamically adjusted according to the actual data length. It is generally recommended not to exceed the default maximum memory space of the data page, except in special cases, such as manual optimization by the user.

[0094] In stream computing mode:

[0095] By default, Pax data pages store only one data record. To improve the throughput of stream computing, the number of data records stored in a data page can be increased appropriately.

[0096] Adjust the N value based on whether back pressure (data backlog) occurs.

[0097] 1) If back pressure is generated, adjust the number of data records N = N*2, recalculate the actual memory size of the data page, process more data at a time, and reduce data backlog;

[0098] 2) If the system recovers from backpressure or executes normally, the value of N should be appropriately reduced (N = N / 2) to recalculate the actual memory size of the data page, thereby processing data faster and reducing data latency.

[0099] To ensure the lowest possible latency, the system uses two dimensions: the number of data records and the timeout period.

[0100] R is set to whether the data page is sent downstream, true for sending, false for not sending and continuing to write data, P size T represents the number of records in the current data page, S is the maximum number of records that the data page can hold, and T represents the maximum number of records that the data page can hold. timeout The maximum allowable waiting time is calculated using the following formula:

[0101]

[0102] For example, if the maximum wait time is 100 milliseconds and a data page can hold 100 data records, then when the wait time exceeds 100 milliseconds, the data will be sent downstream regardless of whether there are actually 100 records on the page, thus preventing excessively long wait times. If the number of data records reaches 100, the data will still be sent downstream even if the maximum wait time has not yet been reached, reducing data processing latency.

[0103] In batch processing mode:

[0104] The focus is on overall throughput, minimizing the overall execution time of a single job. There is no need to worry about the latency of processing each piece of data. The goal is to maximize data throughput as much as possible. The default maximum memory space of the data page is used as the actual memory size of the data page. N has no default value. Data is continuously written to the data page until the data page is full.

[0105] The maximum wait time parameter is not effective in batch processing. Pax data pages will only stop being generated and sent downstream for batch data processing after the data processing is completed and a message indicating that the data input is complete is received.

[0106] 2. Memory Abstraction

[0107] Flink employs its own internal management system. MemorySegment is the smallest unit of memory allocation in Flink. This object represents a 32K block of off-heap memory by default (which can be configured to other sizes as needed).

[0108] After data records are serialized, they are organized in memory according to the Pax data page format, and MemorySegment is used as the container for Pax data pages. A Pax data page may store a large number of records (in batch processing mode), so Pax data pages may use multiple MemorySegments for storage.

[0109] To support Pax row-column mixed memory structures, Flink needs to be improved to support the following Pax data page operations:

[0110] (1) Insert

[0111] When data is inserted, a new data page is first generated, and then different sub-data pages are allocated according to the length of the attribute value. Note the variable-length attribute values ​​here, because their length is not fixed. Therefore, for variable-length attributes, the system-set default length is used first, and the average length of the attribute value is continuously calculated as the job is executed. The average length is used as the basis for subsequent space allocation.

[0112] by Figure 4 For example, when inserting a data record, the attributes (columns) of that data record are copied to different sub-data pages. If a data record can still be inserted into this data page, but the data for a certain attribute of that data record cannot be inserted into the corresponding sub-data page, the boundaries of the different sub-data pages will be readjusted. If a data page is full, a new Pax data page will be generated to store the new data.

[0113] (2) Update

[0114] When data is updated, the offset of the attribute value to be updated in the data record within different sub-data pages is first calculated. Figure 4 For example, for variable-length attributes, if the size of the updated data exceeds the available space of the child data page, such as when the field "John" is updated to a 1K-length string, exceeding the available space of the child data page, the child data page will try to request space from surrounding child data pages. If the adjacent child data pages do not have enough extra space, a new Pax data page will be requested, and the data record to be updated will be moved to the newly requested data page. The record to be updated will be marked as deleted in the old data page.

[0115] (3) Delete

[0116] At the beginning of each data page is a bitmap used to mark deleted data. When data is deleted, it is not actually removed, but rather the deleted data record is marked in the bitmap.

[0117] As the number of deletion markers increases, performance may be affected because many fragments appear in the data pages, making it difficult to efficiently utilize the CPU cache. Therefore, during the shuffle process, if the number of records to be deleted in the Pax data page exceeds a certain threshold, the data marked for deletion is deleted, and a new data page is generated.

[0118] Number of records P in the Pax data page size The cleanup threshold is P. threshold The number of records marked as deleted, P. del_size .

[0119] When (P) del_size / P size )>P threshold If the number of deleted data in the data page exceeds the expected amount, then cleanup will be performed and a new data page will be generated; otherwise, no cleanup will be performed.

[0120] 3. Serializer / Deserializer

[0121] Since development is done using Flink's existing DataStream API, developers need to write business logic processing function code, with the development languages ​​being Java, Scala, and Python.

[0122] (1) The role of serialization / deserialization in Flink

[0123] First, let me explain the role of serialization / deserialization in Flink. The three programming languages ​​mentioned above are all object-oriented, meaning their code parameters are primitive data types and object types defined within that language. User-written code also manipulates objects defined within that language. However, network transmission uses binary data, so serialization / deserialization is needed as an intermediary. During data processing, binary data is deserialized into objects (built-in or user-defined Event objects), and during data transmission, data is serialized into a binary data stream.

[0124] In existing Flink implementations, event serialization and deserialization are implemented using its built-in serializer and user-defined serializers. For example, the string serializer (StringSerializer) is used to serialize the string "HelloWorld!" into a binary data stream, which is then stored in a MemorySegment for network transmission. When needed, deserialization is performed, reading the binary data stream from the MemorySegment and deserializing it back into the string "HelloWorld!".

[0125] (2) Flink's existing line-based serialization

[0126] In Flink, a memory segment object is called a MemorySegment, which represents a contiguous block of memory addresses. When using row-based serialization, Flink serializes the field values ​​into binary data according to the field order of the object and stores them contiguously.

[0127] (3) Pax-based adaptation to Flink serialization / deserialization

[0128] The serialization / deserialization of a single attribute (i.e., column) can still reuse Flink's existing serializer, but Flink's memory data layout method needs to be improved so that the attribute values ​​after event serialization can be written column by column into the sub-data pages of the Pax memory structure, and binary data can be read from the Pax storage structure, deserialized and assembled into an event object.

[0129] like Figure 5 As shown, the left side is the original data, and the right side is the Pax row and column mixed memory storage structure. Event 1 and Event 2 on the left are stored according to the attribute column using the storage structure defined by Pax.

[0130] It is understandable that, compared with the prior art, the data organization method is completely different based on the improved serialization. In the prior art, data is organized by row, while in the embodiments of this application, data is organized by column.

[0131] The Pax data layout approach is combined with Flink's MemorySegment-based memory management, where MemorySegment is responsible for memory allocation and deallocation, and Pax is responsible for data layout within MemorySegment.

[0132] like Figure 6 As shown in the diagram, the left side shows Flink's existing row-based data organization method, which uses data records as units. After the data is serialized in sequence, it is stored in MemorySegment one by one. Deserialization is similar, reading data records one by one in sequence.

[0133] The right side of the diagram illustrates Flink's columnar data organization based on Pax. Pax data pages are used as units. Each Pax data page instance is created by allocating a memory segment from a MemorySegment. When memory is insufficient, the system allocates a new MemorySegment and assigns it to the Pax data page. After data serialization, it is written sequentially to the memory segment allocated by Pax. Similarly, data is read in units of Pax data pages, by reading a complete Pax data page from a MemorySegment and deserializing it.

[0134] Serialization / deserialization in streaming mode:

[0135] Receive one event, serialize its data, and write it to the corresponding sub-data page according to the Pax storage structure. This can be understood as one event occupying one Pax data page. Deserialization is to read the corresponding column values ​​from the Pax storage structure and reassemble them into an event object.

[0136] The computational model is suitable for stream processing. Events are processed as soon as they arrive at the computation node, without having to wait for data like in batch processing mode, where a batch of data is accumulated in a Pax data page before processing, resulting in lower data processing latency.

[0137] Serialization / deserialization in batch mode:

[0138] In batch processing mode, multiple events are received, their data is serialized column by column, and written to the corresponding sub-data pages according to the Pax storage structure. This can be understood as storing multiple events in one data page. Deserialization reads the corresponding column values ​​from the Pax storage structure and reassembles them into event objects.

[0139] The computing model is suitable for batch processing. In this mode, (1) data can be processed in batches, and a single CPU instruction can process multiple data. The specific number of data processed depends on the CPU instruction set. (2) Data can be transmitted in batches over the network, which has better data throughput than the streaming mode.

[0140] 4. Flink SQL code generation

[0141] (1) Existing code generation solutions

[0142] For efficiency, Flink's SQL engine uses code generation to convert SQL logic into Java code, compile it into Java binary code in memory, and then execute it, avoiding the use of multi-level function calls to execute SQL logic.

[0143] The following SQL statement is an example:

[0144] select c1+100,c2 from t where c1<100and c3=10

[0145] Using Flink's code generation, simplified Java expression operations can be directly generated. The WHERE condition in the above SQL is converted into the following Java code, which is dynamically compiled and executed in the JVM: (c1<100&&(c3==10))

[0146] It should be noted that the above is Java pseudocode, mainly to illustrate the core mechanism of dynamic code generation. Because Java is an object-oriented language, in practice, the code still needs to be wrapped in the methods of a class in order to be legally compiled into Java bytecode.

[0147] In Flink's existing code generation logic, data is processed one line at a time by operators. After improvements based on Pax, data is processed in batches (in streaming mode, data is processed as single or multiple lines).

[0148] (2) SQL code generation based on Pax

[0149] Flink's existing SQL engine can directly access binary data, avoiding the overhead of serialization and deserialization during data processing. It reads binary data bit by bit and uses binary data for computation. Therefore, for mixed row and column memory structures, data processing can be performed directly according to different patterns. In SQL code generation, it is necessary to consider code generation for Pax mixed row and column memory structures and replace Flink's existing SQL code generation logic.

[0150] During code generation, the system identifies different Java versions and generates optimal Java code. For Java 16 and above (JDK 16 and later versions support optimization for specified SIMD instruction sets), it uses Java's vectorized computation API and performs compilation optimization for SIMD instruction sets (such as SSE, SSE2, SSE4, AVX, etc.) to leverage the CPU's SIMD capabilities. For Java 15 and earlier versions, code based on vector computation is sufficient, and the JVM can automatically optimize loop calculations in the code. Compilation optimization is combined with the efficient CPU caching provided by the Pax row-column hybrid memory structure. The basic code generation process is as follows:

[0151] 1) Environment initialization section

[0152] Initialize according to different SQL operators, such as initializing the parameters of different Join algorithms; allocate additional resources such as memory, temporary disk storage, and file handles required for external sorting during the calculation process;

[0153] 2) Calculation function

[0154] Generate temporary variables required during the calculation process; for the columns involved in the calculation, extract data from Pax to generate corresponding data structure variables;

[0155] During the calculation process, the data in Pax is accessed column by column, and the calculation is performed by iterating through the data in a loop.

[0156] 3) Output of calculation results

[0157] Write the calculation results back to the Pax structure;

[0158] 4) Environmental Destruction

[0159] Release all relevant resources requested for this computation.

[0160] For example, consider the following SQL statement:

[0161] select c1+100,c2 from t where c1<100and c3=10

[0162] Assuming the Pax storage data structure is old_PaxData, the core pseudocode processing is shown here, omitting the serialization process of reading and writing data from Pax.

[0163] Example of core execution code based on Pax code generation:

[0164] long[]c1_100_arr=new long[old_PaxData.length()];

[0165] long[]c2_arr=new long[old_PaxData.length()];

[0166] / / Read data stored in Pax and perform loop calculations

[0167] for(int i = 0; i <old_PaxData.length();i++){

[0168] / / Read column values ​​from Pax and perform calculations in a loop.

[0169] if (c1 < 100 && (c3 == 10))

[0170] c1_100_arr[i] = c1 + 100

[0171] c2_arr[i] = c2;

[0172] }

[0173] / / Write the calculation results to Pax in batches, assuming the new PaxData is new_PaxData

[0174] / / Write the calculated column value arrays c1_100_arr and c2_arr to...

[0175] writePaxData(newPaxData,c1_100_arr,c2)

[0176] During the above calculation process, the CPU caches a data page, eliminating the need to load data from memory during the calculation, which can greatly improve Flink's computational efficiency.

[0177] 5. Improvements to Flink data processing mode

[0178] After Pax, Flink's data processing model changed, such as... Figure 7 As shown, data processing has been improved from processing by row to processing by data page. Operators are containers for Flink's data processing and are responsible for executing business logic code (i.e., the dynamically generated code in the SQL code generation mentioned in this embodiment). Figure 7 The left side shows Flink's existing data processing mode, where data enters the operator one by one, is calculated, and then outputs to the downstream in rows. Figure 7 The right side is a computational model based on Pax. A group of data records in a data page enters the operator together. The business logic code retrieves the data records from the data page, processes multiple data records in batches in a loop, and then writes them to a new data page. The data page is then sent to the downstream operator for processing.

[0179] Compared with the prior art, the embodiments of this application adopt a row-column hybrid memory data structure, which supports columnar computation and makes full use of the SIMD optimization of modern CPUs. Compared with the existing Flink implementation, it takes into account the low latency of stream processing, provides higher batch processing throughput, and optimizes the efficiency of window computation.

[0180] The batch-stream integrated data processing method provided in the embodiments of this application is described below. The batch-stream integrated data processing method described below can be referred to in correspondence with the batch-stream integrated data processing system described above.

[0181] Please see Figure 8 This application provides an integrated data processing method, including:

[0182] S1. Within the smallest unit of Flink memory allocation, the memory abstraction module manages the serialized memory data according to the current data computation mode. The data computation mode includes stream computing mode and batch processing mode, and the memory data uses the Pax storage structure.

[0183] S2. Use the serialization and deserialization modules to convert between data objects and memory data;

[0184] S3. The SQL code generation module generates vector-based SQL execution code based on the Pax storage structure from the memory data to be processed according to the current data calculation mode.

[0185] S4. Using the operator module and a data processing mode based on the Pax storage structure, execute the vector-based SQL execution code to process the memory data to be processed.

[0186] It is understood that the above method embodiments correspond to the system embodiments of this application. The batch-stream integrated data processing method provided by the embodiments of this application can realize the batch-stream integrated data processing system provided by any system embodiment of this application.

[0187] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call a computer program stored in the memory 930 to execute the steps of the batch-stream integrated data processing method, such as including:

[0188] S1. Within the smallest unit of Flink memory allocation, the memory abstraction module manages the serialized memory data according to the current data computation mode. The data computation mode includes stream computing mode and batch processing mode, and the memory data uses the Pax storage structure.

[0189] S2. Use the serialization and deserialization modules to convert between data objects and memory data;

[0190] S3. The SQL code generation module generates vector-based SQL execution code based on the Pax storage structure from the memory data to be processed according to the current data calculation mode.

[0191] S4. Using the operator module and a data processing mode based on the Pax storage structure, execute the vector-based SQL execution code to process the memory data to be processed.

[0192] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0193] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the batch-stream integrated data processing method provided in the above embodiments.

[0194] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments.

[0195] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A batch-stream integrated data processing system, characterized in that, include: The memory abstraction module is used to manage the memory of serialized memory data within the smallest unit of Flink memory allocation, according to the current data computation mode; wherein, the data computation mode includes stream computing mode and batch processing mode, and the memory data adopts the Pax storage structure; The serialization and deserialization modules are used to convert between data objects and in-memory data; The SQL code generation module is used to generate vectorized SQL execution code based on the Pax storage structure from the memory data to be processed according to the current data calculation mode. The operator module is used to execute the vectorized SQL execution code in a data processing mode based on the Pax storage structure to process the memory data to be processed. The memory abstraction module includes a data insertion management unit, a data update management unit, and a data deletion management unit, wherein: The data insertion management unit is used to generate a new data page and allocate sub-data pages to the new data page according to the length of the data attribute value when inserting data, and then copy the attributes of the data record to be inserted to different sub-data pages respectively. The data update management unit is used to, when updating data, if it is determined that the available space of the sub-data page is exceeded based on the offset of the attribute value to be updated in different sub-data pages, then apply for space from the adjacent sub-data page of the sub-data page; if it is not possible to apply for space from the adjacent sub-data page, then a new data page is applied for, and the data record to be updated is moved to the newly applied data page, while the updated data record in the old data page is marked as deleted. The data deletion management unit is used to mark the data to be deleted in a preset bitmap of the data page when data is deleted. When the number of marked data records to be deleted exceeds a preset threshold, the marked data records to be deleted are deleted and a new data page is generated for data storage.

2. The batch-stream integrated data processing system according to claim 1, characterized in that, The operator module includes multiple operator units, each of which is used to: acquire the input data stream, extract data records from the data pages of the data stream based on the vectorized SQL execution code, process multiple data records in batches in a loop, write them into a new data page, and then output a data stream in units of data pages.

3. The batch-stream integrated data processing system according to claim 1, characterized in that, The memory abstraction module is further configured to: select appropriate data page storage parameters to manage the memory data according to the current data calculation mode; wherein, the data page storage parameters include the number of data records, the maximum waiting time, and the default maximum memory space of the data page.

4. The batch-stream integrated data processing system according to claim 3, characterized in that, The memory abstraction module is further used for: In stream computing mode, the number of data records in the memory data is adaptively adjusted in real time according to the current data backlog. Under the constraint of the default maximum memory space of the data page, memory management is performed on the memory data based on the maximum waiting time and the adjusted number of data records. In batch processing mode, memory management is performed on the memory data based on the default maximum memory space of the data page.

5. The batch-stream integrated data processing system according to claim 1, characterized in that, The serialization and deserialization module includes a first serialization and deserialization unit and a second serialization and deserialization unit, wherein: The first serialization and deserialization unit is used to serialize the received event data object and write it into the corresponding sub-data page according to the Pax storage structure in stream computing mode; it is also used to read the corresponding column value from the Pax storage structure and reassemble it into an event data object; wherein, in stream computing mode, each sub-data page stores a sequence of event data objects. The second serialization and deserialization unit is used to serialize multiple received event data objects by column in batch processing mode and write them into the corresponding sub-data pages according to the Pax storage structure; it is also used to read the corresponding column values ​​from the Pax storage structure and reassemble them into event data objects; wherein, in batch processing mode, each sub-data page stores a sequence of multiple event data objects.

6. The batch-stream integrated data processing system according to claim 1, characterized in that, The SQL code generation module is specifically used for: applying for the resources required for calculation after initialization according to the parameters of the SQL operator; generating temporary variables required during the calculation process, extracting data from the Pax storage structure for the columns involved in the calculation, traversing the calculation to generate the corresponding data structure variables in a loop calculation manner; writing the calculation results back to the memory data of the Pax storage structure, and releasing the resources required for calculation applied for in this calculation.

7. A batch-stream integrated data processing method, characterized in that, The batch-stream integrated data processing system as described in any one of claims 1 to 6 includes: Within the smallest unit of Flink memory allocation, the memory abstraction module manages the serialized memory data according to the current data computation mode. The data computation mode includes stream processing mode and batch processing mode, and the memory data uses the Pax storage structure. Use the serialization and deserialization modules to convert between data objects and in-memory data; The SQL code generation module generates vectorized SQL execution code based on the Pax storage structure from the memory data to be processed according to the current data calculation mode. The operator module employs a data processing mode based on the Pax storage structure to execute the vectorized SQL execution code, thereby processing the memory data to be processed.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the batch-stream integrated data processing method of claim 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the batch-stream integrated data processing method as described in claim 7.

Images