A method for managing operator temporary tables

Abstract

The application discloses a management method of an operator temporary table, and relates to the technical field of data management; the method comprises the following steps: step 1, creating a temporary table according to operator requirements; step 2, dividing the block size of the temporary table to correspond to the block size of physical memory division; step 3, defining that the block of the temporary table comprises a 4-byte integer bit and a data block DataChunk; step 4, defining that the DataChunk adopts columnar storage layout; step 5, writing data in a block unit through a writing interface of the temporary table; step 6, reading temporary table data through a reading interface of the temporary table; and step 7, deleting the temporary table and releasing the memory occupied by the temporary table when operator execution ends or the temporary table is no longer needed.

Description

Technical Field

[0001] This invention discloses a method for managing operator temporary tables, which relates to the field of data management technology. Background Technology

[0002] Temporary tables are generally used to store temporary data or intermediate results during computation. In database operations, especially blocking operations such as Sort and GroupBy, if the amount of data being processed is large, a temporary table is typically created to store the data on disk. The data is then written to the temporary table, read from it when needed, and finally deleted when no longer required. Existing temporary tables generally provide file read / write interfaces. When writing data, the operator needs to serialize the memory structure of the data to be written and then call the temporary table file write interface to write it to disk. When reading data, the operator calls the temporary table file read interface to read the data from disk and then deserializes it into the memory structure required for the operator's operation. However, serialization and deserialization are inefficient and time-consuming operations, which are detrimental to improving the efficiency of reading and writing data, especially when the data structure of the operator's operation and the data structure of the temporary table's operation do not match. Summary of the Invention

[0003] This invention addresses the problems of existing technologies by providing a method for managing operator temporary tables, thereby improving read and write performance.

[0004] The specific solution proposed in this invention is as follows:

[0005] This invention provides a method for managing operator temporary tables, comprising:

[0006] Step 1: Create a temporary table according to the operator requirements;

[0007] Step 2: The size of the temporary table blocks should correspond to the size of the physical memory blocks;

[0008] Step 3: Define a temporary table block containing a 4-byte integer and a data block DataChunk. The 4-byte integer records the number of rows of data actually stored in the DataChunk.

[0009] Step 4: Define the DataChunk to use a columnar storage layout and store different rows of data in the same column together. Calculate the maximum number of rows that the DataChunk can store based on the size of the DataChunk and the data type of the columns to be stored.

[0010] Step 5: Write data in blocks through the temporary table's write interface: First, obtain a free block. If the free block is not full, it has been converted into a DataChunk for operator operations using the positioning new method and placed in the cache. Calculate the number of remaining data rows in the free block and copy the data to be written to the free block. If this write causes the free block to become full, the full free block is removed from the cache; otherwise, the not-full free block is kept in the cache for future writes. If the free block is empty, it is converted into a DataChunk for operator operations using the positioning new method, and the data to be written is copied to the free block. If the empty block is filled at once, it is not placed in the cache; otherwise, it is placed in the cache for future writes.

[0011] Step 6: Read temporary table data through the temporary table read interface: Methods for reading temporary table data include sequential read and random read.

[0012] Using a sequential read method, the operator increments the block number to be read one by one, and then the temporary table read interface is called to obtain the block to be read.

[0013] Using a random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows of data that the temporary table can store, and then calls the temporary table read interface to obtain the block to be read;

[0014] Step 7: When the operator finishes execution or the temporary table is no longer needed, delete the temporary table and release the memory occupied by the temporary table.

[0015] Furthermore, step 1 of the method for managing temporary tables for operators includes creating temporary tables according to operator requirements, which includes:

[0016] The system determines whether the amount of data that the operator needs to temporarily store has reached or exceeded a pre-set threshold. If so, it either creates a temporary table or enables an existing temporary table.

[0017] Furthermore, in step 4 of the aforementioned method for managing operator-based temporary tables, based on the maximum number of rows of data that a DataChunk in a block of the temporary table can store, the method estimates when the block of the temporary table will be full, and during writing, it splits the DataChunk to be written until the block of the temporary table is full.

[0018] When defining a DataChunk using a columnar storage layout, each column is set to start with an N-bit BitMap to identify whether the data in the corresponding row of the column is NULL. Data with NULL values ​​does not occupy storage space.

[0019] Furthermore, in step 4 of the aforementioned method for managing operator temporary tables, when calculating the value of N, the value of N is: (Capacity / 8+1)*8, where Capacity is the maximum number of rows of data that DataChunk can store.

[0020] Furthermore, in step 6 of the operator temporary table management method, when reading temporary table data through the temporary table read interface, the block of the temporary table to be read is converted into a DataChunk of the operator operation using the positioning new method, and then the block of the temporary table to be read is put into the cache for easy reading next time.

[0021] This invention provides a management device for operator temporary tables, including a creation module, a management module, a writing module, and a reading module.

[0022] The creation module creates temporary tables based on operator requirements;

[0023] The size of the temporary table allocated by the management module corresponds to the size of the physical memory block.

[0024] The management module defines that a temporary table block contains a 4-byte integer and a data block DataChunk. The 4-byte integer records the number of rows of data actually stored in the DataChunk.

[0025] The management module defines that DataChunk adopts a columnar storage layout and stores different rows of data in the same column together. It calculates the maximum number of rows that DataChunk can store based on the size of DataChunk and the data type of the data column to be stored.

[0026] The write module writes data in blocks through the temporary table's write interface: First, it acquires a free block. If the free block is not full, it is converted into a DataChunk for operator operations using the positioning new method and placed in the cache. The remaining number of data rows stored in the free block is calculated, and the data to be written is copied to the free block. If this write causes the free block to become full, the full free block is removed from the cache; otherwise, the not-full free block is kept in the cache for future writes. If the free block is empty, it is converted into a DataChunk for operator operations using the positioning new method, and the data to be written is copied to the free block. If the empty block is filled in one go, it is not placed in the cache; otherwise, it is placed in the cache for future writes.

[0027] The read module reads data from the temporary table through the temporary table's read interface. Reading methods for the temporary table include sequential read and random read.

[0028] Using a sequential read method, the operator increments the block number to be read one by one, and then calls the temporary table read interface to obtain the block to be read.

[0029] Using a random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows of data that the temporary table can store, and then calls the temporary table read interface to obtain the block to be read;

[0030] When the operator finishes execution or the temporary table is no longer needed, the management module deletes the temporary table and releases the memory occupied by the temporary table.

[0031] Furthermore, the creation module of the operator temporary table management device creates a temporary table according to operator requirements, including:

[0032] The system determines whether the amount of data that the operator needs to temporarily store has reached or exceeded a pre-set threshold. If so, it either creates a temporary table or enables an existing temporary table.

[0033] Furthermore, the management module of the operator temporary table management device estimates when the temporary table block will be full based on the maximum number of rows that the DataChunk in the temporary table block can store, and splits the DataChunk to be written until the temporary table block is full during the writing process.

[0034] When defining a DataChunk using a columnar storage layout, each column is set to start with an N-bit BitMap to identify whether the data in the corresponding row of the column is NULL. Data with NULL values ​​does not occupy storage space.

[0035] Furthermore, when the management module of the management device for the operator temporary table calculates the value of N, the value of N is: (Capacity / 8+1)*8, where Capacity is the maximum number of rows of data that DataChunk can store.

[0036] Furthermore, when the reading module of the operator temporary table management device reads temporary table data through the temporary table reading interface, it uses the positioning new method to convert the block of the temporary table to be read into the DataChunk of the operator operation, and then puts the block of the temporary table to be read into the cache for easy reading next time.

[0037] The advantages of this invention are:

[0038] This invention innovates the temporary table implementation structure, defines the temporary table's block and DataChunk structure, as well as the read and write interface, replacing inefficient and time-consuming operations such as serialization and deserialization, which helps improve the efficiency of reading and writing data. Attached Figure Description

[0039] Figure 1 This is a mapping diagram of temporary table disk blocks (Blocks) and data chunks (Chunks).

[0040] Figure 2 It is a mapping diagram of temporary tables with virtual memory and physical memory.

[0041] Figure 3 This is a flowchart of the temporary table application process. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0043] Example 1

[0044] This invention provides a method for managing operator temporary tables, comprising:

[0045] Step 1: Create a temporary table according to the operator requirements.

[0046] The system determines whether the amount of data that the operator needs to temporarily store has reached or exceeded a pre-set threshold. If so, it either creates a temporary table or enables an existing temporary table.

[0047] Before enabling a temporary table, the operator temporarily stores the data that needs to be temporarily stored in memory and calculates the size of the temporarily stored data. As the operator executes, the amount of data that needs to be temporarily stored accumulates, eventually reaching a pre-set threshold, at which point the operator decides to enable the temporary table. If, by the time the operator finishes execution, the amount of data that needs to be temporarily stored has not reached the pre-set threshold, the operator may not enable the temporary table. For guidance on enabling temporary tables, please refer to [reference needed]. Figure 3 In the process position ①, the mmap system call can be called to open the temporary table in file mapping mode. If the operator has data that needs to be temporarily stored, the temporary table write interface can be called directly to store it in the temporary table.

[0048] Step 2: The size of the temporary table blocks must correspond to the size of the physical memory blocks. The mapping relationship between temporary table blocks, virtual memory blocks, and physical memory blocks can be found in [reference needed]. Figure 2 .

[0049] Step 3: Define a temporary table block containing a 4-byte integer and a data block DataChunk. The 4-byte integer records the number of rows of data actually stored in DataChunk.

[0050] The layout of the DataChunk in the temporary table block can be referenced. Figure 1The section with the black border in the middle. Its layout is identical to the memory DataChunk used for operator computation and inter-operator data transfer. This not only enables efficient writing and reading of temporary tables but also simplifies operator implementation and unifies operator operations on memory and disk data structures. However, the DataChunk within the temporary table block and the memory DataChunk used by the operator do not necessarily need to be the same size.

[0051] Step 4: Define the DataChunk to use a columnar storage layout and store different rows of data in the same column together. Calculate the maximum number of rows that the DataChunk can store based on the size of the DataChunk and the data type of the data column to be stored.

[0052] This involves estimating when the temporary table's blocks will be full based on the maximum number of rows that a DataChunk in a block can store, and then splitting the DataChunk to be written into segments until the temporary table's blocks are full during the write process.

[0053] When defining a DataChunk using a columnar storage layout, each column begins with an N-bit Bitmap to identify whether the data in the corresponding row is NULL. NULL values ​​do not occupy storage space. The value of N is calculated as (Capacity / 8 + 1) * 8, where Capacity is the maximum number of rows that the DataChunk can store.

[0054] Step 5: Write data in blocks using the temporary table's write interface:

[0055] The write interface for a temporary table can be represented as:

[0056] void Append(DataChunk*chunk);

[0057] Here, chunk is the DataChunk data structure used by the operator, representing the temporary data that the operator needs to write.

[0058] First, a free block is acquired. If the free block is not full, it has been converted into a DataChunk for operator operations using the positioning new method and placed in the cache. The number of data rows remaining in the free block is calculated, and the data to be written is copied to the free block. If this write causes the free block to become full, the full free block is removed from the cache; otherwise, the not-full free block is kept in the cache for future writes. If the free block is empty, it is converted into a DataChunk for operator operations using the positioning new method, and the data to be written is copied to the free block. If the empty block is filled in one go, it is not placed in the cache; otherwise, it is placed in the cache for future writes.

[0059] For details on write operations, please refer to [reference]. Figure 3 The process is located at step ②. Because the DataChunk is converted using a location-based `new` approach, no memory is allocated to the system during the write process. This not only improves efficiency but also simplifies error handling logic. Writes are performed using block copying, avoiding time-consuming serialization operations.

[0060] Free blocks include both empty and partially filled blocks. If there are no free blocks, the temporary table can be expanded first. The number of blocks expanded at a time depends on the specific situation: one block can be expanded at a time, maximizing disk utilization and minimizing memory usage, but this increases the number of temporary table file remapping operations, as each expansion requires remapping, which is a time-consuming process; alternatively, several, dozens, or even hundreds of blocks can be expanded at once, reducing the number of file remapping operations, but some of the expanded blocks may ultimately be unused, wasting disk space and memory.

[0061] Step 6: Read data from the temporary table through its read interface: The temporary table read interface can be represented as follows:

[0062] void Get(int chunkIdx, DataChunk**chunk);

[0063] Where chunkIdx is the block number of the temporary table to be read; chunk is the identifier of the data to be read, which is used by the operator.

[0064] Temporary tables can be read in two ways: sequential read and random read.

[0065] Using a sequential read method, the operator increments the block number to be read one by one, and then the temporary table read interface is called to obtain the block to be read.

[0066] Using a random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that the temporary table can store, and then calls the temporary table read interface to obtain the block to be read.

[0067] See read operation Figure 3 The process is described in step ③. During operation, the `new` method is used to convert the temporary table block to be read into a `DataChunk` for the operator operation, and then it is placed in the cache for easy retrieval later. This completely avoids the tedious and time-consuming deserialization operation, and no memory is requested from the system during the operation.

[0068] Step 7: When the operator finishes execution or the temporary table is no longer needed, delete the temporary table and release the memory it occupies. (See...) Figure 3 In step ④ of the process, when deleting a temporary table, the operator can first call the munmap system call to end the file mapping, release the memory occupied by the temporary table, and then delete the temporary table file to release the disk space occupied by the temporary table.

[0069] Example 2

[0070] This invention provides a management device for operator temporary tables, including a creation module, a management module, a writing module, and a reading module.

[0071] The creation module creates temporary tables based on operator requirements;

[0072] The size of the temporary table allocated by the management module corresponds to the size of the physical memory block.

[0073] The management module defines that a temporary table block contains a 4-byte integer and a data block DataChunk. The 4-byte integer records the number of rows of data actually stored in the DataChunk.

[0074] The management module defines that DataChunk adopts a columnar storage layout and stores different rows of data in the same column together. It calculates the maximum number of rows that DataChunk can store based on the size of DataChunk and the data type of the data column to be stored.

[0075] The write module writes data in blocks through the temporary table's write interface: First, it acquires a free block. If the free block is not full, it is converted into a DataChunk for operator operations using the positioning new method and placed in the cache. The remaining number of data rows stored in the free block is calculated, and the data to be written is copied to the free block. If this write causes the free block to become full, the full free block is removed from the cache; otherwise, the not-full free block is kept in the cache for future writes. If the free block is empty, it is converted into a DataChunk for operator operations using the positioning new method, and the data to be written is copied to the free block. If the empty block is filled in one go, it is not placed in the cache; otherwise, it is placed in the cache for future writes.

[0076] The read module reads data from the temporary table through the temporary table's read interface. Reading methods for the temporary table include sequential read and random read.

[0077] Using a sequential read method, the operator increments the block number to be read one by one, and then calls the temporary table read interface to obtain the block to be read.

[0078] Using a random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows of data that the temporary table can store, and then calls the temporary table read interface to obtain the block to be read;

[0079] When the operator finishes execution or the temporary table is no longer needed, the management module deletes the temporary table and releases the memory occupied by the temporary table.

[0080] The information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0081] Similarly, the device of this invention innovates the temporary table implementation structure, defines the block and DataChunk structure of the temporary table, as well as the read and write interface, replacing inefficient and time-consuming operations such as serialization and deserialization, which helps to improve the efficiency of reading and writing data.

[0082] It should be noted that not all steps and modules in the above processes and device structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structure described in the above embodiments can be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0083] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method of managing an operator temporary table, characterized by Comprise: Step 1: create a temporary table according to the operator requirements; Step 2: divide the block size of the temporary table corresponding to the block size of the physical memory division; Step 3: define the block of the temporary table contains a 4 byte integer bit and a data block DataChunk, and record the actual storage data row number of DataChunk through the 4 byte integer bit; Step 4: define DataChunk adopts column storage layout, and stores different row data of the same column together, and calculate the maximum number of rows that DataChunk can store data according to the size of DataChunk and the data type of the data column to be stored; Step 5: write data in block units through the write interface of the temporary table: first get an idle block, if the idle block is not full block, the idle block has been converted into the data block DataChunk of the operator operation by positioning new, and is put into the cache cache, calculate the remaining storage data row number of the idle block, copy the data to be written to the idle block, if this time writing makes the idle block full, the idle block full is eliminated from the cache cache, otherwise the idle block not full is continued to be placed in the cache cache to facilitate the next write; if the idle block is empty block, the idle block is converted into the data block DataChunk of the operator operation by positioning new, and then the data to be written is copied to the idle block, if the empty block is written full at one time, it is not put into the cache cache, otherwise it is put into the cache cache to facilitate the next write; Step 6: read the temporary table data through the read interface of the temporary table: the reading mode of the temporary table includes sequential reading and random reading, Using sequential reading mode, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read, Using random reading mode, the operator calculates the block number where the data to be read is located according to the row number of the data to be read and the maximum number of rows that the block of the temporary table can store data, and then calls the temporary table reading interface to obtain the block to be read; Step 7: when the operator execution ends or no longer needs the temporary table, delete the temporary table and release the memory occupied by the temporary table.

2. The method of claim 1, wherein In step 1, the temporary table is created according to the operator requirements, including: According to the amount of data required by the operator to determine whether it reaches or is greater than the pre-set threshold, if yes, create a temporary table or enable the created temporary table.

3. The management method of the operator temporary table according to claim 1, characterized in that in step 4, according to the maximum number of rows that the DataChunk in the block of the temporary table can store data, it is estimated when the block of the temporary table is full, and the DataChunk to be written is divided until the block of the temporary table is full, When defining DataChunk to adopt column storage layout, set the start of each column to store data as N Bit BitMap, which is used to identify whether the data on the column corresponding row is NULL value, and the data with NULL value does not occupy storage space.

4. The method of claim 1, wherein the value of N is (Capacity / 8+1)*8, and Capacity is the maximum number of rows of data that can be stored in a DataChunk.

5. The method of claim 1, wherein the operator temporary table is managed by In step 6, when reading the data of the temporary table through the reading interface of the temporary table, the block of the temporary table to be read is converted into a DataChunk of the operator operation in the positioning new mode, and then the block of the temporary table to be read is placed in the cache for the next reading.

6. An apparatus for managing an operator temporary table, characterized by The system comprises a creating module, a managing module, a writing module and a reading module, The creating module creates the temporary table according to the requirement of the operator. The managing module divides the block size of the temporary table to correspond to the block size of the physical memory. The managing module defines that the block of the temporary table comprises a 4-byte integer bit and a DataChunk, and records the number of rows of data actually stored in the DataChunk through the 4-byte integer bit. The managing module defines that the DataChunk adopts a columnar storage layout, and stores different rows of data of the same column together, and calculates the maximum number of rows of data that can be stored in the DataChunk according to the size of the DataChunk and the data type of the data column to be stored. The writing module writes data in the block unit through the writing interface of the temporary table: first, an idle block is obtained, if the idle block is an unwritten block, the idle block has been converted into a DataChunk of the operator operation in the positioning new mode and placed in the cache, the number of rows of data remaining to be stored in the idle block is calculated, the data to be written is copied to the idle block, if the writing causes the idle block to be full, the idle block full of data is removed from the cache, otherwise, the idle block not full of data is placed in the cache for the next writing; if the idle block is an empty block, the idle block is converted into a DataChunk of the operator operation in the positioning new mode, and then the data to be written is copied to the idle block, if the empty block is full of data at one time, the empty block is not placed in the cache, otherwise, the empty block is placed in the cache for the next writing. The reading module reads the data of the temporary table through the reading interface of the temporary table: the reading mode of the temporary table comprises sequential reading and random reading, In the sequential reading mode, the operator increments the block number to be read one by one, and then calls the reading interface of the temporary table to obtain the block to be read, In the random reading mode, the operator calculates the block number of the data to be read according to the row number of the data to be read and the maximum number of rows of data that can be stored in the block of the temporary table, and then calls the reading interface of the temporary table to obtain the block to be read. The managing module deletes the temporary table and releases the memory occupied by the temporary table when the execution of the operator ends or the temporary table is no longer needed.

7. The apparatus for managing an operator temporary table of claim 6, wherein The creating module The creating module creates the temporary table according to the requirement of the operator, including: According to the amount of data required to be temporarily stored by the operator, it is judged whether the amount of data reaches or is greater than a pre-set threshold, if yes, the temporary table is created or the created temporary table is enabled.

8. The apparatus for managing an operator temporary table of claim 6, wherein The management module estimates when the blocks of the temporary table are full according to the maximum number of rows that the DataChunk can store data, and divides the DataChunk to be written until the blocks of the temporary table are full when writing, When the columnar storage layout is defined for the DataChunk, a BitMap of N bits is set for each column to indicate whether the data on the corresponding row is NULL, and the data with NULL value does not occupy storage space.

9. The apparatus for managing an operator temporary table of claim 6, wherein When the management module calculates the value of N, the value of N is (Capacity / 8+1)*8, and Capacity is the maximum number of rows that the DataChunk can store data.

10. The apparatus for managing an operator temporary table of claim 6, wherein When the reading module reads the data of the temporary table through the reading interface of the temporary table, the blocks of the temporary table to be read are converted into the DataChunk of the operator operation in the positioning new mode, and then the blocks of the temporary table to be read are put into the cache for the next reading.

Images