Management method of operator temporary table

Abstract

The invention discloses an operator temporary table management method, and relates to the technical field of data management. Comprising the steps of 1, creating a temporary table according to operator requirements; 2, enabling the size of a block for dividing the temporary table to correspond to the size of a block for dividing the physical memory; step 3, defining that a block of the temporary table comprises an integer bit of 4 bytes and a data block DataChunk; 4, defining that the DataChunk adopts a column type storage layout; 5, writing data by taking a block as a unit through a writing interface of the temporary table; 6, reading data of the temporary table through a reading interface of the temporary table; and 7, when the operator execution is finished or the temporary table is not needed any more, deleting the temporary table, and releasing the memory occupied by the temporary table.

Description

Technical Field

[0001] The invention discloses a management method for a temporary operator table and relates to the technical field of data management. Background Art

[0002] Temporary tables are generally used to store temporary data or intermediate results during the calculation process. In database operators, especially blocking operators such as Sort and GroupBy, if the amount of data being operated is large, it is generally necessary to create a temporary table and store the data on disk. Then write the operated data into the temporary table and read it from the temporary table when needed; finally, delete the temporary table when not in use. Existing temporary tables generally provide file read and write interfaces. When writing, the operator needs to serialize the memory structure of the data to be written, and then call the temporary table file write interface to write it to the disk; when reading, the operator calls the temporary table file read interface to read the data from the disk, and then deserialize it into the memory structure of the operator operation for operation. However, serialization and deserialization are inefficient and time-consuming operations, which are not conducive to improving the efficiency of reading and writing data, especially when the data structure of the operator operation and the data structure of the temporary table operation do not match. Summary of the invention

[0003] In view of the problems in the prior art, the present invention provides a method for managing a temporary table of an operator to manage the temporary table of the operator and improve the reading and writing performance.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a method for managing an operator temporary table, comprising:

[0006] Step 1: Create a temporary table according to operator requirements;

[0007] Step 2: The block size of the temporary table is divided into blocks corresponding to the block size of the physical memory;

[0008] Step 3: Define the block of the temporary table to contain a 4-byte integer and a data block DataChunk. The 4-byte integer is used to record the number of rows of data actually stored in DataChunk.

[0009] Step 4: Define DataChunk to use columnar storage layout, store different rows of data in the same column together, and calculate the maximum number of rows that DataChunk can store based on the size of DataChunk and the data type of the data column to be stored;

[0010] Step 5: Write data in blocks through the write interface of the temporary table: first obtain a free block. If the free block is not full, the free block has been converted into a data block DataChunk for operator operation by positioning new, and is placed in the cache. Calculate the number of data rows remaining in the free block, and copy the data to be written to the free block. If the free block is full due to this writing, remove the full free block from the cache. Otherwise, keep the unfull free block in the cache to facilitate the next writing. If the free block is an empty block, use the positioning new method to convert the free block into a data block DataChunk for operator operation, and then copy the data to be written to the free block. If the empty block is full at one time, it will not be placed in the cache. Otherwise, it will be placed in the cache to facilitate the next writing.

[0011] Step 6: Read temporary table data through the temporary table read interface: The temporary table can be read in sequential and random ways.

[0012] Using the sequential reading method, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read.

[0013] Using the random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that can be stored in the temporary table block, and then calls the temporary table read interface to obtain the block to be read;

[0014] Step 7: When the operator execution ends or the temporary table is no longer needed, delete the temporary table and release the memory occupied by the temporary table.

[0015] Furthermore, in step 1 of the method for managing a temporary operator table, creating a temporary table according to operator requirements includes:

[0016] Determine whether the amount of data that the operator needs to temporarily store reaches or exceeds the preset threshold. If so, create a temporary table or enable the created temporary table.

[0017] Furthermore, in step 4 of the method for managing a temporary table of an operator, based on the maximum number of rows of data that can be stored in a DataChunk in a block of the temporary table, it is estimated when the block of the temporary table is full, and the DataChunk to be written is split during writing until the block of the temporary table is full.

[0018] When defining DataChunk to use a columnar storage layout, set the beginning of each column to store data as a BitMap of N bits, which is used to identify whether the data on the row corresponding to the column is a NULL value. Data with a NULL value does not occupy storage space.

[0019] Furthermore, when calculating the N value in step 4 of the method for managing a temporary operator table, the value of N is: (Capacity / 8+1)*8, where Capacity is the maximum number of rows of data that can be stored in a DataChunk.

[0020] Furthermore, in step 6 of the method for managing an operator temporary table, when reading temporary table data through the temporary table read interface, the block of the temporary table to be read is converted into a DataChunk of the operator operation by positioning new, and then the block of the temporary table to be read is put into the cache to facilitate the next reading.

[0021] The present invention provides a management device for an operator temporary table, comprising a creation module, a management module, a writing module and a reading module.

[0022] The creation module creates a temporary table according to the operator requirements;

[0023] The block size of the temporary table divided by the management module corresponds to the block size of the physical memory;

[0024] The management module defines that the block of the temporary table includes a 4-byte integer and a data block DataChunk. The 4-byte integer is used to record the number of rows of data actually stored in DataChunk.

[0025] The management module defines that DataChunk adopts a columnar storage layout and stores different rows of data in the same column together. The maximum number of rows that DataChunk can store is calculated based on the size of DataChunk and the data type of the data column to be stored.

[0026] The write module writes data in blocks through the write interface of the temporary table: first, a free block is obtained. If the free block is not full, the free block has been converted into a data block DataChunk for operator operation by positioning new, and is placed in the cache. The number of data rows remaining in the free block is calculated, and the data to be written is copied to the free block. If the free block is full due to this writing, the full free block is removed from the cache, otherwise the unfull free block is kept in the cache to facilitate the next writing. If the free block is an empty block, the free block is converted into a data block DataChunk for operator operation by positioning new, and then the data to be written is copied to the free block. If the empty block is full at one time, it is not placed in the cache, otherwise it is placed in the cache to facilitate the next writing.

[0027] The read module reads temporary table data through the temporary table read interface: the ways to read temporary tables include sequential reading and random reading.

[0028] Using the sequential reading method, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read.

[0029] Using the random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that can be stored in the temporary table block, and then calls the temporary table read interface to obtain the block to be read;

[0030] When the operator execution is completed or the temporary table is no longer needed, the management module deletes the temporary table and releases the memory occupied by the temporary table.

[0031] Furthermore, the creation module of the management device of the operator temporary table creates a temporary table according to the operator demand, including:

[0032] Determine whether the amount of data that the operator needs to temporarily store reaches or exceeds the preset threshold. If so, create a temporary table or enable the created temporary table.

[0033] Furthermore, the management module of the management device for the operator temporary table estimates when the block of the temporary table is full according to the maximum number of rows that can be stored in the DataChunk in the block of the temporary table, and divides the DataChunk to be written when writing until the block of the temporary table is full.

[0034] When defining DataChunk to use a columnar storage layout, set the beginning of each column to store data as a BitMap of N bits, which is used to identify whether the data on the row corresponding to the column is a NULL value. Data with a NULL value does not occupy storage space.

[0035] Furthermore, when the management module of the management device of the operator temporary table calculates the N value, the value of N is: (Capacity / 8+1)*8, and Capacity is the maximum number of rows of data that can be stored in DataChunk.

[0036] Furthermore, when the reading module of the management device of the operator temporary table reads the temporary table data through the reading interface of the temporary table, the block of the temporary table to be read is converted into the DataChunk of the operator operation by positioning new method, and then the block of the temporary table to be read is put into the cache cache for convenience of reading next time.

[0037] The benefits of the present invention are:

[0038] The present invention innovates the temporary table implementation structure, defines the block and DataChunk structure of the temporary table, and the read-write interface, replaces inefficient and time-consuming operations such as serialization and deserialization, and is conducive to improving the efficiency of reading and writing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a mapping diagram between temporary table disk blocks Block and DataChunk.

[0040] Figure 2 It is a mapping diagram between temporary tables, virtual memory, and physical memory.

[0041] Figure 3 It is a temporary table application flow chart. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0043] Example 1

[0044] The present invention provides a method for managing an operator temporary table, comprising:

[0045] Step 1: Create a temporary table based on operator requirements.

[0046] The amount of data that the operator needs to temporarily store is judged to see whether it reaches or exceeds a preset threshold. If so, a temporary table is created or an already created temporary table is enabled.

[0047] Before the operator enables the temporary table, it temporarily stores the data that needs to be temporarily stored in memory and counts the size of the temporarily stored data. As the operator execution progresses, the amount of data that the operator needs to temporarily store continues to accumulate and may eventually reach the pre-set threshold. The operator decides to enable the temporary table. If the amount of data that the operator needs to temporarily store does not reach the pre-set threshold at the end of the operator execution, the operator may not enable the temporary table. When the operator enables the temporary table, you can refer to Figure 3 In the process position of ①, the mmap system call can be called to open the temporary table in file mapping mode. If the operator has data that needs to be temporarily stored, the temporary table write interface can be directly called to store it directly in the temporary table.

[0048] Step 2: The block size of the temporary table should correspond to the block size of the physical memory. The mapping relationship between the temporary table block, the virtual memory block, and the physical memory can be found in Figure 2 .

[0049] Step 3: Define a block of the temporary table that contains a 4-byte integer and a data block DataChunk. The 4-byte integer is used to record the number of rows of data actually stored in the DataChunk.

[0050] The layout of DataChunk in the temporary table block can be referred to Figure 1The black border part. It is exactly the same as the memory DataChunk layout used by operator calculation and data transmission between operators. This not only enables efficient writing and reading of temporary tables, but also simplifies operator implementation and unifies operator operations on memory data structures and disk data structures. However, the DataChunk in the block of the temporary table and the memory DataChunk used by the operator do not need to be the same size.

[0051] Step 4: Define the DataChunk to use a columnar storage layout and store different rows of data in the same column together. Calculate the maximum number of rows that the DataChunk can store based on the size of the DataChunk and the data type of the data column to be stored.

[0052] According to the maximum number of rows that can be stored in the DataChunk in the temporary table block, it is possible to estimate when the temporary table block will be full, and split the DataChunk to be written when writing until the temporary table block is full.

[0053] When defining DataChunk to use columnar storage layout, set the beginning of each column to store data as a BitMap of N bits, which is used to identify whether the data on the corresponding row of the column is a NULL value. Data with a NULL value does not occupy storage space. When calculating the N value, the value of N is: (Capacity / 8+1)*8, where Capacity is the maximum number of rows that DataChunk can store data.

[0054] Step 5: Write data in blocks through the write interface of the temporary table:

[0055] The write interface of the temporary table can be expressed as:

[0056] void Append(DataChunk*chunk);

[0057] Among them, chunk is the memory DataChunk data structure used by the operator, which represents the temporary data to be written by the operator.

[0058] First, get a free block. If the free block is not full, the free block has been converted into a data block DataChunk for operator operation by positioning new, and is placed in the cache. The number of data rows remaining to be stored in the free block is calculated, and the data to be written is copied to the free block. If the free block is full due to this writing, the full free block is removed from the cache, otherwise the unfull free block continues to be placed in the cache to facilitate the next writing. If the free block is an empty block, the free block is converted into a data block DataChunk for operator operation by positioning new, and then the data to be written is copied to the free block. If the empty block is full at one time, it is not placed in the cache, otherwise it is placed in the cache to facilitate the next writing.

[0059] For write operations, see Figure 3 The process position of ② in the figure. Since DataChunk is converted by positioning new, no memory is requested from the system during the writing process, which not only improves efficiency but also simplifies the error handling logic. Writing is performed in block copy mode, avoiding time-consuming serialization operations.

[0060] Free blocks include empty blocks and unfilled blocks. If there are no free blocks, you can expand the temporary table first. When expanding, the number of blocks to be expanded at a time depends on the specific situation: you can expand one block at a time, which maximizes disk utilization and minimizes memory usage, but increases the number of temporary table file remappings, because each expansion requires temporary table file remapping, and file mapping operations are a relatively time-consuming operation; you can also expand several blocks, dozens of blocks, or even hundreds of blocks at a time, which can reduce the number of file remappings, but some of the expanded blocks may not be used in the end, which will waste some disk space and memory.

[0061] Step 6: Read the temporary table data through the temporary table read interface: The temporary table read interface can be expressed as:

[0062] void Get(int chunkIdx, DataChunk**chunk);

[0063] Among them, chunkIdx is the block number of the temporary table block to be read; chunk is the identifier of the read data, which is used by the operator.

[0064] There are two ways to read temporary tables: sequential reading and random reading.

[0065] Using the sequential reading method, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read.

[0066] Using the random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that can be stored in the temporary table block, and then calls the temporary table read interface to obtain the block to be read.

[0067] Read operation see Figure 3 The process position of ③ in the figure. During the operation, the block of the temporary table to be read is converted into the DataChunk of the operator operation by positioning new, and then put into the cache for the convenience of reading next time. This completely avoids the tedious and time-consuming deserialization operation, and no memory is requested from the system during the operation.

[0068] Step 7: When the operator execution ends or the temporary table is no longer needed, delete the temporary table and release the memory occupied by the temporary table. Figure 3 In the process position ④, when deleting a temporary table, the operator can first call the munmap system call to end file mapping and release the memory occupied by the temporary table, and then delete the temporary table file to release the disk space occupied by the temporary table.

[0069] Example 2

[0070] The present invention provides a management device for an operator temporary table, comprising a creation module, a management module, a writing module and a reading module.

[0071] The creation module creates a temporary table according to the operator requirements;

[0072] The block size of the temporary table divided by the management module corresponds to the block size of the physical memory;

[0073] The management module defines that the block of the temporary table includes a 4-byte integer and a data block DataChunk. The 4-byte integer is used to record the number of rows of data actually stored in DataChunk.

[0074] The management module defines that DataChunk adopts a columnar storage layout and stores different rows of data in the same column together. The maximum number of rows that DataChunk can store is calculated based on the size of DataChunk and the data type of the data column to be stored.

[0075] The write module writes data in blocks through the write interface of the temporary table: first, a free block is obtained. If the free block is not full, the free block has been converted into a data block DataChunk for operator operation by positioning new, and is placed in the cache. The number of data rows remaining in the free block is calculated, and the data to be written is copied to the free block. If the free block is full due to this writing, the full free block is removed from the cache, otherwise the unfull free block is kept in the cache to facilitate the next writing. If the free block is an empty block, the free block is converted into a data block DataChunk for operator operation by positioning new, and then the data to be written is copied to the free block. If the empty block is full at one time, it is not placed in the cache, otherwise it is placed in the cache to facilitate the next writing.

[0076] The read module reads temporary table data through the temporary table read interface: the ways to read temporary tables include sequential reading and random reading.

[0077] Using the sequential reading method, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read.

[0078] Using the random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that can be stored in the temporary table block, and then calls the temporary table read interface to obtain the block to be read;

[0079] When the operator execution is completed or the temporary table is no longer needed, the management module deletes the temporary table and releases the memory occupied by the temporary table.

[0080] As the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0081] Similarly, the device of the present invention innovates the temporary table implementation structure, defines the block and DataChunk structure of the temporary table, and the read-write interface, replaces inefficient and time-consuming operations such as serialization and deserialization, and is conducive to improving the efficiency of reading and writing data.

[0082] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0083] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for managing a temporary operator table, characterized in that include: Step 1: Create a temporary table according to operator requirements; Step 2: The block size of the temporary table is divided into blocks corresponding to the block size of the physical memory; Step 3: Define the block of the temporary table to contain a 4-byte integer and a data block DataChunk. The 4-byte integer is used to record the number of rows of data actually stored in DataChunk. Step 4: Define DataChunk to use columnar storage layout, store different rows of data in the same column together, and calculate the maximum number of rows that DataChunk can store based on the size of DataChunk and the data type of the data column to be stored; Step 5: Write data in blocks through the write interface of the temporary table: first obtain a free block. If the free block is not full, the free block has been converted into a data block DataChunk for operator operation by positioning new, and is placed in the cache. Calculate the number of data rows remaining in the free block, and copy the data to be written to the free block. If the free block is full due to this writing, remove the full free block from the cache. Otherwise, keep the unfull free block in the cache to facilitate the next writing. If the free block is an empty block, use the positioning new method to convert the free block into a data block DataChunk for operator operation, and then copy the data to be written to the free block. If the empty block is full at one time, it will not be placed in the cache. Otherwise, it will be placed in the cache to facilitate the next writing. Step 6: Read temporary table data through the temporary table read interface: The temporary table can be read in sequential and random ways. Using the sequential reading method, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read. Using the random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that can be stored in the temporary table block, and then calls the temporary table read interface to obtain the block to be read; Step 7: When the operator execution ends or the temporary table is no longer needed, delete the temporary table and release the memory occupied by the temporary table.

2. The method for managing a temporary operator table according to claim 1, characterized in that In step 1, a temporary table is created according to operator requirements, including: Determine whether the amount of data that the operator needs to temporarily store reaches or exceeds the preset threshold. If so, create a temporary table or enable the created temporary table.

3. A method for managing an operator temporary table according to claim 1, characterized in that in step 4, based on the maximum number of rows that can be stored in the DataChunk in the block of the temporary table, it is estimated when the block of the temporary table is full, and the DataChunk to be written is split when writing until the block of the temporary table is full. When defining DataChunk to use a columnar storage layout, set the beginning of each column to store data as a BitMap of N bits, which is used to identify whether the data on the row corresponding to the column is a NULL value. Data with a NULL value does not occupy storage space.

4. According to the method for managing an operator temporary table of claim 1, the characteristic is that when calculating the N value in step 4, the value of N is: (Capacity / 8+1)*8, and Capacity is the maximum number of rows of data that DataChunk can store.

5. The method for managing a temporary operator table according to claim 1, characterized in that When reading temporary table data through the temporary table read interface in step 6, the block of the temporary table to be read is converted into DataChunk for operator operation by positioning new method, and then the block of the temporary table to be read is put into cache for easy reading next time.

6. A management device for a temporary operator table, characterized in that Including creation module, management module, writing module and reading module, The creation module creates a temporary table according to operator requirements; The block size of the temporary table divided by the management module corresponds to the block size of the physical memory; The management module defines that the block of the temporary table includes a 4-byte integer and a data block DataChunk. The 4-byte integer is used to record the number of rows of data actually stored in DataChunk. The management module defines that DataChunk adopts a columnar storage layout and stores different rows of data in the same column together. The maximum number of rows that DataChunk can store is calculated based on the size of DataChunk and the data type of the data column to be stored. The write module writes data in blocks through the write interface of the temporary table: first, a free block is obtained. If the free block is not full, the free block has been converted into a data block DataChunk for operator operation by positioning new, and is placed in the cache. The number of data rows remaining in the free block is calculated, and the data to be written is copied to the free block. If the free block is full due to this writing, the full free block is removed from the cache, otherwise the unfull free block is kept in the cache to facilitate the next writing. If the free block is an empty block, the free block is converted into a data block DataChunk for operator operation by positioning new, and then the data to be written is copied to the free block. If the empty block is full at one time, it is not placed in the cache, otherwise it is placed in the cache to facilitate the next writing. The read module reads temporary table data through the temporary table read interface: the ways to read temporary tables include sequential reading and random reading. Using the sequential reading method, the operator increments the block number to be read one by one, and then calls the temporary table reading interface to obtain the block to be read. Using the random read method, the operator calculates the block number of the data to be read based on the row number of the data to be read and the maximum number of rows that can be stored in the temporary table block, and then calls the temporary table read interface to obtain the block to be read; When the operator execution is completed or the temporary table is no longer needed, the management module deletes the temporary table and releases the memory occupied by the temporary table.

7. The management device for a temporary operator table according to claim 6, characterized in that Creating a Module Create a temporary table based on operator requirements, including: Determine whether the amount of data that the operator needs to temporarily store reaches or exceeds the preset threshold. If so, create a temporary table or enable the created temporary table.

8. The management device for a temporary operator table according to claim 6, characterized in that The management module estimates when the block of the temporary table will be full based on the maximum number of rows that can be stored in the DataChunk in the block of the temporary table, and splits the DataChunk to be written when writing until the block of the temporary table is full. When defining DataChunk to use a columnar storage layout, set the beginning of each column to store data as a BitMap of N bits, which is used to identify whether the data on the row corresponding to the column is a NULL value. Data with a NULL value does not occupy storage space.

9. The management device for a temporary operator table according to claim 6, characterized in that When the management module calculates the N value, the value of N is: (Capacity / 8+1)*8, where Capacity is the maximum number of rows that DataChunk can store.

10. The management device for operator temporary table according to claim 6, characterized in that When the reading module reads temporary table data through the temporary table's reading interface, it uses the positioning new method to convert the temporary table block to be read into the DataChunk of the operator operation, and then puts the temporary table block to be read into the cache for easy reading next time.

Images