Temporary file management method and device, electronic equipment and storage medium

Abstract

The specification provides a temporary file management method and device, electronic equipment and storage medium, which comprises: allocating a memory temporary page to a temporary file to be stored, and writing data of the temporary file into the memory temporary page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier; whenever a memory temporary page is written by the data of the temporary file to a remaining capacity less than a first threshold, a new memory temporary page is allocated to the temporary file, the identifier of the new memory temporary page is recorded in the memory temporary page with a remaining capacity less than the first threshold, and the data of the temporary file is continuously written into the new memory temporary page; and if the data of the temporary file is completely written into the memory, the identifier of at least one memory temporary page storing the temporary file is recorded in the memory.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of database, and particularly relate to a temporary file management method and apparatus, electronic device and storage medium. BACKGROUND

[0002] A database generates temporary files in the process of data query and index construction. The temporary files refer to files of temporary nature generated in the process of a process performing a computing task, which have a relatively short life cycle and will perish with the process.

[0003] The number of temporary files generated in the daily operation of a database is large, and some temporary files occupy a large space. The management effect of the temporary files in the related technology is poor, for example, the temporary files occupy a large space due to complex metadata and other information. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide a temporary file management method and apparatus, electronic device and storage medium.

[0005] To achieve the above object, one or more embodiments of the present specification provide the technical solutions as follows.

[0006] According to a first aspect of one or more embodiments of the present specification, a temporary file management method is provided, and the method comprises:

[0007] allocating a memory temporary page to a temporary file to be stored, and writing data of the temporary file into the memory temporary page, wherein the space in the memory of a database for storing temporary files is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier;

[0008] Whenever a memory temporary page is written by data of the temporary file to a remaining capacity less than a first threshold, a new memory temporary page is allocated to the temporary file, the identifier of the new memory temporary page is recorded in the memory temporary page with a remaining capacity less than the first threshold, and the data of the temporary file is continuously written into the new memory temporary page;

[0009] If the data of the temporary file is completely written into the memory, the identifier of at least one memory temporary page storing the temporary file is recorded in the memory.

[0010] In one possible embodiment of the present specification, the method further comprises:

[0011] If data of the temporary file is completely written into the memory, an end identifier is recorded in the last memory temporary page storing the temporary file, wherein the end identifier is used to represent that the memory temporary page is the last memory temporary page storing the temporary file.

[0012] In one possible implementation of the present disclosure, the recording in the memory of the identifier of the at least one memory temporary page storing the temporary file comprises:

[0013] The identifier of the first memory temporary page storing the temporary file and the number of the at least one memory temporary page storing the temporary file are recorded in the memory.

[0014] In one possible implementation of the present disclosure, the method further comprises:

[0015] If the remaining capacity of the space in the memory of the database for storing temporary files is less than a third threshold value, at least one temporary file stored in the memory is flushed to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold value, and the memory temporary page storing the at least one temporary file is released.

[0016] In one possible implementation of the present disclosure, the flushing of the at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold value comprises:

[0017] The at least one temporary file stored in the memory is flushed to the persistent storage device in a manner of preferentially flushing the temporary file with the largest data volume until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold value.

[0018] In one possible implementation of the present disclosure, the flushing of the at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold value comprises:

[0019] The at least one temporary file stored in the memory is flushed to the persistent storage device in a manner of preferentially flushing the temporary file with the lowest access frequency until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold value.

[0020] In one possible implementation of the present disclosure, the flushing of the at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold value comprises:

[0021] If any temporary file with a data volume not less than the fifth threshold value is flushed to the persistent storage device, the temporary file is divided into at least one data block in a manner of dividing each fifth threshold value into a data block, and the at least one data block is uploaded to the object storage layer;

[0022] If any temporary file with a data volume less than the fifth threshold value is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

[0023] In one possible embodiment of the present specification, the method further comprises:

[0024] The data flushed to the local disk of the database is divided into at least one data block in a manner of dividing each fifth threshold value into a data block;

[0025] If the local disk meets the preset uploading condition, the data block in the local disk is uploaded to the object storage layer.

[0026] In one possible embodiment of the present specification, the method further comprises:

[0027] Receiving a data reading instruction for any temporary file, and reading part or all of the data of the temporary file from the persistent storage device and / or the memory based on the data position range indicated by the data reading instruction.

[0028] In one possible embodiment of the present specification, the reading part or all of the data of the temporary file from the persistent storage device and / or the memory based on the data position range indicated by the data reading instruction comprises:

[0029] According to the identification of the memory temporary page storing the temporary file recorded in the memory, determining the data position range of the data of the temporary file stored in the memory in the temporary file;

[0030] If the data position range indicated by the data reading instruction is within the data position range of the data of the temporary file stored in the memory in the temporary file, part or all of the data of the temporary file is read from the memory;

[0031] If the data position range indicated by the data reading instruction has no intersection with the data position range of the data of the temporary file stored in the memory in the temporary file, part or all of the data of the temporary file is read from the persistent storage device;

[0032] If the data position range indicated by the data reading instruction is partially within the data position range of the data of the temporary file stored in the memory in the temporary file, part or all of the data of the temporary file is read from the persistent storage device and the memory.

[0033] In one possible implementation of the present specification, the data range indicated by the data read instruction is used to determine a target data block in the persistent storage device, and part or all of the data of the temporary file is read from the target data block.

[0034] In one possible implementation of the present specification, the data range indicated by the data read instruction is used to determine a target data block in the persistent storage device, and part or all of the data of the temporary file is read from the target data block.

[0035] In one possible implementation of the present specification, the data range indicated by the data read instruction is used to determine a target data block in the persistent storage device, and part or all of the data of the temporary file is read from the target data block.

[0036] In one possible implementation of the present specification, the data range indicated by the data read instruction is used to determine a target data block in the persistent storage device, and part or all of the data of the temporary file is read from the target data block.

[0037] According to a second aspect of one or more embodiments of the present specification, a temporary file management apparatus is provided, and the apparatus comprises:

[0038] a distribution module configured to allocate a memory temporary page to a temporary file to be stored, and write data of the temporary file into the memory temporary page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier;

[0039] a linking module configured to, whenever a memory temporary page is written with data of the temporary file and the remaining capacity is less than a first threshold, allocate a new memory temporary page to the temporary file, record the identifier of the new memory temporary page in the memory temporary page with the remaining capacity less than the first threshold, and continue to write data of the temporary file into the new memory temporary page;

[0040] a metadata module configured to, if the data of the temporary file is completely written into the memory, record the identifier of at least one memory temporary page storing the temporary file in the memory.

[0041] According to a third aspect of one or more embodiments of the present specification, a computer program product is provided, comprising computer programs / instructions which, when executed by a processor, implement the steps of the method of the first aspect.

[0042] According to a fourth aspect of one or more embodiments of the present specification, an electronic device is provided, and the device comprises:

[0043] a processor;

[0044] a memory for storing processor-executable instructions;

[0045] The processor implements the method according to the first aspect by running the executable instructions.

[0046] According to a fifth aspect of one or more embodiments of the present specification, a computer readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the method according to the first aspect.

[0047] The technical solutions provided by the embodiments of the present specification can include the following beneficial effects:

[0048] In the temporary file management method provided by the embodiments of the present specification, the space in the memory of the database for storing temporary files is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier; when storing a temporary file, a memory temporary page is first allocated to the temporary file to be stored, and the data of the temporary file is written into the memory temporary page, and whenever a memory temporary page is written by the data of the temporary file and the remaining capacity is less than a first threshold, a new memory temporary page is allocated to the temporary file, the identifier of the new memory temporary page is recorded in the memory temporary page with a remaining capacity less than the first threshold, and the data of the temporary file is continuously written into the new memory temporary page; if the data of the temporary file is completely written into the memory, the identifier of at least one memory temporary page storing the temporary file is recorded in the memory. The space for storing temporary files is divided into a plurality of memory temporary pages, and the temporary file is written into the memory by applying for memory temporary pages one by one, and the temporary file is formed into a page linked list by recording the identifier of the next memory temporary page in each memory temporary page, so that the management of the temporary file is simple and efficient, and the form of the page linked list simplifies the metadata of the temporary file and occupies less space, thereby improving the management effect of the temporary file and improving the utilization rate of the memory. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 FIG. 1 is a schematic diagram of the architecture of a data system provided by an example embodiment.

[0050] Figure 2 FIG. 2 is a flowchart of a temporary file management method provided by an example embodiment.

[0051] Figure 3 FIG. 3 is a schematic diagram of the metadata of a temporary file provided by an example embodiment.

[0052] Figure 4 FIG. 4 is a management method of a temporary file provided by an example embodiment.

[0053] Figure 5 FIG. 5 is a linked list of a temporary file provided by an example embodiment.

[0054] Figure 6 is control logic of a history file write phase provided by an example embodiment.

[0055] Figure 7 is control logic of a history file flush phase provided by an example embodiment.

[0056] Figure 8 is control logic of a history file read phase provided by an example embodiment.

[0057] Figure 9 is a structural diagram of a device provided by an example embodiment.

[0058] Figure 10 is a block diagram of a temporary file management apparatus provided by an example embodiment. DETAILED DESCRIPTION

[0059] The example embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to designate the same elements, unless otherwise indicated. The embodiments described in the following example embodiments are not representative of all embodiments consistent with one or more embodiments of the present specification. Rather, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present specification, as detailed in the appended claims.

[0060] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in the present specification in other embodiments. In some other embodiments, the steps included in the methods can be more or less than those described in the present specification. In addition, a single step described in the present specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the present specification can be combined into a single step for description in other embodiments.

[0061] First, some concepts related to the present specification are explained.

[0062] LSM-Tree: A high-efficiency storage structure widely used in database systems, designed to optimize write performance and disk space utilization. It reduces the overhead of random writes by caching write operations in memory and periodically merging data to disk.

[0063] Temporary file: A file of temporary nature generated in the process of a process performing a computing task. These files have a relatively short life cycle and will also perish with the process.

[0064] Object storage: Object storage is a data management architecture that efficiently and flexibly stores and retrieves massive amounts of unstructured data by treating data as objects rather than files or blocks. Its "everything is an object" concept emphasizes that all data, whether documents, pictures, videos, or logs, can be uniformly processed as objects, helping to improve access efficiency and scalability.

[0065] Based on the above technical problems in the background art, at least one embodiment of the present specification provides a temporary file management method, which can be applied to a database, such as a database implemented based on an LSM-Tree storage engine. The database can be an append-only database, and the temporary file management method can be applied to the database. Figure 1 As shown in the data system, the data system includes an object storage layer and a plurality of databases connected in communication with the object storage layer, and each database instance locally has a disk and a memory. The object storage has the characteristics of low cost (the unit price is several tenths of the cloud disk), payment according to actual usage (capacity and read-write request amount), read-only, and sharing. Using object storage as shared storage can significantly reduce storage costs. However, object storage has limitations such as large read-write delay, and is only suitable for businesses that are not sensitive to time delay in the past. Using the memory and the local disk to cache the data and the metadata on the object storage can make the shared storage adapt to most OLTP businesses.

[0066] The method can efficiently manage the temporary files generated by the database, can ensure that the write operation and the read operation of the temporary files are efficiently executed, can avoid that the management of the temporary files is too complex to affect the operation of the database, and can reduce the space occupation of the database by simplifying the metadata of the temporary files.

[0067] Please refer to the accompanying drawings Figure 2 which exemplarily shows a flowchart of the method, including steps S201 to S203.

[0068] In step S201, a memory temporary page is allocated to a temporary file to be stored, and data of the temporary file is written into the memory temporary page, wherein the space in the memory of the database for storing the temporary file is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier.

[0069] The database can allocate a part of the write cache in memory for storing temporary files, and can divide the part of space into multiple pages in units of a certain capacity value, referred to as memory temporary pages, each of which has an identifier. After the database is initialized or restarted, all the memory temporary pages are idle, and during the running of the database, no memory temporary page is idle. The idle memory temporary pages can form a linked list, and each memory temporary page in the linked list can record an identifier of a next memory temporary page. For example, the idle memory temporary pages can be sorted according to the identifiers to form the linked list.

[0070] In this step, when the temporary file is written into the memory, a memory temporary page can be applied for from the linked list formed by the idle memory temporary pages, and data of the temporary file can be written into the allocated memory temporary page. For example, the memory temporary page at the head or tail of the linked list formed by the idle memory temporary pages can be allocated to the temporary file.

[0071] In step S202, whenever a memory temporary page is written by data of the temporary file to have a remaining capacity less than a first threshold value, a new memory temporary page is allocated to the temporary file, an identifier of the new memory temporary page is recorded in the memory temporary page with the remaining capacity less than the first threshold value, and data of the temporary file is continuously written into the new memory temporary page.

[0072] In the process of writing data of the temporary file into the memory temporary page allocated in step S201, the memory temporary page is gradually occupied by the data to reduce the remaining space, and when the remaining space is reduced to a certain extent (for example, less than the first threshold value), a new memory temporary page can be applied for from the linked list formed by the idle memory temporary pages to continue writing data of the temporary file. In this way, when the remaining capacity of the previous memory temporary page is less than the first threshold value, a new memory temporary page is continuously applied for to continue writing data of the temporary file, until data of the temporary file is completely written into the memory.

[0073] After each new memory temporary page is applied for (i.e., allocated), an identifier of the new memory temporary page can be written into the previous memory temporary page, so that all the memory temporary pages in which data of the temporary file is written are linked into a page linked list.

[0074] In step S203, if data of the temporary file is completely written into the memory, an identifier of at least one memory temporary page in which the temporary file is stored is recorded in the memory.

[0075] In the memory, the identification of the at least one memory temporary page storing the temporary file is recorded, which is the generation and saving of the metadata of the temporary file in the memory. For example, the identification of the first memory temporary page storing the temporary file and the number of the at least one memory temporary page storing the temporary file can be recorded in the memory. The metadata of the temporary file cached in the memory in the method is simple, convenient to manage and occupies less space. For example, the metadata of the temporary file is shown in the attached Figure 3 Figure 1, which records the tmp_file_id and meta_data of each temporary file in the form of a table, where the tmp_file_id is the id of the temporary file, and the meta_data is the metadata of the temporary file, including the identification of the first memory temporary page begin_page_id and the number of the memory temporary file page_nums.

[0076] In addition, if the data of the temporary file is completely written into the memory, the end identification is recorded in the last memory temporary page storing the temporary file, where the end identification is used to represent that the memory temporary page is the last memory temporary page storing the temporary file. Thus, the end flag is identified when the temporary file is read.

[0077] Please refer to the attached Figure 4 which shows the temporary file management method provided by the embodiment.

[0078] The space in the write cache for storing the temporary file is divided into multiple Pages with 8KB as a unit, and each Page is maintained by a PageEntry. For the chain table formed by the multiple Pages storing a temporary file, the PageEntry of each Page in the chain table records the tem_file_id, page_id, next_page_id and page_buffer in the Page, where the tem_file_id is the identification of the temporary file; the page_id is the identification of the Page; the next_page_id is the identification of the next Page, and the next of each Page except the last Page records the identification of the next Page, and the next of the last Page records the end identification, for example, -1; and the page_buffer is the data cached in the Page. All the idle Pages form a chain table free list.

[0079] The temporary file management method provided by the embodiments of the present specification divides the space for storing temporary files into a plurality of memory temporary pages, and the temporary files write data to the memory by applying for memory temporary pages one by one, and form a page linked list by recording the identifier of the next memory temporary page in each memory temporary page, so that the management of the temporary files is simple and efficient, and the form of the page linked list simplifies the metadata of the temporary files and occupies less space, thereby improving the management effect of the temporary files as a whole and improving the utilization rate of the memory.

[0080] In some embodiments of the present specification, if the remaining capacity of the space in the memory of the database for storing temporary files is less than a third threshold, at least one temporary file stored in the memory is flushed to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold, and the memory temporary page storing the at least one temporary file is released.

[0081] The third threshold can be equal to the fourth threshold. Alternatively, the third threshold can be greater than the fourth threshold, so that the memory space fluctuates around the third threshold to avoid the flushing operation being too frequent.

[0082] For example, at least one temporary file stored in the memory is flushed to the persistent storage device in a manner of preferentially flushing the temporary file with the largest data amount until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold.

[0083] The temporary file with a larger data amount occupies more memory temporary pages, and flushing the temporary file with a larger data amount to the persistent storage device can release more memory temporary pages to alleviate the capacity shortage of the space in the memory for storing temporary files.

[0084] For example, all temporary files can be divided into two groups according to the data amount of the temporary files in the memory, the temporary files in the first group have a data amount in the memory greater than or equal to a certain threshold, and the temporary files in the second group have a data amount in the memory less than a certain threshold. Therefore, when flushing the temporary files, the temporary files in the first group can be preferentially selected for flushing.

[0085] For example, at least one temporary file stored in the memory is flushed to the persistent storage device in a manner of preferentially flushing the temporary file with the lowest access heat until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold.

[0086] The temporary file with a higher heat has a higher probability of being accessed, and the temporary file with a lower heat has a lower probability of being accessed, so eliminating the temporary file with the lowest heat can minimize the impact on the hit rate of the temporary files in the memory.

[0087] For example, an LRU list for embodying the hotness can be constructed, each node in the list representing a temporary file, and the nodes corresponding to the temporary files that have been recently used and modified are at the tail of the LRU list, so the temporary files corresponding to the nodes at the head of the LRU list can be preferentially selected for disk flushing.

[0088] For example, if any temporary file with a data amount not less than a fifth threshold value is flushed to the persistent storage device, the temporary file is divided into at least one data block in a manner of dividing every fifth threshold value into a data block, and the at least one data block is uploaded to the object storage layer; if any temporary file with a data amount less than the fifth threshold value is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

[0089] In this example, the temporary file with a data amount sufficient to form at least one data block segment is directly flushed to the object storage layer, which can reduce the storage pressure of the local disk, simplify the process of transferring data from the memory to the disk, and reduce the system IO pressure.

[0090] It should be understood that if the data amount of the temporary file forms at least one data block and there is no remaining data (i.e., data that is not enough to form one data block), the at least one data block described above can be flushed to the object storage; if the data amount of the temporary file forms at least one data block and there is remaining data (i.e., data that is not enough to form one data block), the at least one data block described above is flushed to the object storage, and the remaining data is flushed to the local disk.

[0091] Uploading to the object storage in units of data blocks can reduce the delay and improve the upload efficiency due to the high bandwidth characteristics of the object storage.

[0092] It should also be understood that when flushing data to the local disk of the database, the data flushed to the local disk of the database can be divided into at least one data block in a manner of dividing every fifth threshold value into a data block; if the local disk satisfies a preset upload condition, the data block in the local disk is uploaded to the object storage layer. That is, when flushing data to the local disk, the data block is divided synchronously, so that the data written into the local disk is in the form of a data block, facilitating subsequent uploading to the object storage; and uploading the data block in the local disk to the object storage in a timely manner can release the space of the local disk in a timely manner and improve the space utilization rate.

[0093] Since the data on the persistent storage device, i.e., the local disk and the object storage, is in the form of data blocks, the data amount of the data block is fixed, and therefore the metadata of the temporary file in the persistent storage device can be recorded by the disk offset, which can be positioned to a specific data block and the data offset in the data block.

[0094] The above three examples provide specific ways for the process of flushing the temporary file to disk from different angles, thereby improving the effect of the process. The above three embodiments can be combined with each other to form the following further preferred manner:

[0095] All temporary files can be divided into two groups according to the amount of data of the temporary files in the memory, the temporary files in the first group have a data amount in the memory greater than or equal to a fifth threshold (i.e., the data amount of the data block uploaded to the object storage), and the temporary files in the second group have a data amount in the memory less than the fifth threshold; then, an lru list for reflecting the heat of each group of temporary files is constructed, each node in the list is used to represent a temporary file in the group, and the nodes corresponding to the temporary files that have been recently used and modified are at the tail of the lru list, i.e., as shown in lots_lru_list and less_lru_list in FIG. 10, the temporary files in the two lists correspond to the temporary files shown in FIG. 11. Figure 5 Figure 3 When flushing, the temporary files corresponding to the nodes at the head of the lru list of the first group can be preferentially selected, and the selected temporary files are divided into at least one data block according to the fifth threshold, and the obtained data block is flushed to the object storage, and the remaining data is flushed to the local disk.

[0096] In this embodiment, the performance jitter caused by the elimination and the memory utilization are considered to control the elimination water line with the fourth threshold and the fifth threshold. Eliminating a large amount of memory data at a time can quickly free up resources, but it will significantly slow down the SQL statement that triggers the elimination, causing performance jitter. In addition, such behavior is not friendly to read operations, which can cause requests that could have hit in memory to miss. Therefore, we set the memory elimination water line to keep the memory usage rate at about 80%, balancing the overall performance of read and write.

[0097] Please refer to FIG. 12, which shows the control logic of the temporary file writing stage combined with the above multiple embodiments. Figure 6

[0098] Please refer to FIG. 13, which shows the control logic of the temporary file writing stage combined with the above multiple embodiments. Figure 7

[0099] When the space for storing temporary files in the memory is tight, this embodiment selects part of the temporary files to flush to the object storage or the local disk, thereby releasing part of the memory temporary pages for subsequent temporary files to use, so that the limited space for storing temporary files in the memory can be recycled in the scenario of continuous generation of temporary files, thereby improving the utilization of the memory and the success rate of temporary file writing.

[0100] ​​​Some embodiments of the present specification further comprise: receiving a data read instruction for any temporary file, reading part or all of the data of the temporary file from the persistent storage device and / or the memory based on the data location range indicated by the data read instruction.

[0101] When reading the data in the temporary file, all of the data of the temporary file can be in the memory; all of the data of the temporary file can also be in the persistent storage device; or part of the data of the temporary file can be in the memory and part of the data of the temporary file can be in the persistent storage device.

[0102] When reading the data in the temporary file, all of the data of the temporary file can be read or part of the data of the temporary file can be read.

[0103] The data read instruction can be characterized by a data offset, i.e. the position of the data in the temporary file that is desired to be read. For example, the data location range is from the start of the data to the position of a certain data offset, or the data location range is from the position of a certain data offset to the end of the data, or the data location range is from the position of a certain data offset to the position of another data offset.

[0104] For example, the data reading step of the present embodiment is performed in the following manner:

[0105] First, the data location range of the data of the temporary file stored in the memory in the temporary file is determined according to the identification of the memory temporary page storing the temporary file recorded in the memory.

[0106] As described above, the identification of the memory temporary page storing the temporary file recorded in the memory is the metadata of the temporary file in the memory, which records the identification of the first memory temporary page occupied by the temporary file and the number of memory temporary pages occupied. The amount of data of the temporary file in the memory can be calculated by the capacity of the memory temporary page (for example, 8KB in the foregoing example) and the number of memory temporary pages occupied by the temporary file, and then the data location range of the data of the temporary file in the temporary file is determined based on the amount of data of the temporary file.

[0107] The data location range of the data of the temporary file stored in the memory in the temporary file can be characterized by a flushed offset, i.e. the data from 0 to flushed offset in the temporary file is in the persistent storage device, and the data from flushed offset to tem_file_length in the temporary file is in the memory, where tem_file_length is the amount of data of the temporary file.

[0108] Next, if the data location range indicated by the data read instruction is within the data location range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the memory.

[0109] That is, if the data location range indicated by the data read instruction is between flushed_offset and tem_file_length (including flushed_offset and tem_file_length), the corresponding data of the temporary file is read from the memory. For example, based on the data location range indicated by the data read instruction, a target memory temporary page is determined in the memory, and part or all of the data of the temporary file is read from the target memory temporary page. For example, if the capacity of the memory temporary page is 8 KB, and the data location range indicated by the data read instruction is flushed_offset to flushed_offset+8n KB, the first n memory temporary pages of the temporary file in the memory are taken as the target memory temporary page, and the data is read from the first n memory temporary pages.

[0110] Next, if the data location range indicated by the data read instruction has no intersection with the data location range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device.

[0111] That is, if the data location range indicated by the data read instruction is between 0 and flushed_offset (including 0 and flushed_offset), the corresponding data of the temporary file is read from the persistent storage device. For example, based on the data location range indicated by the data read instruction, a target data block is determined in the persistent storage device, and part or all of the data of the temporary file is read from the target data block. For example, if the data amount of the data block is 2 MB, and the data location range indicated by the data read instruction is 10 MB to flushed_offset, the 6th data block to the last data block in the persistent storage device are all determined as the target data block, and the data of the temporary file in the target data block is read.

[0112] Next, if the data location range indicated by the data read instruction is partially within the data location range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device and the memory.

[0113] When the data location range indicated by the data read instruction crosses the flushed_offset, the corresponding data is read from the persistent storage device and the memory. For example, based on the start position of the data location range indicated by the data read instruction to the flushed_offset, a target data block is determined in the persistent storage device, based on the flushed_offset to the end position of the data location range indicated by the data read instruction, a target memory temporary page is determined in the memory, and data of the temporary file is read from the target data block and the target memory temporary page. For example, if the data amount of a data block is 2 MB, the capacity of a memory temporary page is 8 KB, the data location range indicated by the data read instruction is 10 MB to flushed_offset+8n KB, the 6th data block to the last data block in the persistent storage device are all determined as the target data block, the first n memory temporary pages of the temporary file in the memory are determined as the target memory temporary page, and data of the temporary file is read from the target data block and the target memory temporary page.

[0114] FIG. 1 shows a schematic structural diagram of a device according to an example embodiment. Figure 8 FIG. 2 shows a control logic of a temporary file reading stage according to an example embodiment.

[0115] In summary, the temporary file management method provided in the present specification directly stores temporary file data in data blocks instead of a temporary directory of a file system, so that the memory, IO, disk, and other resource management and isolation capabilities of a database system can be utilized to reduce the impact of temporary files on normal user operations. The space in the memory for storing temporary files in the temporary file management method provided in the present specification is strictly controllable, further controlling the impact of the temporary file module on the overall system. The simplified and efficient metadata management manner in the temporary file management method provided in the present specification takes into account the unified storage and management of large temporary files and small temporary files, reducing additional system memory overhead and storage overhead.

[0116] Figure 9 FIG. 1 shows a schematic structural diagram of a device according to an example embodiment. Figure 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and can also include other hardware required by tasks. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 910 into the memory 908 by the processor 902 and then running. Of course, in addition to the software implementation manner, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0117] Please refer to Figure 10 , the temporary file management device can be applied to the device as Figure 9 shown to achieve the technical solutions of the specification. The temporary file management device can include:

[0118] The allocation module 1001 is configured to allocate a memory temporary page to a temporary file to be stored, and write data of the temporary file into the memory temporary page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier.

[0119] The linking module 1002 is configured to allocate a new memory temporary page to the temporary file whenever a memory temporary page is written by the data of the temporary file to a remaining capacity less than a first threshold, record the identifier of the new memory temporary page in the memory temporary page with a remaining capacity less than the first threshold, and continue to write the data of the temporary file into the new memory temporary page.

[0120] The metadata module 1003 is configured to record the identifier of at least one memory temporary page storing the temporary file in the memory if the data of the temporary file is completely written into the memory.

[0121] In one possible embodiment of the specification, the device further comprises a cutoff module configured to:

[0122] If the data of the temporary file is completely written into the memory, record an end identifier in the last memory temporary page storing the temporary file, wherein the end identifier is used to represent that the memory temporary page is the last memory temporary page storing the temporary file.

[0123] In one possible embodiment of the specification, the metadata module is configured to:

[0124] Record the identifier of the first memory temporary page storing the temporary file and the number of at least one memory temporary page storing the temporary file in the memory.

[0125] In one possible embodiment of the specification, the device further comprises a disk flushing module configured to:

[0126] If the remaining capacity of the space in the memory of the database for storing temporary files is less than a third threshold, flush at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold, and release the memory temporary page storing the at least one temporary file.

[0127] In a possible implementation of the present specification, the disk flushing module is configured to:

[0128] flushing, to the persistent storage device, at least one temporary file stored in the memory in a manner that prioritizes temporary files with the largest data volume, until the remaining capacity of the memory of the database for storing temporary files is not less than a fourth threshold value.

[0129] In a possible implementation of the present specification, the disk flushing module is configured to:

[0130] flushing, to the persistent storage device, at least one temporary file stored in the memory in a manner that prioritizes temporary files with the lowest access frequency, until the remaining capacity of the memory of the database for storing temporary files is not less than a fourth threshold value.

[0131] In a possible implementation of the present specification, the disk flushing module is configured to:

[0132] if any temporary file with a data volume not less than a fifth threshold value is flushed to the persistent storage device, dividing the temporary file into at least one data block in a manner that each fifth threshold value is divided into one data block, and uploading the at least one data block to the object storage layer;

[0133] if any temporary file with a data volume less than the fifth threshold value is flushed to the persistent storage device, flushing the temporary file to the local disk of the database.

[0134] In a possible implementation of the present specification, the apparatus further includes an uploading module configured to:

[0135] dividing data flushed to the local disk of the database into at least one data block in a manner that each fifth threshold value is divided into one data block;

[0136] if the local disk satisfies a preset uploading condition, uploading the data block in the local disk to the object storage layer.

[0137] In a possible implementation of the present specification, the apparatus further includes a reading module configured to:

[0138] receiving a data reading instruction for any temporary file, and reading part or all of the data of the temporary file from the persistent storage device and / or the memory based on a data location range indicated by the data reading instruction.

[0139] In a possible implementation of the present specification, the reading module is configured to:

[0140] determining a data location range of the data of the temporary file stored in the memory in the temporary file according to an identifier of a memory temporary page storing the temporary file recorded in the memory;

[0141] if the data location range indicated by the data read instruction is within the data location range of the temporary file stored in the memory, reading the partial or entire data of the temporary file from the memory;

[0142] if the data location range indicated by the data read instruction has no intersection with the data location range of the temporary file stored in the memory, reading the partial or entire data of the temporary file from the persistent storage device;

[0143] if the data location range indicated by the data read instruction is partially within the data location range of the temporary file stored in the memory, reading the partial or entire data of the temporary file from the persistent storage device and the memory.

[0144] In a possible implementation of the present specification, when the reading module is configured to read the partial or entire data of the temporary file from the persistent storage device based on the data location range indicated by the data read instruction, the reading module is configured to:

[0145] determining a target data block in the persistent storage device based on the data location range indicated by the data read instruction, and reading the partial or entire data of the temporary file from the target data block.

[0146] In a possible implementation of the present specification, when the reading module is configured to read the partial or entire data of the temporary file from the memory based on the data location range indicated by the data read instruction, the reading module is configured to:

[0147] determining a target memory temporary page in the memory based on the data location range indicated by the data read instruction, and reading the partial or entire data of the temporary file from the target memory temporary page.

[0148] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an e-mail device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0149] In a typical configuration, a computer includes one or more processors (CPU), input / output interface, network interface, and memory.

[0150] Memory can include, without being limited to, non- transitory computer-readable media, random access memory (RAM), and / or read-only memory (ROM), such as flash memory, among others. Memory is an example of computer readable media.

[0151] Computer readable media includes permanent and non- permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, without being limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition provided herein, a computer readable medium does not include transitory computer readable media, such as modulated data signals and carrier waves.

[0152] It is also important to note that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0153] The above description of certain examples of the application has been presented for the purposes of illustration and description. Other examples are within the scope and range of equivalents of the claims. In some cases, acts or steps can be performed in an order different from that of the examples, and / or at least partially concurrently, and still accomplish the desired results. Also, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0154] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments described in this description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0155] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the specification are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0156] It should be understood that, although the terms first, second, third, etc. can be employed in this description to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, a first information can also be referred to as a second information without departing from the scope of one or more embodiments described in this description, and similarly, a second information can also be referred to as a first information. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "upon" or "in response to determining".

[0157] The above descriptions are only preferred embodiments of one or more embodiments of the specification and are not intended to limit one or more embodiments of the specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the specification shall be included in the scope of protection of one or more embodiments of the specification.

Claims

1. A temporary file management method, the method comprising: allocating a memory temporary page to a temporary file to be stored, and writing data of the temporary file into the memory temporary page, wherein a space in a memory of a database for storing temporary files is divided into a plurality of memory temporary pages, each memory temporary page having an identifier; whenever a memory temporary page is written by the data of the temporary file to a remaining capacity less than a first threshold, allocating a new memory temporary page to the temporary file, recording an identifier of the new memory temporary page in the memory temporary page with the remaining capacity less than the first threshold to form a page chain table, and continuing to write the data of the temporary file into the new memory temporary page; if the data of the temporary file is completely written into the memory, recording in the memory an identifier of at least one memory temporary page storing the temporary file; wherein the recording in the memory the identifier of the at least one memory temporary page storing the temporary file comprises: recording in the memory metadata of the temporary file, the metadata comprising an identifier of a first memory temporary page allocated to store the temporary file and a total number of memory temporary pages allocated to store the temporary file. 2.The temporary file management method of claim 1, further comprising: if the data of the temporary file is completely written into the memory, recording in a last memory temporary page storing the temporary file an end identifier, wherein the end identifier is used to represent that the memory temporary page is the last memory temporary page storing the temporary file. 3.The temporary file management method of claim 1, wherein the recording in the memory the identifier of the at least one memory temporary page storing the temporary file comprises: recording in the memory the identifier of the first memory temporary page storing the temporary file and a number of the at least one memory temporary page storing the temporary file. 4.The temporary file management method of claim 1, further comprising: if a remaining capacity of the space in the memory of the database for storing temporary files is less than a third threshold, flushing at least one temporary file stored in the memory to a persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold, and releasing memory temporary pages storing the at least one temporary file. 5.The temporary file management method of claim 4, wherein the flushing the at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold comprises: flushing the at least one temporary file stored in the memory to the persistent storage device in a manner that a temporary file with a largest amount of data to be flushed is preferentially flushed until the remaining capacity of the space in the memory of the database for storing temporary files is not less than the fourth threshold.

6. The temporary file management method of claim 4, wherein the step of flushing the at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the memory space for storing temporary files in the database is not less than the fourth threshold comprises: flushing the at least one temporary file stored in the memory to the persistent storage device in a manner that the temporary file with the lowest access frequency is preferentially flushed until the remaining capacity of the memory space for storing temporary files in the database is not less than the fourth threshold.

7. The temporary file management method of claim 4, wherein the step of flushing the at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the memory space for storing temporary files in the database is not less than the fourth threshold comprises: if the temporary file with any data amount not less than the fifth threshold is flushed to the persistent storage device, dividing the temporary file into at least one data block in a manner that each fifth threshold is a data block, and uploading the at least one data block to the object storage layer; if the temporary file with any data amount less than the fifth threshold is flushed to the persistent storage device, flushing the temporary file to the local disk of the database.

8. The temporary file management method of claim 7, further comprising: dividing the data flushed to the local disk of the database into at least one data block in a manner that each fifth threshold is a data block; if the local disk satisfies a preset uploading condition, uploading the data block in the local disk to the object storage layer.

9. The temporary file management method of claim 4, further comprising: receiving a data reading instruction for any temporary file, and reading part or all of the data of the temporary file from the persistent storage device and / or the memory based on the data location range indicated by the data reading instruction.

10. The temporary file management method of claim 9, wherein the step of reading part or all of the data of the temporary file from the persistent storage device and / or the memory based on the data location range indicated by the data reading instruction comprises: determining the data location range of the data of the temporary file stored in the memory in the temporary file according to the identifier of the memory temporary page storing the temporary file recorded in the memory; if the data location range indicated by the data reading instruction is within the data location range of the data of the temporary file stored in the memory in the temporary file, reading part or all of the data of the temporary file from the memory; if the data location range indicated by the data reading instruction has no intersection with the data location range of the data of the temporary file stored in the memory in the temporary file, reading part or all of the data of the temporary file from the persistent storage device; if the data location range indicated by the data reading instruction is partially within the data location range of the data of the temporary file stored in the memory in the temporary file, reading part or all of the data of the temporary file from the persistent storage device and the memory. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 11.The temporary file management method of claim 9, wherein the reading the partial or entire data of the temporary file from the data location range indicated by the data read instruction comprises: determining a target data block in the persistent storage according to the data location range indicated by the data read instruction, and reading the partial or entire data of the temporary file from the target data block. 12.The temporary file management method of claim 9, wherein the reading the partial or entire data of the temporary file from the data location range indicated by the data read instruction comprises: determining a target memory temporary page in the memory according to the data location range indicated by the data read instruction, and reading the partial or entire data of the temporary file from the target memory temporary page. 13.A temporary file management apparatus, comprising: an allocation module configured to allocate a memory temporary page to a temporary file to be stored, and write data of the temporary file into the memory temporary page, wherein a memory space of a database for storing temporary files is divided into a plurality of memory temporary pages, and each memory temporary page has an identifier; a linking module configured to, whenever a memory temporary page is written by data of the temporary file to a remaining capacity less than a first threshold, allocate a new memory temporary page to the temporary file, record an identifier of the new memory temporary page in the memory temporary page with the remaining capacity less than the first threshold to form a page chain table, and continue to write data of the temporary file into the new memory temporary page; a metadata module configured to, if the data of the temporary file is completely written into the memory, record identifiers of at least one memory temporary page storing the temporary file in the memory; wherein the recording the identifiers of the at least one memory temporary page storing the temporary file in the memory comprises: recording metadata of the temporary file in the memory, the metadata comprising an identifier of a first memory temporary page allocated to store the temporary file and a total number of memory temporary pages allocated to store the temporary file. 14.The temporary file management apparatus of claim 13, further comprising a flushing module configured to: if a remaining capacity of the memory space of the database for storing temporary files is less than a third threshold, flush at least one temporary file stored in the memory to the persistent storage until the remaining capacity of the memory space of the database for storing temporary files is not less than a fourth threshold, and release memory temporary pages storing the at least one temporary file. 15.The temporary file management apparatus of claim 14, wherein the flushing module is configured to: if any temporary file with a data amount not less than a fifth threshold is flushed to the persistent storage, divide the temporary file into at least one data block in a manner of dividing each fifth threshold into one data block, and upload the at least one data block to an object storage layer. ​ ​ ​ ​ ​ If any data volume of the temporary file is less than the fifth threshold value, the temporary file is flushed to the local disk of the database. 16.The temporary file management apparatus of claim 15, further comprising an uploading module configured to: divide the data flushed to the local disk of the database into at least one data block in a manner of dividing into a data block per fifth threshold value; if the local disk meets a preset uploading condition, upload the data block in the local disk to the object storage layer. 17.The temporary file management apparatus of claim 14, further comprising a reading module configured to: receive a data reading instruction for any temporary file, and read part or all of the data of the temporary file from the persistent storage device and / or the memory based on a data location range indicated by the data reading instruction. 18.A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 12. 19.An electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 1 to 12 by running the executable instructions. 20.A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 12.

Images