Temporary file management method and device, electronic equipment and storage medium

Abstract

The invention provides a temporary file management method and device, electronic equipment and a storage medium, and the method comprises the steps: distributing a memory temporary page to a to-be-stored temporary file, and writing the data of the temporary file into the memory temporary page, a space used for storing temporary files in a memory of the database is divided into a plurality of memory temporary pages, and each memory temporary page is provided with an identifier; when one memory temporary page is written by the data of the temporary file until the residual capacity is smaller than a first threshold value, allocating a new memory temporary page to the temporary file, and recording an identifier of the new memory temporary page in the memory temporary page of which the residual capacity is smaller than the first threshold value, continuing to write the data of the temporary file into the new memory temporary page; and if the data of the temporary file is completely written into the memory, recording an identifier of at least one memory temporary page for storing the temporary file in the memory.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a temporary file management method and device, an electronic device, and a storage medium. Background Art

[0002] The database generates temporary files during the execution of data query and index building. Temporary files refer to temporary files generated during the process of executing computing tasks. These files have a short life cycle and will disappear when the process disappears.

[0003] A large number of temporary files are generated during the daily operation of the database, and some temporary files occupy a large amount of space. The management effect of temporary files in related technologies is poor. For example, temporary files occupy a large amount of space due to complex information such as metadata. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a temporary file management method and device, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a temporary file management method is proposed, the method comprising: Allocating a temporary memory page to a temporary file to be stored, and writing data of the temporary file into the temporary memory page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of temporary memory pages, each of which has an identifier; Whenever a temporary memory page is written with data of the temporary file until the remaining capacity is less than a first threshold, a new temporary memory page is allocated to the temporary file, an identifier of the new temporary memory page is recorded in the temporary memory page whose remaining capacity is less than the first threshold, and the data of the temporary file continues to be written into the new temporary memory page; If the data of the temporary file is completely written into the memory, an identifier of at least one temporary memory page storing the temporary file is recorded in the memory.

[0006] In a possible embodiment of this specification, the method further includes: If the data of the temporary file is completely written into the memory, an end mark is recorded in the last temporary memory page storing the temporary file, wherein the end mark is used to indicate that the temporary memory page is the last temporary memory page storing the temporary file.

[0007] In a possible embodiment of the present specification, recording in the memory an identifier of at least one temporary memory page storing the temporary file includes: The identifier of the first temporary memory page storing the temporary file and the number of at least one temporary memory page storing the temporary file are recorded in the memory.

[0008] In a possible embodiment of this specification, the method further includes: If the remaining capacity of the space in the database's memory for storing temporary files is less than a third threshold, at least one temporary file stored in the memory is flushed to the persistent storage device until the remaining capacity of the space in the database's memory for storing temporary files is not less than a fourth threshold, and the temporary memory page storing the at least one temporary file is released.

[0009] In a possible embodiment of the present specification, flushing at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space for storing temporary files in the memory of the database is not less than a fourth threshold includes: At least one temporary file stored in the memory is flushed to the persistent storage device in a manner of giving priority to temporary files with the largest amount of data, until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold.

[0010] In a possible embodiment of the present specification, flushing at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space for storing temporary files in the memory of the database is not less than a fourth threshold includes: At least one temporary file stored in the memory is flushed to the persistent storage device in a manner that prioritizes temporary files with the lowest disk access heat, until the remaining capacity of the space in the database memory for storing temporary files is not less than a fourth threshold.

[0011] In a possible embodiment of the present specification, flushing at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space for storing temporary files in the memory of the database is not less than a fourth threshold includes: If any temporary file with a data volume not less than the fifth threshold is flushed to the persistent storage device, the temporary file is divided into at least one data block in a manner of dividing each fifth threshold into one data block, and the at least one data block is uploaded to the object storage layer; If any temporary file whose data volume is less than the fifth threshold is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

[0012] In a possible embodiment of this specification, the method further includes: Divide the data of the local disk flushed to the database into at least one data block in a manner of dividing each fifth threshold into one data block; If the local disk meets the preset upload condition, the data blocks in the local disk are uploaded to the object storage layer.

[0013] In a possible embodiment of this specification, the method further includes: A data read instruction for any temporary file is received, and part or all of the data of the temporary file is read from a persistent storage device and / or a memory based on a data location range indicated by the data read instruction.

[0014] In a possible embodiment of the present specification, reading part or all of the data of the temporary file from a persistent storage device and / or a memory based on the data location range indicated by the data reading instruction includes: Determine, according to the identifier of the temporary memory page storing the temporary file recorded in the memory, a data location range of the data of the temporary file stored in the memory in the temporary file; If the data position range indicated by the data reading instruction is within the data position range of the temporary file stored in the memory, then part or all of the data of the temporary file is read from the memory; If the data position range indicated by the data reading instruction has no intersection with the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device; If the data position range indicated by the data read instruction is partially within the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device and the memory.

[0015] In a possible embodiment of the present specification, reading part or all of the data of the temporary file from the persistent storage device based on the data location range indicated by the data reading instruction includes: A target data block is determined in the persistent storage device based on the data location range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target data block.

[0016] In a possible embodiment of the present specification, the reading part or all of the data of the temporary file from the memory based on the data location range indicated by the data reading instruction includes: A target memory temporary page is determined in the memory based on the data position range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target memory temporary page.

[0017] According to a second aspect of one or more embodiments of this specification, a temporary file management device is provided, the device comprising: An allocation module, used to allocate a temporary memory page to a temporary file to be stored, and write the data of the temporary file into the temporary memory page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of temporary memory pages, and each temporary memory page has an identifier; A link module, configured to allocate a new temporary memory page to the temporary file whenever a temporary memory page is written by the data of the temporary file to a remaining capacity less than a first threshold, record an identifier of the new temporary memory page in the temporary memory page whose remaining capacity is less than the first threshold, and continue to write the data of the temporary file into the new temporary memory page; The metadata module is used to record in the memory an identifier of at least one temporary memory page storing the temporary file if the data of the temporary file is completely written into the memory.

[0018] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0019] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor implements the method as described in the first aspect by running the executable instructions.

[0020] According to a fifth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0021] The technical solutions provided by the embodiments of this specification may have the following beneficial effects: In the temporary file management method provided by the embodiment of the present specification, the space for storing temporary files in the memory of the database is divided into multiple temporary memory pages, each temporary memory page has an identifier; when storing a temporary file, first allocate a temporary memory page to the temporary file to be stored, and write the data of the temporary file into the temporary memory page, and whenever a temporary memory page is written by the data of the temporary file to a remaining capacity less than a first threshold, a new temporary memory page is allocated to the temporary file, and the identifier of the new temporary memory page is recorded in the temporary memory page whose remaining capacity is less than the first threshold, and the data of the temporary file continues to be written into the new temporary memory page; if the data of the temporary file is completely written into the memory, the identifier of at least one temporary memory page storing the temporary file is recorded in the memory. The method divides the space for storing temporary files into multiple temporary memory pages, and the temporary file writes data to the memory by applying for temporary memory pages one by one, and forms a page list of temporary files by recording the identifier of the next temporary memory page in each temporary memory page, so that the management of temporary files is simple and efficient, and the form of the page list simplifies the metadata of temporary files and occupies less space, which improves the management effect of temporary files and improves the utilization rate of memory as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the architecture of a data system provided by an exemplary embodiment.

[0023] Figure 2 is a flowchart of a temporary file management method provided by an exemplary embodiment.

[0024] Figure 3 It is a metadata diagram of a temporary file provided by an exemplary embodiment.

[0025] Figure 4 This is a temporary file management method provided by an exemplary embodiment.

[0026] Figure 5 It is a linked list of temporary files provided by an exemplary embodiment.

[0027] Figure 6 This is the control logic of the history file writing phase provided by an exemplary embodiment.

[0028] Figure 7 This is the control logic of the historical file flushing stage provided by an exemplary embodiment.

[0029] Figure 8 This is the control logic of the history file reading phase provided by an exemplary embodiment.

[0030] Fig. 9It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0031] Fig.10 is a block diagram of a temporary file management device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0033] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0034] First, some concepts involved in this specification are explained.

[0035] LSM-Tree: An efficient storage structure widely used in database systems that aims to optimize write performance and disk space utilization. It reduces the overhead of random writes by caching write operations in memory and periodically merging data to disk.

[0036] Temporary files: Temporary files generated when a process executes computing tasks. These files have a short life cycle and will disappear when the process disappears.

[0037] Object storage: Object storage is a data management architecture that treats data as objects rather than files or blocks to efficiently and flexibly store and retrieve massive amounts of unstructured data. Its "everything is an object" concept emphasizes that all data, whether it is documents, pictures, videos or logs, can be processed as objects, which helps improve access efficiency and scalability.

[0038] Based on the above technical problems in the background technology, at least one embodiment of this specification provides a temporary file management method, which can be applied to a database, such as a database implemented based on an LSM-Tree storage engine. Figure 1A database instance in the data system shown in the figure. The data system includes an object storage layer and multiple databases that communicate with it. Each database instance has a local disk and memory. Object storage has the characteristics of low cost (the unit price is a fraction of that of cloud disk), payment based on actual usage (capacity and read and write request volume), read-only, and sharing. Using object storage as shared storage can significantly reduce storage costs. However, object storage has limitations such as large read and write latency, and was previously only suitable for businesses that are not very sensitive to latency. Using memory and local disk to cache data and metadata on object storage can make shared storage suitable for most OLTP businesses.

[0039] This method can efficiently manage temporary files when they are generated in the database, ensuring that the write and read operations of the temporary files can be executed efficiently, avoiding the impact of excessive complexity in the management of temporary files on database operations, and reducing the space occupied by temporary files in the database by simplifying the metadata of temporary files.

[0040] Please refer to the attached Figure 2 , which exemplarily shows a flow chart of the method, including steps S201 to S203.

[0041] In step S201, a temporary memory page is allocated to a temporary file to be stored, and data of the temporary file is written into the temporary memory page, wherein the space in the database memory for storing temporary files is divided into multiple temporary memory pages, each of which has an identifier.

[0042] The database can allocate a portion of the write cache in the memory to store temporary files. This space can be divided into multiple pages in units of a certain capacity value, which are called temporary memory pages. Each temporary memory page has an identifier. After the database is initialized or restarted, all temporary memory pages are idle. During the operation of the database, all temporary memory pages are idle; the idle temporary memory pages can form a linked list, and each temporary memory page in the linked list can record the identifier of the next temporary memory page. For example, the idle temporary memory pages can be sorted in the order of the identifiers to form a linked list.

[0043] In this step, when writing a temporary file into the memory, a temporary memory page may be requested from the linked list formed by the idle temporary memory pages, and the data of the temporary file may be written into the allocated temporary memory page. For example, the temporary memory page at the head or tail of the linked list formed by the idle temporary memory pages may be allocated to the temporary file.

[0044] In step S202, whenever a temporary memory page is written with data from the temporary file to a remaining capacity less than a first threshold, a new temporary memory page is allocated to the temporary file, an identifier of the new temporary memory page is recorded in the temporary memory page whose remaining capacity is less than the first threshold, and the data of the temporary file continues to be written into the new temporary memory page.

[0045] In the process of writing the data of the temporary file to the temporary memory page allocated in step S201, the temporary memory page will be gradually occupied by the data and the remaining space will be reduced. When the remaining space is reduced to a certain extent (for example, less than the first threshold), a new temporary memory page can be applied for from the linked list formed by the above-mentioned idle temporary memory pages to continue writing the data of the temporary file. Similarly, after the remaining capacity of the previous temporary memory page is less than the first threshold, a new temporary memory page will be applied for to continue writing the data of the temporary file until the data of the temporary file is completely written into the memory.

[0046] After each application (i.e., allocation) of a new temporary memory page, the identifier of the new temporary memory page can be written into the previous temporary memory page, so that all temporary memory pages of data written into the temporary file are linked into a page linked list.

[0047] In step S203, if the data of the temporary file is completely written into the memory, an identifier of at least one temporary memory page storing the temporary file is recorded in the memory.

[0048] Among them, recording the identifier of at least one temporary memory page storing the temporary file in the memory is to generate and save the metadata of the temporary file in the memory. For example, the identifier of the first temporary memory page storing the temporary file and the number of at least one temporary memory page storing the temporary file can be recorded in the memory. In this method, the metadata of the temporary file cached in the memory is simple, easy to manage, and occupies less space. For example, the metadata of the temporary file is as shown in the attached Figure 3 As shown, it records the tmp_file_id and meta_data of each temporary file in the form of a table, where tmp_file_id is the id of the temporary file, and meta_data is the metadata of the temporary file, including the identifier begin_page_id of the first temporary memory page and the number page_nums of temporary memory files.

[0049] In addition, if the data of the temporary file is completely written into the memory, an end mark is recorded in the last temporary memory page storing the temporary file, wherein the end mark is used to indicate that the temporary memory page is the last temporary memory page storing the temporary file, so as to facilitate the identification of the end mark when reading the temporary file.

[0050] Please refer to the attached Figure 4 , which exemplarily shows the temporary file management method provided by this embodiment.

[0051] The space used to store temporary files in the write cache is divided into multiple Pages in units of 8KB, and each Page is maintained by a PageEntry. For the linked list formed by multiple Pages storing a temporary file, the PageEntry of each Page in the linked list records tem_file_id, page_id, next_page_id, and page_buffer in the Page. tem_file_id is the identifier of the temporary file to which it belongs; page_id is the identifier of the Page; next_page_id is the identifier of the next Page. The next of each Page except the last Page records the identifier of the next Page, and the next of the last Page records the end identifier, such as -1; page_buffer is the data cached in the Page. All idle Pages form a linked list free list.

[0052] The temporary file management method provided in the embodiments of the present specification divides the space for storing temporary files into multiple temporary memory pages. The temporary files write data to the memory by applying for temporary memory pages one by one, and the temporary files are formed into a page list by recording the identifier of the next temporary memory page in each temporary memory page, thereby making the management of temporary files simple and efficient. In addition, the form of page list simplifies the metadata of temporary files and occupies less space, which improves the overall management effect of temporary files and improves the utilization rate of memory.

[0053] Some embodiments of the present specification also include: if the remaining capacity of the space for storing temporary files in the memory of the database is less than a third threshold, at least one temporary file stored in the memory is flushed to a persistent storage device until the remaining capacity of the space for storing temporary files in the memory of the database is not less than a fourth threshold, and the temporary memory page storing the at least one temporary file is released.

[0054] The third threshold value may be equal to the fourth threshold value. Alternatively, the third threshold value may be greater than the fourth threshold value, thereby preventing the memory space from fluctuating near the third threshold value and causing the disk flushing operation to be too frequent.

[0055] Exemplarily, at least one temporary file stored in the memory is flushed to the persistent storage device by giving priority to temporary files with the largest amount of data until the remaining capacity of the space in the database memory for storing temporary files is no less than a fourth threshold.

[0056] Temporary files with large amounts of data occupy more temporary memory pages. Flushing temporary files with large amounts of data to persistent storage devices can free up more temporary memory pages to alleviate the capacity shortage of space in the memory used to store temporary files.

[0057] For example, all temporary files can be divided into two groups according to the amount of data in the memory of the temporary files, the amount of data in the memory of the temporary files in the first group is greater than or equal to a certain threshold, and the amount of data in the memory of the temporary files in the second group is less than a certain threshold. Therefore, when flushing temporary files to disk, the temporary files in the first group can be preferentially selected for flushing.

[0058] As another example, at least one temporary file stored in the memory is flushed to the persistent storage device by giving priority to temporary files with the lowest disk access popularity until the remaining capacity of the space for storing temporary files in the database memory is not less than a fourth threshold.

[0059] Temporary files with higher temperatures are more likely to be accessed, while temporary files with lower temperatures are less likely to be accessed. Therefore, eliminating temporary files with the lowest temperature can minimize the impact on the hit rate of temporary files in memory.

[0060] For example, an lru linked list can be constructed to reflect popularity. Each node in the linked list is used to represent a temporary file. The nodes corresponding to the most recently used and modified temporary files are at the end of the lru linked list. Therefore, the temporary files corresponding to the nodes at the head of the lru linked list can be preferentially selected for disk flushing.

[0061] As another example, if any temporary file whose data volume is not less than the fifth threshold is flushed to the persistent storage device, the temporary file is divided into at least one data block in such a manner that each fifth threshold is divided into one data block, and the at least one data block is uploaded to the object storage layer; if any temporary file whose data volume is less than the fifth threshold is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

[0062] In this example, temporary files with enough data to form at least one data block segment are directly flushed to the object storage layer, which can reduce the storage pressure of the local disk and simplify the redundant process of transferring data from memory to disk and from disk to object storage, thereby reducing system IO pressure.

[0063] It should be understood that if the data volume of the temporary file forms at least one data block and there is no remaining data (that is, the data is not enough to form a data block), then the at least one data block can be flushed to the object storage; if the data volume of the temporary file forms at least one data block and there is remaining data (that is, the data is not enough to form a data block), then after flushing the at least one data block to the object storage, the remaining data can be flushed to the local disk.

[0064] Uploading to object storage in blocks can reduce latency and take advantage of the high bandwidth characteristics of object storage to improve upload efficiency.

[0065] It should also be understood that when flushing data to the current disk of the database, the data flushed to the local disk of the database can be divided into at least one data block in a manner that every fifth threshold is divided into one data block; if the local disk meets the preset upload conditions, the data blocks in the local disk are uploaded to the object storage layer. That is, when flushing data to the local disk, the data blocks are synchronously divided so that the data written to the local disk is in the form of data blocks, which is convenient for subsequent upload to the object storage; and uploading the data blocks in the local disk to the object storage in a timely manner can release the space of the local disk in a timely manner and improve its space utilization.

[0066] Since data on persistent storage devices, i.e., local disks and object storage, are in the form of data blocks, and the amount of data in a data block is fixed, the metadata of temporary files in persistent storage devices can be recorded as disk offsets, which can be used to locate specific data blocks and data offsets within data blocks.

[0067] The above three examples provide specific methods for the process of flashing temporary files from different perspectives, thereby improving the effect of the process. The above three embodiments can be combined with each other to form the following further preferred methods: All temporary files can be divided into two groups according to the amount of data in the memory of the temporary files. The amount of data in the memory of the temporary files in the first group is greater than or equal to the fifth threshold (that is, the amount of data blocks uploaded to the object storage), and the amount of data in the memory of the temporary files in the second group is less than the fifth threshold. Then, an lru linked list for reflecting the popularity is constructed for the temporary files in each group. Each node in the linked list is used to represent a temporary file in the group. The nodes corresponding to the most recently used and modified temporary files are at the end of the lru linked list, as shown in the attached figure. Figure 5 The lots_lru_list and less_lru_list shown in the figure correspond to the temporary files in the attached Figure 3When flushing the disk, the temporary file corresponding to the node at the head of the lru linked list of the first group can be preferentially selected, and the selected temporary file can be divided into at least one data block according to the fifth threshold, and the obtained data block can be flushed to the object storage, and the remaining data can be flushed to the local disk.

[0068] In this embodiment, the performance jitter caused by elimination and memory utilization are taken into account, and the elimination waterline is controlled by the fourth threshold and the fifth threshold. Although eliminating a large amount of memory data at one time can quickly free up resources, it will cause the SQL statement that triggers the elimination to slow down significantly, resulting in performance jitter. In addition, such behavior is not friendly to read operations, and will cause requests that may have been hit in memory to fail. Therefore, we set the memory elimination waterline to keep the memory utilization rate at around 80% as much as possible to balance the overall performance of reading and writing.

[0069] Please refer to the attached Figure 6 , which shows the control logic of the temporary file writing stage obtained by combining the above multiple embodiments.

[0070] Please refer to the attached Figure 7 , which shows the control logic of the temporary file storage stage obtained by combining the above multiple embodiments.

[0071] When the space in the memory for storing temporary files is limited, this embodiment selects some temporary files to be flushed to the object storage or local disk, thereby releasing some temporary memory pages for use by subsequent written temporary files. This allows the limited space in the memory for storing temporary files to be recycled in scenarios where temporary files are constantly generated, thereby improving memory utilization and the success rate of temporary file writing.

[0072] Some embodiments of the present specification also include: receiving a data read instruction for any temporary file, and reading part or all of the data of the temporary file from a persistent storage device and / or memory based on a data location range indicated by the data read instruction.

[0073] When reading data in a temporary file, all the data of the temporary file may be in the memory; all the data of the temporary file may also be in a persistent storage device; it is also possible that part of the data of the temporary file is in the memory and part of the data is in a persistent storage device.

[0074] When reading data in a temporary file, all or part of the data in the temporary file may be read.

[0075] The data read instruction can represent the data position range with a data offset, that is, the position of the data to be read in the temporary file. For example, the data position range is the range from the data start position to the position of a certain data offset, or the data position range is the range from the position of a certain data offset to the data end position, or the data position range is the range from the position of a certain data offset to the position of another data offset.

[0076] Exemplarily, the data reading step of this embodiment is performed in the following manner: First, according to the identifier of the temporary memory page storing the temporary file recorded in the memory, the data position range of the data of the temporary file stored in the memory in the temporary file is determined.

[0077] As mentioned above, the identifier of the temporary memory page storing the temporary file recorded in the memory is the metadata of the temporary file in the memory, which records the identifier of the first temporary memory page occupied by the temporary file and the number of temporary memory pages occupied. The amount of data of the temporary file in the memory can be calculated by the capacity of the temporary memory page (for example, 8KB in the above example) and the number of temporary memory pages occupied by the temporary file, and then the data location range of the data of the temporary file in the temporary file can be determined based on the data amount of the temporary file.

[0078] The data location range of the temporary file stored in the memory can be represented by the disk offset flushed_offset, that is, the data from 0 to flushed_offset in the temporary file is in the persistent storage device, and the data from flushed_offset to tem_file_length is in the memory, where tem_file_length is the amount of data in the temporary file.

[0079] Next, if the data position range indicated by the data reading instruction is within the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the memory.

[0080] That is, when the data position range indicated by the data read instruction is between flushed_offset and tem_file_length (including flushed_offset and tem_file_length), the corresponding data of the temporary file is read from the memory. For example, the target memory temporary page is determined in the memory based on the data position range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target memory temporary page. If the capacity of the memory temporary page is 8KB, and the data position range indicated by the data read instruction is from flushed_offset to flushed_offset+8nKB, the first n memory temporary pages of the temporary file in the memory are used as the target memory temporary pages, and data is read from the first n memory temporary pages.

[0081] Next, if the data position range indicated by the data read instruction has no intersection with the data position range of the data of the temporary file stored in the memory in the temporary file, part or all of the data of the temporary file is read from the persistent storage device.

[0082] That is, when the data position range indicated by the data read instruction is between 0 and flushed_offset (including 0 and flushed_offset), the corresponding data of the temporary file is read from the persistent storage device. For example, the target data block is determined in the persistent storage device based on the data position range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target data block. If the data size of the data block is 2MB, and the data position range indicated by the data read instruction is 10MB to flushed_offset, the 6th data block to the last data block in the persistent storage device are determined as target data blocks, and the data of the temporary file in the target data block is read.

[0083] Next, if the data position range indicated by the data read instruction is partially within the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device and the memory.

[0084] That is, when the data position range indicated by the data read instruction spans flushed_offset, the corresponding data is read from the persistent storage device and the memory. For example, the target data block is determined in the persistent storage device based on the starting position of the data position range indicated by the data read instruction to flushed_offset, the target memory temporary page is determined in the memory based on flushed_offset to the end position of the data position range indicated by the data read instruction, and the data of the temporary file is read from the target data block and the target memory temporary page. If the data volume of the data block is 2MB, the capacity of the memory temporary page is 8KB, and the data position range indicated by the data read instruction is 10MB to flushed_offset+8nKB, then the 6th data block to the last data block in the persistent storage device are all determined as target data blocks, the first n memory temporary pages of the temporary file in the memory are used as target memory temporary pages, and the data of the temporary file is read from the target data block and the target memory temporary page.

[0085] Please refer to the attached Figure 8 , which shows the control logic of the temporary file reading stage obtained by combining the above multiple embodiments.

[0086] In summary, the temporary file management method provided in this specification stores temporary file data directly in the data block, rather than in a temporary directory of the file system. Therefore, the memory, IO, disk and other resource management and isolation capabilities of the database system can be utilized to reduce the impact of temporary files on normal user operations. In the temporary file management method provided in this specification, the space used to store temporary files in the memory is strictly controllable, further controlling the impact of the temporary file module on the overall system. The streamlined and efficient metadata management method in the temporary file management method provided in this specification takes into account the unified storage and management of large temporary files and small temporary files, reducing additional system memory overhead and storage overhead.

[0087] Fig. 9 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig. 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 902 reading the corresponding computer program from the non-volatile memory 910 into the memory 908 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0088] Please refer to Fig.10 The temporary file management device can be applied to Fig. 9 The temporary file management device may include: The allocation module 1001 is used to allocate a temporary memory page to a temporary file to be stored, and write the data of the temporary file into the temporary memory page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of temporary memory pages, and each temporary memory page has an identifier; The link module 1002 is used for allocating a new temporary memory page to the temporary file whenever a temporary memory page is written by the data of the temporary file to a remaining capacity less than a first threshold, recording an identifier of the new temporary memory page in the temporary memory page whose remaining capacity is less than the first threshold, and continuing to write the data of the temporary file into the new temporary memory page; The metadata module 1003 is used to record in the memory an identifier of at least one temporary memory page storing the temporary file if the data of the temporary file is completely written into the memory.

[0089] In a possible embodiment of this specification, the device further includes a cutoff module, which is used to: If the data of the temporary file is completely written into the memory, an end mark is recorded in the last temporary memory page storing the temporary file, wherein the end mark is used to indicate that the temporary memory page is the last temporary memory page storing the temporary file.

[0090] In a possible embodiment of this specification, the metadata module is used to: The identifier of the first temporary memory page storing the temporary file and the number of at least one temporary memory page storing the temporary file are recorded in the memory.

[0091] In a possible embodiment of the present specification, the device further includes a brush module for: If the remaining capacity of the space in the database's memory for storing temporary files is less than a third threshold, at least one temporary file stored in the memory is flushed to the persistent storage device until the remaining capacity of the space in the database's memory for storing temporary files is not less than a fourth threshold, and the temporary memory page storing the at least one temporary file is released.

[0092] In a possible embodiment of this specification, the brushing module is used for: At least one temporary file stored in the memory is flushed to the persistent storage device in a manner of giving priority to temporary files with the largest amount of data, until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold.

[0093] In a possible embodiment of this specification, the brushing module is used for: At least one temporary file stored in the memory is flushed to the persistent storage device in a manner that prioritizes temporary files with the lowest disk access heat, until the remaining capacity of the space in the database memory for storing temporary files is not less than a fourth threshold.

[0094] In a possible embodiment of this specification, the brushing module is used for: If any temporary file with a data volume not less than the fifth threshold is flushed to the persistent storage device, the temporary file is divided into at least one data block in a manner of dividing each fifth threshold into one data block, and the at least one data block is uploaded to the object storage layer; If any temporary file whose data volume is less than the fifth threshold is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

[0095] In a possible embodiment of the present specification, the device further includes an uploading module, which is used to: Divide the data of the local disk flushed to the database into at least one data block in a manner of dividing each fifth threshold into one data block; If the local disk meets the preset upload condition, the data blocks in the local disk are uploaded to the object storage layer.

[0096] In a possible embodiment of the present specification, the device further includes a reading module, which is used to: A data read instruction for any temporary file is received, and part or all of the data of the temporary file is read from a persistent storage device and / or a memory based on a data location range indicated by the data read instruction.

[0097] In a possible embodiment of this specification, the reading module is used to: Determine, according to the identifier of the temporary memory page storing the temporary file recorded in the memory, a data location range of the data of the temporary file stored in the memory in the temporary file; If the data position range indicated by the data reading instruction is within the data position range of the temporary file stored in the memory, then part or all of the data of the temporary file is read from the memory; If the data position range indicated by the data reading instruction has no intersection with the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device; If the data position range indicated by the data read instruction is partially within the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device and the memory.

[0098] In a possible embodiment of the present specification, when the reading module is used to read part or all of the data of the temporary file from the persistent storage device based on the data location range indicated by the data reading instruction, it is used to: A target data block is determined in the persistent storage device based on the data location range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target data block.

[0099] In a possible embodiment of the present specification, when the reading module is used to read part or all of the data of the temporary file from the memory based on the data position range indicated by the data reading instruction, it is used to: A target memory temporary page is determined in the memory based on the data position range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target memory temporary page.

[0100] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0101] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0103] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0107] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0108] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0109] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A temporary file management method, the method comprising: Allocating a temporary memory page to a temporary file to be stored, and writing data of the temporary file into the temporary memory page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of temporary memory pages, each of which has an identifier; Whenever a temporary memory page is written with data of the temporary file until the remaining capacity is less than a first threshold, a new temporary memory page is allocated to the temporary file, an identifier of the new temporary memory page is recorded in the temporary memory page whose remaining capacity is less than the first threshold, and the data of the temporary file continues to be written into the new temporary memory page; If the data of the temporary file is completely written into the memory, an identifier of at least one temporary memory page storing the temporary file is recorded in the memory.

2. The temporary file management method according to claim 1, further comprising: If the data of the temporary file is completely written into the memory, an end mark is recorded in the last temporary memory page storing the temporary file, wherein the end mark is used to indicate that the temporary memory page is the last temporary memory page storing the temporary file.

3. The temporary file management method according to claim 1, wherein recording in the memory an identifier of at least one temporary memory page storing the temporary file comprises: The identifier of the first temporary memory page storing the temporary file and the number of at least one temporary memory page storing the temporary file are recorded in the memory.

4. The temporary file management method according to claim 1, further comprising: If the remaining capacity of the space in the database's memory for storing temporary files is less than a third threshold, at least one temporary file stored in the memory is flushed to the persistent storage device until the remaining capacity of the space in the database's memory for storing temporary files is not less than a fourth threshold, and the temporary memory page storing the at least one temporary file is released.

5. The temporary file management method according to claim 4, wherein the step of flushing at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold comprises: At least one temporary file stored in the memory is flushed to the persistent storage device in a manner of giving priority to temporary files with the largest amount of data, until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold.

6. The temporary file management method according to claim 4, wherein the step of flushing at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold comprises: At least one temporary file stored in the memory is flushed to the persistent storage device in a manner that prioritizes temporary files with the lowest disk access heat, until the remaining capacity of the space in the database memory for storing temporary files is not less than a fourth threshold.

7. The temporary file management method according to claim 4, wherein the step of flushing at least one temporary file stored in the memory to the persistent storage device until the remaining capacity of the space in the memory of the database for storing temporary files is not less than a fourth threshold comprises: If any temporary file with a data volume not less than the fifth threshold is flushed to the persistent storage device, the temporary file is divided into at least one data block in a manner of dividing each fifth threshold into one data block, and the at least one data block is uploaded to the object storage layer; If any temporary file whose data volume is less than the fifth threshold is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

8. The temporary file management method according to claim 7, further comprising: Divide the data of the local disk flushed to the database into at least one data block in a manner of dividing each fifth threshold into one data block; If the local disk meets the preset upload condition, the data blocks in the local disk are uploaded to the object storage layer.

9. The temporary file management method according to claim 4, further comprising: A data read instruction for any temporary file is received, and part or all of the data of the temporary file is read from a persistent storage device and / or a memory based on a data location range indicated by the data read instruction.

10. The temporary file management method according to claim 9, wherein the step of reading part or all of the data of the temporary file from a persistent storage device and / or a memory based on the data location range indicated by the data reading instruction comprises: Determine, according to the identifier of the temporary memory page storing the temporary file recorded in the memory, a data location range of the data of the temporary file stored in the memory in the temporary file; If the data position range indicated by the data reading instruction is within the data position range of the temporary file stored in the memory, then part or all of the data of the temporary file is read from the memory; If the data position range indicated by the data reading instruction has no intersection with the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device; If the data position range indicated by the data read instruction is partially within the data position range of the temporary file stored in the memory, part or all of the data of the temporary file is read from the persistent storage device and the memory.

11. The temporary file management method according to claim 9, wherein the step of reading part or all of the data of the temporary file from the persistent storage device based on the data location range indicated by the data reading instruction comprises: A target data block is determined in the persistent storage device based on the data location range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target data block.

12. The temporary file management method according to claim 9, wherein the step of reading part or all of the data of the temporary file from the memory based on the data position range indicated by the data reading instruction comprises: A target memory temporary page is determined in the memory based on the data position range indicated by the data read instruction, and part or all of the data of the temporary file is read from the target memory temporary page.

13. A temporary file management device, the device comprising: An allocation module, used to allocate a temporary memory page to a temporary file to be stored, and write the data of the temporary file into the temporary memory page, wherein the space in the memory of the database for storing temporary files is divided into a plurality of temporary memory pages, and each temporary memory page has an identifier; A link module, configured to allocate a new temporary memory page to the temporary file whenever a temporary memory page is written by the data of the temporary file to a remaining capacity less than a first threshold, record an identifier of the new temporary memory page in the temporary memory page whose remaining capacity is less than the first threshold, and continue to write the data of the temporary file into the new temporary memory page; The metadata module is used to record in the memory an identifier of at least one temporary memory page storing the temporary file if the data of the temporary file is completely written into the memory.

14. The temporary file management device according to claim 13, further comprising a disk flushing module, for: If the remaining capacity of the space in the database's memory for storing temporary files is less than a third threshold, at least one temporary file stored in the memory is flushed to the persistent storage device until the remaining capacity of the space in the database's memory for storing temporary files is not less than a fourth threshold, and the temporary memory page storing the at least one temporary file is released.

15. The temporary file management device according to claim 14, wherein the disk flushing module is used for: If any temporary file with a data volume not less than the fifth threshold is flushed to the persistent storage device, the temporary file is divided into at least one data block in a manner of dividing each fifth threshold into one data block, and the at least one data block is uploaded to the object storage layer; If any temporary file whose data volume is less than the fifth threshold is flushed to the persistent storage device, the temporary file is flushed to the local disk of the database.

16. The temporary file management device according to claim 15, further comprising an upload module, configured to: Divide the data of the local disk flushed to the database into at least one data block in a manner of dividing each fifth threshold into one data block; If the local disk meets the preset upload condition, the data blocks in the local disk are uploaded to the object storage layer.

17. The temporary file management device according to claim 14, further comprising a reading module, configured to: A data read instruction for any temporary file is received, and part or all of the data of the temporary file is read from a persistent storage device and / or a memory based on a data location range indicated by the data read instruction.

18. A computer program product, comprising a computer program / instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 12.

19. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 12 by running the executable instructions.

20. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images