OpenGauss-based table partition hot unloading and hot loading method

Abstract

The invention belongs to the technical field of databases, and particularly discloses a table partition hot unloading and hot loading method based on openGauss, which comprises the following steps: when a target partition is unloaded, obtaining a corresponding physical file unique identifier by querying metadata; searching a physical file corresponding to the target partition based on the physical file unique identifier corresponding to the target partition, and backing up the searched physical file to obtain physical file backup data corresponding to the target partition; clearing the physical file corresponding to the target partition; when the target partition is loaded, obtaining the unique identifier of the corresponding physical file by querying the metadata; searching physical file backup data corresponding to the target partition based on the physical file unique identifier corresponding to the target partition; and recovering the physical file corresponding to the target partition. According to the method, backup or recovery is performed by operating the physical file, the generated I / O load is relatively low, and efficient surface partition unloading or loading can be realized.

Description

Technical Field

[0001] The present application belongs to the field of database technology, and more specifically, relates to a table partition hot unloading and hot loading method based on openGauss. Background Art

[0002] In the openGauss database, partition tables are widely used because they can improve retrieval efficiency, have high availability, facilitate maintenance, and balance I / O. As the amount of data in a partition table increases dramatically, it may be necessary to delete old partitions and add new partitions. If you need to read old partition data that has been backed up but deleted, you need to add new partitions and load the backed-up partition data.

[0003] In the prior art, for the openGauss partition table, the specific methods of unloading partitions and loading old partitions are as follows: (1) Backing up the partitions of the partition table, generally using tools such as gs_dump that perform logical backup (logical backup tools) to export metadata and data; (2) Unloading the partitions in the partition table (including removing the metadata of the corresponding partitions in the database). (3) Recovering and loading old partitions from the backup, generally using tools such as gs_restore that perform logical recovery (logical recovery tools) to import metadata and data.

[0004] However, using logical backup tools such as gs_dump to back up metadata and data, and using logical recovery tools such as gs_restore to restore old partitions, takes up a lot of database IO resources, seriously affecting other operations in the database that require IO resources. When removing an old partition, it is necessary to delete the metadata of the old partition. When restoring an old partition, it is necessary to add a new partition and then load the backed-up partition data (including reconfiguring the metadata of the old partition). The operation is cumbersome. For the openGauss database, how to achieve efficient surface partition unloading or loading is a technical problem that needs to be solved in this field. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of this application is to achieve efficient surface partition unloading or loading for the openGauss database.

[0006] To achieve the above-mentioned objectives, in a first aspect, the present application provides a table partition hot unloading and hot loading method based on openGauss, comprising: when unloading a target partition, obtaining a physical file unique identifier corresponding to the target partition by querying metadata; based on the physical file unique identifier corresponding to the target partition, searching for the physical file corresponding to the target partition, and backing up the found physical file to obtain physical file backup data corresponding to the target partition; based on the physical file unique identifier corresponding to the target partition, clearing the physical file corresponding to the target partition.

[0007] In a possible implementation, before backing up the found physical file, the method further includes: locking the target partition, and prohibiting DDL operations and DML operations on the partition in the locked state.

[0008] In a possible implementation, before backing up the found physical file, the method further includes: writing the memory data corresponding to the target partition into the physical file through a checkpoint mechanism of the database.

[0009] In a possible implementation, after clearing the physical files corresponding to the target partition, the method further includes: when mounting the target partition, obtaining the physical file unique identifier corresponding to the target partition by querying metadata; searching for the physical file backup data corresponding to the target partition based on the physical file unique identifier corresponding to the target partition; and restoring the physical files corresponding to the target partition based on the found physical file backup data.

[0010] In a possible implementation, after the physical file corresponding to the target partition is restored, the method further includes: clearing data from a partition cache of the partition table.

[0011] In a possible implementation, after clearing data from the partition cache of the partition table, the method further includes: unlocking the target partition, and the partition in the unlocked state can perform DDL operations and DML operations.

[0012] In a second aspect, the present application provides a table partition hot unloading and hot loading device based on openGauss, comprising: The metadata query module is used to obtain the physical file unique identifier corresponding to the target partition by querying the metadata when the target partition is uninstalled; A physical file backup module is used to search for a physical file corresponding to a target partition based on a unique physical file identifier corresponding to the target partition, and to back up the found physical file to obtain physical file backup data corresponding to the target partition; The physical file clearing module is used to clear the physical file corresponding to the target partition based on the unique identifier of the physical file corresponding to the target partition.

[0013] In a possible implementation, it also includes: a backup data search module and a physical file recovery module; The metadata query module is also used to obtain a unique physical file identifier corresponding to the target partition by querying the metadata when the target partition is mounted; A backup data search module, used to search for physical file backup data corresponding to a target partition based on a unique identifier of a physical file corresponding to the target partition; The physical file recovery module is used to recover the physical files corresponding to the target partition based on the found physical file backup data.

[0014] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0016] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: (1) The working process of gs_dump in the prior art includes obtaining metadata, generating backup scripts, and exporting data. Its backup method is logical backup (backup according to the operation logic of the database), which requires reading data and generating SQL statements, which will generate a high I / O load. However, the present application directly backs up physical files. The backup method of the present application is physical backup (backup by operating physical files). The copying of physical files can be quickly completed through the file copy command of the operating system. There is no need to parse the table structure and generate SQL statements. The generated I / O load is relatively low. Therefore, the physical backup method of the present application is faster than the logical backup method in the prior art.

[0017] (2) The working process of gs_restore in the prior art includes: reading backup files, restoring database objects and restoring data. Its recovery method is logical recovery (recovery according to the operation logic of the database), which requires parsing the SQL statements and metadata in the backup files generated by gs_dump and performing a large number of insert operations, which will generate a high I / O load. However, the present application restores (copies) the physical files to the original location based on the backup of the physical files. The backup method of the present application is physical recovery (recovery by operating the physical files). The copying of physical files can be quickly completed through the file copy command of the operating system. There is no need to parse SQL statements and metadata, nor is there a need to perform a large number of insert operations. The generated I / O load is relatively low. Therefore, the physical backup recovery method of the present application is faster than the logical recovery method in the prior art.

[0018] (3) In the prior art, deleting an old partition will remove the metadata of the old partition in the database. When adding an old partition, the metadata of the old partition needs to be reconfigured, which is cumbersome and inefficient. However, in this application, when uninstalling an old partition, only the physical files corresponding to the old partition are removed. When loading the old partition, the physical files are restored (copied) to the original location. There is no need to reconfigure the metadata of the old partition. The operation is simpler and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is one of the flow diagrams of the openGauss-based table partition hot unloading and hot loading method provided in the embodiment of the present application; Figure 2 It is a schematic diagram of the original structure of the partition table provided in the embodiment of the present application; Figure 3 It is a schematic diagram of hot unloading of a partition table provided in an embodiment of the present application; Figure 4 This is the second flow chart of the openGauss-based table partition hot unloading and hot loading method provided in the embodiment of the present application; Figure 5 It is a schematic diagram of hot loading of a partition table provided in an embodiment of the present application; Figure 6 This is one of the structural schematic diagrams of the table partition hot unloading and hot loading device based on openGauss provided in the embodiment of the present application; Figure 7 This is the second structural schematic diagram of the table partition hot unloading and hot loading device based on openGauss provided in the embodiment of the present application; Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0022] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0023] First, the technical terms involved in the embodiments of the present application are introduced.

[0024] (1) openGauss is an open source, high-performance relational database management system. It is designed to meet the needs of enterprises in large-scale data processing and high-concurrency scenarios.

[0025] (2) The partition table in the openGauss database is a database table design method that allows a large table to be divided into multiple smaller, more manageable parts, which are called partitions. A large table with multiple partitions is called a partitioned table. A partition table is a logical structure that defines how to divide data into multiple partitions. It contains metadata for all partitions but does not directly store data. Each partition contains a subset of the table data and can be divided according to a specific partition key value. Each partition is actually a subtable of the partition table. Partitioned tables can improve the performance, manageability, and availability of the database. A table partition refers to a partition of a partition table.

[0026] (3) Physical files: In a database system, physical files refer to actual data files stored on disk. They contain rows, indexes, toast tables (used to store compressed data of large fields) and data of other database objects in the database table. Physical files are usually located on the database's file system, which can be a local disk, network storage or a specific storage device. In database systems such as openGauss, each physical file has a unique identifier called relfilenode. This identifier allows the database management system (DBMS) to accurately locate the physical file corresponding to each object in the database (such as table, index, toast table, etc.). Physical files contain the actual data of the database. For example, for a table, the corresponding physical file will contain the row data of the table; for an index, the corresponding physical file will contain the index entries; for a toast table, the corresponding physical file contains compressed or decomposed large field data. When performing database operations such as querying, updating, inserting or deleting data, the DBMS will find and access the corresponding physical file through relfilenode to read or modify the data.

[0027] (4) In the openGauss database, relfilenode is an important concept that represents the physical file identifier of each relation in the database. Each table and index has a unique relfilenode value in the database, which is used to locate the specific file that stores the table or index data on the file system. relfilenode enables the database system to track and manage data files stored on disk. The actual data of each relation is stored in one or more files, and the names of these files are usually associated with relfilenode. When executing queries, the database engine uses relfilenode to locate and access the corresponding physical files to read or write data. relfilenode is often used with other fields in system catalog tables (such as pg_class) to provide detailed information about database objects. For example, there is a relfilenode column in the pg_class table that records the relfilenode value of each table or index.

[0028] (5) TOAST (The Oversized-Attribute Storage Technique) is a technology used by PostgreSQL and its derivative databases (including openGauss) to store large fields (such as large text or large binary data). When the size of a field exceeds a certain threshold, the database will store the data of the field in the TOAST table. Each Toast table will also have its own independent relfilenode.

[0029] (6) gs_dump is a backup tool provided by openGauss, similar to PostgreSQL's pg_dump. It can be used to export the structure and data of database objects, including partitioned tables and their partitions. The working process of gs_dump includes obtaining metadata, generating backup scripts, and exporting data. Obtaining metadata: The gs_dump tool queries the system catalog to obtain information about the object to be backed up, including table structure, indexes, constraints, triggers, etc.; if the object to be backed up is a partitioned table, gs_dump will identify all related partitions and prepare to back up the data of each partition. Generate backup scripts: After obtaining the necessary metadata, gs_dump will start generating backup scripts. For example, the gs_dump tool will generate SQL DDL (data definition language) statements for creating tables, indexes, constraints, etc. (these statements are used to rebuild the structure of database objects during recovery). For example, for each table, gs_dump will generate SQL DML (data manipulation language) statements for inserting data (these statements will be used to insert the backup data into the database during recovery). Data export: The tool will read the data in the specified table sequentially, and the read data will be written to the specified backup file.

[0030] (7) gs_restore is a recovery tool provided by openGauss, similar to PostgreSQL's pg_restore. It can be used to restore database objects, including partitioned tables and their partitions, from the backup files generated by gs_dump. The working process of gs_restore includes: reading backup files, restoring database objects, and restoring data. Reading backup files: gs_restore will read the backup files generated by gs_dump and parse the SQL statements and metadata therein. Restoring database objects: When creating database objects, gs_restore will send DDL statements to the database. Restoring data: When restoring data, gs_restore will perform a large number of insert operations.

[0031] (8) DDL (Data Definition Language), which is used to define and manage the structure of the database, including creating, modifying, and deleting database objects (such as tables, indexes, views, etc.).

[0032] (9) DML (Data Manipulation Language) is used to operate data in the database, including inserting, updating, deleting and querying data.

[0033] (10) Checkpoint: A mechanism in the database that writes data in memory to disk to ensure data persistence and consistency.

[0034] (11) TRUNCATE statement: In a database, the TRUNCATE statement is used to quickly delete all records in a table without recording the deletion of each row. For partitioned tables, the TRUNCATE statement can be executed at the partition level to safely clear physical files.

[0035] (12) Hot Unloading: Usually refers to removing certain components (such as tablespaces, data files, log files, etc.) from a running database without affecting the overall operation of the database.

[0036] (13) Hot Loading: usually refers to dynamically adding new components (such as tablespaces, data files, log files, etc.) to the database during the operation of the database system without affecting the overall operation of the database.

[0037] (14) Memory and cache in computers. Memory usually refers to random access memory (RAM), which is used to temporarily store data and programs in use. Memory is the main working area of ​​a computer, where data is quickly read and written. Compared to memory, cache is a faster storage device located between the CPU and memory, used to store frequently used data and instructions. The access speed of cache is much faster than that of memory, and its purpose is to reduce the time it takes for the CPU to access memory.

[0038] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0039] Figure 1 This is one of the flow diagrams of the table partition hot unloading and hot loading method based on openGauss provided in the embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101, S102 and S103.

[0040] Step S101, when a target partition is unmounted, a physical file unique identifier (relfilenode, or referred to as a physical file ID) corresponding to the target partition is obtained by querying metadata.

[0041] For example, the metadata is queried through the target partition name to obtain the unique physical file identifier corresponding to the target partition.

[0042] Step S102, based on the physical file unique identifier corresponding to the target partition, searching for the physical file corresponding to the target partition, and backing up the found physical file to obtain physical file backup data corresponding to the target partition.

[0043] Step S103: based on the unique identifier of the physical file corresponding to the target partition, clear the physical file corresponding to the target partition.

[0044] Specifically, the target partition can be any partition in the partition table. When the target partition needs to be uninstalled, the unique physical file identifier corresponding to the target partition can be obtained by querying the metadata information of the database. This unique identifier is a unique code used to identify the physical file in the database, ensuring that the physical file of the target partition can be accurately found in subsequent operations.

[0045] Based on the obtained unique identifier of the physical file, the physical files corresponding to the target partition can be found. After finding, these physical files are backed up to generate physical file backup data corresponding to the target partition. The backup process can ensure that the data will not be lost after the partition is uninstalled, and can be restored when needed.

[0046] After the backup is completed, the physical files corresponding to the target partition are cleared based on the unique identifier of the physical file. This step will free up storage space and "unload" the target partition from the database. The above process of unloading the target partition only clears the physical files corresponding to the target partition, which does not affect the overall operation of the database. In addition, the unloading method of directly operating the physical files takes a short time, so the partition unloading method provided in this application can achieve "hot unloading".

[0047] For example, Figure 2 This is a schematic diagram of the original structure of the partition table. Figure 3 This is a schematic diagram of hot unloading of the partition table, such as Figure 2-3 As shown, the target partition may be partition 2. By querying the metadata information of the database, the physical file unique identifier corresponding to partition 2 may be obtained. According to the obtained physical file unique identifier, the physical file corresponding to partition 2 may be searched, such as the physical file corresponding to the partition data of partition 2, the physical file corresponding to the index data of partition 2, the physical file corresponding to the toast data of partition 2, and the physical file corresponding to other data of partition 2. After being found, these physical files are packaged and backed up to generate physical file backup data corresponding to the target partition.

[0048] It is understandable that the working process of gs_dump in the prior art includes obtaining metadata, generating backup scripts, and exporting data. Its backup method is logical backup (backup according to the operation logic of the database), which requires reading data and generating SQL statements, which will generate a high I / O load. However, the present application directly backs up physical files. The backup method of the present application is physical backup (backup by operating physical files). The copying of physical files can be quickly completed through the file copy command of the operating system. It does not need to parse the table structure and generate SQL statements. The generated I / O load is relatively low. Therefore, the physical backup method of the present application is faster than the logical backup method in the prior art.

[0049] In a possible implementation, before backing up the found physical file, the method further includes: locking the target partition, and prohibiting DDL operations and DML operations on the partition in the locked state.

[0050] For example, Figure 3 As shown, the target partition may be partition 2, and before backing up the physical files corresponding to partition 2, partition 2 may be locked.

[0051] In a possible implementation, before backing up the found physical file, the method further includes: writing the memory data corresponding to the target partition into the physical file through a checkpoint mechanism of the database.

[0052] Figure 4This is the second flow chart of the table partition hot unloading and hot loading method based on openGauss provided in the embodiment of the present application, such as Figure 4 As shown, after clearing the physical files corresponding to the target partition, the method further includes the following steps S201, S202 and S203.

[0053] Step S201, when mounting a target partition, obtaining a physical file unique identifier corresponding to the target partition by querying metadata.

[0054] Step S202: searching for physical file backup data corresponding to the target partition based on the physical file unique identifier corresponding to the target partition.

[0055] Step S203: restore the physical file corresponding to the target partition based on the found physical file backup data.

[0056] The physical file unique identifier corresponding to the target partition can be used not only to locate the storage location of the physical file corresponding to the target partition on the disk, but also to locate the storage location of the physical file backup data corresponding to the target partition on the disk. For example, a first correspondence between the physical file unique identifier and the storage location of the physical file on the disk can be established, and the physical file can be found at the corresponding storage location based on the first correspondence and the physical file unique identifier. For example, a second correspondence between the physical file unique identifier and the storage location of the physical file backup data on the disk can be established, and the physical file backup data can be found at the corresponding storage location based on the second correspondence and the physical file unique identifier.

[0057] Specifically, when it is necessary to load the target partition, the unique physical file identifier corresponding to the target partition can be obtained by querying the metadata information of the database. Then, based on the obtained unique physical file identifier, the physical file backup data corresponding to the target partition can be found. Then, based on the found physical file backup data, the physical file corresponding to the target partition is restored. This step will allow the target partition to be re-added to the database to achieve "loading" of the data. The above-mentioned process of unloading the target partition only requires restoring (copying) the physical file corresponding to the target partition to the original location, which does not affect the overall operation of the database, and the time required for directly operating the loading method of the physical file is short, so the partition loading method provided in the present application can achieve "hot loading".

[0058] Figure 5 Schematic diagram of hot loading of partition table provided in the embodiment of the present application. Figure 5As shown, the target partition may be partition 2. By querying the metadata information of the database, the physical file unique identifier corresponding to partition 2 may be obtained. Based on the obtained physical file unique identifier, the physical file backup data corresponding to partition 2 may be searched, and the backup data includes: the physical file backup corresponding to the partition data of partition 2, the physical file backup corresponding to the index data of partition 2, the physical file backup corresponding to the toast data of partition 2, and the physical file backup corresponding to other data of partition 2. Then, based on the found physical file backup data, the physical file corresponding to partition 2 is restored to the original position.

[0059] It is understandable that the working process of gs_restore in the prior art includes: reading backup files, restoring database objects and restoring data, and its recovery method is logical recovery (recovery according to the operation logic of the database), which requires parsing the SQL statements and metadata in the backup files generated by gs_dump, and performing a large number of insert operations, which will generate a high I / O load. However, the present application restores (copies) the physical files to the original location based on the backup of the physical files. The backup method of the present application is physical recovery (recovery by operating the physical files). The copying of physical files can be quickly completed through the file copy command of the operating system, and there is no need to parse SQL statements and metadata, nor to perform a large number of insert operations. The generated I / O load is relatively low, so the physical backup recovery method of the present application is faster than the logical recovery method in the prior art.

[0060] In addition, in the prior art, when uninstalling an old partition, the metadata of the old partition in the database will be removed. When loading the old partition, the metadata of the old partition needs to be reconfigured, which is cumbersome and inefficient. However, in this application, when uninstalling an old partition, only the physical files corresponding to the old partition are removed. When loading the old partition, the physical files are restored (copied) to the original location. There is no need to reconfigure the metadata of the old partition. The operation is simpler and more efficient.

[0061] In a possible implementation, after the physical file corresponding to the target partition is restored, the method further includes: clearing data from a partition cache of the partition table.

[0062] In a possible implementation, after clearing data from the partition cache of the partition table, the method further includes: unlocking the target partition, and the partition in the unlocked state can perform DDL operations and DML operations.

[0063] For example, Figure 5 As shown, the target partition may be partition 2. After the physical files corresponding to partition 2 are restored, the partition cache of the partition table is cleared, and then partition 2 is unlocked.

[0064] The following is an example to illustrate the openGauss-based table partition hot unloading and hot loading method provided by the present application.

[0065] Partition table partition hot unloading and hot loading mainly include two processes: hot unloading and hot loading.

[0066] The hot unloading process is as follows.

[0067] (1) Lock the unloaded partition of the partition table. DDL and DML operations can no longer be performed on the unloaded partition of the partition table. After the partition is locked, if any DDL and DML operations are performed on the partition, the partition will report a partition lock error.

[0068] (2) Checkpoint the database to ensure that all data in memory is written to disk (including writing data in memory to corresponding physical files) and that the physical files of the partitions in the partition table contain all data without any loss.

[0069] (3) After completing the checkpoint, search the metadata for the relfilenode corresponding to the partition table partition.

[0070] (4) According to relfilenode, the physical files corresponding to the partition (including the physical files corresponding to the partition data, the physical files corresponding to the partition index data, the physical files corresponding to the partition toast data, and the physical files corresponding to other data) are packaged and backed up.

[0071] (5) Execute the truncate statement on the partition to safely clear the physical files.

[0072] The hot loading process is as follows.

[0073] (1) Query the corresponding relfilenode in the metadata according to the partition that needs to be hot-loaded.

[0074] (2) Use the relfilenode in the previous step to restore the physical files of the hot uninstall package backup to their original locations.

[0075] (3) Clear the partition cache of the partition table to ensure that data is loaded from the physical file and to prevent the data remaining in the cache from affecting the accuracy of the loaded data.

[0076] (4) Unlock the partitions of the partitioned table, and the partitions of the partitioned table can perform DDL and DML operations.

[0077] It can be understood that this application reduces the IO consumption of the database by directly backing up and packaging the physical files, eliminating the need for backup and recovery tools to import and export data. It also retains the metadata of the original partition, and the original partition can be used by simply reloading, achieving the effect of hot unloading and hot loading.

[0078] The table partition hot unloading and hot loading device based on openGauss provided in the present application is described below. The table partition hot unloading and hot loading device based on openGauss described below and the table partition hot unloading and hot loading method based on openGauss described above can refer to each other.

[0079] Figure 6 This is one of the schematic diagrams of the structure of the table partition hot unloading and hot loading device based on openGauss provided in the embodiment of the present application, such as Figure 6 As shown, the device includes: a metadata query module 10, a physical file backup module 20 and a physical file clearing module 30. Among them: The metadata query module 10 is used to obtain a physical file unique identifier corresponding to the target partition by querying metadata when the target partition is uninstalled; The physical file backup module 20 is used to search for the physical file corresponding to the target partition based on the physical file unique identifier corresponding to the target partition, and back up the found physical file to obtain the physical file backup data corresponding to the target partition; The physical file clearing module 30 is used to clear the physical file corresponding to the target partition based on the physical file unique identifier corresponding to the target partition.

[0080] Figure 7 This is the second structural diagram of the table partition hot unloading and hot loading device based on openGauss provided in the embodiment of the present application, such as Figure 7 As shown, the device also includes: a backup data search module 40 and a physical file recovery module 50; The metadata query module 10 is also used to obtain a physical file unique identifier corresponding to the target partition by querying the metadata when the target partition is mounted; A backup data search module 40, configured to search for physical file backup data corresponding to a target partition based on a physical file unique identifier corresponding to the target partition; The physical file recovery module 50 is used to recover the physical file corresponding to the target partition based on the found physical file backup data.

[0081] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0082] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0083] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0084] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0085] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0086] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0087] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0088] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0089] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0090] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0091] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A table partition hot unloading and hot loading method based on openGauss, characterized in that: include: When the target partition is unmounted, the unique physical file identifier corresponding to the target partition is obtained by querying metadata; Based on the unique identifier of the physical file corresponding to the target partition, the physical file corresponding to the target partition is searched, and the found physical file is backed up to obtain the physical file backup data corresponding to the target partition; Based on the unique identifier of the physical file corresponding to the target partition, the physical file corresponding to the target partition is cleared.

2. According to the openGauss-based table partition hot unloading and hot loading method of claim 1, it is characterized in that: Before backing up the found physical files, it also includes: Lock the target partition. DDL and DML operations are prohibited on the locked partition.

3. According to the openGauss-based table partition hot unloading and hot loading method of claim 2, it is characterized in that: Before backing up the found physical files, it also includes: Through the database checkpoint mechanism, the memory data corresponding to the target partition is written into the physical file.

4. The table partition hot unloading and hot loading method based on openGauss according to any one of claims 1 to 3, characterized in that: After clearing the physical files corresponding to the target partition, it also includes: When mounting the target partition, the unique physical file identifier corresponding to the target partition is obtained by querying the metadata; Based on the unique identifier of the physical file corresponding to the target partition, searching for the physical file backup data corresponding to the target partition; Based on the found physical file backup data, restore the physical files corresponding to the target partition.

5. According to the openGauss-based table partition hot unloading and hot loading method of claim 4, it is characterized in that: After the physical files corresponding to the target partition are restored, the method further includes: clearing data from the partition cache of the partition table.

6. According to the openGauss-based table partition hot unloading and hot loading method of claim 5, it is characterized in that: After clearing the partition cache of the partition table, it also includes: Unlock the target partition. The unlocked partition can be used for DDL and DML operations.

7. A table partition hot unloading and hot loading device based on openGauss, characterized in that: include: The metadata query module is used to obtain the physical file unique identifier corresponding to the target partition by querying the metadata when the target partition is uninstalled; A physical file backup module is used to search for a physical file corresponding to a target partition based on a unique physical file identifier corresponding to the target partition, and to back up the found physical file to obtain physical file backup data corresponding to the target partition; The physical file clearing module is used to clear the physical file corresponding to the target partition based on the unique identifier of the physical file corresponding to the target partition.

8. The table partition hot unloading and hot loading device based on openGauss according to claim 7 is characterized in that: Also includes: Backup data search module and physical file recovery module; The metadata query module is also used to obtain a unique physical file identifier corresponding to the target partition by querying the metadata when the target partition is mounted; A backup data search module, used to search for physical file backup data corresponding to a target partition based on a unique identifier of a physical file corresponding to the target partition; The physical file recovery module is used to recover the physical files corresponding to the target partition based on the found physical file backup data.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images