A method for hot unloading and hot loading of table partitions based on openGauss

Abstract

This application belongs to the field of database technology, and specifically discloses a method for hot unloading and hot loading of table partitions based on openGauss, including: when unloading a target partition, obtaining the corresponding physical file unique identifier by querying metadata; based on the physical file unique identifier corresponding to the target partition, finding the physical file corresponding to the target partition, and backing up the found physical file to obtain the physical file backup data corresponding to the target partition; clearing the physical file corresponding to the target partition; when loading the target partition, obtaining the corresponding physical file unique identifier by querying metadata; based on the physical file unique identifier corresponding to the target partition, finding the physical file backup data corresponding to the target partition; restoring the physical file corresponding to the target partition. This application performs backup or restoration by operating on physical files, resulting in relatively low I / O load, and can achieve efficient unloading or loading of table partitions.

Description

Technical Field

[0001] The present application belongs to the field of database technology, and more specifically, relates to a table partition hot unloading and hot loading method based on openGauss. Background Art

[0002] In the openGauss database, partition tables are widely used because they can improve retrieval efficiency, have high availability, facilitate maintenance, and balance I / O. As the amount of data in a partition table increases dramatically, it may be necessary to delete old partitions and add new partitions. If you need to read old partition data that has been backed up but deleted, you need to add new partitions and load the backed-up partition data.

[0003] In the prior art, for the openGauss partition table, the specific methods of unloading partitions and loading old partitions are as follows: (1) Backing up the partitions of the partition table, generally using tools such as gs_dump that perform logical backup (logical backup tools) to export metadata and data; (2) Unloading the partitions in the partition table (including removing the metadata of the corresponding partitions in the database). (3) Recovering and loading old partitions from the backup, generally using tools such as gs_restore that perform logical recovery (logical recovery tools) to import metadata and data.

[0004] However, using logical backup tools such as gs_dump to back up metadata and data, and using logical recovery tools such as gs_restore to restore old partitions, takes up a lot of database IO resources, seriously affecting other operations in the database that require IO resources. When removing an old partition, it is necessary to delete the metadata of the old partition. When restoring an old partition, it is necessary to add a new partition and then load the backed-up partition data (including reconfiguring the metadata of the old partition). The operation is cumbersome. For the openGauss database, how to achieve efficient surface partition unloading or loading is a technical problem that needs to be solved in this field. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of this application is to achieve efficient surface partition unloading or loading for the openGauss database.

[0006] To achieve the above-mentioned objectives, in a first aspect, the present application provides a table partition hot unloading and hot loading method based on openGauss, comprising: when unloading a target partition, obtaining a physical file unique identifier corresponding to the target partition by querying metadata; based on the physical file unique identifier corresponding to the target partition, searching for the physical file corresponding to the target partition, and backing up the found physical file to obtain physical file backup data corresponding to the target partition; based on the physical file unique identifier corresponding to the target partition, clearing the physical file corresponding to the target partition.

[0007] In a possible implementation, before backing up the found physical file, it further includes: locking the target partition, and the partition in the locked state is prohibited from performing DDL operations and DML operations.

[0008] In a possible implementation, before backing up the found physical file, it further includes: writing the memory data corresponding to the target partition into the physical file through the Checkpoint mechanism of the database.

[0009] In a possible implementation, after clearing the physical file corresponding to the target partition, it further includes: when loading the target partition, obtaining the unique identifier of the physical file corresponding to the target partition by querying the metadata; based on the unique identifier of the physical file corresponding to the target partition, searching for the backup data of the physical file corresponding to the target partition; based on the found backup data of the physical file, restoring the physical file corresponding to the target partition.

[0010] In a possible implementation, after restoring the physical file corresponding to the target partition, it further includes: clearing the data in the partition cache of the partition table.

[0011] In a possible implementation, after clearing the data in the partition cache of the partition table, it further includes: unlocking the target partition, and the partition in the unlocked state can perform DDL operations and DML operations.

[0012] In a second aspect, the present application provides a table partition hot unload and hot load device based on openGauss, including:

[0013] A metadata query module, configured to obtain the unique identifier of the physical file corresponding to the target partition by querying the metadata when unloading the target partition;

[0014] A physical file backup module, configured to search for the physical file corresponding to the target partition based on the unique identifier of the physical file corresponding to the target partition, and back up the found physical file to obtain the backup data of the physical file corresponding to the target partition;

[0015] A physical file clearing module, configured to clear the physical file corresponding to the target partition based on the unique identifier of the physical file corresponding to the target partition.

[0016] In a possible implementation, it further includes: a backup data search module and a physical file restoration module;

[0017] The metadata query module is further configured to obtain the unique identifier of the physical file corresponding to the target partition by querying the metadata when loading the target partition;

[0018] A backup data search module, configured to search for physical file backup data corresponding to a target partition based on a physical file unique identifier corresponding to the target partition;

[0019] A physical file restoration module, configured to restore a physical file corresponding to a target partition based on the found physical file backup data.

[0020] In a third aspect, the present application provides an electronic device, including: at least one memory for storing a program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, and when the computer program runs on a processor, it causes the processor to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0022] Generally speaking, compared with the prior art through the above technical solutions conceived by the present application, the following beneficial effects are achieved:

[0023] (1) The working process of gs_dump in the prior art includes obtaining metadata, generating a backup script, and data export. Its backup method is logical backup (backing up according to the running logic of the database), which requires reading data and generating SQL statements, resulting in a relatively high I / O load. In contrast, the present application directly backs up physical files, and its backup method is physical backup (backing up by operating physical files). The copying of physical files can be quickly completed through the file copy command of the operating system, without the need to parse table structures and generate SQL statements, and the resulting I / O load is relatively low. Therefore, the physical backup method of the present application is faster than the logical backup method in the prior art.

[0024] (2) The working process of gs_restore in the prior art includes: reading a backup file, restoring database objects, and restoring data. Its restoration method is logical restoration (restoring according to the running logic of the database), which requires parsing SQL statements and metadata in the backup file generated by gs_dump and performing a large number of insert operations, resulting in a relatively high I / O load. In contrast, the present application restores (copies) the physical file to its original location based on the backup of the physical file. Its backup method is physical restoration (restoring by operating physical files). The copying of physical files can be quickly completed through the file copy command of the operating system, without the need to parse SQL statements and metadata, nor perform a large number of insert operations, and the resulting I / O load is relatively low. Therefore, the physical backup and restoration method of the present application is faster than the logical restoration method in the prior art.

[0025] In the prior art, deleting an old partition will remove the metadata of the old partition in the database. When adding an old partition, it is also necessary to reconfigure the metadata of the old partition, which is a cumbersome operation and has low efficiency. In contrast, in this application, when unloading an old partition, only the physical file corresponding to the old partition is removed. When loading the old partition, the physical file is restored (copied) to the original location, without the need to reconfigure the metadata of the old partition, and the operation is relatively simple and the efficiency is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 FIG. is one of the schematic flowcharts of the method for hot unloading and hot loading of table partitions based on openGauss provided by an embodiment of the present application;

[0027] Figure 2 FIG. is a schematic diagram of the original structure of a partitioned table provided by an embodiment of the present application;

[0028] Figure 3 FIG. is a schematic diagram of hot unloading of a partitioned table provided by an embodiment of the present application;

[0029] Figure 4 FIG. is another schematic flowchart of the method for hot unloading and hot loading of table partitions based on openGauss provided by an embodiment of the present application;

[0030] Figure 5 FIG. is a schematic diagram of hot loading of a partitioned table provided by an embodiment of the present application;

[0031] Figure 6 FIG. is one of the schematic structural diagrams of the device for hot unloading and hot loading of table partitions based on openGauss provided by an embodiment of the present application;

[0032] Figure 7 FIG. is another schematic structural diagram of the device for hot unloading and hot loading of table partitions based on openGauss provided by an embodiment of the present application;

[0033] Figure 8 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] In the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0036] In the description of the embodiments of this application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units, etc.; a plurality of elements refers to two or more elements, etc.

[0037] First, the technical terms involved in the embodiments of this application are introduced.

[0038] (1) openGauss is an open-source high-performance relational database management system, which is designed specifically to meet the needs of enterprises in large-scale data processing and high-concurrency scenarios.

[0039] (2) A partitioned table in the openGauss database is a design method of a database table that allows a large table to be split into multiple smaller and more manageable parts, which are called partitions. A large table with multiple partitions is called a partitioned table. A partitioned table is a logical structure that defines how to divide data into multiple partitions. It contains the metadata of all partitions but does not directly store data. Each partition contains a subset of the table data and can be divided according to specific partition key values. Each partition is actually a sub-table of the partitioned table. Partitioned tables can improve the performance, manageability, and availability of the database. Table partitioning refers to a certain partition of a partitioned table.

[0040] (3) Physical Files: In a database system, physical files refer to the actual data files stored on disk. They contain the rows of tables in the database, indexes, TOAST tables (used to store compressed data for large fields), and data for other database objects. Physical files are typically located on the database's file system, which can be a local disk, network storage, or a specific storage device. In database systems such as openGauss, each physical file has a unique identifier called relfilenode. This identifier allows the database management system (DBMS) to accurately locate the physical file corresponding to each object (such as tables, indexes, TOAST tables, etc.) in the database. Physical files contain the actual data of the database. For example, for a table, the corresponding physical file will contain the row data of the table; for an index, the corresponding physical file will contain the index entries; for a TOAST table, the corresponding physical file contains the compressed or decomposed large field data. When performing database operations such as querying, updating, inserting, or deleting data, the DBMS will find and access the corresponding physical file through relfilenode to read or modify the data.

[0041] (4) In the openGauss database, relfilenode is an important concept. It represents the physical file identifier for each relation in the database. Each table and index has a unique relfilenode value in the database, and this value is used to locate the specific file storing the data of that table or index on the file system. relfilenode enables the database system to track and manage the data files stored on disk. The actual data of each relation is stored in one or more files, and the names of these files are usually associated with relfilenode. When executing a query, the database engine uses relfilenode to locate and access the corresponding physical file to read or write data. relfilenode is usually used together with other fields in the system catalog tables (such as pg_class) to provide detailed information about database objects. For example, the pg_class table has a relfilenode column that records the relfilenode value of each table or index.

[0042] (5) TOAST (The Oversized-Attribute Storage Technique) is a technique used by PostgreSQL and its derivatives (including openGauss) to store large fields (such as large text or large binary data). When the size of a certain field exceeds a certain threshold, the database will store the data of that field in a TOAST table. Each TOAST table also has its own independent relfilenode.

[0043] (6) gs_dump is a backup tool provided by openGauss, similar to pg_dump of PostgreSQL. It can be used to export the structure and data of database objects, including partitioned tables and their partitions. The working process of gs_dump includes obtaining metadata, generating backup scripts, and data export. Obtaining metadata: The gs_dump tool queries the system catalog to obtain information about the objects to be backed up, including table structures, indexes, constraints, triggers, etc.; if the object to be backed up is a partitioned table, gs_dump will identify all related partitions and prepare to back up the data of each partition. Generating backup scripts: After obtaining the necessary metadata, gs_dump starts to generate backup scripts. For example, the gs_dump tool generates SQL DDL (Data Definition Language) statements for creating tables, indexes, constraints, etc. (these statements are used to reconstruct the structure of database objects during recovery). For example, for each table, gs_dump generates SQL DML (Data Manipulation Language) statements for inserting data (these statements will be used to insert backup data into the database during recovery). Data export: The tool sequentially reads the data in the specified table, and the read data is written to the specified backup file.

[0044] (7) gs_restore is a recovery tool provided by openGauss, similar to pg_restore of PostgreSQL. It can be used to recover database objects from the backup files generated by gs_dump, including partitioned tables and their partitions. The working process of gs_restore includes: reading the backup file, recovering database objects, and recovering data. Reading the backup file: gs_restore reads the backup file generated by gs_dump and parses the SQL statements and metadata in it. Recovering database objects: When creating database objects, gs_restore sends DDL statements to the database. Recovering data: When recovering data, gs_restore performs a large number of insert operations.

[0045] (8) DDL (Data Definition Language) is used to define and manage the structure of the database, including creating, modifying, and deleting database objects (such as tables, indexes, views, etc.).

[0046] (9) DML (Data Manipulation Language) is used to operate on the data in the database, including inserting, updating, deleting, and querying data.

[0047] (10) Checkpoint: A mechanism in the database used to write the data in memory to disk to ensure data persistence and consistency.

[0048] (11)TRUNCATE statement: In a database, the TRUNCATE statement is used to quickly delete all records in a table without logging the deletion operation for each row. For partitioned tables, the TRUNCATE statement can be executed at the partition level to safely purge physical files.

[0049] (12)Hot Unloading: Generally refers to the process of removing certain components (such as tablespaces, data files, log files, etc.) from a running database system without affecting the overall operation of the database.

[0050] (13)Hot Loading: Generally refers to the process of dynamically adding new components (such as tablespaces, data files, log files, etc.) to a database during the operation of the database system without affecting the overall operation of the database.

[0051] (14)Memory and cache in a computer. Memory usually refers to random access memory (RAM), which is used to temporarily store data and programs in use. Memory is the main working area of the computer where data is quickly read and written. Compared to memory, cache is a faster memory located between the CPU and memory, used to store frequently accessed data and instructions. The access speed of cache is much faster than that of memory, aiming to reduce the time for the CPU to access memory.

[0052] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

[0053] Figure 1 is one of the flow diagrams of the table partition hot unloading and hot loading method based on openGauss provided by the embodiments of the present application. As Figure 1 shown, the method includes the following steps S101, step S102, and step S103.

[0054] Step S101, when unloading the target partition, obtain the unique physical file identifier (relfilenode, or the ID of the physical file) corresponding to the target partition by querying metadata.

[0055] For example, query the metadata by the target partition name to obtain the unique physical file identifier corresponding to the target partition.

[0056] Step S102, based on the unique physical file identifier corresponding to the target partition, find the physical file corresponding to the target partition and back up the found physical file to obtain the physical file backup data corresponding to the target partition.

[0057] Step S103: Based on the unique physical file identifier corresponding to the target partition, clear the physical files corresponding to the target partition.

[0058] Specifically, the target partition can be any partition in the partition table. In the case where the target partition needs to be unmounted, by querying the metadata information of the database, the unique physical file identifier corresponding to the target partition can be obtained. This unique identifier is the unique code used in the database to identify physical files, ensuring that the physical files of the target partition can be accurately found in subsequent operations.

[0059] According to the obtained unique physical file identifier, the physical files corresponding to the target partition can be located. After finding them, these physical files are backed up to generate the physical file backup data corresponding to the target partition. The backup process can ensure that data will not be lost after the partition is unmounted and can be restored when needed.

[0060] After the backup is completed, based on the unique physical file identifier, the physical files corresponding to the target partition are cleared. This step will release storage space and "unmount" the target partition from the database. The above process of unmounting the target partition only clears the physical files corresponding to the target partition, does not affect the overall operation of the database, and the direct operation of physically unmounting the files takes a short time. Therefore, the partition unmounting method provided in this application can achieve "hot unmounting".

[0061] Exemplarily, Figure 2 is a schematic diagram of the original structure of the partition table, Figure 3 is a schematic diagram of hot unmounting of the partition table. As Figure 2-3 shown, the target partition can be Partition 2. By querying the metadata information of the database, the unique physical file identifier corresponding to Partition 2 can be obtained. According to the obtained unique physical file identifier, the physical files corresponding to Partition 2 can be located, such as the physical files corresponding to the partition data of Partition 2, the physical files corresponding to the index data of Partition 2, the physical files corresponding to the Toast data of Partition 2, and the physical files corresponding to other data of Partition 2. After finding them, these physical files are packaged and backed up to generate the physical file backup data corresponding to the target partition.

[0062] It can be understood that the working process of gs_dump in the prior art includes obtaining metadata, generating a backup script, and exporting data. Its backup method is logical backup (backup according to the running logic of the database), which requires reading data and generating SQL statements, resulting in a relatively high I / O load. In contrast, the present application directly backs up physical files, and the backup method of the present application is physical backup (backup by operating physical files). The copying of physical files can be quickly completed through the file copying command of the operating system, without the need to parse the table structure and generate SQL statements, resulting in a relatively low I / O load. Therefore, the physical backup method of the present application is faster than the logical backup method in the prior art.

[0063] In a possible implementation, before backing up the found physical file, it further includes: locking the target partition, and DDL operations and DML operations are prohibited on the partition in the locked state.

[0064] Exemplarily, as Figure 3 shown, the target partition can be partition 2. Before backing up the physical file corresponding to partition 2, partition 2 can be locked.

[0065] In a possible implementation, before backing up the found physical file, it further includes: writing the in-memory data corresponding to the target partition into the physical file through the checkpoint mechanism of the database.

[0066] Figure 4 FIG. 2 is a second schematic flowchart of the method for hot unloading and hot loading of table partitions based on openGauss provided by the embodiments of the present application. As Figure 4 shown, after clearing the physical file corresponding to the target partition, the method further includes the following steps S201, S202, and S203.

[0067] Step S201, when loading the target partition, obtain the unique identifier of the physical file corresponding to the target partition by querying the metadata.

[0068] Step S202, based on the unique identifier of the physical file corresponding to the target partition, find the backup data of the physical file corresponding to the target partition.

[0069] Step S203, based on the found backup data of the physical file, restore the physical file corresponding to the target partition.

[0070] The physical file unique identifier corresponding to the target partition can not only be used to locate the storage location of the physical file corresponding to the target partition on the disk, but also be used to locate the storage location of the backup data of the physical file corresponding to the target partition on the disk. For example, a first correspondence can be established between the physical file unique identifier and the storage location of the physical file on the disk. According to this first correspondence and the physical file unique identifier, the physical file can be found at the corresponding storage location. For example, a second correspondence can be established between the physical file unique identifier and the storage location of the backup data of the physical file on the disk. According to this second correspondence and the physical file unique identifier, the backup data of the physical file can be found at the corresponding storage location.

[0071] Specifically, when it is necessary to load the target partition, by querying the metadata information of the database, the physical file unique identifier corresponding to the target partition can be obtained. Then, based on the obtained physical file unique identifier, the backup data of the physical file corresponding to the target partition can be searched for. Then, based on the found backup data of the physical file, the physical file corresponding to the target partition is restored. This step will enable the target partition to be re-added to the database, realizing the "loading" of data. In the above process of unloading the target partition, only the physical file corresponding to the target partition needs to be restored (copied) to the original location, which does not affect the overall operation of the database, and the loading method that directly operates on the physical file takes a short time. Therefore, the partition loading method provided by this application can achieve "hot loading".

[0072] Figure 5 is a schematic diagram of the hot loading of the partition table provided by the embodiment of this application, as Figure 5 shown, the target partition can be partition 2. By querying the metadata information of the database, the physical file unique identifier corresponding to partition 2 can be obtained. According to the obtained physical file unique identifier, the backup data of the physical file corresponding to partition 2 can be searched for. The backup data includes: the physical file backup corresponding to the partition data of partition 2, the physical file backup corresponding to the index data of partition 2, the physical file backup corresponding to the Toast data of partition 2, and the physical file backup corresponding to other data of partition 2. Then, based on the found backup data of the physical file, the physical file corresponding to partition 2 is restored to the original location.

[0073] It can be understood that the working process of gs_restore in the prior art includes: reading the backup file, restoring database objects, and restoring data. The restoration method is logical restoration (restoring according to the running logic of the database), which requires parsing the SQL statements and metadata in the backup file generated by gs_dump and performing a large number of insert operations, resulting in a relatively high I / O load. In contrast, in the present application, based on the physical file backup, the physical file is restored (copied) to the original location. The backup method of the present application is physical restoration (restoring by operating on physical files). The copying of physical files can be quickly completed through the file copy command of the operating system, without the need to parse SQL statements and metadata, nor to perform a large number of insert operations, resulting in a relatively low I / O load. Therefore, the physical backup and restoration method of the present application is faster than the logical restoration method in the prior art.

[0074] In addition, in the prior art, when unloading an old partition, the metadata of the old partition in the database is removed. When loading the old partition, it is also necessary to reconfigure the metadata of the old partition, which is a cumbersome operation and has low efficiency. In the present application, when unloading an old partition, only the physical file corresponding to the old partition is removed. When loading the old partition, the physical file is restored (copied) to the original location, without the need to reconfigure the metadata of the old partition, and the operation is relatively simple and efficient.

[0075] In a possible implementation, after restoring the physical file corresponding to the target partition, it further includes: clearing the data in the partition cache of the partition table.

[0076] In a possible implementation, after clearing the data in the partition cache of the partition table, it further includes: unlocking the target partition. A partition in the unlocked state can perform DDL operations and DML operations.

[0077] Exemplarily, as Figure 5 shown, the target partition can be partition 2. After restoring the physical file corresponding to partition 2, the data in the partition cache of the partition table is cleared, and then partition 2 is unlocked.

[0078] The method for hot unloading and hot loading of table partitions based on openGauss provided by the present application will be exemplarily described below through an example.

[0079] The hot unloading and hot loading of table partitions mainly include two processes: hot unloading and hot loading.

[0080] The hot unloading process is as follows.

[0081] (1) Lock the unloading partition of the partition table. The unloading partition of the partition table cannot perform DDL and DML operations anymore. After the partition is locked, if any DDL and DML operations are performed on this partition, then this partition will report an error of partition locking.

[0082] (2) Execute a Checkpoint on the database to ensure that all data in memory is written to disk (including writing the data in memory to the corresponding physical files), and ensure that the physical files of the partitions in the partition table contain all data without any loss.

[0083] (3) On the basis of completing the checkpoint, search for the corresponding relfilenode of the partition of the partition table in the metadata.

[0084] (4) Pack and back up the physical files corresponding to the partition according to the relfilenode (including the physical files corresponding to the partition data, the physical files corresponding to the partition index data, the physical files corresponding to the partition Toast data, and the physical files corresponding to other data).

[0085] (5) Execute the truncate statement on the partition to safely clear the physical files.

[0086] The hot loading process is as follows.

[0087] (1) Query the corresponding relfilenode in the metadata according to the partition to be hot loaded.

[0088] (2) Use the relfilenode in the previous step to restore the physically file packed and backed up during hot unloading to its original position.

[0089] (3) Clear the data in the partition cache of the partition table to ensure that data is loaded from the physical file and avoid the data stored in the cache affecting the accuracy of the loaded data.

[0090] (4) Release the partition lock of the partition table, and DDL and DML operations can be performed on the partition of the partition table.

[0091] It can be understood that in this application, by directly backing up and packing the physical files, there is no need to import and export data using a backup and recovery tool, which reduces the IO consumption of the database. The metadata of the original partition will also be retained, and the original partition can be used simply by reloading, achieving the effects of hot unloading and hot loading.

[0092] Next, the table partition hot unloading and hot loading device based on openGauss provided by this application will be described. The table partition hot unloading and hot loading device based on openGauss described below can be mutually referred to with the table partition hot unloading and hot loading method based on openGauss described above.

[0093] Figure 6 is one of the schematic structural diagrams of the table partition hot unloading and hot loading device based on openGauss provided by the embodiments of this application, as Figure 6As shown in the figure, the device includes: a metadata query module 10, a physical file backup module 20, and a physical file deletion module 30. Among them:

[0094] The metadata query module 10 is configured to, when unloading a target partition, obtain the unique identifier of the physical file corresponding to the target partition by querying metadata;

[0095] The physical file backup module 20 is configured to, based on the unique identifier of the physical file corresponding to the target partition, find the physical file corresponding to the target partition, and back up the found physical file to obtain the physical file backup data corresponding to the target partition;

[0096] The physical file deletion module 30 is configured to, based on the unique identifier of the physical file corresponding to the target partition, delete the physical file corresponding to the target partition.

[0097] Figure 7 is the second schematic diagram of the structure of the table partition hot-unloading and hot-loading device based on openGauss provided by an embodiment of the present application. As Figure 7 shown in the figure, the device further includes: a backup data search module 40 and a physical file restoration module 50;

[0098] The metadata query module 10 is further configured to, when loading a target partition, obtain the unique identifier of the physical file corresponding to the target partition by querying metadata;

[0099] The backup data search module 40 is configured to, based on the unique identifier of the physical file corresponding to the target partition, find the physical file backup data corresponding to the target partition;

[0100] The physical file restoration module 50 is configured to, based on the found physical file backup data, restore the physical file corresponding to the target partition.

[0101] It can be understood that for the detailed function implementation of each of the above units / modules, reference can be made to the introduction in the foregoing method embodiments, and details are not described herein again.

[0102] It should be understood that the above device is used to execute the method in the above embodiment. For the corresponding program modules in the device, their implementation principles and technical effects are similar to those described in the above method. The working process of the device can refer to the corresponding process in the above method, and details are not described herein again.

[0103] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device. Figure 8 is the schematic diagram of the structure of the electronic device provided by an embodiment of the present application. As Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logical instructions in the memory 830 to execute the method in the above embodiments.

[0104] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application.

[0105] Based on the method in the above embodiments, an embodiment of this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on a processor, the processor is caused to execute the method in the above embodiments.

[0106] Based on the method in the above embodiments, an embodiment of this application provides a computer program product. When the computer program product runs on a processor, the processor is caused to execute the method in the above embodiments.

[0107] It can be understood that the processor in the embodiments of this application may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0108] The method steps in the embodiments of the present application can be implemented in a hardware manner or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), register, hard disk, removable hard disk, CD-ROM, or any other form of storage medium well-known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0109] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0110] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0111] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for hot unloading and hot loading of table partitions based on openGauss, characterized in that Including: When unloading the target partition, obtain the unique physical file identifier corresponding to the target partition by querying metadata; Based on the unique physical file identifier corresponding to the target partition, locate the physical file corresponding to the target partition, and back up the located physical file to obtain the physical file backup data corresponding to the target partition; Based on the unique physical file identifier corresponding to the target partition, clear the physical file corresponding to the target partition; After clearing the physical file corresponding to the target partition, it further includes: When loading the target partition, obtain the unique physical file identifier corresponding to the target partition by querying metadata; Based on the unique physical file identifier corresponding to the target partition, locate the physical file backup data corresponding to the target partition; Based on the located physical file backup data, restore the physical file corresponding to the target partition to its original location.

2. The method for hot unloading and hot loading of table partitioning based on openGauss according to claim 1, wherein Before backing up the located physical file, it further includes: Lock the target partition. A partition in the locked state is prohibited from performing DDL operations and DML operations.

3. The method for hot unloading and hot loading of table partitions based on openGauss according to claim 2, wherein Before backing up the located physical file, it further includes: Through the database's Checkpoint mechanism, write the memory data corresponding to the target partition into the physical file.

4. The method for hot unloading and hot loading of table partitioning based on openGauss according to claim 1, wherein After restoring the physical file corresponding to the target partition, it further includes: clearing the data in the partition cache of the partition table.

5. The method for hot unloading and hot loading of table partitioning based on openGauss according to claim 4, wherein After clearing the data in the partition cache of the partition table, it further includes: Unlock the target partition. A partition in the unlocked state can perform DDL operations and DML operations.

6. A table partition hot unloading and hot loading device based on openGauss, characterized in that, Including: A metadata query module, used to obtain the unique physical file identifier corresponding to the target partition by querying metadata when unloading the target partition; A physical file backup module, used to locate the physical file corresponding to the target partition based on the unique physical file identifier corresponding to the target partition, and back up the located physical file to obtain the physical file backup data corresponding to the target partition; A physical file clearing module, used to clear the physical file corresponding to the target partition based on the unique physical file identifier corresponding to the target partition; It further includes: a backup data search module and a physical file restoration module; The metadata query module is further used to obtain the unique physical file identifier corresponding to the target partition by querying metadata when loading the target partition; The backup data search module is used to locate the physical file backup data corresponding to the target partition based on the unique physical file identifier corresponding to the target partition; The physical file restoration module is used to restore the physical file corresponding to the target partition to its original location based on the located physical file backup data.

7. An electronic device, characterized in that, Including: At least one memory for storing computer programs; At least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program runs on the processor, the processor is caused to execute the method according to any one of claims 1-5.

Images