Data processing method, readable storage medium and electronic equipment

Abstract

The invention relates to the field of databases, in particular to a data processing method, a readable storage medium and electronic equipment. The data processing method is applied to a database system in the electronic equipment, and when the database system cleans dirty data in a first heap file in a target column storage table, raw data except the dirty data in the first heap file is copied to a second heap file. And after the second raw data is copied, if a first service in the database system is accessing the first heap file, the database system clears the first heap file after waiting for the first service to finish accessing the first heap file. Therefore, when the database system cleans the heap file in the column storage table, an exclusive lock does not need to be added to the heap file, so that other businesses of the database system for accessing the first heap file are not influenced, and the parallel capability of the database system for cleaning the businesses of the first heap file in the column storage table and other businesses for accessing the first heap file is improved.

Description

Technical Field

[0001] The present application relates to the field of databases, and in particular to a data processing method, a readable storage medium and an electronic device. Background Art

[0002] PostgreSQL database (PG) is an open source, practical, efficient and widely used general database management system. In PG-based database systems (such as GreenPlum database (GP), etc.), when updating or deleting data, the data to be updated or deleted (hereinafter referred to as dirty data) will not be directly deleted, but only a mark will be made for the dirty data, and the space occupied by these dirty data (hereinafter referred to as garbage space) will still exist. In addition, if there are indexes corresponding to the dirty data (hereinafter referred to as dirty indexes) in the PG database, the dirty index data will also remain. Therefore, the PG database system provides an automatic cleanup process for dirty data and dirty indexes (such as the AutoVacuum process) to automatically clean up the garbage space to avoid the continuous expansion of dirty data or dirty indexes. This ensures the data query performance of the database and the performance of other data manipulation languages ​​(DML).

[0003] The automatic cleanup process can reuse the garbage space marked as dirty data (it only cleans the dirty data in the garbage space and does not reclaim the space). The garbage collection operation will not lock the table of dirty data to be cleaned in the PG database (hereinafter referred to as the source table), so the automatic cleanup process can be parallel with other services of the PG system for the source table. However, the automatic cleanup process can only clean up the data in the row storage table (the data of a table is stored in rows, and a row of data is saved together). The data in the column storage table (the data of a table is stored in columns, and a row of data is stored separately) cannot be cleaned up by the automatic cleanup process due to the storage structure and data writing method. Therefore, the column storage table can only be cleaned up by the PG native VACUUMFULL command (a command for cleaning data in the PG database).

[0004] However, when VACUUM FULL is used to clean up data, an exclusive lock (access exclusive lock) will be added to the source table to be cleaned up in the database to prevent the system from modifying the source table, resulting in all operations on the source table being blocked during the system's cleanup of the source table. Therefore, when VACUUM FULL is used to clean up dirty data in a source table, the entire system's business on the source table will be interrupted, affecting the quality of business services. Summary of the invention

[0005] Embodiments of the present application provide a data processing method, a readable storage medium, and an electronic device.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to an electronic device, comprising: detecting a first instruction, the first instruction being used to instruct cleaning of a first part of data in a first pile of files in a database; copying a second part of data in the first pile of files to a second pile of files, and modifying the physical address of the second part of data from a first physical address corresponding to the first pile of files to a second physical address corresponding to the second pile of files; deleting the first pile of files, and when a business for the first data in the second part of the data is detected, accessing the first data in the second pile of files based on the second physical address.

[0007] Exemplarily, the data processing method provided in the present application is applied to a database system of an electronic device. In some embodiments of the present application, the first part of the data may also be referred to as dirty data, and the second part of the data may also be referred to as raw data. The first instruction may also be referred to as an instruction for cleaning dirty data in other embodiments. When the database system in the electronic device detects the first instruction for cleaning the first part of the data in the first pile of files, the second part of the data in the first pile of files is first copied to the second pile of files, and the second pile of files and the first pile of files belong to the same file in a target column storage table in the database. Moreover, the second pile of files is the last pile of files in the target column storage table. The process of copying the second part of the data is equivalent to re-inserting the data into the database system, so that when the database copies the second part of the data, it will not block other services of the database system from accessing the first pile of files. Moreover, when the database system has copied the second part of the data, any service in the database system that accesses the second part of the data will access the second pile of files. If there is still a first service accessing the first pile of files in the database system, the database system will wait for the first service to end accessing the first pile of files before deleting the first pile of files, thereby completing the cleaning of the first pile of files.

[0008] That is to say, the database system can copy the second part of the data in parallel with other businesses in the database system, and after copying the second part of the data, the database system will wait until no business in the database system accesses the first pile of files before deleting the first pile of files, so the process of deleting the first pile of files will not block other businesses in the database system. Therefore, the business of clearing the data of the files in the column storage table in the database system can be executed in parallel with other businesses, improving the concurrency of the database system.

[0009] In a possible implementation of the first aspect, the first pile of files and the second pile of files are pile files in the same target column storage table in a PostgreSQL database or a modified database based on the PostgreSQL database.

[0010] For example, in some embodiments of the present application, the database system in the electronic device is a PostgreSQL database or a database based on a variant of the PostgreSQL database, such as a GreenPlum database. Both the first pile of files and the second pile of files are files in a target column storage table in the database system. That is, when the data processing method in the embodiment of the present application cleans up the pile files in the column storage table in the database system, the cleaning process will not be blocked by other services in the database system.

[0011] In a possible implementation of the first aspect, the copying of the second part of the data in the first stack file to the second stack file includes: when the remaining storage space of the last stack file corresponding to the target column storage table is greater than the storage space occupied by the data of the second part of the data, the last stack file of the target column storage table is used as the second stack file; when the last stack file of the target column storage table is full of data, a second stack file is established in the target column storage table.

[0012] For example, in some embodiments of the present application, if the remaining storage space in the last heap file of the target column storage table is larger than the storage space occupied by the second part of the data, the database system will copy the second part of the data to the last heap file of the target column storage table, that is, the last heap file of the target column storage table is the second heap file. When the last heap file of the target column storage table is full of data, the database system will create a new heap file at the end of the target column storage table as the second heap file to store the second part of the data. This process of copying data is similar to the process of inserting data into the target column storage table, so the copying process will not block other services of the database system for the first heap file.

[0013] In a possible implementation of the first aspect above, the first stack file also includes a third part of data in addition to the first part of data and the second part of data, and the method also includes: the remaining storage space of the last stack file corresponding to the target column storage table is greater than the storage space occupied by the second part of data and the third part of data, and the third part of data is stored in the second stack file; the remaining storage space of the last stack file corresponding to the target column storage table is less than the storage space occupied by the second part of data and the third part of data, a third stack file is established in the target column storage table, and the third part of data is stored in the third stack file.

[0014] That is to say, the last heap file of the target column storage table still has remaining storage space for storing the second part of the data copied by the database system, but the first heap file contains the third part of the data (also the raw data) in addition to the first part of the data (also the dirty data). Since the storage space of the last heap file (i.e. the second heap file) in the target column storage table is full after storing the second part of the data, the remaining raw data (i.e. the third part of the data) in the first heap file will be stored in the third heap file newly created by the database system at the end of the target column storage table.

[0015] In a possible implementation of the first aspect above, the first instruction is also used to instruct cleaning of first index data corresponding to the first portion of data; and the method also includes: in response to the first instruction, cleaning of the first index data corresponding to the first portion of data.

[0016] That is, when the first part of the data in the database system needs to be cleaned, the first dirty index data corresponding to the first part of the data will also be cleaned. Therefore, the first instruction is also used to instruct to clean the first index data corresponding to the first part of the data.

[0017] In a possible implementation of the first aspect above, the above-mentioned first instruction includes an identifier of the first part of the data; and in response to the first instruction, the first index data corresponding to the first part of the data is cleaned up, including: determining the first physical address of the first part of the data based on the identifier of the first part of the data; determining the first index data pointing to the first physical address of the first part of the data based on the first physical address; traversing the index files in the search database, and cleaning up the first index data corresponding to the first physical address.

[0018] That is to say, when the database system cleans up the index file, it will not lock the index file, but determine the physical storage address of the first part of the file (i.e., the first physical address) according to the identifier of the first part of the data, and then determine the first index data pointing to the first physical address according to the first physical address. Then the database system traverses the index file and cleans up the first index data. Therefore, when the database system cleans up the first index data, it will not block other services of the database system from accessing the index file.

[0019] In a possible implementation of the first aspect above, the method further includes: scanning the storage space occupied by the first part of the data in the target column storage table in the database; and generating a first instruction when the storage space occupied by the first part of the data exceeds a space threshold.

[0020] For example, in some embodiments of the present application, the database system scans the table files in the database in real time, and generates a first instruction when the storage space occupied by the first part of the data (that is, dirty data) of the first pile file in the target column storage table exceeds the space threshold. The space threshold can be determined, for example, according to the capacity of the pile file, for example, the space threshold is 30% of the storage space of the first pile file. In other embodiments, the space threshold can also be other values, and the embodiments of the present application do not limit the size of the space threshold.

[0021] In a possible implementation of the first aspect above, the above-mentioned deletion of the first stack of files includes: before changing the physical address of the second part of the data from the first physical address corresponding to the first stack of files to the second physical address corresponding to the second stack of files, detecting a first business that accesses the second data in the second part of the data in the first stack of files based on the first physical address, and after changing the physical address of the second part of the data from the first physical address corresponding to the first stack of files to the second physical address corresponding to the second stack of files, determining that the first business has not ended accessing the second data, waiting for the first business to end accessing the second data, and then deleting the first stack of files.

[0022] That is, before the database system copies the second part of the data, there is a first service in the database system that accesses the second data in the first pile of files. And after the database system copies the second part of the data, the first service has not finished accessing the second data in the first pile of files. The database system will wait until the first service finishes accessing the second data in the first pile of files before deleting the first pile of files to ensure the normal operation of the first service. That is to say, the database system will not block the service that accesses the first pile of files when cleaning up the first pile of files.

[0023] In a possible implementation of the first aspect above, the above-mentioned deleting the first stack of files also includes: after changing the physical address of the second part of the data from the first physical address corresponding to the first stack of files to the second physical address corresponding to the second stack of files, determining that no business is accessing the first stack of files; and deleting the first stack of files.

[0024] That is, after the database system has copied the second part of the data, there is no business in the database system accessing the first pile of files. The database system can directly delete the first pile of files to complete the cleaning of the first pile of files.

[0025] In a second aspect, the present application provides an electronic device, the electronic device comprising: a memory for storing instructions; at least one processor for executing instructions so that the device implements the first aspect and any possible implementation method of the first aspect. The beneficial effects that can be achieved in the second aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here.

[0026] In a third aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and when the instructions are executed by a device, the computer implements the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects that can be achieved in the third aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here.

[0027] In a fourth aspect, the present application provides a computer program product, which, when running on a device, enables the device to implement the method provided in the first aspect and any possible implementation of the first aspect. The beneficial effects that can be achieved in the fourth aspect can refer to the beneficial effects of the method provided in any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a table file structure in a PG database system is shown;

[0029] Figure 2 A flowchart of cleaning data by VACUUM FULL is shown;

[0030] Figure 3 According to some embodiments of the present application, an architecture diagram of a database system;

[0031] Figure 4 According to some embodiments of the present application, an architecture diagram of a distributed database system is shown;

[0032] Figure 5 According to some embodiments of the present application, a flowchart for implementing cleaning up heap file data is shown;

[0033] Figure 6 According to some embodiments of the present application, a schematic diagram of a cudesc table of a column storage table copying data is shown;

[0034] Figure 7 According to some embodiments of the present application, a schematic diagram of any snapshot of the cudesc table is shown;

[0035] Figure 8 According to some embodiments of the present application, a flowchart of cleaning a column storage table in a database system 100 is shown;

[0036] Fig. 9 According to some embodiments of the present application, a schematic diagram of a database system 100 cleaning up a first pile of files is shown;

[0037] Fig.10According to some embodiments of the present application, a schematic diagram of a database system 100 cleaning dirty indexes is shown;

[0038] Fig.11 According to some embodiments of the present application, a schematic diagram of the structure of an electronic device 400 is shown. DETAILED DESCRIPTION

[0039] The illustrative embodiments of the present application include, but are not limited to, a data processing method, a readable storage medium, and an electronic device.

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0041] Figure 1 A schematic diagram of a table file structure in a PG database system is shown.

[0042] It should be understood that in the PG database system, all tables are managed based on the library dimension, that is, a table always belongs to a library. Both databases and tables can be named using object identifiers (oid).

[0043] It should be understood that in the PG database system, each table is stored as one or more heap files, each heap file can be called a block, and the maximum capacity of each block can be the block capacity (e.g. 1GB). If the size of the data in a table is less than or equal to the block capacity, the table can be stored as one heap file; if the size of the data in a table is greater than the block capacity, the data in the table can be stored as multiple heap files, and the size of the data in each heap file is less than or equal to the block capacity.

[0044] For example, Figure 1 As shown, a library file named test (hereinafter referred to as test library) is established in the database system, and the oid of the test library is 1234. Among them, a table file named student (hereinafter referred to as student table) is established in the test library, and the oid of the student table is 12345. Exemplarily, each 1GB block of the table is stored in a separate heap file. When the table file has reached 1GB, when the user stores data in the table file again (for example, when inserting data into the student table through the insert into instruction), the PG database system will recreate a new heap file (hereinafter referred to as new file), and the new file name format is: table oid+".+serial number id (serial number id starts from 1 and increases once). For example Figure 112345.1, 12345.2, 12345.3, etc.

[0045] When the dirty data (or the first part of data) in the student table is cleaned up by VACUUM FULL in the database system, an exclusive lock will be added to the student table, so that other businesses in the database system cannot access the data in the student table, resulting in business interruption. The following introduces the process of cleaning up data by VACUUM FULL in the database system.

[0046] Figure 2 A flowchart for cleaning data using VACUUM FULL is shown.

[0047] It should be understood that the executor of each of the following processes is the PG database system. When introducing each process, the executor of each process will not be repeated.

[0048] like Figure 2 As shown, the process includes:

[0049] S201, adding an exclusive lock to the source table.

[0050] For example, when cleaning dirty data in the source table through VACUUM FULL, it is necessary to add an exclusive lock to the entire source table to prevent the PG database system from performing other operations on the source table.

[0051] It should be understood that the exclusive lock is used to ensure that the business holding the lock is the only business accessing the source table. That is, only the VACUUM FULL business can access the source table. In other words, after the source table is locked by a business, other businesses cannot access the source table.

[0052] S202, creating a temporary table.

[0053] For example, after an exclusive lock is added to the source table, the database system will create a temporary table, and no data is stored in the temporary table at this time.

[0054] S203, copying data from the source table to the temporary table.

[0055] Exemplarily, after the temporary table is established, the data in the source table can be copied to the temporary table. It should be understood that in the process of copying data, only the data other than the dirty data in the source table (hereinafter referred to as raw data or the second part of data) will be copied to the temporary table. Therefore, dirty data will not be stored in the temporary table, and there will be no garbage space.

[0056] S204, exchanging the contents of the source table and the temporary table.

[0057] For example, after all raw data in the source table is copied to the temporary table, the data in the source table can be exchanged with the data in the temporary table. In this way, no dirty data is stored in the source table, no garbage space exists, and the garbage space in the source table is cleaned up.

[0058] S205: Rebuild the index on the source table.

[0059] For example, since the temporary table only copies the raw data in the source table, the storage location of the data in the temporary table will be different from the storage location in the source table. After the data in the source table and the temporary table are exchanged, the data on the source table needs to be re-indexed.

[0060] S206, delete the temporary table.

[0061] Exemplarily, after exchanging the data of the temporary table and the source table, the temporary table can be deleted to release the storage space of the PG database system.

[0062] It should be understood that there is no order between S205 and S206. In other embodiments, S206 may be performed first and then S205.

[0063] S207, releasing the exclusive lock of the source table.

[0064] For example, after the data in the source table is indexed, the dirty data cleaning and garbage space recovery in the source table have been completed. At this time, the exclusive lock of the source table can be released so that the PG database system can execute other services for the source table.

[0065] It should be understood that when the source table data is cleaned up through the VACUUM FULL service, an exclusive lock will be added to the source table first to prevent other services in the PG database system from modifying the source table. As a result, all operations on the source table in the PG database system during the source table cleanup process, such as DML-based data operations and data definition language (DDL) data operations, will be blocked, causing serious business interruptions and affecting business service quality.

[0066] In some embodiments, DML data operations can be divided into two categories: data query and data update, and data update is further divided into three operations: insert, delete, and modify. DDL is used to define database modes, basic tables, views, and index creation and cancellation operations.

[0067] In other embodiments, when cleaning up garbage space, the database system may also clean up only a certain heap file in the source table (e.g., the file corresponding to 12345.2 in the student table), so that an exclusive lock is set only for the heap file. In this way, the database system can execute corresponding DML services for other heap files in the source table, thereby improving the parallel ability of the database system to clean up data and execute services. However, when the database system reclaims space for a certain heap file, the DML statement for the heap file will be blocked, so the concurrency of the solution of cleaning dirty data by adding an exclusive lock to the heap file is low.

[0068] In order to solve the problem of low concurrency when the above-mentioned database system cleans up the space of the column storage table, the present application provides a data processing method, which is applied to the database system. The method includes that when the database system detects an instruction to clean up the dirty data in the first pile of files corresponding to the source table, the database system can first copy the other data (hereinafter referred to as raw data) in the first pile of files except the dirty data to the second pile of files corresponding to the source table, and modify the first physical address of the raw data recorded in the source table pointing to the first pile of files to the second physical address pointing to the second pile of files (that is, the physical address of the storage space storing the copied raw data). In this way, for the business that needs to access the raw data after the database system updates the physical address of the raw data, the raw data in the second pile of files can be accessed.

[0069] In addition, the first business that has been accessing the raw data when the database system starts to copy the raw data can continue to access the raw data in the first pile of files, and the database system can delete the first pile of files after the first business completes accessing the first pile of files to complete the cleaning of dirty data in the first pile of files. If no business accesses the raw data in the first pile of files after the raw data is copied, the database system can delete the first pile of files, directly clean up the dirty data in the first pile of files, and retain the raw data in the first pile of files in the second pile of files.

[0070] In the above-mentioned cleaning process of the first pile of files, the access of the services in the database system to the raw data in the first pile of files will not be blocked. For example, for the services that are already accessing the raw data when the database system starts to copy the raw data, the raw data in the first pile of files can continue to be accessed. For another example, for the services that need to access the raw data during the period from when the database system starts to copy the raw data to when the database system updates the physical address of the raw data, the raw data in the first pile of files can be accessed. For another example, for the services that need to access the raw data after the database system updates the physical address of the raw data, the raw data in the second pile of files can be accessed. Moreover, after the database system copies the raw data, if there is a first service that is accessing the first pile of files, the database system will wait until the first service completes the access before deleting the first pile of files to avoid interruption of the first service's access to the raw data. In other words, the cleaning process of the first pile of files will not block the DML services of the database system for the raw data, thereby improving the concurrency of the database system cleaning the column storage table and other services in the database system.

[0071] Through this solution, when the database system cleans up the garbage space of the column-stored table, it does not need to add an exclusive lock to the entire column-stored table or the heap file in the column-stored table. Therefore, the database system will not block other DML services for the column-stored table, and the concurrency of cleaning up the garbage space and executing the DML services of the database system is improved, thereby improving the service quality of the database system.

[0072] It should be understood that when the data is generated into the second pile file, if the space of the second pile file is full, a new pile file can be created to continue the data (for example, create a 12345.4 file in the student table).

[0073] It should be understood that when cleaning dirty indexes, all dirty data in the column storage table can be traversed to determine the dirty index information pointing to the location of the dirty data based on the location information of the dirty data, thereby deleting the dirty index data to complete the cleaning of the dirty index.

[0074] The following introduces an architecture of a database system.

[0075] Figure 3 According to some embodiments of the present application, an architecture diagram of a database system is shown.

[0076] like Figure 3 As shown, the database system 100 includes a database 110 and a database management system (DBMS) 130 .

[0077] Among them, the database 110 is a data set stored in the data storage 120, that is, a collection of associated data organized, stored and used according to a specific data model. According to the different data models used to organize data, data can be divided into multiple types, such as relational data, graph data, time series data, etc. Relational data is data modeled using a relational model, usually represented as a table. Exemplarily, in some row storage tables, the rows in the table represent a set of related values ​​of an object or entity. In some column storage tables, the related values ​​of an object or entity are stored separately (stored by column). Graph data, referred to as "graph", is used to represent the relationship between objects or entities, such as social relationships. Time series data, referred to as time series data, is a data column recorded and indexed in chronological order, used to describe the state change information of an object in the time dimension. It should be understood that in the embodiment of the present application, taking relational data as an example, the cleaning process for dirty data and dirty indexes in column storage tables is introduced.

[0078] The database management system 130 is used to organize, store and maintain data. The client 200 can access the database 110 through the database management system 130, and the database administrator can also perform maintenance work on the database 110 through the database management system 130. The database management system 130 provides multiple functions for the client 200 to establish, modify and query the database, wherein the client 200 can be an application or a user device. The functions provided by the database management system 130 may include but are not limited to the following items: (1) Data definition function. The database management system 130 provides DDL to define the structure of the database 110. DDL is used to characterize the database framework and can be saved in the data dictionary. (2) Data access function. The database management system 130 provides DML to implement basic access operations on the database 110, such as retrieval, insertion, modification and deletion. (3) Database operation management function. The database management system 130 provides data control functions to effectively control and manage the operation of the database 110 to ensure that the data is correct and valid. (4) Database establishment and maintenance functions, including loading of initial data into the database 110, dumping, restoring, and reorganizing the database 110, and system performance monitoring and analysis. (5) Database transmission: the database management system 130 provides transmission of processed data to achieve communication between the client 200 and the database management system 130, which is usually coordinated with the operating system.

[0079] Exemplarily, in some embodiments of the present application, the database management system 130 may also perform operations such as real-time scanning of the data storage 120 (e.g., scanning the data in the source table, obtaining information about dirty data in the source table (including information such as location and size) and dirty index information), cleaning the data storage 120 (e.g., cleaning dirty data in the source table in the storage 120), and reclaiming garbage space. For example, in some embodiments of the present application, the database system 100 may scan the column storage table in the data node 330A in real-time based on the data management system 130, and determine whether to clean up the heap file based on the storage space occupied by dirty data in the heap file in the corresponding column storage table. Exemplarily, in an embodiment of the present application, when cleaning up dirty data in the first heap file. The database system 100 may copy the raw data in the first heap file to be cleaned to the last heap file of the corresponding column storage table based on the data management system 130 deployed on the data node 330A, and then modify the first physical address of the copied raw data pointing to the first heap file to the second physical address. After the physical address of the generated data is modified, if a first service is accessing the first pile of files, the data management system 130 waits for the first service access to end before clearing the first pile of files, so that the database system 100 does not block other services in the database system 100 that target the first pile of files when clearing the first pile of files.

[0080] The data storage 120 includes, but is not limited to, solid state drives (SSDs), disk arrays, cloud storage, or other types of non-transient computer-readable storage media. Figure 3 Fewer or more components than those shown in the figure, or including Figure 3 The components shown are different components, Figure 3 Only components more relevant to the implementation disclosed in the embodiment of the present invention are shown.

[0081] For example, the database system 100 provided in the embodiment of the present application may be a distributed database system (DDBS). In the process of transaction processing, the DDBS usually uses a global transaction manager (GTM) to manage transactions in order to achieve concurrency control between transactions.

[0082] For example, Figure 4 According to an embodiment of the present application, an architecture diagram of a distributed database system is shown.

[0083] like Figure 4As shown, the distributed database system includes at least one coordinator node (CN) 310, multiple data nodes (DN) 330 (e.g., data node 330A, data node 330B, data node 330C), and at least one global transaction manager 320. The coordinator node 310 and the data node 330 communicate through a network channel. In some embodiments, the network channel can be composed of network devices such as switches, routers, and gateways. The coordinator node 310, the data node 330, and the global transaction manager 320 jointly implement the functions of the database management system 130, and provide the client 200 with services such as retrieval, insertion, modification, and deletion of the database 110.

[0084] Exemplarily, in some embodiments, a database management system 130 is deployed on each coordination node 310, data node 330, and global transaction manager 320. The data nodes 330 have their own exclusive hardware resources (such as data storage 340, including data storage 340A, data storage 340B, data storage 340C, etc.), operating system and database 110. Under this system, data will be allocated to each data node 330 according to the database 110 model and application characteristics, and query tasks will be divided into several parts by the data nodes 330, which will be executed in parallel on all data nodes 330, and they will cooperate with each other to provide database services as a whole, and all communication functions will be implemented on a high-bandwidth network interconnection system.

[0085] Exemplarily, taking data node 330A as an example, when cleaning dirty data and dirty indexes in data node 330A, the dirty data in each source table (including row storage table and column storage table) in data node 330A can be scanned based on the data management system 130 deployed on data node 330A, and the automatic cleaning process or VACUUM FULL command can be called to clean the dirty data in data node 330A, and the garbage space in data node 330A can be recovered (wherein the automatic cleaning process cannot clean the dirty data in the column storage table). In some embodiments of the present application, when the database system 100 (or referred to as a distributed database system) cleans the column storage table in data node 330A, the raw data in the first pile file to be cleaned can be copied to the last pile file of the corresponding column storage table based on the data management system 130 deployed on data node 330A, and then the first physical address of the copied raw data pointing to the first pile file is modified to the second physical address. After the physical address of the raw data is modified, if a first business is accessing the first pile file, the data management system 130 waits for the first business access to end, and then cleans the first pile file. This ensures that the database system 100 does not block other services in the database system 100 that are directed to the first pile of files when clearing the first pile of files.

[0086] The coordination node 310, data node 330, and global transaction manager 320 in the distributed database system can be a physical machine, such as a database server, or a virtual machine (VM) or container running on abstract hardware resources. In some embodiments, the coordination node 310, data node 330, and global transaction manager 320 are virtual machines or containers, and the network channel is a virtual switching network, which includes a virtual switch. The database management system 130 deployed in the coordination node 310, data node 330, and global transaction manager 320 is a DBMS instance, which can be a process or a thread, and these DBMSs cooperate to complete the functions of the database relational system. In another embodiment, the coordination node 310, data node 330, and global transaction manager 320 are physical machines, and the network channel includes one or more switches, and the switch is a storage area network (SAN) switch, an Ethernet switch, a fiber switch or other physical switching equipment.

[0087] Next, taking the heap file in the data node 330A as an example, the process of cleaning up the heap file in the embodiment of the present application is introduced.

[0088] For example, Figure 5 According to some embodiments of the present application, a flowchart for implementing cleaning up heap file data is shown.

[0089] It should be understood that a database management system 130 is deployed in the data node 330A in the database system 100, so the DML business for the data node 330A can be executed by the database management system 130 in the data node 330A. In some embodiments of the present application, the execution entity for cleaning up dirty data in the heap file is the database system 100 (or the data management system 130 in the database system 100). When introducing the following processes, the execution entity will not be repeated.

[0090] like Figure 5 As shown, the process includes:

[0091] S501 , scanning the column storage table in the database 110 in real time.

[0092] Exemplarily, the database system 100 can scan each column storage table in the database 110 in real time. The smallest storage unit of the column storage table is CU (compress unit, CU), and the size of each CU is an integer multiple of 8KB (it should be noted that CU is an independent storage unit). Referring to Table 1, Table 1 shows a storage structure of a column storage table. The column identifier of each column CU in the column storage table is column_id (hereinafter referred to as col_id). In each column, the CU is identified according to the number of rows of CU. For example, the identifier cu_id of the CU in the first row is 1001. Therefore, the identification information of a CU needs to be determined by col_id and cu_id. For example, the identification information of the CU in the second column of the 1001th row in Table 1 is: col_id=2, cu_id=1001. In the column storage table, multiple CUs in the same column or multiple CUs in different columns of the same cu_id are continuously stored in a heap file. When the size of the heap file exceeds 1G, it will automatically switch to a new heap file. For example, if col_id=1, CUs from cu_id=1001 to cu_id=1010 (i.e., 10 rows of CUs in the same column) store 1G of data, the CUs will be stored in a heap file. In other embodiments, multiple CUs of 1G in size may be stored in a heap file by row.

[0093] Table 1

[0094] col_id=1 col_id=2 cu_id=1001 CU CU cu_id=1002 CU CU cu_id=1003 CU CU … … …

[0095] The database system 100 also stores a row storage table cudesc table corresponding to each column storage table for recording auxiliary and management information of the CU, for example, refer to Table 2. Table 2 shows the structure of a cudesc table. Each row record in Table 2 corresponds to the information of a CU, and the information of the CU includes the identification information of the CU (i.e., col_id and cu_id) and the pointer cu_pointer of the storage address of the CU. The pointer cu_pointer is used to indicate the physical storage address of the corresponding CU. For example, the cu_pointer information in the CU with col_id=2 and cu_id=1001 is P21001, and P21001 indicates the physical storage address of the CU with col_id=2 and cu_id=1001.

[0096] In some embodiments, the cudesc table further stores a row recording information of dirty data in the corresponding CU.

[0097] For example, there is a row with col_id=-10 in the cudesc table, which corresponds to a virtual CU (hereinafter referred to as VCU). The cu_pointer corresponding to the VCU records the deletion of data in a group of CUs with the same cu_id (i.e., CUs in the same row in the column storage table). For example, P1001 records the physical storage address of each dirty data in each column of the CU corresponding to the row with cu_id=1001.

[0098] Table 2

[0099]

[0100] Exemplarily, when the database system 100 scans the column storage table, it may determine which data in the column storage table needs to be deleted based on the information of the VCU in the cudesc table corresponding to the column storage table.

[0101] S502: Based on the scanning result, determine whether to perform data cleaning on the heap files in the column storage table.

[0102] Exemplarily, the database system 100 can determine the size of the storage space occupied by the dirty data of the corresponding heap file in the column storage table based on the scanning results, and determine whether to clean up the heap file based on whether the storage space occupied by the dirty data in the heap file is greater than a threshold.

[0103] For example, the pointer (cu_pointer) corresponding to the VCU of cu_id=1001 is P1001, and P1001 indicates that there are 6000 rows of dirty data in the CU of each column of cu_id=1001 (i.e., the columns of col_id=1 and col_id=2), and each row of dirty data occupies 8KB of storage space, with a total of 6000×8KB=48000KB of dirty data, i.e., 46.88MB of dirty data. In other words, there are 46.88MB of dirty data in the CU of each column of cu_id=1001. The database system 100 continues to scan the pointer corresponding to the VCU of cu_id=1002, thereby continuing to determine the size of the storage space occupied by the dirty data in the heap file.

[0104] It should be understood that in the embodiment of the present application, the threshold of the storage space occupied by dirty data in the heap file may be, for example, 0.2G. In other embodiments, the threshold may also be other values, such as 0.3G, 0.5G, etc. The embodiment of the present application does not limit the threshold of the storage space occupied by dirty data in the heap file.

[0105] If the judgment result is yes, for example, it is determined that the storage space occupied by the dirty data of the heap file in the column storage table is greater than the threshold, the database system 100 issues an instruction to clean up the corresponding heap file (hereinafter referred to as the first heap file) and executes S503, detecting the instruction to clean up the dirty data of the first heap file.

[0106] If the judgment result is no, for example, it is determined that the storage space occupied by the dirty data of no heap file in the column storage table is greater than the threshold, S501 is executed to scan the column storage table in the database 110 in real time.

[0107] S503: An instruction to clean dirty data from the first pile of files is detected.

[0108] Exemplarily, after the database system 100 determines to clean up the first pile of files based on the scanning result, it generates an instruction (i.e., a first instruction) to clean up the first pile of files. The instruction may, for example, include the oid of the row storage table corresponding to the first pile of files and the oid corresponding to the first pile of files. The database system 100 cleans up the first pile of files based on the instruction.

[0109] For example, in some other embodiments, the instruction to clean up dirty data in the first pile of files detected by the database system 100 may also be determined by the user. That is, the user may trigger the instruction to clean up dirty data in the first pile of files through the client 200. The database system 100 cleans up the first pile of files according to the instruction. That is, when the user determines to issue the instruction to clean up dirty data in the first pile of files, the database system 100 does not execute S501 and S502.

[0110] S504, copying the raw data in the first pile of files to the second pile of files.

[0111] Exemplarily, after detecting an instruction to clean dirty data on a first file, the database system 100 may scan the first pile of files, determine raw data in the first pile of files, and copy the determined raw data to the second pile of files.

[0112] The database system 100 can determine the information of dirty data and raw data in the first pile of files based on the cudesc table of the source table corresponding to the first pile of files. For example, the VCU in the cudesc table records the information of dirty data in the first pile of files, and the data in the first pile of files other than the dirty data is raw data. The database system 100 copies the raw data in the first pile of files to the second pile of files.

[0113] It should be understood that the database system 100 copies the raw data in the first pile file to the second pile file, for example, by reinserting the raw data into the corresponding column storage table through an insert instruction (insert). The corresponding second pile file is the last pile file in the column storage table. If the second pile file is full of 1G data, the database system 100 will create a new pile file in the column storage table (hereinafter referred to as the newly created pile file as the third pile file) to continue inserting the raw data in the first pile file. In other words, if the remaining storage space size of the second pile file is larger than the size of the storage space occupied by the raw data. The database system 100 inserts all the raw data into the second pile file. If the remaining storage space size of the second pile file is smaller than the size of the storage space occupied by the raw data. After the second pile file is full of raw data, the database system 100 creates a third pile file in the corresponding column storage table and inserts the remaining raw data into the third pile file. If the second pile file is just full of 1G data, the database system 100 creates a third pile file in the corresponding column storage table and inserts the raw data into the third pile file.

[0114] For example, Figure 6 According to some embodiments of the present application, a schematic diagram of a cudesc table of a column storage table copying data is shown.

[0115] Reference Figure 6 In the cudesc table before copying, the first pile of files stores the data of CUs with col_id=1, cu_id=1001 to cu_id=1010, and the data of CUs with col_id=2, cu_id=1001 to cu_id=1010. (That is, the pointers of the above CUs point to the physical addresses corresponding to the first pile of files)

[0116] Reference Figure 6 The cudesc table copied in the first pile of files copies the raw data of the CU as follows: insert the raw data of the CU in the first pile of files into the column storage table, and then insert the data into the last pile file (i.e., the second pile file) of the column storage table. For example, the pointers of the data of the CUs with col_id=1, cu_id=1001 to cu_id=1010, and the pointers of the data of the CUs with col_id=2, cu_id=1001 to cu_id=1010 both point to the physical addresses corresponding to the second pile of files.

[0117] It should be understood that when the raw data in the first pile of files is copied to the second pile of files, the status of the VCU in the first pile of files (i.e., the CU corresponding to the -10 row) will not be changed, that is, the cudesc table still records the information of the dirty data in the first pile of files, and the above process of copying the raw data in the first pile of files will not change the information of the VCU in the first pile of files. In other words, the above process of copying the raw data will not mark the raw data in the first pile of files as deleted. Therefore, the information of the VCU corresponding to the first pile of files will not change.

[0118] S505: Update the physical address of the storage space of the copied raw data to a second physical address.

[0119] Exemplarily, after the database system 100 completes copying the raw data, it updates the first physical address of the raw data to the second physical address. Figure 7 After the raw data of the first pile of files is copied to the second pile of files, the physical address of the raw data is updated in the cudesc table of the column storage table to point to the second physical address of the second pile of files. For example, the database system 100 can point the pointer of the corresponding copied CU in the cudesc table to the physical address corresponding to the second pile of files.

[0120] It should be understood that the raw data in the first pile of files has not been deleted at this time, so the first business can continue to query the raw data in the first pile of files. That is, the first business for the first pile of files before the database system 100 copies the first raw data can continue to access the first pile of files. After the raw data is copied, the business corresponding to the raw data in the database system 100 will access the second pile of files based on the second physical address of the raw data. Therefore, the process of copying the raw data by the database system 100 will not block other businesses in the database system 100 that are directed to the first pile of files.

[0121] For example, Figure 7 According to an embodiment of the present application, a schematic diagram of any snapshot of the cudesc table is shown.

[0122] The any snapshot represents the information of all CUs in the cudesc table, that is, the CU information before copying in the first pile of files can also be retained on the any snapshot of the cudesc table (under normal circumstances, refer to Figure 6 After the first physical address of the raw data is updated to the second physical address, the first physical address of the raw data in the first pile file will not be saved in the cudesc table. ).

[0123] Reference Figure 7In any snapshot before cleaning the cudesc table, the cudesc table also includes the first physical address of the raw data pointing to the first pile of files. This is because the database system 100 has not cleaned up the first pile of files. When a new business in the database system 100 accesses the raw data, it will access the second pile of files based on the second physical address of the raw data. In other words, before the raw data is copied, the first business in the database system 100 accesses the first pile of files based on the first physical address of the raw data. After the raw data is copied, the database system 100 updates the first physical address of the raw data pointing to the first pile of files to the second physical address pointing to the second pile of files, and the new business in the database system 100 will access the raw data in the second pile of files based on the second physical address.

[0124] Exemplarily, in some embodiments, the database system 100 may update the physical address of the corresponding raw data once for each raw data copy when copying the raw data. For example, after copying a row of raw data, insert it to the end of the second pair of files, and then update the first physical address of the row of raw data to the second physical address. In other embodiments, the database system 100 may also copy all the raw data in the first pile of files to the second pile of files, and then update the first physical address of all the raw data to the second physical address.

[0125] S506: Determine whether there is a first service accessing the first pile of files.

[0126] Exemplarily, after updating the physical address of the generated data, the database system 100 determines whether there is still a first service accessing the first pile of files in the database system 100 .

[0127] It should be understood that the process of the database system 100 copying the raw data of the first pile of files will not block the first service of the database system 100 accessing the first pile of files. Moreover, after the database system 100 updates the physical address of the raw data, the data in the first pile of files has not been deleted, so the first service can continue to be executed. Therefore, when the database system 100 clears the first pile of files, it needs to determine whether there is still the first service accessing the first pile of files.

[0128] If the judgment result is yes, then S507 is executed to wait for the first service to finish accessing the first pile of files.

[0129] If the judgment result is no, then S508 is executed to clean up the first pile of files.

[0130] S507, waiting for the first service to finish accessing the first pile of files.

[0131] For example, since there is still a first service for the first pile of files in the database system 100, that is, before the raw data in the first pile of files is copied, there is a first service for the first pile of files accessing the first pile of files. And after the raw data in the first pile of files is copied, the first service for the first pile of files has not completed the access, so before cleaning up the first pile of files, it is necessary to wait for the access of the first service to end, so as to ensure that the first service can be executed smoothly.

[0132] S508, clearing the first pile of files.

[0133] Exemplarily, when there is no business targeting the first pile of files in the database system 100, the database system 100 can clean up the first pile of files. For example, the database system 100 can first clear all dirty data pointing to the first pile of files, and then empty the first pile of files, thereby completing the space recovery of the first pile of files. Since the database system 100 has no business accessing the first pile of files, it is not necessary to lock the first pile of files during the cleaning process.

[0134] Reference Figure 7 In the any snapshot before cleaning the cudesc table, before cleaning the first pile of files, the first pile of files still retains live data. Therefore, after the first business of the database system 100 is executed, the first pile of files can be cleaned up to complete the space recovery of the first pile of files.

[0135] It should be understood that the cudesc table is a row storage table. When cleaning the cudesc table, the cleaning process of the row storage table can be used, such as cleaning the cudesc table based on the automatic cleaning process (AutoVacuum process). The process of cleaning the cudesc table by the automatic cleaning process (AutoVacuum process) is not described in detail here.

[0136] It should be understood that after the database system 100 completes cleaning of dirty data, it can continue to execute S501 to scan the column-stored table in the database 110 .

[0137] It should be understood that the above process of cleaning up the first pile of files copies the raw data in the first pile of files to the tail of the last pile of files (the second pile of files), and only updates the data physical address (cu_pointer) of the CU on the cudesc table, without updating the VCU. This design method ensures that the copying of the raw data in the first pile of files will not be blocked by the delete service, because the delete service of the column storage table is implemented by marking the VCU, and the copying of the raw data does not change the VCU, so the delete service will not be blocked. Since the insert service of the column storage table is implemented by appending to the tail of the data file, it will not be blocked naturally. The essence of the update service is deletion + insertion, so it will not be affected. Because the raw data in the first pile of files still exists, the query service (select) for the first pile of files (the first service) can also query the data normally. Therefore, in the embodiment of the present application, the database system 100 will not block the DML service of the database system 100 when cleaning up the first pile of files.

[0138] The following describes the process of cleaning dirty data and dirty indexes in an embodiment of the present application.

[0139] For example, Figure 8 According to some embodiments of the present application, a flowchart of cleaning a column-stored table in a database system 100 is shown.

[0140] Exemplarily, in some embodiments of the present application, the execution entity for cleaning dirty data and dirty indexes in the column-stored table is the database system 100 (or the data management system 130 in the database system 100), and the execution entity will not be repeated when introducing the following processes.

[0141] like Figure 8 As shown, the process includes:

[0142] S801 , scanning the column storage table in the database 110 in real time.

[0143] S802: Based on the scanning result, determine whether to perform data cleanup on the heap files in the column storage table.

[0144] S803, reclaiming the storage space occupied by the dirty data in the corresponding heap file.

[0145] For example, the contents of S801 to S803 above can refer to Figure 5 The process corresponding to the embodiment.

[0146] S804: Clean up dirty indexes corresponding to dirty data in the column storage table.

[0147] Exemplarily, the database system 100 can scan the dirty data in the column storage table to determine the location information of the dirty data, and based on the location information of the dirty data, determine the dirty index information pointing to the location of the dirty data, thereby deleting the dirty index data to complete the cleaning of the dirty index. The dirty index cleaning process is described below.

[0148] It should be understood that the execution order of S803 and S804 is not limited in this embodiment. In some embodiments, S804 can be executed first to clean up the index corresponding to the dirty data in the column storage table, and then S803 can be executed to reclaim the storage space occupied by the dirty data in the corresponding heap file. Or S803 and S804 can be executed at the same time.

[0149] It should be understood that after completing the cleaning of dirty data and dirty indexes, the database system 100 may continue to perform S801 to scan the column storage table in the database 110 in real time, so as to continue to clean up the heap files in the column storage table whose storage space occupied by dirty data is greater than the threshold.

[0150] It should be understood that in the embodiment of the present application, when the database system 100 cleans up the dirty data of the heap file, it will not block other DML services for the corresponding heap file in the database system 100. That is, the DML services of the database system 100 for the heap file (for example, querying data, inserting data, deleting data, updating data, etc.) and the services for cleaning up the heap file can be carried out in parallel. Since the DDL services in the database system 100 are used to define the creation and cancellation operations of the database 110 mode, basic tables, views, and indexes, etc., and have higher permissions (exclusive locks will be added to the corresponding tables or files), the automatic cleaning process (AutoVacuum process) of the database system 100 for the data cleaning service of the heap file will give locks to high-authority operations. In the embodiment of the present application, when the database system 100 automatically cleans up the heap file in the column storage table, it will also give locks to the DDL services with high permissions, so the data cleaning service of the database system 100 for the heap file will not block the DDL services of the database system 100.

[0151] It should be understood that in some embodiments of the present application, when a user actively triggers an instruction to clean up dirty data in a heap file in a column storage table through the client 200, the instruction has a higher authority and will not give up the lock to the DDL service. At this time, the dirty data cleaning service of the heap file in the column storage table will block the DDL service in the database system.

[0152] Next, we'll describe the process of cleaning up the first pile of files.

[0153] Fig. 9 According to an embodiment of the present application, a schematic diagram of a database system 100 cleaning up a first pile of files is shown.

[0154] like Fig. 9As shown, cleaning of the first pile of files may include processes such as file determination, file copying, and file cleaning.

[0155] Exemplarily, the database system 100 scans the first pile of files in real time to obtain dirty data information of the first pile of files. When the first pile of files meets the cleaning condition (that is, the storage space occupied by the dirty data in the first pile of files is greater than a threshold), the database system 100 determines to clean the first pile of files.

[0156] After the database system 100 determines to clean up the first pile file, it copies the raw data in the first pile file to the last pile file in the column storage table, for example, to the second pile file. If the second pile file is full of 1G data, the database system 100 will create a new pile file in the column storage table (hereinafter referred to as the new pile file is the third pile file).

[0157] It should be understood that after the database system 100 copies the raw data in the first pile of files to the third pile of files, the raw data in the first pile of files is not cleared. After the database system 100 copies the raw data, it updates the first physical address of the raw data pointing to the first pile of files to the second physical address (i.e., the physical address pointing to the third pile of files). Thereafter, the business for raw data in the database system 100 can access the third pile of files based on the second physical address. In other words, there is still raw data in the first pile of files. It should be understood that the raw data in the first pile of files is accessed by the first business, and the first business is the business before the database system 100 copies the raw data.

[0158] When the database system 100 completes the replication of the raw data, the first pile of files can be cleared after the first business has accessed the first pile of files. In the process of clearing the first pile of files, since no business of the database system 100 will access the first pile of files, there is no need to lock the first pile of files.

[0159] It should be understood that the above process of cleaning up the first pile of files will not block other services in the database system 100 for the raw data in the first pile of files, thereby improving the parallel capability of the database system 100 in cleaning up the column storage table and other services.

[0160] Next, taking the column storage table in the data node 330A as an example, the process of cleaning up dirty indexes in the column storage table in the embodiment of the present application is introduced.

[0161] For example, Fig.10 According to an embodiment of the present application, a schematic diagram of cleaning dirty indexes of a database system 100 is shown.

[0162] like Fig.10 As shown, the cleaning process of dirty indexes in the column storage table 10 may include, for example, dirty data judgment and dirty data index cleaning processes.

[0163] Exemplarily, after the database system 100 detects an instruction to clean up dirty indexes, it starts to clean up dirty index data in the column storage table based on the instruction. Exemplarily, the instruction may include, for example, an identifier of the column storage table, such as the column storage table 10. The database system 100 traverses all data in the column storage table 10 and determines the dirty data in the column storage table 10.

[0164] Exemplarily, the process of the database system 100 determining dirty data in the column storage table 10 may be based on VCU in the cudesc table of the column storage table 10. The database system 100 obtains dirty data information based on VCU, and the dirty data information may include, for example, physical address of the dirty data.

[0165] The database system 100 determines the index data pointing to the physical address location based on the physical address and other information of the dirty data. The index data pointing to the physical location of the dirty data is the dirty index data. The database system 100 traverses the index files in the column storage table and cleans up the index data corresponding to the physical location of the dirty data.

[0166] It should be understood that the above process of cleaning dirty index data can clean the index files in the column storage table through the automatic cleaning process (AutoVacuum process). The automatic cleaning process can only clean the dirty index data in the index file, and cannot recycle the storage space corresponding to the dirty index data, but can reuse the storage space corresponding to the dirty index data. In other words, after the dirty index data is cleaned, the data inserted into the index file can continue to use the space corresponding to the dirty index data.

[0167] It should be understood that dirty index cleaning is at the table level (for column-stored tables). During the cleaning of dirty indexes, only the shared lock of the table is held, which will not affect DML and other services on the table. After the cleaning is completed, the dirty data in the index on the table will be removed, and the subsequent insertion of data can reuse the cleaned space, effectively controlling the space occupied by the index file, alleviating the index expansion problem, and improving the performance of index queries.

[0168] Furthermore, the embodiment of the present application also provides an electronic device 400 for implementing the data processing methods provided in the aforementioned embodiments.

[0169] For example, Fig.11 According to some embodiments of the present application, a schematic diagram of the structure of an electronic device 400 is shown. The electronic device 400 corresponds to the coordination node 310, the data node 330, or the global transaction manager 320 in the database system 100. The electronic device 400 of the embodiment of the present application is introduced below using the data node 330 as an example.

[0170] like Fig.11As shown, electronic device 400 includes a processor 410 , a memory 420 , a communication interface 430 , and a bus 440 .

[0171] The processor 410 may include one or more processing units, for example, the processor 410 may include a central processing unit (CPU), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a microcontroller unit (MCU), a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), a field programmable gate array (FPGA), etc. In some embodiments, different processing units may be independent devices or integrated into one or more processors.

[0172] In some embodiments, the processor 410 may be configured to execute one or more programs to implement the dirty data and dirty index cleaning methods provided in the aforementioned embodiments.

[0173] For example, the processor 410 can be used to execute instructions based on the database system 100 in the electronic device 400, such as executing DML instructions (including inserting data, deleting data, updating data, querying data, etc.), traversing dirty data in a column storage table, issuing instructions for cleaning dirty data based on the traversal results, and executing data cleaning instructions (such as executing an automatic cleaning process, a VACUUM FULL command, and dirty data and dirty index cleaning instructions in an embodiment of the present application), etc.

[0174] The memory 420 may include one or more memories for storing data or program codes. For example, in some embodiments, the memory 420 may be used to store data, such as data in the data node 330 in various embodiments of the present application. For another example, in some embodiments, the memory 420 may be used to store instructions corresponding to the data processing methods provided in the aforementioned embodiments.

[0175] In some embodiments, the memory 420 may include a hard disk drive, a solid state drive, a flash memory, an optical disk, or a magneto-optical disk.

[0176] In some embodiments, memory 420 may include removable or non-removable or fixed media.

[0177] In some embodiments, the memory 420 may be internal or external to the electronic device 400 .

[0178] The communication interface 430 is used to implement communication between the electronic device 400 and other devices (eg, the coordination node 310, the global transaction manager 320, and other data nodes 330). In some embodiments, the communication interface 430 may include a wireless communication interface and a wired communication interface.

[0179] In some embodiments, the wireless communication interface can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (blue tooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc., which are applied to the electronic device 400. The wireless communication module can be one or more devices integrating at least one communication processing module.

[0180] The wired communication interface may include an Ethernet interface (such as a fiber optic interface, an RJ-45 interface, etc.), a universal serial bus (USB), a power line carrier communication (PLC) interface, a high definition multimedia interface (HDMI) interface, a digital audio interface, and other wired communication interfaces, for providing a wired communication solution for application on the electronic device 400.

[0181] The bus 440 is used to connect the processor 410 , the memory 420 , the communication interface 430 , and other possible modules or circuits.

[0182] It should be understood that Fig.11 The structure of the electronic device 400 is only an example. In other embodiments, the electronic device 400 may include more or fewer modules, which is not limited here.

[0183] An embodiment of the present application also provides a program product, which, when executed on an electronic device, can enable the electronic device to implement the data processing methods provided by the aforementioned embodiments.

[0184] An embodiment of the present application further provides a readable storage medium, in which one or more programs are stored. When the one or more programs are executed by an electronic device, the electronic device implements the data processing methods provided by the aforementioned embodiments.

[0185] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0186] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0187] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0188] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A data processing method, applied to an electronic device, characterized in that: include: A first instruction is detected, where the first instruction is used to instruct to clean up a first portion of data in a first pile of files in a database; Copying the second part of the data in the first pile of files to the second pile of files, and modifying the physical address of the second part of the data from the first physical address corresponding to the first pile of files to the second physical address corresponding to the second pile of files; delete the first pile of files, and In a case where a service for the first data in the second portion of data is detected, the first data in the second heap file is accessed based on the second physical address.

2. The method according to claim 1, characterized in that The first pile of files and the second pile of files are pile files in the same target column storage table in a PostgreSQL database or a modified database based on the PostgreSQL database.

3. The method according to claim 2, characterized in that The step of copying the second part of the data in the first pile of files to the second pile of files comprises: The remaining storage space of the last heap file corresponding to the target column storage table is larger than the storage space occupied by the data of the second part of data, and the last heap file of the target column storage table is used as the second heap file; The last stack file corresponding to the target column storage table is full of data, and the second stack file is created in the target column storage table.

4. The method according to claim 3, characterized in that The first stack of files also includes a third portion of data in addition to the first portion of data and the second portion of data, and The method further comprises: The remaining storage space of the last heap file corresponding to the target column storage table is larger than the storage space occupied by the second part of data and the third part of data, and the third part of data is further stored in the second heap file; The remaining storage space of the last stack file corresponding to the target column storage table is smaller than the storage space occupied by the second part of data and the third part of data, a third stack file is established in the target column storage table, and the third part of data is stored in the third stack file.

5. The method according to claim 2, characterized in that: The first instruction is also used to instruct to clean up the first index data corresponding to the first part of data; and The method further comprises: In response to the first instruction, first index data corresponding to the first portion of data is cleared.

6. The method according to claim 5, characterized in that The first instruction includes an identification of the first portion of data; and In response to the first instruction, clearing the first index data corresponding to the first portion of data includes: determining the first physical address of the first portion of data based on the identifier of the first portion of data; Determine the first index data of the first physical address pointing to the first portion of data based on the first physical address; The index files in the database are traversed and the first index data corresponding to the first physical address is cleared.

7. The method according to claim 2, characterized in that: Also includes: Scanning the storage space occupied by the first part of the data in the target column storage table in the database; The first instruction is generated corresponding to the storage space occupied by the first part of data exceeding the space threshold.

8. The method according to claim 1, characterized in that The deleting the first pile of files includes: Before modifying the physical address of the second portion of data from the first physical address corresponding to the first stack file to the second physical address corresponding to the second stack file, a first service accessing second data in the second portion of data in the first stack file based on the first physical address is detected, and, After the physical address of the second portion of data is modified from the first physical address corresponding to the first stack of files to the second physical address corresponding to the second stack of files, it is determined that the first service has not ended accessing the second data, After the first service finishes accessing the second data, the first pile of files is deleted.

9. The method according to claim 1, characterized in that: The deleting the first pile of files further includes: After modifying the physical address of the second portion of data from a first physical address corresponding to the first stack of files to a second physical address corresponding to the second stack of files, determining that no business is accessing the first stack of files; Delete the first pile of files.

10. An electronic device, characterized in that: The electronic device comprises a memory for storing instructions; At least one processor is used to execute the instructions so that the electronic device implements the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images