Waste database page recovery method and device, equipment and storage medium

Abstract

The invention discloses a waste database page recovery method and device, equipment and a storage medium, and relates to the technical field of database management, and the method comprises the steps of obtaining a waste page list by using waste database pages meeting a preset marking rule when a database page memory space is scanned; when a new database page memory space is needed, multiplexing the abandoned database page meeting a preset multiplexing condition; monitoring a state of a target data table corresponding to the abandoned database page and a current system resource, and if a monitoring result shows that the state of the target data table is not an access state and the current system resource meets a preset sufficient condition, copying a non-abandoned database page in the target data table based on the tree data structure; and if the monitoring result shows that the target data table state is not the access state and the current system resource does not meet the preset sufficient condition, covering the target abandoned database page based on a preset coverage rule. The utilization rate of database resources is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database management, and particularly relates to a method, device, equipment and storage medium for recycling abandoned database pages. Background Art

[0002] Currently, when a database is running, a lot of abandoned pages will be generated. Abandoned pages refer to those database pages that no longer contain valid data or are no longer needed. Abandoned pages are usually generated after multiple data addition, deletion and modification operations. When abandoned pages are generated, they cannot be immediately recycled by the database because abandoned pages are not continuous in space but are scattered. Recycling abandoned pages involves moving other normal pages. To ensure the normal recycling and prevent the loss of user data, the current table is locked during recycling, and the entire table is not allowed to be accessed, resulting in access blocking, which greatly reduces the read and write efficiency of the database and affects user experience.

[0003] As can be seen from the above, how to reduce the impact of abandoned page resource recycling on database performance and improve the utilization rate of database resources is an urgent problem to be solved at present. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for recycling abandoned database pages, which can reduce the impact of abandoned page resource recycling on database performance and improve the utilization rate of database resources. The specific solutions are as follows:

[0005] In the first aspect, the present application provides a method for recycling abandoned database pages, including:

[0006] Scanning the memory space of database pages using a preset thread task, and marking the database pages that meet the preset marking rules as abandoned database pages during the scanning process to obtain an abandoned page list using the abandoned database pages;

[0007] When it is detected that new database page memory space is needed in the database, reusing the abandoned database pages that meet the preset reuse conditions in the abandoned page list, and determining the abandoned database pages that have not been reused as target abandoned database pages;

[0008] Monitoring the status of the target data table corresponding to the target abandoned table database page in the database and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, copying the non-abandoned database pages in the target data table to the target database page memory space through a database page manager and based on a tree data structure, and deleting the target data table containing the target abandoned database pages;

[0009] If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, then the scanned target abandoned database page is overwritten through the database page manager and based on the preset overwrite rule to implement the recycling operation of the abandoned database page.

[0010] Optionally, scanning the database page memory space using a preset thread task, and marking the database pages that meet the preset marking rule as abandoned database pages during the scanning process to obtain an abandoned page list using the abandoned database pages, including:

[0011] Scanning the database page memory space using a preset thread task and based on a preset index order, and judging whether the current database page is invalid data based on the scanning result;

[0012] If the current database page is invalid data, then mark the current database page as an abandoned database page and save the abandoned database page to the abandoned page list.

[0013] Optionally, judging whether the current database page is invalid data based on the scanning result includes:

[0014] If the scanning result judges whether there is a transaction operation for the data table corresponding to the current database page;

[0015] If there is no transaction operation for the data table corresponding to the current database page, then judge whether the expiration fields of each data row corresponding to the current database page are all preset values;

[0016] If the expiration fields of each data row corresponding to the current database page are all preset values, it indicates that the current database page is invalid data;

[0017] If there is an expiration field that is not a preset value among the expiration fields of each data row corresponding to the current database page, it indicates that the current database page is not invalid data.

[0018] Optionally, when it is detected that new database page memory space is needed in the database, reusing the abandoned database pages that meet the preset reuse conditions in the abandoned page list, and determining the abandoned database pages that have not been reused as target abandoned database pages, including:

[0019] Obtaining the historical data of the abandoned database pages in the abandoned page list, and using an association rule mining algorithm to determine the association relationship between different data fields in the historical data;

[0020] Perform clustering analysis on the historical data based on the association relationship and a preset clustering algorithm, and during the clustering analysis process, use an autoencoder in deep learning technology to extract features and reduce the dimension of the historical data to obtain first feature information;

[0021] Obtain the load metrics corresponding to the current system, and perform multi-dimensional fusion based on the load metrics and the first feature information to obtain the fused information;

[0022] Detect the database through a database manager;

[0023] If the detection result indicates that the database requires new database page memory space, then use a preset similarity calculation method to determine the similarity result between the first feature information of the discarded database page and the second feature information corresponding to the data to be migrated;

[0024] If the similarity result meets the preset similarity condition, determine the discarded database page as a reusable data page, then migrate the data to be migrated to the reusable data page, and determine the discarded database pages that have not been reused in the discarded page list as target discarded database pages.

[0025] Optionally, before copying the non-discarded database pages in the target data table to the target database page memory space through a database page manager and based on a tree data structure, and deleting the target data table containing the target discarded database page, further include:

[0026] Monitor the current database using a database monitoring tool, and determine whether there are transaction operations in the target data table based on the monitoring result;

[0027] If there are no transaction operations in the target data table, copy the non-discarded database pages in the target data table to the target database page memory space through a database page manager and based on a tree data structure, and delete the target data table containing the target discarded database page.

[0028] Optionally, monitor the status of the target data table corresponding to the target discarded table database page in the database and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, then copy the non-discarded database pages in the target data table to the target database page memory space through a database page manager and based on a tree data structure, and delete the target data table containing the target discarded database page, including:

[0029] Determine the target data table corresponding to the target discarded database page in the database, and monitor the status of the target data table and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, add a table lock to the target data table;

[0030] Through the database page manager and based on the B+ tree structure and the preset replication order, copy the non-discarded database pages in the target data table to the pre-created target database page memory space, and determine whether the replication result indicates that the non-discarded database pages are successfully replicated;

[0031] If the replication result indicates that the non-discarded database pages are successfully replicated, update the node information in the target database page memory space, and determine whether the updated data is consistent with the data information in the non-discarded database pages in the target data table;

[0032] If the updated data is consistent with the data information in the non-discarded database pages in the target data table, delete the target data table containing the target discarded database page;

[0033] Wherein, the preset sufficient conditions include that the memory usage rate of the current system is lower than the first usage threshold and the CPU usage rate is lower than the second usage threshold.

[0034] Optionally, if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, cover the scanned target discarded database page through the database page manager and based on the preset covering rule to implement the recovery operation of the discarded database page, including:

[0035] If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, add a table lock to the target data table, and use the non-discarded database pages behind the scanned target discarded database page to cover the target discarded database page through the database page manager to implement the recovery operation of the discarded database page.

[0036] In a second aspect, the present application provides a discarded database page recovery device, including:

[0037] A database page marking module, configured to scan the database page memory space by using a preset thread task, and mark the database pages that meet the preset marking rules as discarded database pages during the scanning process, so as to obtain a discarded page list by using the discarded database pages;

[0038] A database page reuse module, which is used to reuse the discarded database pages that meet the preset reuse conditions in the discarded page list when it is detected that new database page memory space is required in the database, and determine the unused discarded database pages as target discarded database pages;

[0039] A database page copying module, which is used to monitor the status of the target data table corresponding to the target discarded table database page in the database and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, copy the non-discarded database pages in the target data table to the target database page memory space through the database page manager and based on the tree data structure, and delete the target data table containing the target discarded database pages;

[0040] A database page overwriting module, which is used to overwrite the scanned target discarded database pages through the database page manager and based on a preset overwriting rule if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, so as to implement the recycling operation of the discarded database pages.

[0041] In a third aspect, the present application provides an electronic device, including:

[0042] A memory, which is used to store a computer program;

[0043] A processor, which is used to execute the computer program to implement the foregoing method for recycling discarded database pages.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store a computer program, wherein the computer program, when executed by a processor, implements the foregoing method for recycling discarded database pages.

[0045] This application scans the memory space of database pages using a preset thread task, and marks the database pages that meet the preset marking rules as discarded database pages during the scanning process, so as to obtain a discarded page list using the discarded database pages; when it is detected that new database page memory space is needed in the database, the discarded database pages that meet the preset reuse conditions in the discarded page list are reused, and the discarded database pages that are not reused are determined as target discarded database pages; the status of the target data table corresponding to the target discarded table database page in the database and the current system resources are monitored. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, the non-discarded database pages in the target data table are copied to the target database page memory space through the database page manager and based on the tree data structure, and the target data table containing the target discarded database page is deleted; if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, the scanned target discarded database pages are overwritten through the database page manager and based on the preset overwrite rule to implement the recycling operation of the discarded database pages.

[0046] As can be seen from the above, this application marks the discarded database pages in the database using a preset marking rule and constructs a discarded page list. When it is detected that new database page memory space is needed in the database, it is judged whether there are discarded database pages that meet the preset reuse conditions in the discarded page list. If so, the discarded database pages in the discarded page list are reused, and the discarded database pages that are not reused in the discarded page list are determined as target discarded database pages; then the status of the target data table corresponding to the target discarded table database page in the database and the current system resources are monitored. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, the non-discarded database pages in the target data table corresponding to the target discarded database page are copied to the target database page memory space based on the tree data structure, and the target data table is deleted; if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, the target discarded database pages are overwritten based on the preset overwrite rule. In this way, not only the reuse of discarded database pages is realized, but also the discarded database pages that are not reused are cleaned up, thereby improving the utilization rate of database resources. Description of the Drawings

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0048] Figure 1 It is a flowchart of a method for recycling abandoned database pages disclosed in this application;

[0049] Figure 2 It is a schematic diagram of the arrangement of database page space provided by this application;

[0050] Figure 3 It is a specific schematic diagram of recycling abandoned database pages provided by this application;

[0051] Figure 4 It is a schematic diagram of the structure of a device for recycling abandoned database pages disclosed in this application;

[0052] Figure 5 It is a structural diagram of an electronic device disclosed in this application. Specific embodiments

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0054] Currently, when a database runs, abandoned pages will be generated, that is, database pages that no longer contain valid data or are no longer needed, and they are scattered after multiple insert, delete, and update operations. Recycling abandoned pages involves moving normal pages. To ensure the normal recycling and the non-loss of user data, the current table will be locked during recycling, and the entire table is not allowed to be accessed and published, resulting in access blocking, which will reduce the read and write efficiency of the database and affect the user experience. For this reason, this application provides a method for recycling abandoned database pages, which not only realizes the reuse of abandoned database pages, but also cleans up the abandoned database pages that have not been reused, thereby improving the utilization rate of database resources.

[0055] See Figure 1 As shown, the embodiments of the present invention disclose a method for recycling abandoned database pages, including:

[0056] Step S11: Scan the database page memory space using a preset thread task, and mark the database pages that meet the preset marking rules as discarded database pages during the scanning process, so as to obtain a discarded page list using the discarded database pages.

[0057] In this embodiment, when a deletion operation is performed on the client side, a large number of discarded database pages will be generated. The preset thread task of the database kernel scans the database page memory space according to a preset time period and index order to determine whether the current database page is invalid data during the scanning process, and marks the database pages that meet the preset marking rules as discarded database pages based on the judgment result, so as to form a discarded page list through the discarded database pages. Specifically, the step of scanning the database page memory space using a preset thread task, and marking the database pages that meet the preset marking rules as discarded database pages during the scanning process, so as to obtain a discarded page list using the discarded database pages, includes: scanning the database page memory space using a preset thread task and based on a preset index order, and judging whether the current database page is invalid data based on the scanning result; if the current database page is invalid data, then mark the current database page as a discarded database page, and save the discarded database page to the discarded page list. Among them, the preset time period can be adjusted according to the actual situation and will not be specifically limited here.

[0058] It can be understood that in database operations, the deletion of data is not equivalent to data invalidation. Because the deletion operation in the database is usually a logical deletion rather than a physical deletion. This means that the data actually still exists in the database, but is marked in a certain way to indicate that it has been deleted. Therefore, it cannot be simply determined that the data is invalid based on the fact that all the data on the current page has been deleted, but rather it is necessary to consider whether the data is truly invalid. Therefore, after obtaining the scanning result, it is determined whether there is a transaction operation on the data table corresponding to the current database page based on the scanning result; if there is no transaction operation on the data table corresponding to the current database page, and each expiration field of each data row corresponding to the current database page is a non-zero value, it indicates that all the data in the current database page is invalid data; if there is no transaction operation on the data table corresponding to the current database page, and there is an expiration field that is not the preset value among the expiration fields of each data row corresponding to the current database page, it indicates that not all the data in the current data page is invalid data. Specifically, determining whether the current database page is invalid data based on the scanning result includes: determining whether there is a transaction operation on the data table corresponding to the current database page according to the scanning result; if there is no transaction operation on the data table corresponding to the current database page, then determining whether each expiration field of each data row corresponding to the current database page is the preset value; if each expiration field of each data row corresponding to the current database page is the preset value, it indicates that the current database page is invalid data; if there is an expiration field that is not the preset value among the expiration fields of each data row corresponding to the current database page, it indicates that the current database page is not invalid data.

[0059] In a specific embodiment, when there is no transaction operation on the data table corresponding to the current database page, it is determined whether each expiration field expired_xid of each data row corresponding to the current database page is a non-zero value; if so, it indicates that all the data rows of the current database page have been deleted, that is, all the data in the current database page is invalid data, further indicating that the current database page meets the preset marking rule, and the current database page is marked as a discarded database page. Among them, the expiration field expired_xid defaults to zero, that is, access is allowed. When expired_xid is not zero, it indicates that the current row of data has been deleted by a certain transaction. If the current row of data has been deleted by a certain transaction and the transaction has not been committed, other transactions can still access the data of the current row. In addition, the transaction may also roll back and reset the expired_xid of this row to zero again.

[0060] Step S12, when it is detected that new database page memory space is needed in the database, the discarded database pages that meet the preset reuse conditions in the discarded page list are reused, and the discarded database pages that are not reused are determined as target discarded database pages.

[0061] In this embodiment, historical data of the discarded database pages in the discarded page list is obtained; the historical data includes the data content, access frequency, and update information of the discarded database pages; then, an association rule mining algorithm is used to analyze the association relationships between different data fields in the historical data, and a preset clustering algorithm is used to perform clustering analysis on the historical data according to the association relationships. During the clustering analysis process, an autoencoder is used to extract the features of the historical data and perform dimensionality reduction to obtain first feature information; then, the load metrics of the current system are obtained; the load metrics include the CPU (Central Processing Unit) usage rate and the memory occupancy rate; the load metrics and the first feature information are fused in multiple dimensions to obtain fused information, and a database manager is used to detect the status of the database; if the detection result indicates that the database requires new memory space, a cosine similarity calculation method or an Euclidean distance calculation method is used to determine the similarity result between the first feature information of the discarded database page and the second feature information corresponding to the data to be migrated; if the similarity result meets the preset similarity threshold, the discarded database page is determined as a reusable data page, and the data to be migrated is migrated to the reusable data page, and the discarded database pages that have not been reused in the discarded page list are determined as target discarded database pages. Among them, the association mining algorithm can be Apriori (i.e., the association rule mining algorithm) or FP-Growth (Frequent Pattern Growth); the preset clustering algorithm can be K-means (K-means algorithm), DBSCAN (Density-Based Spatial Clustering of Applications with Noise, i.e., the density-based clustering algorithm). It can be understood that when it is detected that the database requires new database page memory space and there are no discarded databases in the discarded page list that meet the preset reuse conditions, a new database page is created through the database page manager. It is worth mentioning that the preset similarity threshold can be adjusted accordingly according to the actual situation and is not specifically limited here.

[0062] Specifically, when it is detected that new database page memory space is required in the database, the discarded database pages in the discarded page list that meet the preset reuse conditions are reused, and the discarded database pages that are not reused are determined as target discarded database pages, including: obtaining the historical data of the discarded database pages in the discarded page list, and using an association rule mining algorithm to determine the association relationship between different data fields in the historical data; performing cluster analysis on the historical data based on the association relationship and a preset clustering algorithm, and in the process of performing cluster analysis, using an autoencoder in deep learning technology to extract features and reduce the dimension of the historical data to obtain first feature information; obtaining the load metrics corresponding to the current system, and performing multi-dimensional fusion based on the load metrics and the first feature information to obtain fused information; detecting the database through a database manager; if the detection result indicates that the database requires new database page memory space, then using a preset similarity calculation method to determine the similarity result between the first feature information of the discarded database page and the second feature information corresponding to the data to be migrated; if the similarity result meets the preset similarity condition, then determining the discarded database page as a reusable data page, then migrating the data to be migrated to the reusable data page, and determining the discarded database pages in the discarded page list that are not reused as target discarded database pages.

[0063] Step S13: Monitor the status of the target data table corresponding to the target discarded table database page in the database and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, then copy the non-discarded database pages in the target data table to the target database page memory space through a database page manager based on a tree data structure, and delete the target data table containing the target discarded database page.

[0064] In this embodiment, before performing the database page copying operation, it is necessary to first monitor the current database using a database monitoring tool to determine whether there are transaction operations in the target data table based on the monitoring results. If there are no transaction operations in the target data table, the non-abandoned database pages in the target data table are copied to a new target database page memory space through the database page manager and based on the B+ tree structure, and then the target data table containing the target abandoned database pages is deleted; wherein, the database monitoring tool can be a monitoring tool built into the database or other tools with monitoring functions. Specifically, before copying the non-abandoned database pages in the target data table to the target database page memory space through the database page manager and based on the tree data structure and deleting the target data table containing the target abandoned database pages, it also includes: using the database monitoring tool to monitor the current database and determining whether there are transaction operations in the target data table based on the monitoring results; if there are no transaction operations in the target data table, copying the non-abandoned database pages in the target data table to the target database page memory space through the database page manager and based on the tree data structure, and deleting the target data table containing the target abandoned database pages.

[0065] It can be understood that Figure 2A schematic diagram of the arrangement of database page space provided by this embodiment. In the B+ tree structure, the bottom layer is the leaf node, which mainly stores data and also records the location information of the parent node (i.e., the internal node). Above the leaf node is the internal node, which does not store data but stores the location information of the child nodes. The top layer is the root node, which is used to store the metadata information of the database. When there is only one node in the entire tree structure, the root node may be either a leaf node or an internal node. When there are multiple nodes in the entire tree structure, the root node is an internal node. Before performing database page recycling, it is necessary to first determine the target data table corresponding to the target database page in the data, and then judge the status of the target data table and the current system resource situation. If the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, a table lock is added to the target data table to stop the user end from performing insert, delete, and update operations on the data table. Then, through the database page manager and using the B+ data structure, the non-discarded database pages in the target data table are sequentially copied to the pre-created target database page memory space. If the copy operation is successful, the index number of the target database page memory space is updated, and it is judged whether the updated data is consistent with the data of the non-discarded database pages in the target data table. If the updated data is consistent with the data of the non-discarded database pages in the target data table, the target data table containing the target discarded database page is deleted. In a specific implementation manner, when the memory utilization rate is lower than 30%, the CPU utilization rate is lower than 20%, and the disk I / O is at a low level, it is considered that the current system resources meet the preset sufficient conditions. The preset sufficient conditions can be adjusted according to the actual situation.

[0066] Specifically, monitor the status of the target data table corresponding to the target discarded database page in the database and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, then copy the non-discarded database pages in the target data table to the target database page memory space through the database page manager and based on the tree data structure, and delete the target data table containing the target discarded database page, including: determining the target data table corresponding to the target discarded database page in the database, and monitoring the status of the target data table and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, then add a table lock to the target data table; copy the non-discarded database pages in the target data table to the pre-created target database page memory space through the database page manager and based on the B+ tree structure and the preset copy order, and determine whether the copy result indicates that the non-discarded database pages are copied successfully; if the copy result indicates that the non-discarded database pages are copied successfully, then update the node information in the target database page memory space, and determine whether the updated data is consistent with the data information in the non-discarded database pages in the target data table; if the updated data is consistent with the data information in the non-discarded database pages in the target data table, then delete the target data table containing the target discarded database page; wherein, the preset sufficient conditions include that the memory usage rate of the current system is lower than the first usage threshold and the CPU usage rate is lower than the second usage threshold.

[0067] Step S14, if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, then cover the scanned target discarded database page through the database page manager and based on the preset coverage rule to implement the recovery operation of the discarded database page.

[0068] In this embodiment, if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, a table lock is added to the target data table, and then the target data table is scanned in sequence according to the index order through the database page manager. If a target obsolete database page is scanned, the non-obsolete database pages behind the target obsolete database page are moved to cover the target obsolete database page, and compression is performed based on the remaining free pages to implement the recycling operation of the obsolete database page. Specifically, if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, the target obsolete database page scanned is covered through the database page manager based on a preset covering rule to implement the recycling operation of the obsolete database page, including: if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, a table lock is added to the target data table, and the non-obsolete database pages behind the target obsolete database page scanned are used by the database page manager to cover the target obsolete database page to implement the recycling operation of the obsolete database page.

[0069] In a specific implementation manner, Figure 3 FIG. 5 is a schematic diagram of recycling of a specific obsolete database page provided in this embodiment. The target data table is composed of 7 database pages with a size of 8 KB. Among them, the page with index 0 is the root node page, which will not be marked as an obsolete database page and will not be deleted. Indexes 1, 4, and 6 are obsolete database pages (Obsolete Page), and indexes 2, 3, and 5 are non-obsolete database pages, that is, normal pages (Normal Page). When the database page manager needs to allocate new page space, it preferentially searches among the obsolete database pages to implement the reuse of the obsolete database pages. Among them, the reused obsolete database pages will have the obsolete mark removed and will be allowed to be accessed again. If there are no reusable obsolete database pages in the obsolete page list, a new database page is opened. If the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, a table lock is added to the target data table, the non-obsolete database pages behind the target obsolete database page scanned are used by the database page manager to cover the target obsolete database page, and the target obsolete database page is cleaned by a preset cleaning thread, so as to implement the compression of the space resources and release the remaining space resources.

[0070] As can be seen from the above, the present application uses a preset marking rule to mark the discarded database pages in the database and constructs a discarded page list. When it is detected that new database page memory space is needed in the database, it is determined whether there are discarded database pages in the discarded page list that meet the preset reuse conditions. If so, the discarded database pages in the discarded page list are reused, and the discarded database pages that are not reused in the discarded page list are determined as target discarded database pages; then, the status of the target data table corresponding to the target discarded table database page in the database and the current system resources are monitored. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, the non-discarded database pages in the target data table are copied to the target database page memory space based on the tree data structure through the database page manager, and the target data table containing the target discarded database page is deleted; if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, the target discarded database page is overwritten based on the preset overwrite rule. In this way, not only the reuse of discarded database pages is realized, but also the discarded database pages that are not reused are cleaned up, thereby improving the utilization rate of database resources.

[0071] Correspondingly, referring to Figure 4 as shown, the present application also provides a discarded database page recycling device, including:

[0072] A database page marking module 11, configured to scan the database page memory space using a preset thread task, and mark the database pages that meet the preset marking rule as discarded database pages during the scanning process, so as to obtain a discarded page list using the discarded database pages;

[0073] A database page reuse module 12, configured to reuse the discarded database pages that meet the preset reuse conditions in the discarded page list when it is detected that new database page memory space is needed in the database, and determine the discarded database pages that are not reused as target discarded database pages;

[0074] A database page copying module 13, configured to monitor the status of the target data table corresponding to the target discarded table database page in the database and the current system resources. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, the non-discarded database pages in the target data table are copied to the target database page memory space based on the tree data structure through the database page manager, and the target data table containing the target discarded database page is deleted;

[0075] A database page overwrite module 14, which is used to, if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, overwrite the scanned target discarded database pages through the database page manager and based on a preset overwrite rule, so as to implement the recycling operation of the discarded database pages.

[0076] As can be seen from the above, the present application uses a preset marking rule to mark the discarded database pages in the database and constructs a discarded page list. When it is detected that new database page memory space is needed in the database, it is judged whether there are discarded database pages in the discarded page list that meet the preset reuse conditions. If so, the discarded database pages in the discarded page list are reused, and the discarded database pages that are not reused in the discarded page list are determined as target discarded database pages; then, the status of the target data table corresponding to the target discarded database page in the database and the current system resources are monitored. If the monitoring result indicates that the status of the target data table is not in the access state and the current system resources meet the preset sufficient conditions, the non-discarded database pages in the target data table corresponding to the target discarded database page are copied to the target database page memory space based on the tree data structure, and the target data table is deleted; if the monitoring result indicates that the status of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, the target discarded database pages are overwritten based on the preset overwrite rule. In this way, not only the reuse of the discarded database pages is realized, but also the discarded database pages that are not reused are cleaned up, thereby improving the utilization rate of database resources.

[0077] In some specific embodiments, the database page marking module 11 may specifically include:

[0078] A memory space scanning unit, which is used to scan the database page memory space by using a preset thread task and based on a preset index order, and judge whether the current database page is invalid data based on the scanning result;

[0079] A database page marking completion unit, which is used to, if the current database page is invalid data, mark the current database page as a discarded database page and save the discarded database page to the discarded page list.

[0080] In some specific embodiments, the database page marking module 11 may specifically include:

[0081] A data table judgment unit, which is used to judge whether there is a transaction operation on the data table corresponding to the current database page if the scanning result judges;

[0082] An expired field judgment unit, configured to judge whether the expired fields of each data row corresponding to the current database page are all preset values if there is no transaction operation for the data table corresponding to the current database page;

[0083] An invalid data determination unit, configured to characterize the current database page as invalid data if the expired fields of each data row corresponding to the current database page are all preset values;

[0084] A non-invalid data determination unit, configured to characterize that the current database page is not invalid data if there is an expired field that is not a preset value among the expired fields of each data row corresponding to the current database page.

[0085] In some specific embodiments, the database page reuse module 12 may specifically include:

[0086] An association relationship determination unit, configured to obtain the historical data of the discarded database page in the discarded page list, and use an association rule mining algorithm to determine the association relationship between different data fields in the historical data;

[0087] A data clustering analysis unit, configured to perform clustering analysis on the historical data based on the association relationship and a preset clustering algorithm, and in the process of performing clustering analysis, use an autoencoder in deep learning technology to perform feature extraction and data dimensionality reduction on the historical data to obtain first feature information;

[0088] An information fusion unit, configured to obtain the load metrics corresponding to the current system, and perform multi-dimensional fusion based on the load metrics and the first feature information to obtain fused information;

[0089] A database detection unit, configured to detect the database through a database manager;

[0090] A similarity determination unit, configured to use a preset similarity calculation method to determine the similarity result between the first feature information of the discarded database page and the second feature information corresponding to the data to be migrated if the detection result indicates that the database requires new database page memory space;

[0091] A reusable data page determination unit, configured to determine the discarded database page as a reusable data page if the similarity result meets a preset similarity condition, then migrate the data to be migrated to the reusable data page, and determine the discarded database pages that have not been reused in the discarded page list as target discarded database pages.

[0092] In some specific embodiments, the discarded database page recycling device may specifically further include:

[0093] A database monitoring unit, which is used to monitor the current database by using a database monitoring tool, and judge whether there is a transaction operation in the target data table based on the monitoring result;

[0094] A non-discarded database page copying unit, which is used to copy the non-discarded database pages in the target data table to the target database page memory space through a database page manager and based on a tree data structure if there is no transaction operation in the target data table, and delete the target data table containing the target discarded database page.

[0095] In some specific embodiments, the database page copying module 13 may specifically include:

[0096] A first monitoring result determining unit, which is used to determine the target data table corresponding to the target discarded database page in the database, monitor the state of the target data table and the current system resources, and add a table lock to the target data table if the monitoring result indicates that the state of the target data table is not in the access state and the current system resources meet the preset sufficient conditions;

[0097] A non-discarded database page judging unit, which is used to copy the non-discarded database pages in the target data table to a pre-created target database page memory space through the database page manager and based on a B+ tree structure and a preset copying order, and judge whether the copying result indicates that the non-discarded database page is successfully copied;

[0098] A node information updating unit, which is used to update the node information in the target database page memory space if the copying result indicates that the non-discarded database page is successfully copied, and judge whether the updated data is consistent with the data information in the non-discarded database pages in the target data table;

[0099] A target data table deleting unit, which is used to delete the target data table containing the target discarded database page if the updated data is consistent with the data information in the non-discarded database pages in the target data table.

[0100] In some specific embodiments, the database page covering module 14 may specifically include:

[0101] A second monitoring result determining unit, which is used to add a table lock to the target data table if the monitoring result indicates that the state of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions, and cover the target discarded database page with the non-discarded database pages behind the scanned target discarded database page through the database page manager to implement the recovery operation of the discarded database page.

[0102] Further, an embodiment of the present application also discloses an electronic device. Figure 5 FIG. Figure 5 is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be regarded as any limitation on the scope of use of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the abandoned database page recovery method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0103] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0104] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0105] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the abandoned database page recovery method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0106] Further, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the abandoned database page recovery method disclosed above is implemented. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

[0107] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0108] Those skilled in the art can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0109] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0110] Finally, it should also be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0111] The technical solutions provided in this application have been introduced in detail above. Specific examples are used herein to elaborate on the principles and implementation manners of this application. The descriptions of the above embodiments are only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for recycling discarded database pages, characterized in that: include: Scanning the database page memory space by using a preset thread task, and marking the database pages satisfying the preset marking rule as discarded database pages during the scanning process, so as to obtain a discarded page list by using the discarded database pages; When it is detected that new database page memory space is needed in the database, the discarded database pages in the discarded page list that meet the preset reuse condition are reused, and the discarded database pages that are not reused are determined as target discarded database pages; The state of the target data table corresponding to the target abandoned table database page in the database and the current system resources are monitored, and if the monitoring result indicates that the state of the target data table is not in an access state and the current system resources meet the preset sufficient condition, the non-abandoned database pages in the target data table are copied to the target database page memory space through the database page manager and based on the tree data structure, and the target data table containing the target abandoned database page is deleted; If the monitoring result indicates that the target data table is not in an access state and the current system resources do not meet the preset sufficient condition, the scanned target abandoned database page is overwritten by the database page manager based on the preset overwriting rule to implement the recycling operation of the abandoned database page.

2. The method for recycling discarded database pages according to claim 1, characterized in that: The method of scanning the database page memory space by using a preset thread task and marking the database pages satisfying a preset marking rule as discarded database pages during the scanning process, so as to obtain a discarded page list by using the discarded database pages, includes: Scanning the database page memory space using a preset thread task and based on a preset index order, and determining whether the current database page is invalid data based on the scanning result; If the current database page is invalid data, the current database page is marked as a discarded database page, and the discarded database page is saved in the discarded page list.

3. The method for recycling discarded database pages according to claim 2, characterized in that: The step of judging whether the current database page is invalid data based on the scanning result includes: If the scanning result determines whether there is a transaction operation in the data table corresponding to the current database page; If there is no transaction operation in the data table corresponding to the current database page, determining whether the expiration fields of each data row corresponding to the current database page are all preset values; If all the expiration fields of the data rows corresponding to the current database page are preset values, it indicates that the current database page is invalid data; If there is an expiration field whose value is not a preset value among the expiration fields of the data rows corresponding to the current database page, it indicates that the current database page is not invalid data.

4. The method for recycling discarded database pages according to claim 1, characterized in that: When it is detected that a new database page memory space is needed in the database, the discarded database pages in the discarded page list that meet the preset reuse condition are reused, and the discarded database pages that are not reused are determined as target discarded database pages, including: Acquire the historical data of the discarded database pages in the discarded page list, and determine the association relationship between different data fields in the historical data by using an association rule mining algorithm; Performing cluster analysis on the historical data based on the association relationship and a preset clustering algorithm, and in the process of performing cluster analysis, using an autoencoder in deep learning technology to perform feature extraction and data dimension reduction on the historical data to obtain first feature information; Obtaining a load indicator corresponding to the current system, and performing multi-dimensional fusion based on the load indicator and the first feature information to obtain fused information; Detect the database through the database manager; If the detection result indicates that the database needs new database page memory space, a preset similarity calculation method is used to determine a similarity result between the first characteristic information of the discarded database page and the second characteristic information corresponding to the data to be migrated; If the similarity result meets the preset similarity condition, the discarded database page is determined as a reusable data page, and then the data to be migrated is migrated to the reusable data page, and the discarded database page that is not reused in the discarded page list is determined as a target discarded database page.

5. The method for recycling discarded database pages according to claim 1, characterized in that: Before the non-discarded database pages in the target data table are copied to the target database page memory space through the database page manager based on the tree data structure and the target data table containing the target discarded database pages is deleted, the method further includes: Using a database monitoring tool to monitor the current database, and judging whether there is a transaction operation in the target data table based on the monitoring result; If there is no transaction operation in the target data table, the non-discarded database pages in the target data table are copied to the target database page memory space through the database page manager based on the tree data structure, and the target data table containing the target discarded database pages is deleted.

6. The method for recycling discarded database pages according to any one of claims 1 to 5, characterized in that: The state of the target data table corresponding to the target abandoned table database page in the database and the current system resources are monitored, and if the monitoring result indicates that the state of the target data table is not in an access state and the current system resources meet the preset sufficient condition, the non-abandoned database page in the target data table is copied to the target database page memory space through the database page manager and based on the tree data structure, and the target data table containing the target abandoned database page is deleted, including: Determine a target data table in the database corresponding to the target abandoned database page, and monitor the state of the target data table and current system resources, and if the monitoring result indicates that the state of the target data table is not in an access state and the current system resources meet preset sufficient conditions, add a table lock to the target data table; Copying the non-discarded database pages in the target data table to a pre-created target database page memory space through the database page manager based on the B+ tree structure and a preset copy order, and determining whether the copy result indicates that the non-discarded database pages are successfully copied; If the copy result indicates that the non-discarded database page is successfully copied, the information of each node in the target database page memory space is updated, and it is determined whether the updated data is consistent with the data information in the non-discarded database page in the target data table; If the updated data is consistent with the data information in the non-abandoned database page in the target data table, the target data table including the target abandoned database page is deleted; The preset sufficient condition includes that the memory usage of the current system is lower than a first usage threshold and the CPU usage is lower than a second usage threshold.

7. The method for recycling discarded database pages according to claim 6, characterized in that: If the monitoring result indicates that the state of the target data table is not in an access state and the current system resources do not meet the preset sufficient condition, the scanned target abandoned database page is overwritten by the database page manager based on a preset overwriting rule to implement the recycling operation of the abandoned database page, including: If the monitoring result indicates that the state of the target data table is not in an access state and the current system resources do not meet the preset sufficient conditions, a table lock is added to the target data table, and the non-discarded database page behind the scanned target discarded database page is used by the database page manager to overwrite the target discarded database page to implement the recycling operation of the discarded database page.

8. A discarded database page recovery device, characterized in that: include: A database page marking module is used to scan the database page memory space using a preset thread task, and mark the database pages that meet the preset marking rules as discarded database pages during the scanning process, so as to obtain a discarded page list using the discarded database pages; A database page reuse module, for reusing the discarded database pages in the discarded page list that meet a preset reuse condition when it is detected that a new database page memory space is needed in the database, and determining the discarded database pages that have not been reused as target discarded database pages; A database page copy module is used to monitor the state of a target data table corresponding to the target abandoned table database page in the database and current system resources, and if the monitoring result indicates that the state of the target data table is not in an access state and the current system resources meet a preset sufficient condition, then the non-abandoned database page in the target data table is copied to the target database page memory space through the database page manager and based on the tree data structure, and the target data table containing the target abandoned database page is deleted; The database page overwriting module is used to overwrite the scanned target abandoned database page through the database page manager based on the preset overwriting rules to realize the recycling operation of the abandoned database page if the monitoring result indicates that the state of the target data table is not in the access state and the current system resources do not meet the preset sufficient conditions.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the method for recycling discarded database pages according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method for recycling discarded database pages according to any one of claims 1 to 7 is implemented.

Images