Database data block recovery method and device, equipment and medium

Abstract

The invention provides a data block recovery method and device of a database, equipment and a medium, and can be applied to the technical field of computers. The data block recovery method for the database comprises the steps that the log generation speed of the database within a first preset duration and the available space proportion of log storage space are monitored, and the available space proportion is the ratio of the number of data blocks without storing logic logs to the total number of N data blocks; optimizing a preset occupancy rate threshold value by utilizing the log generation speed to obtain an optimized occupancy rate threshold value; under the condition that the available space proportion is smaller than or equal to an optimized occupancy rate threshold value, obtaining the transaction meeting a preset recovery condition in the transactions of the target type to obtain a target transaction; determining a target associated data block associated with the logic log of the target transaction; and storing the rollback log of the target transaction in the backup storage space, and releasing the association relationship between the target associated data block and the target transaction so as to recover the target associated data block.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a database data block recycling method, device, equipment and medium. BACKGROUND

[0002] In the process of database system running, logical log plays a core role in maintaining data consistency and reliability, which provides a key basis for transaction rollback and data recovery after system crash by recording the insertion, update and deletion operations of transactions on data.

[0003] In related technologies, logical log usually adopts a circular writing mechanism, and log files are divided into fixed-size blocks, and transaction operations are continuously written in order, and when the last block is full, the system will start from the first block and overwrite. However, when there is a long transaction that has not been committed, since the generated log may span multiple log files, the system needs to continuously retain these data blocks until the transaction is completed, resulting in a large amount of space that can be recycled being occupied for a long time, and the data blocks cannot be reused in time, causing idle waste of space resources. SUMMARY

[0004] In view of the above problems, the present application provides a database data block recycling method, device, equipment and medium.

[0005] According to a first aspect of the present application, a database data block recycling method is provided, the database comprising a log storage space, the log storage space comprising N data blocks, the data blocks being used to store logical logs generated in the transaction processing process, N being an integer greater than 1, comprising: monitoring the log generation speed of the database within a first preset time length and the available space ratio of the log storage space, the available space ratio being the ratio of the number of data blocks not storing logical logs to the total number of N data blocks; optimizing the preset occupancy rate threshold value using the log generation speed to obtain an optimized occupancy rate threshold value; in the case that the available space ratio is less than or equal to the optimized occupancy rate threshold value, obtaining transactions of a target type that meet a preset recycling condition to obtain target transactions; determining target associated data blocks associated with the logical logs of the target transactions; storing the rollback logs of the target transactions to a backup storage space and removing the association between the target associated data blocks and the target transactions, so as to recycle the target associated data blocks.

[0006] According to an embodiment of the present invention, the above-mentioned N data blocks are linked end-to-end; before obtaining the target transaction, when the available space ratio is less than or equal to the optimized occupancy threshold, the method further includes: when the log storage space is less than or equal to the preset storage space, repeatedly performing the following operations: determining the predicted available space ratio within a second preset time period based on the log generation speed, the available space ratio, and the log storage space, wherein the predicted available space ratio represents the ratio of the number of the above-mentioned data blocks without stored logical logs to the total number of the above-mentioned N data blocks within the future second preset time period; when the predicted available space ratio is less than or equal to the preset predicted occupancy threshold, determining at least one extended data block based on the log storage space and the available space ratio; linking the at least one extended data block to the last data block in the log storage space, and marking the status of the at least one extended data block as idle; and determining extended storage space based on the at least one extended data block.

[0007] According to an embodiment of the present invention, the above-mentioned optimization of the preset occupancy threshold using the above-mentioned log generation speed to obtain the optimized occupancy threshold includes: determining a target optimization correction amount based on the product of the above-mentioned log generation speed and the preset optimization parameters; and correcting the above-mentioned preset occupancy threshold using the above-mentioned target optimization correction amount to obtain the above-mentioned optimized occupancy threshold.

[0008] According to an embodiment of the present invention, the method further includes: determining a recyclable percentage based on the state of each data block among the N data blocks, wherein the recyclable percentage is the ratio of the storage space corresponding to the data block in the pending recyclable state to the log storage space; if the recyclable percentage is greater than or equal to a preset threshold, determining the data block in the pending recyclable state as the target data block to be flushed to disk; and in response to the successful flushing of the target data block to disk, marking the state of the target data block to disk as idle.

[0009] According to an embodiment of the present invention, the method further includes: in response to the fact that the available space ratio is greater than or equal to a preset shutdown threshold, determining the data block in the idle state as the target data block to be written; storing the rollback log of the target transaction in the target data block to be written, and removing the association between the backup storage space and the target transaction.

[0010] According to an embodiment of the present invention, the method further includes: monitoring the space usage ratio of the extended storage space, wherein the space usage ratio is the ratio of the storage space corresponding to the extended data block of the stored logical log to the extended storage space; and suspending transaction processing when the space usage ratio is equal to a preset warning threshold.

[0011] According to an embodiment of the present invention, the above-mentioned determination of the predicted available space ratio within a second preset time period based on the log generation speed, the available space ratio, and the log storage space includes: determining the predicted occupied space based on the product of the log generation speed and the second preset time period; determining the predicted occupied space ratio based on the quotient of the predicted occupied space and the log storage space; and subtracting the available space ratio using the predicted occupied space ratio to obtain the predicted available space ratio.

[0012] A second aspect of the present invention provides a data block reclamation apparatus for a database, wherein the database includes a log storage space, the log storage space includes N data blocks, the data blocks are used to store logical logs generated during transaction processing, and N is an integer greater than 1, comprising: a monitoring module, configured to monitor the log generation speed of the database within a first preset time period, and the available space ratio of the log storage space, the available space ratio being the ratio of the number of data blocks without stored logical logs to the total number of the N data blocks; an optimization module, configured to optimize a preset occupancy rate threshold using the log generation speed, to obtain an optimized occupancy rate threshold; an acquisition module, configured to acquire transactions of a target type that meet preset reclamation conditions when the available space ratio is less than or equal to the optimized occupancy rate threshold, to obtain a target transaction; a determination module, configured to determine a target associated data block associated with the logical log of the target transaction; and a storage module, configured to store the rollback log of the target transaction in a backup storage space and remove the association between the target associated data block and the target transaction, so as to facilitate the reclamation of the target associated data block.

[0013] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0015] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] According to an embodiment of the present invention, by monitoring the log generation speed and available space ratio of the database within a first preset time period, and using the log generation speed to optimize the preset occupancy threshold, an optimized occupancy threshold is dynamically generated. Then, when the available space ratio is less than or equal to the optimized occupancy threshold, the target associated data block corresponding to the target transaction of the target type is determined, and the rollback log of the target transaction is stored in the backup storage space. At the same time, the target associated data block corresponding to the target transaction is reclaimed. Thus, before the log storage space is exhausted, the data block occupied by the target transaction of the target type is accurately reclaimed. Under the premise of ensuring that the target transaction can be rolled back, the available space of the log storage space is guaranteed, and the database is prevented from waiting for transactions or failing to commit transactions due to insufficient log storage space, thereby improving the stability of the database service. Attached Figure Description

[0017] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0018] Figure 1 The diagram illustrates an application scenario of a database data block reclamation method and apparatus according to an embodiment of the present invention.

[0019] Figure 2 A flowchart of a database data block reclamation method according to an embodiment of the present invention is shown;

[0020] Figure 3 A structural block diagram of a database data block recycling device according to an embodiment of the present invention is shown;

[0021] Figure 4 A block diagram of an electronic device suitable for implementing a data block reclamation method for a database, according to an embodiment of the present invention, is shown. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] Current mainstream databases (especially those designed for high-frequency transaction processing) typically employ a circular write mechanism for their logical logs: the log file is divided into fixed-size blocks, transaction operations are written sequentially, and when the last block is full, the system starts overwriting from the first block. However, this mechanism reveals significant space utilization efficiency issues in practical applications.

[0027] On the one hand, the invalidation conditions of logical logs have obvious phased characteristics: for committed transactions, the corresponding logs only fulfill their purpose and are truly ready for reclamation after the data modified by the transaction has been completely flushed to disk. Related technologies trigger reclamation through fixed thresholds. When the log usage rate reaches a set threshold (e.g., 80%), a checkpoint mechanism is forcibly triggered, flushing the data modified by committed transactions in memory to disk, thus invalidating the corresponding logs and releasing space. This approach does not fully consider the actual processing progress of different transactions. When long-running transactions are not committed, their logs may span multiple log files, requiring the system to continuously retain these log blocks until the transaction is completed, resulting in a large amount of reclaimable space being occupied for a long time. Simultaneously, when the total log volume is small, logs that have met the invalidation conditions may occupy space for a long time because they have not reached the trigger threshold, forming an "invalid log residency" phenomenon. This prevents log blocks from being reused in a timely manner, causing idle and wasted space resources.

[0028] On the other hand, in high-frequency transaction scenarios, the log generation speed is much higher than that of traditional business. If the recycling mechanism is slow to respond, it will lead to a large accumulation of invalid log blocks. For example, the timing of triggering a checkpoint requires waiting for the log space utilization rate to reach 80%, but the remaining 20% ​​of available space may be quickly exhausted by high-frequency transactions in a short period of time. At this time, the checkpoint process has not yet completed the data flushing and log invalidation processing, which directly causes "instantaneous exhaustion of log space", forcing the ongoing transaction to be interrupted because it cannot write to the log, causing business processing delays or even failures.

[0029] For databases designed for high-frequency transaction processing, their deployment environments often face the challenge of limited storage resources (such as the limited local storage capacity of edge computing nodes). Traditional log management methods are not optimized for the transaction characteristics of this database (such as log relationships under multi-version concurrency control), causing log space to become a key bottleneck restricting system performance under the dual pressure of high-frequency transactions and resource constraints. Limited storage resources are occupied by a large number of invalid logs, which not only reduces resource utilization efficiency but also frequently triggers transaction waits due to insufficient available log space, significantly prolonging transaction processing time. In extreme cases, transaction commit may even fail due to log block exhaustion, seriously affecting the stability of database services.

[0030] Therefore, the core issue in improving the stability and resource utilization of databases in high-concurrency scenarios is how to combine the dynamic failure conditions of logical logs with the architectural characteristics of databases that are oriented towards high-frequency transaction processing, and design a management strategy that can accurately identify recyclable logs and adaptively adjust the timing and granularity of recycling, so as to maximize the utilization of logical log space and avoid service degradation or interruption due to log space shortage.

[0031] In view of this, embodiments of the present invention provide a database data block reclamation method. The database includes a log storage space, which includes N data blocks. The data blocks are used to store logical logs generated during transaction processing, where N is an integer greater than 1. The method includes: monitoring the log generation speed of the database within a first preset time period and the available space ratio of the log storage space, where the available space ratio is the ratio of the number of data blocks without stored logical logs to the total number of N data blocks; optimizing a preset occupancy threshold using the log generation speed to obtain an optimized occupancy threshold; when the available space ratio is less than or equal to the optimized occupancy threshold, obtaining transactions of the target type that meet preset reclamation conditions to obtain the target transaction; determining the target associated data block associated with the logical log of the target transaction; storing the rollback log of the target transaction in a backup storage space and removing the association between the target associated data block and the target transaction to facilitate the reclamation of the target associated data block.

[0032] Figure 1 The diagram illustrates an application scenario of a database data block recycling method and apparatus according to an embodiment of the present invention.

[0033] like Figure 1As shown, application scenario 100 according to this embodiment may include a first database 101, a second database 102, a third database 103, a network 104, and a server 105. Network 104 serves as a medium for providing communication links between the first database 101, the second database 102, the third database 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0034] The first database 101, the second database 102, and the third database 103 can interact with the server 105 via the network 104. The first database 101, the second database 102, and the third database 103 can run on any terminal or server.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports the website being browsed (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] It should be noted that the database data block reclamation method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the database data block reclamation device provided in this embodiment of the invention can generally be located in server 105. The database data block reclamation method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first database 101, the second database 102, the third database 103, and / or server 105. Correspondingly, the database data block reclamation device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first database 101, the second database 102, the third database 103, and / or server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] The following will be based on Figure 1 The described scene, through Figure 2 The data block reclamation method of the database according to an embodiment of the present invention will be described in detail.

[0039] Figure 2 A flowchart of a database data block reclamation method according to an embodiment of the present invention is shown.

[0040] like Figure 2As shown, the database data block reclamation method of this embodiment includes operations S210 to S250.

[0041] The database includes a log storage space, which consists of N data blocks. These data blocks are used to store logical logs generated during transaction processing, where N is an integer greater than 1.

[0042] During operation S210, monitor the log generation rate of the database within the first preset time period, as well as the percentage of available space in the log storage space.

[0043] The available space percentage is the ratio of the number of data blocks that do not store logical logs to the total number of N data blocks.

[0044] According to an embodiment of the present invention, the status of data blocks in the log storage space can be monitored by a monitoring thread within a first preset time period, thereby calculating the log generation speed and the available space ratio of the log storage space. The first preset time period can be, for example, 1 minute. The monitoring thread executes once every 1 second and counts the number of data blocks in each status by traversing the log storage space. The data block status includes states such as "pending recycling", "in use", "idle", and "redundant retention". The log generation speed represents the processing frequency of the transactions corresponding to the logical log.

[0045] The log generation speed can be calculated using the following formula (1).

[0046] (1);

[0047] in, Indicates the log generation speed. This indicates the number of data blocks that transitioned from an idle state to an in-use state. Indicates the size of the data block storage space. This indicates the first preset duration.

[0048] The percentage of available space can be calculated using the following formula (2).

[0049] (2);

[0050] in, This indicates the percentage of available space in the log storage area. Indicates the number of data blocks in an idle state. Indicates the number of data blocks that are recyclable. Indicates the size of the data block storage space. This indicates the size of the log storage space.

[0051] When operating S220, the preset occupancy threshold is optimized by utilizing the log generation speed to obtain the optimized occupancy threshold.

[0052] According to an embodiment of the present invention, the preset occupancy threshold can be dynamically optimized by the log generation speed. The preset occupancy threshold can be set to 60% of the log storage space. When the log generation speed is fast, the preset occupancy threshold can be lowered to meet the needs of high-frequency transactions.

[0053] In operation S230, when the available space ratio is less than or equal to the optimized occupancy threshold, the transaction that meets the preset reclamation conditions among the transactions of the target type is obtained, and the target transaction is obtained.

[0054] According to embodiments of the present invention, the aforementioned target type of transaction can be, for example, a long transaction, which refers to a transaction that has not been committed or rolled back since its inception, and whose lifespan has exceeded the expected business range. The aforementioned preset reclamation condition can be, for example, that the data block span stored in the logical log corresponding to the long transaction is greater than a preset threshold.

[0055] For example, if there are 10 data blocks in the log storage space, and long transaction A writes logical logs in both the 1st and 8th data blocks, then the corresponding data block span is 7. Assuming the preset threshold is 5, long transaction A can be considered to meet the preset reclamation conditions.

[0056] In operation S240, the target associated data block is determined to be associated with the logical log of the target transaction.

[0057] According to an embodiment of the present invention, a target associated data block can be determined by the identifier of the target transaction. For example, if the identifier of the target transaction is 1, then all data blocks in the log storage space can be scanned, and the data block in which the identifier "1" of the target transaction has appeared can be determined as the target associated data block.

[0058] In operation S250, the rollback log of the target transaction is stored in the backup storage space, and the association between the target associated data block and the target transaction is removed, so as to facilitate the reclamation of the target associated data block.

[0059] A rollback log is a log that records the original values ​​of data before modification in the order of transaction operations before a transaction performs any operation on the data. When a transaction needs to be rolled back, the corresponding data can be restored to its state before modification according to the order of transaction operations in the rollback log to satisfy the atomicity of the transaction.

[0060] According to an embodiment of the present invention, when the available space ratio is less than 5% and there are no idle data blocks in the log storage space, the backup storage space can be enabled. The size of the backup storage space is the product of the current log storage space capacity and the preset backup ratio (which can be set to 10%). After the backup storage space is enabled, it will be merged into the log storage space, and the status of the backup storage space will be marked as "redundant retention".

[0061] According to an embodiment of the present invention, before enabling the backup storage space, the log storage space can be expanded until it reaches the preset storage space size, and the backup storage space can be enabled only after it can not be expanded further.

[0062] According to embodiments of the present invention, a priority queue can be used to manage the writing of rollback logs corresponding to long transactions. The priority of the priority queue can be set to levels 1-10 (level 10 being the highest). Priority can be assigned according to the data block span of the logical log corresponding to the long transaction. Rollback logs corresponding to long transactions with a priority greater than or equal to 6 can be written first, and transaction pause records can be stored in the system log. For example, long transactions with a data block span of 1 can be set to level 1, long transactions with a data block span of 10 can be set to level 10, and so on.

[0063] According to an embodiment of the present invention, by monitoring the log generation speed and available space ratio of the database within a first preset time period, and using the log generation speed to optimize the preset occupancy threshold, an optimized occupancy threshold is dynamically generated. Then, when the available space ratio is less than or equal to the optimized occupancy threshold, the target associated data block corresponding to the target transaction of the target type is determined, and the rollback log of the target transaction is stored in the backup storage space. At the same time, the target associated data block corresponding to the target transaction is reclaimed. Thus, before the log storage space is exhausted, the data block occupied by the target transaction of the target type is accurately reclaimed. Under the premise of ensuring that the target transaction can be rolled back, the available space of the log storage space is guaranteed, and the database is prevented from waiting for transactions or failing to commit transactions due to insufficient log storage space, thereby improving the stability of the database service.

[0064] According to embodiments of the present invention, by independently reclaiming long transactions, the average log storage space occupied by long transactions is reduced by 40%. In tests of multiple complex business scenarios involving long transactions, the situation where a single long transaction blocks the reclamation of the overall log storage space is avoided, thereby improving the database system's ability and compatibility in handling long transactions.

[0065] According to embodiments of the present invention, by adaptively optimizing a preset occupancy threshold, the reclamation of log storage space can be planned in advance based on the real-time status of transaction processing in the database, reducing the probability of instantaneous log storage space exhaustion to below 0.1%. In a simulation test of a transaction system, the transaction interruption rate decreased by 90% compared to before optimization, effectively ensuring the stability and continuity of database services in high-frequency transaction scenarios.

[0066] According to an embodiment of the present invention, the above-mentioned N data blocks are linked end to end; before obtaining the target transaction by acquiring the transaction that meets the preset reclamation condition among the transactions of the target type when the available space ratio is less than or equal to the optimized occupancy threshold, the method further includes: when the log storage space is less than or equal to the preset storage space, repeatedly performing the following operations: determining the predicted available space ratio within a second preset time period based on the log generation speed and the available space ratio, wherein the predicted available space ratio represents the ratio of the number of data blocks without stored logical logs to the total number of N data blocks within the future second preset time period; when the predicted available space ratio is less than or equal to the preset predicted occupancy threshold, determining at least one extended data block based on the log storage space and the available space ratio; linking at least one extended data block to the last data block in the log storage space, and marking the status of at least one extended data block as idle; determining extended storage space based on at least one extended data block.

[0067] According to an embodiment of the present invention, the second preset duration is a fixed-duration time window starting from the current time point, the extended data block is a data block of the same size as the data block in the log storage space, the size of the aforementioned data block can be 1024MB, each data block can be managed through data block metadata, the data block metadata can include fields such as status identifier, associated transaction ID (Identifier), etc., data blocks can be connected through a doubly linked list, and the aforementioned preset predicted occupancy threshold can be, for example, 10%.

[0068] According to an embodiment of the present invention, when the database starts, the log storage space size can be set to 50% of the preset storage space size. The preset storage space size can be set to 20GB, and the second preset duration can be 30 seconds. The size of the extended data block depends on the product of the current log storage space size and the preset ratio, and the difference between the preset storage space size and the current log storage space size. For example, assuming the current log storage space is 15GB, the preset storage space size is 20GB, and the preset ratio is 20%, the size of the extended data block needs to be the smaller value between 15GB×20%=3GB and 20GB-15GB=5GB, i.e., 3GB, and the status of the extended data block is marked as idle at the same time.

[0069] According to an embodiment of the present invention, the predicted occupied space is determined based on the product of the log generation speed and the second preset duration; the predicted occupied space ratio is determined based on the quotient of the predicted occupied space and the log storage space; and the predicted available space ratio is obtained by subtracting the available space ratio using the predicted occupied space ratio.

[0070] According to an embodiment of the present invention, the predicted available space ratio can be calculated using the following formula (3).

[0071] (3);

[0072] in, This indicates the predicted percentage of available space. This indicates the percentage of available space, and `log_gen_speed` indicates the log generation speed. Indicates the second preset duration. This indicates the size of the log storage space.

[0073] According to an embodiment of the present invention, the second preset duration may be, for example, 30 seconds.

[0074] For example, if the log generation speed is 2MB / s, the current available space is 60%, the second preset duration is set to 30s, then the predicted space occupied is 60MB, the log storage space is 1000MB, so the predicted space occupied is 6%, and the predicted available space is 60%-6%=54%.

[0075] According to an embodiment of the present invention, the predicted available space ratio within a second preset time period is determined by the log generation speed, the available space ratio, and the log storage space. Therefore, when the predicted available space is less than or equal to a preset predicted occupancy threshold, extended data blocks are dynamically added to complete the expansion of the log storage space before the transaction reclamation mechanism is triggered. This avoids write interruptions caused by log storage space exhaustion, thereby ensuring the continuous availability of the log storage space and reducing the frequency of subsequent transaction reclamation. Furthermore, by predicting the predicted available space ratio within the second preset time period using the log generation speed, the database can expand in advance in scenarios of sudden increases in logical logs and postpone expansion when logical logs grow gradually. This balances storage resource utilization and write continuity, avoiding resource waste caused by premature expansion or write blocking caused by late expansion, thus enabling the log storage space to adaptively adjust with transaction load.

[0076] According to an embodiment of the present invention, the above-mentioned optimization of the preset occupancy threshold by utilizing the log generation speed to obtain the optimized occupancy threshold includes: determining a target optimization correction amount based on the product of the log generation speed and the preset optimization parameters; and correcting the preset occupancy threshold using the target optimization correction amount to obtain the optimized occupancy threshold.

[0077] The optimized occupancy threshold can be calculated using the following formula (4).

[0078] (4);

[0079] in, This indicates the threshold for optimizing occupancy. This indicates the preset occupancy threshold. This indicates the log generation speed, and p represents the preset optimization parameter (which can be set to 5% here).

[0080] For example, the preset occupancy threshold can be set to 60%. When the log generation speed is 2MB / s, the optimized occupancy threshold can be 60% - 2 × 5% = 50%. When the log generation speed is 0.5MB / s, the optimized occupancy threshold can be 60% - 0.5 × 5% = 57.5%. It should be noted that the aforementioned log generation speed is only used as a numerical value in the calculation of the optimized occupancy threshold, and the units are only for illustrative purposes.

[0081] According to an embodiment of the present invention, the target optimization correction amount is calculated in real time by multiplying the log generation speed and the preset optimization parameters, and the preset occupancy threshold is corrected accordingly to obtain the optimization occupancy threshold. This allows the database system to adaptively adjust the optimization occupancy threshold according to the actual log generation speed. In scenarios with high log generation speed, the threshold is lowered to trigger log recycling as soon as possible, thereby ensuring the stable operation of the database.

[0082] According to embodiments of the present invention, through precise logical log marking and efficient recycling mechanisms, the residence time of invalid logs is reduced by an average of more than 60%. In a business scenario, the log space utilization rate is improved by 30%-50% compared to traditional methods, effectively alleviating the problem of storage resource shortage.

[0083] According to an embodiment of the present invention, the above method further includes: determining the recyclable proportion based on the state of each data block in N data blocks, wherein the recyclable proportion is the ratio of the storage space corresponding to the data block in the pending recyclable state to the log storage space; if the recyclable proportion is greater than or equal to a preset threshold, determining the data block in the pending recyclable state as the target data block to be flushed to disk; and in response to the successful flushing of the target data block to disk, marking the state of the target data block to disk as idle.

[0084] The recyclable percentage can be calculated using the following formula (5).

[0085] (5);

[0086] in, Indicates the percentage of recyclable materials. Indicates the number of data blocks that are recyclable. This represents the storage space for a single data block. This indicates the log storage space.

[0087] Disk flushing refers to the action of the database system synchronously writing the logs or data pages generated by committed transactions that are temporarily residing in the memory buffer to the persistent storage medium. Disk flushing can ensure the persistence of committed transactions.

[0088] According to an embodiment of the present invention, a preset threshold can be set for the recyclable percentage to recycle data blocks in a timely manner. When recycling begins, the log storage space is first locked to prevent concurrent modification. Then, all data blocks in the recyclable state are traversed, and the data in the data blocks is flushed from the volatile cache to the persistent storage. After the flush is successful, the status of the target data block to be flushed is marked as idle, the log storage space is unlocked, and the available space percentage and the recyclable percentage are updated.

[0089] According to an embodiment of the present invention, the method further includes: in response to the available space ratio being greater than or equal to a preset shutdown threshold, determining the data block in the idle state as the target data block to be written; storing the rollback log of the target transaction in the target data block to be written, and removing the association between the backup storage space and the target transaction.

[0090] According to an embodiment of the present invention, the preset shutdown threshold is a proportional threshold compared with the percentage of available space. When the percentage of available space is greater than the preset shutdown threshold, the backup storage space will be cleared and the rollback logs stored in the backup storage space will be migrated to the log storage space. At the same time, the backup storage space will be marked as idle. The preset shutdown threshold may be, for example, 15%.

[0091] According to an embodiment of the present invention, by using the proportion of available space as the condition for enabling and disabling backup storage space, when the proportion of available space is greater than a preset disabling threshold, regression logs are saved through free blocks in log storage space and backup storage space is released, thereby avoiding redundant occupation of backup storage space during periods when log storage space is idle.

[0092] According to an embodiment of the present invention, the above method further includes: monitoring the space usage ratio of the extended storage space, wherein the space usage ratio is the ratio of the storage space corresponding to the extended data block of the stored logical log to the extended storage space; and suspending transaction processing when the space usage ratio is equal to a preset warning threshold.

[0093] According to an embodiment of the present invention, the preset warning threshold is a critical value for the space usage ratio set for the extended storage space. When the space usage ratio is greater than or equal to 90%, in order to ensure data accuracy, transaction processing can be suspended, data block recycling can be waited for, and a warning can be sent to relevant staff at the same time. The preset warning threshold can be, for example, 90%.

[0094] According to embodiments of the present invention, transaction processing can also be paused when the utilization rate of backup storage space exceeds a preset warning threshold in order to wait for data block reclamation. The present invention does not limit this.

[0095] Based on the above-described database data block reclamation method, this invention also provides a database data block reclamation device. The following will be combined with... Figure 3 The device is described in detail.

[0096] Figure 3 A structural block diagram of a database data block recycling device according to an embodiment of the present invention is shown.

[0097] like Figure 3 As shown, the database data block device 300 of this embodiment includes a monitoring module 310, an optimization module 320, an acquisition module 330, a determination module 340, and a storage module 350.

[0098] The monitoring module 310 is used to monitor the log generation rate of the database within a first preset time period, as well as the available space ratio of the log storage space. The available space ratio is the ratio of the number of data blocks without stored logical logs to the total number of N data blocks. In one embodiment, the monitoring module 310 can be used to perform the operation S210 described above, which will not be repeated here.

[0099] The optimization module 320 is used to optimize a preset occupancy threshold using the log generation speed to obtain an optimized occupancy threshold. In one embodiment, the optimization module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0100] The acquisition module 330 is used to acquire transactions of the target type that meet preset reclamation conditions when the available space ratio is less than or equal to the optimized occupancy rate threshold, thereby obtaining the target transaction. In one embodiment, the acquisition module 330 can be used to perform the operation S230 described above, which will not be repeated here.

[0101] The determination module 340 is used to determine the target associated data block that is associated with the logical log of the target transaction. In one embodiment, the determination module 340 may be used to perform the operation S240 described above, which will not be repeated here.

[0102] The storage module 350 is used to store the rollback log of the target transaction to the backup storage space and to remove the association between the target associated data block and the target transaction, so as to facilitate the reclamation of the target associated data block. In one embodiment, the storage module 350 can be used to perform the operation S250 described above, which will not be repeated here.

[0103] According to embodiments of the present invention, any plurality of modules among the monitoring module 310, optimization module 320, acquisition module 330, determination module 340, and storage module 350 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the monitoring module 310, optimization module 320, acquisition module 330, determination module 340, and storage module 350 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the monitoring module 310, optimization module 320, acquisition module 330, determination module 340, and storage module 350 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0104] Figure 4 A block diagram of an electronic device suitable for implementing a data block reclamation method for a database, according to an embodiment of the present invention, is shown.

[0105] like Figure 4 As shown, an electronic device 400 according to an embodiment of the present invention includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0106] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0107] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0108] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0109] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0110] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the database data block reclamation method provided in the embodiments of the present invention.

[0111] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0112] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0113] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0114] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0117] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method of reclaiming data blocks of a database, characterized by, The database includes a log storage space, which comprises N data blocks. These data blocks are used to store logical logs generated during transaction processing, where N is an integer greater than 1. The method includes: Monitor the log generation rate of the database within a first preset time period, and the available space ratio of the log storage space, wherein the available space ratio is the ratio of the number of data blocks that do not store logical logs to the total number of the N data blocks; The preset occupancy rate threshold is optimized using the log generation speed to obtain the optimized occupancy rate threshold; When the available space ratio is less than or equal to the optimized occupancy rate threshold, obtain the target transaction from the transactions of the target type that meet the preset reclamation conditions; Identify the target associated data block that is linked to the logical log of the target transaction; The rollback log of the target transaction is stored in the backup storage space, and the association between the target associated data block and the target transaction is removed, so as to facilitate the reclamation of the target associated data block.

2. The method of claim 1, wherein, The N data blocks are linked end-to-end; If the available space percentage is less than or equal to the optimized occupancy threshold, the process includes obtaining transactions of the target type that meet preset reclamation conditions. Before obtaining the target transaction, the process further includes: If the log storage space is less than or equal to the preset storage space, repeat the following operations: Based on the log generation speed, the available space ratio, and the log storage space, the predicted available space ratio within the second preset time period is determined. The predicted available space ratio represents the ratio of the number of data blocks that do not store logical logs to the total number of the N data blocks within the future second preset time period. If the predicted available space percentage is less than or equal to a preset predicted occupancy rate threshold, at least one extended data block is determined based on the log storage space and the available space percentage. Link the at least one extended data block to the last data block in the log storage space, and mark the status of the at least one extended data block as idle; The extended storage space is determined based on the at least one extended data block.

3. The method of claim 2, wherein, The step of optimizing the preset occupancy threshold using the log generation speed to obtain the optimized occupancy threshold includes: The target optimization correction amount is determined based on the product of the log generation speed and the preset optimization parameters; The preset occupancy rate threshold is corrected using the target optimization correction amount to obtain the optimized occupancy rate threshold.

4. The method of claim 1, wherein, The method further includes: Based on the status of each data block in the N data blocks, the recyclable percentage is determined, whereby the recyclable percentage is the ratio of the storage space corresponding to the data block in the pending recyclable state to the log storage space. If the recyclable percentage is greater than or equal to a preset threshold, the data block in the pending recycling state is determined as the target data block to be flushed to disk. In response to the successful flushing of the target data block to be flushed to disk, the status of the target data block to be flushed to disk is marked as idle.

5. The method of claim 1, wherein, The method further includes: In response to the available space ratio being greater than or equal to a preset closing threshold, a data block in an idle state is determined as a target data block to be written into; The rollback log of the target transaction is stored in the target data block to be written into, and the association between the backup storage space and the target transaction is removed.

6. The method of claim 2, wherein, The method further comprises: monitoring a space usage ratio of the extended storage space, the space usage ratio being a ratio of a storage space corresponding to the extended data block in which a logical log has been stored to the extended storage space; in a case where the space usage ratio is equal to a preset warning threshold, suspending transaction processing.

7. The method of claim 2, wherein, The method further comprises: determining a predicted available space ratio in a second preset time period based on the log generation speed, the available space ratio, and the log storage space, including: determining a predicted occupied space based on a product of the log generation speed and the second preset time period; determining a predicted occupied space ratio based on a quotient of the predicted occupied space and the log storage space; 8. A data block recycling apparatus of a database, characterized by comprising: subtracting the predicted occupied space ratio from the available space ratio to obtain the predicted available space ratio. The database comprises a log storage space, the log storage space comprising N data blocks, the data blocks being used to store logical logs generated in a transaction processing process, N being an integer greater than 1, and the apparatus comprising: a monitoring module configured to monitor a log generation speed of the database in a first preset time period and an available space ratio of the log storage space, the available space ratio being a ratio of a number of the data blocks in which no logical log has been stored to a total number of the N data blocks; an optimization module configured to optimize a preset occupancy rate threshold based on the log generation speed to obtain an optimized occupancy rate threshold; an obtaining module configured to, in a case where the available space ratio is less than or equal to the optimized occupancy rate threshold, obtain a transaction of a target type that meets a preset recovery condition to obtain a target transaction; a determining module configured to determine a target associated data block associated with a logical log of the target transaction; a storage module configured to store a rollback log of the target transaction in a backup storage space and remove an association between the target associated data block and the target transaction, so as to recover the target associated data block.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, 10. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-7. The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-7.

Images