Data processing method and device applied to database and computer storage medium

Abstract

The application discloses a data processing method applied to a database, comprising the following steps: a data storage engine switches a working database into a temporary database for undertaking data update operations during execution of a data compression transaction / snapshot transaction process of the database; and the working database is switched into a compressed database / origin database after the data compression transaction and / or snapshot transaction is executed, wherein the compressed database is created by the data compression transaction, and the origin database is a target database of the snapshot transaction. The application proposes a new temporary database to distinguish newly written data from existing data, so as to ensure that the operation process is safer and more reliable, and the application generates a snapshot more quickly, and lock conflict is less likely to occur, so as to ensure data consistency of multiple storage engines.

Description

Technical Field

[0001] This invention relates to the field of database storage technology, specifically to data processing methods, apparatus, and computer storage media applied to databases. Background Technology

[0002] When using a B+ tree as a database, the database size increases continuously with ongoing writes, but does not decrease even with a large number of deletion operations. Therefore, data compression was developed. Data compression is a data reconstruction process: a new B+ tree database is created, all data is scanned, and data is written to the new database. This ensures that only useful data is written, while deleted data is discarded.

[0003] When a database is working, there are continuous update operations. Imagine a scenario where, during data compression, an update operation for a specific key is still written to the uncompressed B+ tree file. However, during compression, the original value corresponding to this key may have already been written to a new database. Therefore, reading the uncompressed B+ tree file and the new B+ tree file will result in two different values, indicating data inconsistency during the compression process. This is the first technical problem that this invention aims to solve.

[0004] Looking at snapshots, a database is often not a single instance that can handle all read and write operations. Database backups are part of the database system, and snapshots are a database backup solution designed to quickly generate working backup databases.

[0005] A snapshot is a full backup process used to create a backup database. A full backup process consists of two steps: the first step is to generate a full snapshot based on a specific checkpoint, which contains all data up to that checkpoint; the second step is to add incremental data after that checkpoint, which represents the missing parts of the backup database after the full snapshot and needs to be completed to create a usable backup database. When using snapshot backups, it's typically necessary to know which point in the backup is being performed. Data before this point needs to be fully copied and applied, while data after this point can be synchronized with the database master node; this point is called the checkpoint.

[0006] When generating a snapshot, the B+ tree database file needs to be copied. There are two ways to copy files: copy and hard link. The difference between the two methods is that a copy results in two files, each with independent writes; a hard link copies a file node, not the file itself, but writes to the same file, so writes to the source file are reflected in the new file. However, the copy method has a problem: assuming a disk write speed of 500MB / s, copying a 50GB file takes 100 seconds, so generating a snapshot will also take at least 100 seconds. Since snapshot generation blocks data updates, such a long latency is unacceptable for the storage system. Therefore, copying files can only use hard links. Hard links are references to files, allowing the same file to have multiple filenames, which can be distributed across different directories. Simultaneously, modifying the same file will modify all files hard-linked to it. Hard link copying only creates a new file node without copying data, and the time consumption is negligible.

[0007] If hard links are used for copying, write operations to the original database data file will also be reflected in the new copy file. However, when transferring a snapshot to a backup database, the process takes time. If data writes during this time are still written to the original database file, due to the hard link method, although the transferred file is a copy, these write operations are still visible on the copy file. Thus, the copy file contains updates based on the checkpoint after the current checkpoint, which is not allowed by snapshots; therefore, this copy file is actually an erroneous file. This is the second technical problem that this invention aims to solve. In view of this, this invention patent is hereby proposed. Summary of the Invention

[0008] To address the problems described above, this invention proposes a data processing method, apparatus, and computer storage medium for databases. These solutions resolve data security issues under data compression, enable second-level snapshot generation, ensure checkpoint-based data consistency during snapshot transmission, and guarantee snapshot data consistency even across multiple data storage engine instances. Specifically, the following technical solutions are employed:

[0009] Data processing methods applied to databases include:

[0010] During the data storage engine's execution of data compression / snapshot transactions for the database, the working database is switched to a temporary database to handle data update operations during the execution of the data compression / snapshot transactions.

[0011] Once the data compression transaction and / or snapshot transaction are completed, the working database will be switched to the compressed database / original database. The compressed database is created by the data compression transaction, and the original database is the target database of the snapshot transaction.

[0012] As an optional embodiment of the present invention, in the data processing method of the present invention applied to a database, a single storage engine of the database has a swapLock, and the storage engine performs a swapLock locking operation before performing a switching operation of the working database.

[0013] As an optional embodiment of the present invention, in the data processing method of the present invention applied to a database, the database has a task lock within a single storage engine, and the execution flow of the data compression transaction includes:

[0014] The data storage engine performs a task lock operation to switch the working database to a temporary database;

[0015] Perform data compression on the current database. After the data compression is complete, use the compressed database created by the data compression as the current database.

[0016] Switch the working database to the current database and unlock the task lock.

[0017] As an optional embodiment of the present invention, in the data processing method applied to a database, the step of switching the working database to a temporary database includes:

[0018] The data storage engine performs a swapLock locking operation to switch the working database to a temporary database. After the switch is completed, the swapLock is unlocked.

[0019] The switching of the working database to the current database includes:

[0020] The data storage engine performs a swapLock locking operation to switch the working database to the current database. After the switch is completed, the swapLock is unlocked.

[0021] As an optional embodiment of the present invention, in the data processing method of the present invention applied to a database, the single data storage engine of the database has a tryLock function for attempting to acquire a lock, and the execution flow of the snapshot transaction includes:

[0022] The data storage engine calls tryLock to attempt to acquire the lock for the task lock;

[0023] Determine whether the temporary database contains data. If the result is yes, unlock the lock (tryLock) and return failure. If the result is no, perform the task lock (task lock) locking operation.

[0024] The `tryLock` function attempts to acquire the lock against the `swapLock`, and after successfully acquiring the lock on the `swapLock`, the working database is switched to a temporary database.

[0025] Perform a snapshot operation on the current database. After the snapshot is complete, unlock the task lock.

[0026] As an optional embodiment of the present invention, in the data processing method of the present invention applied to a database, the step of calling tryLock to attempt to acquire a lock against swapLock includes:

[0027] Call `tryLock` to attempt to acquire the lock and get the current state of the `swapLock`.

[0028] If the swapLock is currently locked, an attempt to acquire the lock tryLock fails and returns an error; if the swapLock is currently unlocked, the swapLock is locked.

[0029] After a failed attempt to acquire the lock (tryLock), the lock acquisition operation is repeated at regular intervals until the object being acquired is successfully locked, or the snapshot operation is revoked.

[0030] As an optional embodiment of the present invention, in the data processing method of the present invention applied to a database, when a single database contains multiple data storage engine instances, the process of generating a snapshot of the database based on all data storage engine instances includes:

[0031] Each data storage engine instance executes the snapshot preparation process simultaneously.

[0032] When all the data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances. If any data storage engine instance fails to complete the snapshot preparation process, a snapshot failure is returned.

[0033] The snapshot preparation process includes: the data storage engine instance calls tryLock to attempt to acquire the task lock and swapLock. If both the task lock and swapLock are successfully acquired, the snapshot preparation process is successful; otherwise, the snapshot preparation process fails.

[0034] As an optional embodiment of the present invention, in the data processing method of the present invention applied to the database, when a single data storage engine instance fails to execute the snapshot preparation process, a preparation failure return instruction is reported.

[0035] The data processing method includes:

[0036] Each data storage engine instance sequentially checks the results of the snapshot preparation process.

[0037] When all data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances.

[0038] When a snapshot preparation process fails at a certain level of data storage engine instance, a failure return command is reported. Upon receiving the failure return command, the next-level data storage engine instance unlocks the task lock and swap lock, and the snapshot preparation process status changes to snapshot preparation process failure. It then reports a failure return command to the next-level data storage engine instance, and so on, until all data storage engine instances that have successfully completed the snapshot preparation process unlock the task lock and swap lock.

[0039] As an optional embodiment of the present invention, the data processing method of the present invention applied to a database includes:

[0040] Once the data storage engine detects that the requested data in the memory cache has reached the storage threshold, it will perform a data update operation to update the requested data in the memory cache to the current working database.

[0041] The data storage engine performs a swapLock locking operation to obtain the current working database.

[0042] The request data in the memory cache is written in batches to the current working database. After the writing is completed, the swapLock is unlocked.

[0043] As an optional embodiment of the present invention, in the data processing method of the present invention applied to the database, after the data compression transaction and / or snapshot transaction are completed, the request data written by the data update operation in the temporary database is written back to the current working database.

[0044] This invention also provides a data processing apparatus for use in databases, comprising:

[0045] The database switching module, during the data storage engine's execution of data compression / snapshot transactions for the database, switches the working database to a temporary database to handle data update operations during the execution of the data compression / snapshot transactions; after the data compression and / or snapshot transactions are completed, the database switching module switches the working database to a compressed database / original database, wherein the compressed database is created by the data compression transaction, and the original database is the target database of the snapshot transaction.

[0046] The present invention also provides an electronic device, including a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer program is executed by the processor, the processor executes the data processing method applied to a database.

[0047] The present invention also provides a computer-readable storage medium storing a computer-executable program, which, when executed, implements the data processing method applied to a database.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] The data processing method for databases of the present invention introduces a temporary database (dbBuffer) to handle data update operations during data compression / snapshot transactions performed by the data storage engine. This ensures that during data compression, updates to a specific key are written to the temporary database file, and new updates are only applied to the temporary database. Simultaneously, a database file for compression is created, while the original database remains read-only. During data compression, data is read sequentially from the original database and written to the compressed database. Once compression is complete, the compressed database takes effect, and the original database can be deleted, preventing update operations from affecting the data compression process. Furthermore, during compression, read operations prioritize the temporary database because values ​​in the temporary database are always newer than those in the original database. The data processing method for databases of the present invention also utilizes hard-link replication during snapshot transactions performed by the data storage engine to achieve second-level snapshot generation. During snapshot generation, a temporary database (dbBuffer) is introduced to handle data update operations during data compression / snapshot transactions. In this way, when a snapshot is generated, the data updates are written to the temporary database file, while the original database is not updated. Therefore, the transmitted copy file is always consistent with the original database file based on the data after the checkpoint, and there will be no updated data after the checkpoint appearing in the copy file, ensuring that the generated snapshot file is the correct file.

[0050] Therefore, the data processing method for databases in this invention solves the data security problem of snapshot generation / transmission under large data volumes, ensures that snapshot generation can be completed within seconds, and also ensures that data compression is a safe operation, preventing data anomalies caused by new update operations. Addressing the data security issues common to both data compression and snapshots, this invention proposes creating a new temporary database, dbBuffer, to distinguish between newly written data and existing data, ensuring that both data compression and snapshot operations are more secure and reliable.

[0051] The data processing method of this invention applied to databases addresses the more complex snapshot atomication operations faced by multi-storage engine instances by using hard links to accelerate the atomication process, and by incorporating locks and temporary database dbBuffers to make snapshot generation faster, reduce lock conflicts, and ensure data consistency across multiple storage engines. Furthermore, it performs necessary rollbacks for storage engine instances that fail in the snapshot preparation process, ensuring that atomication can be executed safely. Attached image description:

[0052] Figure 1 A flowchart of a data processing method applied to a database according to an embodiment of the present invention;

[0053] Figure 2 The embodiments of the present invention apply to the data processing method of a database to perform update operations, data compression, and snapshots, and provide a comparison;

[0054] Figure 3 A flowchart of the snapshot preparation process for a single data storage engine in a database data processing method according to an embodiment of the present invention;

[0055] Figure 4 A flowchart illustrating the rollback preparation process for a single data storage engine in a database data processing method according to an embodiment of the present invention;

[0056] Figure 5 This invention relates to a snapshot flowchart of multiple data storage engine instances in a database data processing method. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0058] Therefore, the following detailed description of embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0061] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. These terms are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0062] Example 1

[0063] See Figure 1 As shown, the data processing method applied to the database in this embodiment includes:

[0064] During the data storage engine's execution of data compression / snapshot transactions for the database, the working database is switched to a temporary database dbBuffer to handle data update operations during the execution of the data compression / snapshot transactions.

[0065] Once the data compression transaction and / or snapshot transaction are completed, the working database will be switched to the compressed database / original database. The compressed database is created by the data compression transaction, and the original database is the target database of the snapshot transaction.

[0066] This embodiment of the data processing method applied to a database introduces a temporary database (dbBuffer) to handle data update operations during the data compression / snapshot transaction execution process of the data storage engine. This way, during data compression, updates to a specific key are written to the temporary database file, and new updates are only applied to the temporary database. Simultaneously, a database file for compression is created, while the original database remains read-only. During data compression, data is read sequentially from the original database and written to the compressed database. Once compression is complete, the compressed database takes effect, and the original database can be deleted, ensuring that update operations do not affect the data compression process. Furthermore, during compression, read operations prioritize the temporary database because the values ​​in the temporary database are always newer than those in the original database.

[0067] This embodiment of the data processing method applied to a database utilizes a hard-link replication method during the snapshot transaction execution process of the data storage engine to achieve second-level snapshot generation. During snapshot generation, a temporary database `dbBuffer` is introduced to handle data update operations during the data compression / snapshot transaction execution. Thus, when a snapshot is generated, data updates are written to the temporary database file, while the original database remains unchanged. Therefore, the transmitted copy file, based on data after the checkpoint, always maintains consistency with the original database file, ensuring that no updated data after the checkpoint appears in the copy file, thus guaranteeing that the generated snapshot file is correct.

[0068] Therefore, the data processing method applied to databases in this embodiment solves the data security problem of snapshot generation / transmission under large data volumes, ensures that snapshot generation can be completed within seconds, and also ensures that data compression is a safe operation that will not cause data anomalies due to new update operations. Addressing the data security issues common to both data compression and snapshots, this data processing method for databases proposes creating a new temporary database, dbBuffer, to distinguish between newly written data and existing data, ensuring that both data compression and snapshot operations are more secure and reliable.

[0069] Because the data processing method applied to the database in this embodiment requires switching the working database when performing data compression and / or snapshot transactions, in order to ensure the stability and reliability of the data storage engine during the database switching process, as an optional implementation method of this embodiment, the individual storage engine of the database has a swapLock. Before performing the working database switching operation, the storage engine performs a swapLock locking operation. The data storage engine in this embodiment uses the swapLock to implement the database switching process, thereby ensuring the safety and reliability of the database switching process.

[0070] Specifically, when the storage engine applies a swapLock to the current working database, the swapLock will lock the database switching operation, and other operations related to the database switching will no longer be executed. The swapLock specifically locks the current working database, and no new data will be written or existing data will be read during the database switching period.

[0071] As an optional implementation of this embodiment, in the data processing method applied to the database in this embodiment, the database has a task lock within a single storage engine, see [link to relevant documentation]. Figure 2 As shown, the execution flow of the data compression transaction described in this embodiment includes:

[0072] The data storage engine performs a task lock operation, switching the working database to a temporary database dbBuffer;

[0073] Perform data compression on the current database. After the data compression is complete, use the compressed database created by the data compression as the current database.

[0074] Switch the working database to the current database and unlock the task lock.

[0075] In this embodiment, the single data storage engine uses a combination of task lock (taskLock), swap lock (swapLock), and temporary database (dbBuffer) to solve the data anomaly problem caused by data update operations during data compression.

[0076] Specifically, during the data compression process, once the storage engine applies a task lock (taskLock) to the current working database, the data compression process is locked, and other data processing processes related to the data compression process are not allowed to be executed. In other words, the current working database is locked, and no other operations other than data compression are allowed to be executed.

[0077] Furthermore, in the data processing method applied to the database described in this embodiment, switching the working database to a temporary database during the data compression process includes: the data storage engine performing a swapLock locking operation to switch the working database to a temporary database, and after the switching is completed, unlocking the swapLock;

[0078] Switching the working database to the current database includes: the data storage engine performing a swapLock locking operation, switching the working database to the current database, and unlocking the swapLock after the switch is completed.

[0079] As an optional implementation of this embodiment, in the data processing method applied to the database in this embodiment, the single data storage engine of the database has a tryLock function for attempting to acquire a lock. See [link to previous section]. Figure 2 As shown, the execution process of the snapshot transaction in this embodiment includes:

[0080] The data storage engine calls tryLock to attempt to acquire the lock for the task lock;

[0081] Determine whether the temporary database dbBuffer contains data. If the result is yes, unlock the lock tryLock and return failure. If the result is no, perform the task lock operation (because the data in the temporary database dbBuffer is temporary data and needs to be written to the non-temporary working database to be complete).

[0082] The `tryLock` function attempts to acquire the lock against the `swapLock`, and after successfully acquiring the lock on the `swapLock`, the working database is switched to a temporary database.

[0083] Perform a snapshot operation on the current database. After the snapshot is complete, unlock the task lock.

[0084] This embodiment of the data processing method applied to the database uses a hard link approach to generate snapshots. This ensures that snapshot generation is much faster than copying. Simultaneously, a temporary database `dbBuffer` is used to buffer write operations, ensuring that the original database file is not modified during the entire snapshot generation and transmission process, maintaining data integrity and security. Furthermore, based on the `tryLock` call to attempt to acquire a lock, the difference between `tryLock` and `lock` is that `tryLock` immediately acquires a lock if none is available, and returns an error if another task has already acquired the lock. Therefore, `tryLock` can quickly determine whether a lock is locked before it is needed, rather than waiting to acquire the lock. This allows for faster snapshot generation, minimizing the impact on overall database read and write operations.

[0085] Since the swapLock locks the entire update operation, which typically takes seconds and can take minutes in extreme cases, generating a snapshot can be very time-consuming. Therefore, lock cannot be used to replace tryLock.

[0086] Furthermore, in the data processing method applied to the database in this embodiment, the call to attempt to acquire the lock tryLock against the swapLock includes:

[0087] Call `tryLock` to attempt to acquire the lock and get the current state of the `swapLock`.

[0088] If the swapLock is currently locked, an attempt to acquire the lock tryLock fails and returns an error; if the swapLock is currently unlocked, the swapLock is locked.

[0089] After a failed attempt to acquire the lock (tryLock), the lock acquisition operation is repeated at regular intervals until the object being acquired is successfully locked, or the snapshot operation is revoked.

[0090] The data processing method applied to the database in this embodiment involves a batch refresh process for data updates. The data storage engine first writes the requests to a memory cache, and then refreshes them to the B+ tree in batches once the memory cache is full. Therefore, the data processing method applied to the database in this embodiment includes:

[0091] When the data storage engine detects that the requested data in the memory cache has reached the storage threshold, it performs a data update operation to update the data in the memory cache to the current working database (which may be the original database or the temporary database dbBuffer).

[0092] See Figure 2As shown, the data storage engine performs a swapLock locking operation to obtain the current working database;

[0093] The request data in the memory cache is written in batches to the current working database. After the writing is completed, the swapLock is unlocked.

[0094] In this embodiment, the data processing method applied to the database generates a temporary dbBuffer as a transitional database file during the snapshot / data compression process. After the data compression transaction and / or snapshot transaction are completed, the requested data for the data update operation in the temporary database is written back to the current working database. This process is completed asynchronously; before the write-back is complete, both read and write operations need to check the data in the temporary dbBuffer.

[0095] Example 2

[0096] If a single database contains multiple storage engine instances, data compression is not conflicting. However, when generating snapshots based on multiple storage engine instances, this process coordinates each engine instance; locking all storage engine instances is required to ensure snapshot generation is based on a common checkpoint. Therefore, the data processing method applied to the database in this embodiment, when a single database contains multiple data storage engine instances, includes the following process for generating a snapshot of the database based on all data storage engine instances:

[0097] Each data storage engine instance executes the snapshot preparation process simultaneously.

[0098] When all the data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances. If any data storage engine instance fails to complete the snapshot preparation process, a snapshot failure is returned.

[0099] The snapshot preparation process includes: the data storage engine instance calls `tryLock` to attempt to acquire both the task lock and the swap lock. If both the task lock and the swap lock are successfully acquired, the snapshot preparation process succeeds; otherwise, the snapshot preparation process fails. See details... Figure 3 As shown.

[0100] The data processing method applied to the database in this embodiment addresses the more complex snapshot atomication operations faced by multi-storage engine instances. It uses hard links to accelerate the atomication process and incorporates locks and a temporary database dbBuffer to make snapshot generation faster, reduce lock conflicts, and ensure data consistency across multiple storage engines. Furthermore, it performs necessary rollbacks for storage engine instances that fail in the snapshot preparation process, ensuring that atomication can be executed safely.

[0101] Specifically, in the data processing method applied to the database in this embodiment, when a single data storage engine instance fails to execute the snapshot preparation process, a preparation failure return instruction is reported;

[0102] The data processing method includes:

[0103] Each data storage engine instance sequentially checks the results of the snapshot preparation process.

[0104] When all data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances.

[0105] When a snapshot preparation process fails at a certain level of data storage engine instance, a failure return command is reported. Upon receiving the failure return command, the next-level data storage engine instance unlocks the task lock and swap lock, and the snapshot preparation process status changes to snapshot preparation process failure. It then reports a failure return command to the next-level data storage engine instance, and so on, until all data storage engine instances that have successfully completed the snapshot preparation process unlock the task lock and swap lock.

[0106] Specifically, when a data storage engine instance receives a preparation failure return instruction, it executes the PrepareRoll back process. See [link to relevant documentation]. Figure 4 As shown, unlock the task lock and swap lock in sequence.

[0107] For the snapshot process of multiple data storage engine instances, see [link to snapshot process]. Figure 5 As shown (taking two data storage engine instances as an example, data storage engine instance db1 and data storage engine instance db2), the core is that after db2 fails to perform Prepare (snapshot preparation process), it needs to call PrepareRollback of the previous data storage engine instance (data storage engine instance db1) to release the lock of data storage engine instance db1. Otherwise, the lock of data storage engine instance db1 will not be released, resulting in a deadlock. More data storage engine instances follow the same pattern, ensuring that atomicity can be executed safely.

[0108] Example 3

[0109] This embodiment also provides a data processing device for databases, including:

[0110] The database switching module, during the data storage engine's execution of data compression / snapshot transactions for the database, switches the working database to a temporary database to handle data update operations during the execution of the data compression / snapshot transactions; after the data compression and / or snapshot transactions are completed, the database switching module switches the working database to a compressed database / original database, wherein the compressed database is created by the data compression transaction, and the original database is the target database of the snapshot transaction.

[0111] In this embodiment, the data processing device applied to the database, during the data storage engine's data compression transaction for the database, has a database switching module that switches the working database to a temporary database (dbBuffer) to handle data update operations during the data compression / snapshot transaction execution. Thus, during data compression, updates to a specific key are written to the temporary database file, and new updates are only applied to the temporary database. Simultaneously, a database file for compression is created, while the original database remains read-only. During data compression, data is read sequentially from the original database and written to the compressed database. Once the compression operation is complete, the compressed database takes effect, and the original database can be deleted, ensuring that update operations do not affect the data compression process. Furthermore, during compression, read operations prioritize reading the temporary database because the values ​​in the temporary database are always newer than those in the original database.

[0112] In this embodiment, the data processing device applied to the database employs a hard-link replication method during the snapshot transaction execution process of the data storage engine, achieving second-level snapshot generation. During snapshot generation, the database switching module switches the working database to a temporary database (dbBuffer) to handle data update operations during the data compression / snapshot transaction execution. Thus, when a snapshot is generated, data updates are written to the temporary database file, while the original database remains unchanged. Therefore, the transmitted copy file, based on data after the checkpoint, always maintains consistency with the original database file, ensuring that no updated data after the checkpoint appears in the copy file, thus guaranteeing that the generated snapshot file is correct.

[0113] Therefore, the data processing device applied to the database in this embodiment solves the data security problem of snapshot generation / transmission under large data volumes, and also ensures that snapshot generation can be completed in seconds. It also ensures that data compression is a safe operation and will not cause data anomalies due to new update operations. The data processing method applied to the database in this embodiment addresses the data security issues common to both data compression and snapshots by proposing the creation of a new temporary database, dbBuffer, to distinguish between newly written data and existing data, ensuring that both data compression and snapshot operations are more secure and reliable.

[0114] Because the data processing device applied to the database in this embodiment needs to switch working databases when performing data compression and / or snapshot transactions, in order to ensure the stability and reliability of the data storage engine during the database switching process, as an optional implementation of this embodiment, the data processing device applied to the database in this embodiment has a swap lock module inside each storage engine of the database. Before the database switching module performs a swapLock locking operation, the swap lock module performs a swapLock locking operation. The data storage engine in this embodiment uses the swapLock to implement the database switching process, thereby ensuring the safety and reliability of the database switching process.

[0115] Specifically, when the storage engine applies a swapLock to the current working database, the database no longer receives data update operations, and the newly updated data is written to the temporary database dbBuffer.

[0116] As an optional implementation of this embodiment, in the data processing device applied to the database, the database has a task lock module within its single storage engine, and the execution flow of the data compression transaction includes:

[0117] The task lock module performs task lock locking operations, and the database switching module switches the working database to the temporary database dbBuffer.

[0118] Perform data compression on the current database. After the data compression is complete, use the compressed database created by the data compression as the current database.

[0119] The database switching module switches the working database to the current database, and the task lock module unlocks the task lock.

[0120] In this embodiment, the single data storage engine uses a combination of task lock (taskLock), swap lock (swapLock), and temporary database (dbBuffer) to solve the data anomaly problem caused by data update operations during data compression.

[0121] Specifically, during the data compression process, once the storage engine applies a task lock (taskLock) to the current working database, the current working database is locked, and no other operations besides data compression are allowed.

[0122] Furthermore, in the data processing device applied to the database described in this embodiment, during the data compression process, the database switching module switches the working database to a temporary database, which includes: the swap lock module performing a swapLock locking operation, the database switching module switching the working database to a temporary database, and after the switching is completed, the swap lock module unlocking the swapLock.

[0123] The database switching module switches the working database to the current database by: the swap lock module performing a swapLock locking operation, the database switching module switching the working database to the current database, and after the switching is completed, the swap lock module unlocking the swapLock.

[0124] As an optional implementation of this embodiment, the data processing device applied to the database in this embodiment has a lock acquisition attempt module inside a single data storage engine of the database, and the execution flow of the snapshot transaction includes:

[0125] The attempt to acquire a lock module calls the `tryLock` function to attempt to acquire a lock on the task lock.

[0126] Determine whether the temporary database dbBuffer contains data. If the result is yes, unlock the lock tryLock and return failure. If the result is no, perform the task lock operation (because the data in the temporary database dbBuffer is temporary data and needs to be written to the non-temporary working database to be complete).

[0127] The module attempts to acquire the lock by calling tryLock to attempt to lock the swapLock. After successfully acquiring the swapLock, the working database is switched to a temporary database.

[0128] Perform a snapshot operation on the current database. After the snapshot is complete, unlock the task lock.

[0129] The data processing device applied to the database in this embodiment uses a hard link approach to generate snapshots. This ensures that snapshot generation is much faster than copying. A temporary database `dbBuffer` is used to buffer write operations, ensuring that the original database file is not modified during the entire snapshot generation and transmission process, maintaining data integrity and security. Furthermore, based on the `tryLock` call to attempt to acquire a lock, the difference between `tryLock` and `lock` is that `tryLock` immediately acquires a lock if none is available, and returns an error if another task has already locked it. Therefore, `tryLock` can quickly determine whether a lock is locked before it is needed, rather than waiting to acquire the lock. This allows for faster snapshot generation, minimizing the impact on overall database read and write operations.

[0130] Since the swapLock locks the entire update operation, which typically takes seconds and can take minutes in extreme cases, generating a snapshot can be very time-consuming. Therefore, lock cannot be used to replace tryLock.

[0131] Furthermore, in the data processing apparatus applied to the database in this embodiment, the attempt to acquire a lock module calling tryLock to attempt to lock the swapLock includes:

[0132] The module attempts to acquire the lock by calling `tryLock`, which retrieves the current state of the swapLock.

[0133] If the swapLock is currently locked, an attempt to acquire the lock tryLock fails and returns an error; if the swapLock is currently unlocked, the swapLock is locked.

[0134] After the attempt to acquire the lock (tryLock) fails, the lock acquisition module will re-execute the lock acquisition operation at regular intervals until the object being locked is successfully locked, or the snapshot operation is revoked.

[0135] The data processing device applied to the database in this embodiment performs a batch refresh operation. The data storage engine first writes the request to the memory cache, and then refreshes it to the B+ tree in batches after the memory cache is full.

[0136] Therefore, in this embodiment, when the data processing device applied to the database detects that the requested data in the memory cache has reached the storage threshold, the data storage engine performs a data update operation to update the requested data in the memory cache to the data in the current working database (which may be the original database or the temporary database dbBuffer).

[0137] The swap lock module performs a swapLock locking operation to obtain the current working database.

[0138] The request data in the memory cache is written in batches to the current working database. After the writing is completed, the swap lock module unlocks the swap lock.

[0139] In this embodiment, the data processing device applied to the database generates a temporary dbBuffer as a transitional database file during the snapshot / data compression process. After the data compression transaction and / or snapshot transaction is completed, the requested data for the data update operation in the temporary database is written back to the current working database. This process is completed asynchronously; before the write-back is complete, both read and write operations need to check the data in the temporary dbBuffer.

[0140] If a single database contains multiple storage engine instances, data compression is not conflicting. However, when generating snapshots based on multiple storage engine instances, this process coordinates each engine instance; all storage engine instances must be locked to ensure that snapshots are generated based on a common checkpoint. Therefore, in this embodiment of the data processing apparatus applied to a database, when a single database contains multiple data storage engine instances, the process of generating a snapshot of the database based on all data storage engine instances includes:

[0141] Each data storage engine instance executes the snapshot preparation process simultaneously.

[0142] When all the data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances. If any data storage engine instance fails to complete the snapshot preparation process, a snapshot failure is returned.

[0143] The snapshot preparation process includes: the data storage engine instance calls `tryLock` to attempt to acquire both the task lock and the swap lock. If both the task lock and the swap lock are successfully acquired, the snapshot preparation process succeeds; otherwise, the snapshot preparation process fails. See details... Figure 3 As shown.

[0144] To address the more complex snapshot atomication operations faced by multi-storage engine instances, this invention uses hard links to accelerate the atomication process, and incorporates locks and dbBuffers to make snapshot generation faster, reduce lock conflicts, and ensure data consistency across multiple storage engines. Furthermore, it performs necessary rollbacks for storage engine instances that fail during the snapshot preparation process, ensuring that atomication can be executed safely.

[0145] Specifically, in this embodiment, the data processing device applied to the database reports a preparation failure return instruction when a single data storage engine instance fails to execute the snapshot preparation process;

[0146] The data processing method includes:

[0147] Each data storage engine instance sequentially checks the results of the snapshot preparation process.

[0148] When all data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances.

[0149] When a snapshot preparation process fails at a certain level of data storage engine instance, a failure return command is reported. Upon receiving the failure return command, the next-level data storage engine instance unlocks the task lock and swap lock, and the snapshot preparation process status changes to snapshot preparation process failure. It then reports a failure return command to the next-level data storage engine instance, and so on, until all data storage engine instances that have successfully completed the snapshot preparation process unlock the task lock and swap lock.

[0150] Specifically, when a data storage engine instance receives a preparation failure return instruction, it executes the PrepareRoll back process. See [link to relevant documentation]. Figure 4 As shown, unlock the task lock and swap lock in sequence.

[0151] For the snapshot process of multiple data storage engine instances, see [link to snapshot process]. Figure 5 As shown (taking two data storage engine instances as an example, data storage engine instance db1 and data storage engine instance db2), the core is that after db2 fails to perform Prepare (snapshot preparation process), it needs to call PrepareRollback of the previous data storage engine instance (data storage engine instance db1) to release the lock of data storage engine instance db1. Otherwise, the lock of data storage engine instance db1 will not be released, resulting in a deadlock. More data storage engine instances follow the same pattern, ensuring that atomicity can be executed safely.

[0152] Example 4

[0153] This embodiment also provides a computer-readable storage medium storing a computer-executable program, which, when executed, implements the data processing method applied to the database as described above.

[0154] The computer-readable storage medium described in this embodiment may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0155] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer-executable program, and when the computer program is executed by the processor, the processor executes the data processing method applied to the database.

[0156] The electronic device is manifested in the form of a general-purpose computing device. It may contain one or more processors that work collaboratively. This invention also does not preclude distributed processing, meaning that processors may be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0157] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0158] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0159] It should be understood that the electronic device of the present invention may also include elements or components not shown in the examples above. For example, some electronic devices also include display units such as a display screen, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in its memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0160] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by computer software that executes the methods of the present invention, for example, by control software executed by a microprocessor, electronic control unit, client, server, etc. However, it should be noted that the computer software executing the methods of the present invention is not limited to execution in one or a specific set of hardware entities; it can also be implemented in a distributed manner by unspecified hardware. For computer software, the software product can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or distributed across a network, as long as it enables electronic devices to execute the methods according to the present invention.

[0161] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described herein. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the specific embodiments described above. Therefore, any modifications or equivalent substitutions to the present invention, as well as all technical solutions and improvements that do not depart from the spirit and scope of the invention, are covered within the scope of the claims of the present invention.

Claims

1. A data processing method applied to a database, characterized in that, include: During the data storage engine's execution of data compression / snapshot transactions for the database, the working database is switched to a temporary database to handle data update operations during the execution of the data compression / snapshot transactions. After the data compression transaction and / or snapshot transaction are completed, the working database is switched to the compressed database / original database. The compressed database is created by the data compression transaction, and the original database is the target database of the snapshot transaction. The database has a swapLock inside each storage engine. The storage engine performs a swapLock locking operation before performing a switching operation on the working database. The database has a task lock within each of its individual storage engines, and the execution process of the data compression transaction includes: The data storage engine performs a task lock operation, switching the working database to a temporary database; Perform data compression on the current database. After the data compression is complete, use the compressed database created by the data compression as the current database. Switch the working database to the current database and unlock the task lock; The database's single data storage engine internally has a tryLock function for attempting to acquire a lock, and the execution flow of the snapshot transaction includes: The data storage engine calls tryLock to attempt to acquire the lock for the task lock. Determine whether the temporary database contains data. If the result is yes, unlock the lock (tryLock) and return failure. If the result is no, perform the task lock (tasklock) locking operation. The `tryLock` function attempts to acquire the lock against the `swapLock`, and after successfully acquiring the lock on the `swapLock`, the working database is switched to a temporary database. Perform a snapshot operation on the current database. After the snapshot is complete, unlock the task lock.

2. The data processing method applied to a database according to claim 1, characterized in that, The step of switching the working database to a temporary database includes: The data storage engine performs a swapLock locking operation to switch the working database to a temporary database. After the switch is completed, the swapLock is unlocked. The switching of the working database to the current database includes: The data storage engine performs a swapLock locking operation to switch the working database to the current database. After the switch is completed, the swapLock is unlocked.

3. The data processing method applied to a database according to claim 1, characterized in that, The call to attempt to acquire the lock tryLock for the swapLock includes: Call `tryLock` to attempt to acquire the lock and get the current state of the `swapLock`. If the swapLock is currently locked, an attempt to acquire the lock tryLock fails and returns an error; if the swapLock is currently unlocked, the swapLock is locked. After a failed attempt to acquire the lock (tryLock), the lock acquisition operation is repeated at regular intervals until the object being acquired is successfully locked, or the snapshot operation is revoked.

4. The data processing method applied to a database according to claim 1, characterized in that, When a single database contains multiple data storage engine instances, the process of generating a snapshot of the database based on all data storage engine instances includes: Each data storage engine instance executes the snapshot preparation process simultaneously. When all the data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances. If any data storage engine instance fails to complete the snapshot preparation process, a snapshot failure is returned. The snapshot preparation process includes: the data storage engine instance calls tryLock to attempt to acquire the task lock and swapLock. If both the task lock and swapLock are successfully acquired, the snapshot preparation process is successful; otherwise, the snapshot preparation process fails.

5. The data processing method applied to a database according to claim 4, characterized in that, If a single instance of the data storage engine fails to execute the snapshot preparation process, a preparation failure return command shall be reported. The data processing method includes: Each data storage engine instance sequentially checks the results of the snapshot preparation process. When all data storage engine instances successfully complete the snapshot preparation process, a snapshot is generated using the common checkpoint of all data storage engine instances. When a snapshot preparation process fails at a certain level of data storage engine instance, a preparation failure return command is reported. Upon receiving the preparation failure return command, the next-level data storage engine instance unlocks the task lock and swap lock, and the snapshot preparation process status changes to snapshot preparation process failure. It then reports a preparation failure return command to the next-upper-level data storage engine instance, and so on, until all data storage engine instances that have successfully completed the snapshot preparation process unlock the task lock and swap lock.

6. The data processing method applied to a database according to claim 1, characterized in that, include: Once the data storage engine detects that the requested data in the memory cache has reached the storage threshold, it will perform a data update operation to update the requested data in the memory cache to the current working database. The data storage engine performs a swapLock locking operation to obtain the current working database. The request data in the memory cache is written in batches to the current working database. After the writing is completed, the swapLock is unlocked.

7. The data processing method applied to a database according to claim 1, characterized in that, After the data compression transaction and / or snapshot transaction are completed, the request data written by the data update operation in the temporary database is written back to the current working database.

8. A data processing device applied to a database, characterized in that, include: The database switching module, during the data storage engine's execution of data compression / snapshot transactions for the database, switches the working database to a temporary database to handle data update operations during the execution of the data compression / snapshot transactions; after the data compression and / or snapshot transactions are completed, the database switching module switches the working database to a compressed database / original database, wherein the compressed database is created by the data compression transaction, and the original database is the target database of the snapshot transaction; The database has a swapLock inside each storage engine. The storage engine performs a swapLock locking operation before performing a switching operation on the working database. The database has a task lock within each of its individual storage engines, and the execution process of the data compression transaction includes: The data storage engine performs a task lock operation, switching the working database to a temporary database; Perform data compression on the current database. After the data compression is complete, use the compressed database created by the data compression as the current database. Switch the working database to the current database and unlock the task lock; The database's single data storage engine internally has a tryLock function for attempting to acquire a lock, and the execution flow of the snapshot transaction includes: The data storage engine calls tryLock to attempt to acquire the lock for the task lock. Determine whether the temporary database contains data. If the result is yes, unlock the lock (tryLock) and return failure. If the result is no, perform the task lock (tasklock) locking operation. The `tryLock` function attempts to acquire the lock against the `swapLock`, and after successfully acquiring the lock, the working database is switched to a temporary database. Perform a snapshot operation on the current database. After the snapshot is complete, unlock the task lock.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, which, when executed by the processor, performs a data processing method applied to a database as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The system contains a computer-executable program, which, when executed, implements the data processing method applied to a database as described in any one of claims 1-7.

Images