Method and device for cleaning invalid data of global unique index of distributed database
By recording DDL operation logs in the distributed database and using a background cleanup thread to clean up invalid data in the globally unique index, the problems of prolonged DDL execution time and DML uniqueness conflicts are solved, achieving efficient index cleanup and business continuity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA CITIC BANK CO LTD
- Filing Date
- 2025-11-27
- Publication Date
- 2026-04-14
AI Technical Summary
When a partitioning DDL operation occurs in the main table of a distributed database, cleaning up invalid data in the globally unique index prolongs the DDL execution time and affects users' online business. Asynchronous cleaning, on the other hand, can lead to uniqueness conflicts in DML operations.
By logging DDL operations, adding a background cleanup thread to asynchronously clean up invalid data in the globally unique index, and performing uniqueness checks before DML operations, the correctness and efficiency of the index are ensured.
Accelerate DDL execution in high-frequency DDL operation scenarios, reduce the impact on online business, and ensure the correctness of the unique constraints of the globally unique index.
Smart Images
Figure CN121858542A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for cleaning up invalid data in a globally unique index of a distributed database. Background Technology
[0002] Indexed fields refer to specific columns or combinations of columns used to create indexes in a database. These fields, by constructing an ordered data structure, avoid full table scans and directly locate target data, significantly improving data retrieval efficiency. For example, after creating an index on the `user_id` field, a query `WHERE user_id=100` can quickly locate the data. Here, `WHERE` is a core keyword in SQL (Structured Query Language), used to specify filtering conditions in database queries or operations, thereby selecting only data rows that meet specific conditions. Essentially, it filters data through logical expressions and is a fundamental tool for accurate database retrieval and efficient operations.
[0003] Databases commonly encounter scenarios where data filtering cannot be performed using indexed fields, severely impacting query performance. This issue is typically addressed by constructing a mapping table, which maps non-indexed fields (e.g., field 1) to indexed fields (e.g., field 2). For example, if the original table is the main table (containing field 1 and field 2), the table storing the mapping between field 1 and field 2 is the mapping table. If field 1 is directly retrieved from the main table, it's clear that field 1 is a non-indexed field, requiring a full table scan, resulting in low retrieval efficiency. However, by using the mapping table to obtain the corresponding field 2, and then retrieving data from the main table using field 2 (an indexed field), the target data can be directly located, improving data retrieval efficiency. This mapping table is called the global index table. In the global index table, field 2 can be considered a globally unique index.
[0004] Globally unique indexes in distributed databases face a serious problem: when the main table undergoes partitioning DDL operations, such as `drop partition` or `truncate partition`, the corresponding globally unique index needs to clean up invalid data in all its partitions. If this cleanup operation is performed synchronously with the partitioning DDL operation, it will significantly extend the DDL execution time and severely impact online business operations. If the globally unique index uses asynchronous cleanup techniques, it faces the problem of uniqueness conflicts with invalid data caused by DML operations, such as `insert`, `update`, and `delete`. Therefore, a solution is needed that leverages the advantages of cross-partition queries provided by globally unique indexes while optimizing the cleanup of invalid data in the globally unique index caused by frequent partitioning DDL operations on the main table. Currently, there are no publicly available documents or methods to address this issue. Summary of the Invention
[0005] In view of the above problems, this application is made to provide a method, apparatus, and related product for cleaning up invalid data in a globally unique index of a distributed database to overcome or at least partially solve the above problems. The technical solution is as follows: Firstly, a method for cleaning up invalid data in a globally unique index of a distributed database is provided, the method comprising: In response to DDL operations performed on the main table, the DDL operations are recorded in the DDL log. The DDL log is used to record operations that require data cleanup for the globally unique index. A new background cleanup thread for the distributed database has been added. After detecting changes in the DDL log, the background cleanup thread will start a task to clean up invalid data in the globally unique index. It will read one or more change records from the DDL log, clean up the invalid data on the globally unique index according to each change record, and delete the change record from the DDL log after cleaning up the invalid data.
[0006] In one possible implementation, after deleting the change record from the DDL log, the method further includes: Maintain the state of the current globally unique index in memory, and change the state to "cleaned up".
[0007] In one possible implementation, the method further includes: If the distributed database fails during the process of cleaning up invalid data on the global unique index based on each change record, the background cleanup thread will scan the DDL log for any records that have not been deleted after the distributed database restarts. It will then clean up the invalid data on the global unique index based on these records. After cleaning up the invalid data, the invalid record in the DDL log will be deleted.
[0008] In one possible implementation, the method further includes: When the main table performs an insert operation, an index record is constructed to be inserted into the globally unique index during the insert process; The data to be inserted is checked for uniqueness in the global unique index. If there are no conflicts, the uniqueness check passes. If a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, insert the row data into the main table, and insert the index record into the global unique index. If the partition data has not expired, that is, the partition data is valid, the insertion fails and an error of violating the unique constraint is returned.
[0009] In one possible implementation, the method further includes: When the main table performs an update operation, the old record corresponding to the update operation is deleted from the global unique index, and an index record to be inserted into the global unique index is constructed from the new record corresponding to the update operation. Perform a uniqueness check on the data to be updated in the global unique index. If there are no conflicts, the uniqueness check passes. If a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, update the row data in the main table, and update the index record in the global unique index. If the partition data has not expired, that is, the partition data is valid, the update fails and an error of violating unique constraint is returned.
[0010] In one possible implementation, the method further includes: When a delete operation is performed on the main table, the delete operation is performed on both the main table and the global unique index.
[0011] Secondly, a device for cleaning up invalid data in a globally unique index of a distributed database is provided, the device comprising: The recording unit is used to respond to DDL operations performed on the main table and record the DDL operations in the DDL log. The DDL log is used to record operations that require data cleanup for the globally unique index. The cleanup unit is used to add background cleanup threads for the distributed database. After detecting changes in the DDL log, the background cleanup thread will start a task to clean up invalid data in the globally unique index. It reads one or more change records from the DDL log, cleans up invalid data on the globally unique index according to each change record, and deletes the change record from the DDL log after cleaning up the invalid data.
[0012] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method for cleaning up invalid data in a globally unique index of a distributed database as described in any of the preceding claims.
[0013] Fourthly, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute, at runtime, the method for cleaning up invalid data in a globally unique index of a distributed database as described in any of the preceding claims.
[0014] Fifthly, a computer program product is provided, including a computer program configured to execute, at runtime, the method for cleaning up invalid data in a globally unique index of a distributed database as described in any of the preceding claims.
[0015] Using the above technical solution, the method, apparatus, and related products for cleaning invalid data in a globally unique index of a distributed database provided in this application embodiment respond to the execution of DDL operations on the main table by recording the DDL operations in a DDL log; a background cleanup thread for the distributed database is added. After detecting changes in the DDL log, the background cleanup thread initiates a task to clean up invalid data in the globally unique index, reads one or more change records from the DDL log, cleans up the invalid data on the globally unique index according to each change record, and deletes the change record from the DDL log after cleaning up the invalid data. This embodiment can enable the distributed database to accelerate DDL execution and reduce the impact of DDL operations on online services in high-frequency DDL operation scenarios for main tables containing globally unique indexes, while also ensuring the correctness of the unique constraints of the globally unique index. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.
[0017] Figure 1 A flowchart illustrating the method for cleaning up invalid data in a globally unique index of a distributed database, as provided in an embodiment of this application, is shown. Figure 2The software architecture diagram of the method for cleaning up invalid data in a globally unique index of a distributed database provided in an embodiment of this application is shown. Figure 3 This application provides a flowchart illustrating the DML operation process for a distributed database with globally unique indexes independent of DDL operations, as shown in an embodiment of this application. Figure 4 This document illustrates a DML operation flowchart for a globally unique index in a distributed database provided in an embodiment of this application, which involves related DDL operations. Figure 5 This paper shows a structural diagram of a device for cleaning up invalid data in a globally unique index of a distributed database, as provided in an embodiment of this application. Figure 6 This invention provides a structural diagram of a device for cleaning up invalid data in a globally unique index of a distributed database, according to another embodiment of this application. Figure 7 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0018] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.
[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."
[0020] To address the aforementioned technical problems, embodiments of this application provide a method for cleaning up invalid data in a globally unique index of a distributed database, such as... Figure 1 As shown, the method for cleaning up invalid data in the globally unique index of this distributed database may include the following steps S101 and S102: Step S101: In response to the execution of DDL operations on the main table, the DDL operations are recorded in the DDL log. The DDL log is used to record operations that require data cleanup for the globally unique index.
[0021] In this step, the distributed database may include one or more master tables, which are tables that include indexes; different master tables may be partitioned using the same or different partitioning rules, such as partitioning by time or partitioning by column, etc., which are not limited in this embodiment.
[0022] DDL operations can include dropping / truncate partitions, etc., and this embodiment does not limit them.
[0023] Step S102: Add a background cleanup thread for the distributed database. After the background cleanup thread detects a change in the DDL log, it will start a task to clean up invalid data in the global unique index. It will read one or more change records from the DDL log, clean up the invalid data on the global unique index according to each change record, and delete the change record in the DDL log after cleaning up the invalid data.
[0024] In this embodiment, before the main table completes the drop / truncate partition operation, the DDL operations are recorded in the DDL log to record the operations that need to clean up the data of the globally unique index. After recording, the operation of cleaning up the data of the globally unique index does not need to be completed synchronously; the DDL of the main table drop / truncate partition can be completed. A background cleanup thread for the distributed database is added. After the background cleanup thread detects the change in the DDL log, it will start the task of cleaning up invalid data of the globally unique index. This can enable the distributed database to accelerate the execution of DDL and reduce the impact of DDL operations on online business in high-frequency DDL operation scenarios for the main table containing the globally unique index, while also ensuring the correctness of the unique constraint of the globally unique index.
[0025] This application embodiment provides a possible implementation method. After deleting the change record in the DDL log in step S102, it may further include the following step A1: Step A1: Maintain the state of the current globally unique index in memory and change the state to "Cleanup complete".
[0026] In this embodiment, once the status of the globally unique index is updated to "cleaned up," the system can immediately release related memory resources (such as the index structure in the cache), reducing memory usage. At the same time, subsequent queries or operations do not need to repeatedly check the cleanup status of the index, improving efficiency.
[0027] This application provides a possible implementation method, which may further include the following step B1: In step B1, if the distributed database fails during the process of cleaning up invalid data on the global unique index based on each change record, the background cleanup thread will scan the DDL log for records that have not been deleted after the distributed database restarts. It will then clean up the invalid data on the global unique index based on these records and delete the records that have not been deleted from the DDL log after the invalid data has been cleaned up.
[0028] In this embodiment, after a distributed database is interrupted due to a failure, the globally unique index may contain invalid data that has not been cleaned up (e.g., residual data due to incomplete transactions or unsynchronized logs). By scanning the DDL log for undeleted records after restarting, the background cleanup thread can re-trigger the cleanup process, ensuring the consistency between the index data and the underlying data.
[0029] Before the distributed database background cleanup thread completes the cleanup of invalid data in the globally unique index, if a business performs DML operations (insert or update data), the new value needs to be validated against the old value beforehand. This can lead to misjudgments due to the untimely cleanup of invalid data in the old value. This embodiment innovatively introduces an invalid data verification algorithm to solve the misjudgment problem of uniqueness verification, reducing the impact of DDL operations such as drop / truncate partitions on online business while ensuring the correctness of business DML operations.
[0030] This application provides a possible implementation method in which, when the main table performs an insert operation, the following steps C1 to C3 may be included: Step C1: During the insertion process, construct the index record to be inserted into the globally unique index; Step C2: Perform a uniqueness check on the data to be inserted in the global unique index. If there are no conflicts, the uniqueness check passes. In step C3, if a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, insert row data into the main table, and insert an index record into the global unique index; if the partition data has not expired, that is, the partition data is valid, the insertion fails and an error of violating unique constraint is returned.
[0031] This application provides a possible implementation method in which, when the main table performs an update operation, the following steps D1 to D3 may be included: Step D1: Delete the old record corresponding to the update operation from the global unique index, and construct the index record to be inserted into the global unique index from the new record corresponding to the update operation. Step D2: Perform a uniqueness check on the data to be updated in the global unique index. If there are no conflicts, the uniqueness check passes. In step D3, if a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, update the row data in the main table, and update the index record in the global unique index. If the partition data has not expired, that is, the partition data is valid, the update fails and an error of violating unique constraint is returned.
[0032] This application provides a possible implementation method in which, when a deletion operation is performed on the main table, the following steps E1 may be included: Step E1: Perform the deletion operation in the main table and the global unique index respectively.
[0033] This embodiment actively checks the partition status of conflict records when performing DML operations, rather than simply relying on the existence of index data. This mechanism ensures that even if the cleanup thread has not yet finished its work, it can correctly identify truly valid conflicts. Furthermore, by extracting the partition unique identifier from the conflict record and verifying its validity, it can distinguish between real conflicts and invalid data to be cleaned, significantly reducing the false positive rate.
[0034] The above introduces Figure 1 The embodiments shown have various implementation methods for each step. The following will further explain the method for cleaning up invalid data in the globally unique index of a distributed database according to the embodiments of this application through specific examples.
[0035] This embodiment mainly relates to the DDL and DML processes of distributed databases, and mainly includes the following two aspects of the invention: First: Before the main table's drop / truncate partition operation is completed, record the DDL operations in the DDL log to document the data cleanup operations required for the globally unique index. After recording, the global unique index cleanup operation does not need to be synchronized; the main table's drop / truncate partition DDL can complete the process. A background cleanup thread for the distributed database is added. Upon detecting changes in the DDL log, this thread will initiate a task to clean up invalid data in the globally unique index.
[0036] Second: Before the distributed database background cleanup thread completes the cleanup of invalid data in the globally unique index, if the business performs DML operations, such as inserting or updating data, the new value needs to be validated against the old value beforehand. This can lead to misjudgments due to the untimely cleanup of invalid data in the old value. This embodiment innovatively introduces an invalid data verification algorithm to solve the misjudgment problem of uniqueness verification, reducing the impact of DDL operations such as drop / truncate partition on online business, while ensuring the correctness of business DML operations.
[0037] This embodiment is applied to a distributed database system, and its software architecture is as follows: Figure 2 As shown, combined with Figure 2 The description is as follows: 1) The customer business consists of multiple customer applications (APPs). Users use the distributed database through the common ODBC (Open Database Connectivity) and JDBC (Java Database Connectivity) interfaces to concurrently execute drop / truncate partitions and DML operations.
[0038] 2) The compute node cluster consists of multiple middleware DBProxy (Database Proxy). SQL statements are processed in the compute nodes and then distributed to the data node cluster. Figure 2 The diagram illustrates DBProxy1, DBProxy2, ..., DBProxyN, where N is a positive integer.
[0039] 3) Due to the issue of invalid data cleanup caused by drop / truncate partition operations on the main table, a global unique index invalid data cleanup module was added to the data node cluster. A new background thread was added to handle invalid data, and a master-slave synchronization mechanism for invalid data cleanup was implemented.
[0040] Asynchronous cleanup mechanism: (1) Read records from the DDL log and process one drop / truncate partition record at a time.
[0041] (2) Identify the data in the main table that has been dropped / truncated based on the partition unique identifier, process invalid data on the global unique index in batches concurrently, and commit in a timely manner.
[0042] (3) After processing the invalid data, delete this record in the DDL log.
[0043] (4) Maintain the state of the current globally unique index in memory and change the state to "cleaned up".
[0044] If asynchronous execution fails, such as a crash causing asynchronous cleanup to fail, the previously uncleaned data will not be completely cleaned up. When the distributed database service restarts, a background thread scans the DDL log for any uncleaned records and then starts a background thread to clean up the invalid data. The standby server's synchronization mechanism synchronizes DDL data to the standby server's DDL log, and then the standby server's background thread completes the invalid data cleanup.
[0045] If the global unique index invalid data cleanup module does not clean up the data in a timely manner, and there are concurrent DML operations, it may lead to false judgments in global uniqueness detection. Therefore, a DML global uniqueness conflict data false judgment verification module is added to add an invalid data verification mechanism for uniqueness detection.
[0046] The DML operation process for a globally unique index in a distributed database without related DDL operations (such as drop / truncate partition) provided in this embodiment is as follows: Figure 3 As shown, in Figure 3 The process includes the following steps: Step 1: As shown in column T1, the data node receives DML statements such as insert, update, and delete from the compute node.
[0047] Step 2: As shown in column T2, add an MDL (Metadata Lock) read lock to the main table. If a DDL operation has added an MDL write lock at this time, wait for the write lock to be released.
[0048] Step 3: As shown in column T3, the insert operation checks the uniqueness of the value to be inserted in the global unique index. If the unique value constraint is violated, an error is reported and the insertion fails. The update operation first checks whether the value to be updated exists in the table. If it does not exist, it returns with 0 rows updated. If it exists, it continues to check the uniqueness of the new value after the update. If the uniqueness constraint is violated, an error is reported and the update fails. The delete operation does not require uniqueness checking.
[0049] Step 4: As shown in column T4, after passing the global uniqueness check, DML statements can perform insert, update, and delete operations on the main table.
[0050] Step 5: As shown in column T5, after the DML statements perform insert, update, and delete operations on the main table, the corresponding data operations are performed on the globally unique index.
[0051] The DML operation flow for the globally unique index in the distributed database provided in this embodiment, which includes related DDL operations (such as drop / truncate partition), is as follows: Figure 4 As shown, in Figure 4 The process includes the following steps: Step 1: As shown in column T1, the data node receives DML statements such as insert, update, and delete from the compute node.
[0052] Step 2: As shown in column T2, add an MDL read lock to the main table. If a DDL operation has added an MDL write lock at this time, wait for the write lock to be released.
[0053] Step 3: As shown in column T3, the insert operation checks the uniqueness of the value to be inserted in the global unique index. If a unique value constraint is violated, it further checks whether conflicting data is invalid based on the partition unique ID. If the conflicting data is valid, an error is reported confirming the unique constraint violation, and the insert fails. If the conflicting data is invalid, the unique constraint violation is a false alarm, the invalid data is deleted, and it is confirmed that there is no unique constraint violation. The update operation first checks whether the value to be updated exists in the table. If it does not exist, it returns, and the number of updated rows is 0. If it exists, it continues to perform uniqueness checks on the new value after the update. If a unique constraint is violated, it further checks whether conflicting data is invalid based on the partition unique ID. If the conflicting data is valid, an error is reported confirming the unique constraint violation, and the update fails. If the conflicting data is invalid, the unique constraint violation is a false alarm, the invalid data is deleted, and it is confirmed that there is no unique constraint violation. The delete operation does not require uniqueness checks.
[0054] Step 4: As shown in column T4, after passing the global uniqueness check, DML statements can perform insert, update, and delete operations on the main table.
[0055] Step 5: As shown in column T5, after the DML statements perform insert, update, and delete operations on the main table, the corresponding data operations are performed on the globally unique index.
[0056] See also Figure 4 Executing the relevant DDL operations, such as the drop partition / truncate partition statements, includes the following steps: Step 1: As shown in column T1, the data node receives the drop / truncate partition statement issued by the compute node.
[0057] Step 2: As shown in column T2, add an MDL write lock to the main table. If a DML operation has added an MDL read lock at this time, wait for the read lock to be released.
[0058] Step 3: As shown in column T3, dropping a partition in the main table will invalidate the data corresponding to the unique ID of that partition. Truncating a partition in the main table will increment the unique ID of that partition by one (ensuring the ID grows linearly and is unique), thus invalidating the data corresponding to the historical IDs of that partition.
[0059] Step 4: As shown in column T4, the entire DDL process will only end after the drop / truncate partition operation is recorded in the ddl log (i.e., DDL log).
[0060] Step 5: As shown in column T5, the distributed database adds a background cleanup thread to replay records in the DDL log that have not yet been cleaned up with invalid data that have not been cleaned up with a globally unique index, and starts the cleanup task.
[0061] Compared with existing technologies, this embodiment can enable the distributed database to accelerate DDL execution and reduce the impact of DDL operations on online business in high-frequency drop / truncate partition operations on the master table containing a globally unique index; it can also ensure the correctness of the unique constraint of the globally unique index; and at the same time, it can ensure the data consistency of master-slave synchronization.
[0062] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.
[0063] Based on the methods for cleaning up invalid data in globally unique indexes of distributed databases provided in the above embodiments, and based on the same inventive concept, this application also provides a device for cleaning up invalid data in globally unique indexes of distributed databases.
[0064] Figure 5 This is a structural diagram of the device for cleaning up invalid data in a globally unique index of a distributed database, as provided in an embodiment of this application. Figure 5 As shown, the device for cleaning up invalid data in the globally unique index of the distributed database may specifically include a recording unit 510 and a cleaning unit 520.
[0065] Recording unit 510 is used to record DDL operations in the DDL log in response to the execution of DDL operations in the main table. The DDL log is used to record operations that require data cleanup of the globally unique index. Cleanup Unit 520 is used to add a background cleanup thread for the distributed database. After the background cleanup thread detects a change in the DDL log, it will start a task to clean up invalid data in the globally unique index. It will read one or more change records from the DDL log, clean up the invalid data on the globally unique index according to each change record, and delete the change record in the DDL log after cleaning up the invalid data.
[0066] This application embodiment provides a possible implementation, wherein the cleaning unit 520 is further configured to: After deleting the change record from the DDL log, maintain the status of the current globally unique index in memory and change the status to "cleaned up".
[0067] This application embodiment provides a possible implementation, wherein the cleaning unit 520 is further configured to: If the distributed database fails during the process of cleaning up invalid data on the global unique index based on each change record, the background cleanup thread will scan the DDL log for any records that have not been deleted after the distributed database restarts. It will then clean up the invalid data on the global unique index based on these records. After cleaning up the invalid data, the invalid record in the DDL log will be deleted.
[0068] This application provides one possible implementation method, such as... Figure 6 As shown above, Figure 5 The demonstrated device may also include a verification unit 610, used for: When the main table performs an insert operation, an index record is constructed to be inserted into the globally unique index during the insert process; The data to be inserted is checked for uniqueness in the global unique index. If there are no conflicts, the uniqueness check passes. If a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, insert the row data into the main table, and insert the index record into the global unique index. If the partition data has not expired, that is, the partition data is valid, the insertion fails and an error of violating the unique constraint is returned.
[0069] This application embodiment provides a possible implementation, wherein the verification unit 610 is further configured to: When the main table performs an update operation, the old record corresponding to the update operation is deleted from the global unique index, and an index record to be inserted into the global unique index is constructed from the new record corresponding to the update operation. Perform a uniqueness check on the data to be updated in the global unique index. If there are no conflicts, the uniqueness check passes. If a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, update the row data in the main table, and update the index record in the global unique index. If the partition data has not expired, that is, the partition data is valid, the update fails and an error of violating unique constraint is returned.
[0070] This application embodiment provides a possible implementation, wherein the verification unit 610 is further configured to: When a delete operation is performed on the main table, the delete operation is performed on both the main table and the global unique index.
[0071] Based on the same inventive concept, this application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method for cleaning up invalid data in the globally unique index of a distributed database in any of the above embodiments.
[0072] In an exemplary embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The illustrated electronic device 700 includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the electronic device 700 may also include a transceiver 704. It should be noted that in practical applications, the transceiver 704 is limited to one, and the structure of this electronic device 700 does not constitute a limitation on the embodiments of this application.
[0073] Processor 701 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0074] Bus 702 may include a pathway for transmitting information between the aforementioned components. Bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 702 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0075] The memory 703 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0076] The memory 703 stores computer program code that executes the scheme of this application, and its execution is controlled by the processor 701. The processor 701 executes the computer program code stored in the memory 703 to implement the content shown in the foregoing method embodiments.
[0077] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0078] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute the method for cleaning up invalid data in the globally unique index of the distributed database in any of the above embodiments when running.
[0079] Based on the same inventive concept, this application also provides a computer program product, including a computer program configured to execute, at runtime, a method for cleaning up invalid data in a globally unique distributed database index of any of the above embodiments.
[0080] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.
[0081] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0082] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.
[0083] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.
Claims
1. A method for cleaning up invalid data in a globally unique index of a distributed database, characterized in that, The method includes: In response to DDL operations performed on the main table, the DDL operations are recorded in the DDL log. The DDL log is used to record operations that require data cleanup for the globally unique index. A new background cleanup thread for the distributed database has been added. After detecting changes in the DDL log, the background cleanup thread will start a task to clean up invalid data in the globally unique index. It will read one or more change records from the DDL log, clean up the invalid data on the globally unique index according to each change record, and delete the change record from the DDL log after cleaning up the invalid data.
2. The method according to claim 1, characterized in that, After deleting the change record from the DDL log, the method further includes: Maintain the state of the current globally unique index in memory, and change the state to "cleaned up".
3. The method according to claim 1 or 2, characterized in that, The method further includes: If the distributed database fails during the process of cleaning up invalid data on the global unique index based on each change record, the background cleanup thread will scan the DDL log for any records that have not been deleted after the distributed database restarts. It will then clean up the invalid data on the global unique index based on these records. After cleaning up the invalid data, the invalid record in the DDL log will be deleted.
4. The method according to claim 1, characterized in that, The method further includes: When the main table performs an insert operation, an index record is constructed to be inserted into the globally unique index during the insert process; The data to be inserted is checked for uniqueness in the global unique index. If there are no conflicts, the uniqueness check passes. If a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, insert the row data into the main table, and insert the index record into the global unique index. If the partition data has not expired, that is, the partition data is valid, the insertion fails and an error of violating the unique constraint is returned.
5. The method according to claim 1, characterized in that, The method further includes: When the main table performs an update operation, the old record corresponding to the update operation is deleted from the global unique index, and an index record to be inserted into the global unique index is constructed from the new record corresponding to the update operation. Perform a uniqueness check on the data to be updated in the global unique index. If there are no conflicts, the uniqueness check passes. If a conflict occurs, first find the conflicting record in the global unique index, obtain the partition unique identifier from the conflicting record in the global unique index, and determine whether the partition data has expired based on the partition unique identifier. If the partition data has expired, delete it, update the row data in the main table, and update the index record in the global unique index. If the partition data has not expired, that is, the partition data is valid, the update fails and an error of violating unique constraint is returned.
6. The method according to claim 1, characterized in that, The method further includes: When a delete operation is performed on the main table, the delete operation is performed on both the main table and the global unique index.
7. A device for cleaning up invalid data in a globally unique index of a distributed database, characterized in that, The device includes: The recording unit is used to respond to DDL operations performed on the main table and record the DDL operations in the DDL log. The DDL log is used to record operations that require data cleanup for the globally unique index. The cleanup unit is used to add background cleanup threads for the distributed database. After detecting changes in the DDL log, the background cleanup thread will start a task to clean up invalid data in the globally unique index. It reads one or more change records from the DDL log, cleans up invalid data on the globally unique index according to each change record, and deletes the change record from the DDL log after cleaning up the invalid data.
8. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a method for cleaning up invalid data in a globally unique index of a distributed database, as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute, at runtime, a method for cleaning up invalid data in the globally unique index of a distributed database as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, The computer program is configured to execute, at runtime, the method for cleaning up invalid data in the globally unique index of a distributed database as described in any one of claims 1 to 6.