A data processing method and related equipment
By using data pages and verification pages as database storage units, and calculating and storing verification information in real time, the problem of poor reliability in the database is solved, and high reliability and efficiency of data storage are achieved. In particular, it effectively reduces silent errors and resource waste in large-scale storage scenarios.
Patent Information
- Application Number
- CN202411905660.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-12-23
AI Technical Summary
Existing data processing methods suffer from poor reliability in databases, especially in large-scale storage scenarios under distributed architectures. The reliability of storage systems faces enormous challenges and is subject to problems such as write amplification, write holes, and silent errors.
By using data pages and verification pages as the basic units of database storage, verification information is calculated and stored in real time. The verification information is used to detect and repair data corruption, reduce write amplification problems and storage resource waste, and improve the reliability of data storage.
It improves the reliability of data storage, reduces the risk of data corruption caused by silent errors, reduces write amplification problems and storage resource waste, and ensures that data remains intact and correct during storage, especially in analytical databases, effectively balancing storage overhead and data reliability.
Smart Images

Figure CN119828970B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database application technology, and in particular to a data processing method and related equipment. Background Technology
[0002] In today's digital age, databases carry critical business data, making data security paramount. Any loss or corruption of data can lead to severe business disruptions and economic losses. Therefore, stable and reliable storage is essential for databases, ensuring data security and integrity.
[0003] Currently, various storage types are available. Among them, Redundant Array of Independent Disks (RAID) has relatively low construction and maintenance requirements and is widely used in databases. For example, RAID 5 achieves performance improvement and data redundancy through striping and parity disks. However, this type of storage system using redundancy and parity suffers from problems such as write amplification, write holes, and long rebuild times. Furthermore, regardless of the type of storage system, the final storage medium is always a hard drive, either a mechanical hard drive or a solid-state drive. These storage media may experience bit flips due to wear and tear, unstable voltage, electromagnetic interference, etc., leading to silent data corruption. This is especially pronounced in large-scale storage scenarios. When databases upgrade from a centralized architecture to a distributed architecture, the total data volume can even reach petabyte levels, posing a significant challenge to the reliability of the storage system during data processing.
[0004] It is evident that existing data processing methods suffer from poor reliability. Summary of the Invention
[0005] The main purpose of this application is to provide a data processing method and related equipment, which aims to solve the problem of poor reliability of existing data processing methods.
[0006] To achieve the above objectives, this application provides a data processing method, the method comprising:
[0007] Upon receiving a write request for target data, storage space is allocated for the target data. The storage space includes at least one range, which is a basic unit for database storage allocation. The range includes 7 data pages and 1 verification page, and the verification page is located after the data pages within the same range.
[0008] The verification information of the first range is written into the verification page corresponding to the first range. The verification information of the first range is the verification information calculated when the target data is written into the data page corresponding to the first range. The verification information is used for redundancy verification. The first range is any range among the at least one range.
[0009] Write the first range to persistent storage.
[0010] Optionally, before writing the first range to persistent storage, and after writing the verification information of the first range to the verification page corresponding to the first range, the method further includes:
[0011] If multiple dirty pages are detected in the first range, the input and output I / O of the multiple dirty pages are merged to generate an I / O request for the first range. The multiple dirty pages are the data pages corresponding to the first range that have been modified.
[0012] The system responds to the IO request in the data page corresponding to the first range and updates the verification information in the verification page corresponding to the first range according to the IO request.
[0013] Optionally, after writing the first range to persistent storage, the method further includes:
[0014] Upon receiving a read request for the target data, page detection is performed on at least one range stored in the persistent storage to obtain a page detection result for each range;
[0015] If the page detection result of the first range indicates that there is a damaged page in the data page corresponding to the first range, the damaged page is repaired according to the verification page corresponding to the first range to obtain the data page after repair of the first range. The data page after repair of the first range includes the repaired page of the damaged page and other pages in the data page corresponding to the first range.
[0016] The system responds to the target data read request based on the data page after repair in the first range, and replaces the damaged page with the repaired page in the persistent storage.
[0017] Optionally, after performing page detection on at least one range stored in the persistent storage and obtaining page detection results for each range upon receiving a read request for the target data, the method further includes:
[0018] If the page detection results of the first range indicate that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages are repaired according to the data pages corresponding to the backup first range to obtain the data pages after the first range is repaired. The data pages after the first range is repaired include the repaired pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The backup first range is the storage space in the backup database.
[0019] Optionally, when the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, repairing the damaged page according to the verification page corresponding to the first range to obtain the repaired data pages of the first range includes:
[0020] If the page detection results of the first range indicate that there is a damaged page in the data pages corresponding to the first range, then detect the consistency status of each page in the first range.
[0021] If the detection result of the consistency state indicates that at least one page in the first range has a log sequence number (LSN) that is inconsistent with the LSN of other pages, the pages in the first range are restored to the state when they have the same LSN, thus obtaining the first range in a unified state.
[0022] The damaged page is repaired according to the verification page corresponding to the first range of the unified state to obtain the data page after the first range is repaired. In the data page after the first range is repaired, the LSN of the repaired page is the same as the LSN of other pages.
[0023] Optionally, before writing the verification information of the first range into the verification page corresponding to the first range, the method includes:
[0024] After writing the target data into the seven data pages corresponding to the first range, the stored data of each data page in the seven data pages corresponding to the first range is obtained;
[0025] The stored data of each of the seven data pages is XORed to obtain the verification information for the first range.
[0026] To address the aforementioned technical problems, this application also provides a data processing apparatus, the apparatus comprising:
[0027] The application module is used to apply for storage space for the target data when a write request for the target data is received. The storage space includes at least one range, which is a basic unit for database storage allocation. The range includes 7 data pages and 1 verification page, and the verification page is located after the data pages within the same range.
[0028] The first writing module is used to write the verification information of the first range into the verification page corresponding to the first range. The verification information of the first range is the verification information calculated when the target data is written into the data page corresponding to the first range. The verification information is used for redundancy verification. The first range is any range among the at least one range.
[0029] The second write module is used to write the first range to persistent storage.
[0030] To address the aforementioned technical problems, this application also provides an electronic device that employs the following technical solution:
[0031] The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the data processing methods proposed in the embodiments of this application.
[0032] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:
[0033] The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the data processing methods proposed in the embodiments of this application.
[0034] To address the aforementioned technical problems, this application also provides a computer program product that employs the following technical solution:
[0035] The computer program product includes computer instructions that, when executed by a processor, implement the steps of any of the data processing methods proposed in the embodiments of this application.
[0036] Compared with the prior art, the embodiments of this application have the following main advantages:
[0037] This application provides a data processing method that uses a range as the basic unit for database storage allocation. When target data is written to the data page corresponding to the range, verification information is calculated in real time and stored in the verification page corresponding to the range. During subsequent data reading or other operations, the verification information can be used to detect whether the data is corrupted, thereby improving the reliability of data storage, reducing the risk of data corruption due to silent errors and other reasons, and ensuring that the written data remains intact and correct during storage. At the same time, it reduces write amplification problems and reduces the storage space occupied by configuring additional replicas, avoiding excessive waste of storage resources. Especially in scenarios such as analytical databases where storage costs are sensitive and the data volume is large, it can effectively balance storage overhead and data reliability. Attached Figure Description
[0038] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is one of the flowcharts of the data processing method provided in the embodiments of this application;
[0040] Figure 2 This is a schematic diagram of the storage structure provided in the embodiments of this application;
[0041] Figure 3 This is a schematic diagram of the range in the storage structure provided in the embodiments of this application;
[0042] Figure 4 This is the second flowchart of the data processing method provided in the embodiments of this application;
[0043] Figure 5 This is the third flowchart of the data processing method provided in the embodiments of this application;
[0044] Figure 6 This is the fourth flowchart of the data processing method provided in the embodiments of this application;
[0045] Figure 7 This is the fifth flowchart of the data processing method provided in the embodiments of this application;
[0046] Figure 8 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;
[0047] Figure 9 This is a basic structural block diagram of the electronic device in this embodiment. Detailed Implementation
[0048] The data processing methods provided in this application are applied to a data processing apparatus. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application. The terms "comprising" and "having," and any variations thereof, in the description, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the description, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a particular order.
[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0050] refer to Figure 1 This diagram illustrates one of the flowcharts of a data processing method according to this application. The data processing method provided in this application can be applied to a storage system, such as... Figure 1 As shown in the figure, a data processing method provided in this application includes the following steps:
[0051] S110. Upon receiving a write request for target data, apply for storage space for the target data. The storage space includes at least one range, which is a basic unit for database storage allocation. The range includes 7 data pages and 1 verification page, and the verification page is located after the data pages within the same range.
[0052] The database of a storage system can adopt a hierarchical storage structure, which can include tablespaces, segments, extents, and pages, such as... Figure 2 As shown. A tablespace is the highest-level logical storage unit, a segment is a storage unit within a tablespace, and a range is a component of a segment, consisting of a set of contiguous pages. For example, a page can be fixed at 8KB in size; a range can consist of 8 pages, and the range itself is fixed at 64KB in size.
[0053] In this step, upon receiving a write request from the upper-layer executor, the storage system allocates storage space for the target data in 64KB units. This ensures that data within the same range only comes from write requests of the same transaction, meaning all writes are full-range writes, reducing write amplification caused by multiple transactions concurrently writing to different pages within the same range. When the storage space comprises multiple ranges, each range includes 8 pages, such as... Figure 3 As shown, the first seven pages within the same range are data pages, and the last page is a verification page. Data pages store the user's actual data, i.e., the target data, while verification pages store verification data, or verification information. The range serves as the basic unit of database storage allocation, with contiguous storage space and guaranteed 64KB alignment. This allows for quick location of other relevant pages, including verification pages, from any given page during data access, laying the foundation for subsequent data verification and repair operations.
[0054] S120. Write the verification information of the first range into the verification page corresponding to the first range. The verification information of the first range is the verification information calculated when the target data is written into the data page corresponding to the first range. The verification information is used for redundancy verification. The first range is any range among the at least one range.
[0055] In this step, the target data can be written to one or more ranges based on the actual storage space required. For example, if the target data requires less than or equal to 56KB of storage space, it can be written to a data page corresponding to one range; if the target data requires more than 56KB of storage space, it can be written to data pages corresponding to multiple ranges. For instance, if the target data requires 80KB of storage space, it can be written to data pages corresponding to two ranges. For any unfilled ranges, zeros can be added to the unfilled pages within those ranges.
[0056] Taking writing the target data into the first range as an example, before writing the target data into the data page corresponding to the first range, an 8KB (page size) memory space is allocated to temporarily record the verification information. This ensures that there is enough space to gradually calculate and store the verification information during subsequent data writing. Furthermore, since the read and write speed of memory is much faster than that of disk, the efficiency of verification information calculation and processing can be improved. Then, the target data is written into the seven data pages corresponding to the first range in sequence. As each data page is written, the verification information in memory is continuously updated until all seven data pages are written. By performing an XOR operation on the data stored in the seven data pages, the verification information of the first range is obtained.
[0057] When preparing to write the verification page, that is, after writing the target data to the seven data pages corresponding to the first range, the final verification information obtained in memory is written to the verification page corresponding to the first range, completing the entire process of writing data and generating verification information for the first range. If an anomaly occurs during this process (such as system failure, power outage, etc.), since the verification information is updated in real time and temporarily stored in memory, appropriate processing can be performed during recovery based on the written data pages and the status of the verification information in memory (such as data recovery through Write-Ahead Logging (WAL) and verification information), further ensuring the reliability and consistency of data writing. This synchronous calculation of verification information ensures that the verification information can reflect the integrity and accuracy of the currently written data pages in real time, and does not require additional steps to recalculate the verification information after all data pages have been written, saving computing resources and time.
[0058] S130, Write the first range to persistent storage.
[0059] In this step, by writing the first range containing the target data to persistent storage, the data will not be lost even in the event of system failure (such as power outage, software crash, etc.), server restart, or other anomalies, because persistent storage devices (such as hard drives) can retain data in non-volatile storage media. This improves the long-term stability and reliability of data in the storage system, meeting the business's needs for long-term data preservation and readily accessible data.
[0060] Furthermore, data written to persistent storage can be used for subsequent data queries, updates, and deletions. When a data read request is received, the system can quickly retrieve the data from persistent storage and return it to the user; during data update operations, the original data in persistent storage can be used as a basis for modification; and for data deletion operations, the system also needs to mark or actually delete the corresponding data in persistent storage. Writing ranges to persistent storage provides the data foundation for the normal operation and data management of the entire database, enabling the database system to efficiently handle various business operations.
[0061] Compared to related technologies that employ a large amount of redundant information or complex distributed redundant storage schemes, this embodiment uses a range as the basic unit for database storage allocation. When target data is written to the data page corresponding to the range, verification information is calculated in real time and stored in the verification page corresponding to that range. During subsequent data reading or other operations, the verification information can be used to detect whether the data is corrupted, thereby improving the reliability of data storage, reducing the risk of data corruption due to silent errors, and ensuring that the written data remains intact and correct during storage. At the same time, it reduces write amplification problems and reduces the storage space occupied by configuring additional replicas, avoiding excessive waste of storage resources. Especially in scenarios such as analytical databases where storage costs are sensitive and the data volume is large, it can effectively balance storage overhead and data reliability.
[0062] Optionally, before writing the verification information of the first range into the verification page corresponding to the first range, the method includes:
[0063] After writing the target data into the seven data pages corresponding to the first range, the stored data of each data page in the seven data pages corresponding to the first range is obtained;
[0064] The stored data of each of the seven data pages is XORed to obtain the verification information for the first range.
[0065] In this embodiment, the target data is written to the seven data pages corresponding to the first range. For example, the data stored on the first data page is AAA, the data stored on the second data page is BBB, the data stored on the third data page is CCC, and similarly, the data stored on the seventh data page is GGG. Then the verification information of the first range can be represented as HHH=AAA⊕BBB⊕CCC⊕…⊕GGG.
[0066] Before writing the target data into the first range, 8KB of memory space is allocated for temporarily recording verification information. For example, when writing CCC to the third data page, the temporary verification information between the first and second data pages (i.e., AAA⊕BBB) can be calculated simultaneously, and the result can be temporarily stored in memory. This way, when writing CCC to the third data page, the temporary verification information can be quickly read to further calculate the temporary verification information between the first, second, and third data pages (i.e., AAA⊕BBB⊕CCC). Similarly, as data pages are written one by one, the temporary verification information in memory is continuously updated until all seven data pages are written. By performing an XOR operation on the data stored in the seven data pages, the verification information for the first range is obtained.
[0067] The CRC value for each page can be determined using the Cyclic Redundancy Check (CRC) algorithm. The CRC algorithm treats the data to be checked as a sequence of coefficients of a polynomial, performs a division operation with a predefined generator polynomial, and uses the remainder as the checksum. For example, assuming the CRC-32 algorithm is used, its generator polynomial can be expressed as:
[0068] G(x)=x 32 +x 26 +x 23 +x 22 +x 16 +x 12 +x 11 +x 10 +x 8 +x 7 +x 5 +x 4 +x 2 +x+2;
[0069] Thus, during the data retrieval process, after a page is read from persistent storage, the checksum information for that page is recalculated. The page data is then processed again using the polynomial and division methods described above to obtain new checksum information, which is set as the CRC. new Then it is compared with the verification information recorded on the verification page. If the CRC... new Not equal to CRC recorded If this is the case, it indicates that the data is corrupted. If it is determined that a single data page is corrupted, then it is possible to attempt to repair it using the verification information from other uncorrupted data pages and verification pages.
[0070] Through the series of operations based on the CRC algorithm described above, reliable verification information is generated during data writing, and page corruption can be detected during data reading. Furthermore, attempts can be made to repair corrupted pages to some extent, thereby improving the reliability and integrity of database storage. In practical applications, the specific implementation and optimization of the CRC algorithm will be adjusted according to the performance requirements and hardware characteristics of the database system to ensure efficient and accurate completion of verification and repair tasks.
[0071] Optionally, before writing the first range to persistent storage, and after writing the verification information of the first range to the verification page corresponding to the first range, the method further includes:
[0072] If multiple dirty pages are detected in the first range, the input and output I / O of the multiple dirty pages are merged to generate an I / O request for the first range. The multiple dirty pages are the data pages corresponding to the first range that have been modified.
[0073] The system responds to the IO request in the data page corresponding to the first range and updates the verification information in the verification page corresponding to the first range according to the IO request.
[0074] In database systems, a marking mechanism can be used to identify dirty pages during operation. For example, when a data page is modified in the memory buffer, it is marked as a dirty page. The background flush thread periodically scans the memory buffer, finds these marked "dirty pages," and collects them for flushing operations. In traditional storage systems, if each dirty page is handled by a separate I / O operation, the disk head needs to frequently move to different locations for reading and writing, resulting in significant disk seek time and severely impacting I / O performance.
[0075] In this embodiment, when multiple dirty pages are detected within the first range, the I / O operations of these dirty pages are merged to generate a single I / O request. By merging I / O operations, read and write operations of multiple consecutive or adjacent dirty pages are combined into one request, allowing the disk head to complete operations on these pages in a shorter time. This reduces the number of disk seeks and the time spent, thereby significantly improving the efficiency of I / O operations. This performance improvement is particularly pronounced in scenarios involving large amounts of data writing or updating.
[0076] The system responds to IO requests and updates the verification information in the verification page corresponding to the first range of data pages, ensuring that the verification information always remains consistent with the actual state of the data pages. As data pages are modified and IO operations are performed, timely updates to the verification information accurately reflect the latest data state, guaranteeing the timeliness and accuracy of the verification information. During data updates, timely updates to the verification information enhance reliability in transaction processing and data consistency assurance. For example, when multiple data pages are modified within a transaction, IO merging and verification information updates ensure that the atomicity and consistency of these modifications are reflected at the storage level. If a system failure occurs before the transaction is committed, the recovery process can determine the correct state of the data based on accurate verification information and transaction logs, preventing data inconsistencies caused by discrepancies between verification information and the actual data state, further enhancing the database system's data consistency assurance capabilities in complex operating environments. Furthermore, for each collected dirty page, the background flush thread calculates its CRC checksum, enhancing data reliability and detectability, and ensuring the stable operation of the storage system.
[0077] In one example, such as Figure 4 As shown, the data writing operation process is as follows:
[0078] ① Request storage space for the data to be written, and request it in units of range to ensure that data within the same range will only come from the write requests of the same transaction. That is, all writes are full range writes, avoiding write amplification caused by multiple transactions concurrently writing to different pages of the same range.
[0079] ② Write the actual data. Before writing the range, allocate an 8KB (page size) memory space to temporarily record the verification information. Calculate the verification information synchronously for each data page written.
[0080] ③ Write the verification data. After writing the first 7 data pages, write the generated verification information to the verification page. If a range is not filled, fill the remaining space with 0 and then calculate the verification information.
[0081] ④ The background refresh thread collects "dirty pages" and attempts to refresh them. This process involves IO merging and calculating the CRC checksum for each page.
[0082] ⑤ Write to persistent storage.
[0083] By using ranges as the basic unit of database storage allocation, when target data is written to the data page corresponding to the range, verification information is calculated in real time and stored in the verification page corresponding to that range. During subsequent data reading or other operations, the verification information can be used to detect whether the data is corrupted, thereby improving the reliability of data storage, reducing the risk of data corruption due to silent errors and other reasons, and ensuring that the written data remains intact and correct during storage. At the same time, it reduces write amplification problems and reduces the storage space occupied by configuring additional replicas, avoiding excessive waste of storage resources. Especially in scenarios such as analytical databases where storage costs are more sensitive and data volumes are large, it can effectively balance storage overhead and data reliability.
[0084] Optionally, after writing the first range to persistent storage, the method further includes:
[0085] Upon receiving a read request for the target data, page detection is performed on at least one range stored in the persistent storage to obtain a page detection result for each range;
[0086] If the page detection result of the first range indicates that there is a damaged page in the data page corresponding to the first range, the damaged page is repaired according to the verification page corresponding to the first range to obtain the data page after repair of the first range. The data page after repair of the first range includes the repaired page of the damaged page and other pages in the data page corresponding to the first range.
[0087] The system responds to the target data read request based on the data page after repair in the first range, and replaces the damaged page with the repaired page in the persistent storage.
[0088] In this embodiment, when reading target data, page detection is performed on the range stored in persistent storage to obtain the page detection results for each range. If a corrupted page is found, the system can quickly locate the verification page in that range and attempt to repair it based on the verification information. Taking the first range as an example:
[0089] If the page detection result within the first range indicates the presence of a damaged page within the corresponding data page range, the damaged page can be repaired using the corresponding verification page within the first range, resulting in the repaired data page for the first range. This method of repairing damaged pages using verification pages within the same range reduces the overhead of retrieving data from remote backups or other storage locations for repair. Since verification pages and data pages are closely related in storage structure and located within the same range, the latency of reading verification pages is relatively low, enabling rapid retrieval of the information needed for repair. Therefore, the repair process can be completed quickly locally, reducing data read latency and improving the response speed of read requests. Especially for frequent data read operations, this local rapid repair mechanism can significantly improve system performance, meeting the needs of application scenarios with high data read performance requirements.
[0090] After repairing the damaged page, the system directly responds to read requests based on the repaired data page and replaces the damaged page in persistent storage. This eliminates the need for further repair operations when reading data within that range. This optimized read process reduces repetitive detection and repair work, improving data reading efficiency. Over time, the system maintains the data in good condition, further enhancing overall data reading performance and providing users with a faster and smoother data access experience.
[0091] Optionally, after performing page detection on at least one range stored in the persistent storage and obtaining page detection results for each range upon receiving a read request for the target data, the method further includes:
[0092] If the page detection results of the first range indicate that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages are repaired according to the data pages corresponding to the backup first range to obtain the data pages after the first range is repaired. The data pages after the first range is repaired include the repaired pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The backup first range is the storage space in the backup database.
[0093] In this embodiment, as Figure 5 As shown, the data in the standby database (i.e., the backup database) is typically a backup or copy of the primary database (i.e., the master database), and its data status reflects, to some extent, the correct state of the primary database at a certain point in time. By using the first range of data pages from the standby database for repair, a relatively complete and accurate source of data for repair can be obtained, thereby improving the success rate of repairing multiple damaged pages. Even if some data in the primary database suffers serious errors for various reasons, the existence of the standby database provides a reliable alternative for data recovery, increasing the likelihood of correct data recovery and ensuring the stability and reliability of data in the database system.
[0094] If the page detection results for the first range indicate that multiple damaged pages exist within the corresponding data pages, these damaged pages can be repaired using the backup data pages corresponding to the first range. This results in repaired data pages within the first range. Data from the backup database can be used more directly to repair the damaged pages, reducing data recovery time and complexity. Thus, even if the primary database faces severe storage problems, the system can still switch to the backup database resources for data repair, maintaining the normal operation of the database service and preventing complete system crashes or service interruptions due to data corruption. This approach demonstrates greater resilience in the face of various hardware failures, software errors, or external interference, continuously providing support for business operations, reducing the risk of business losses due to system failures, and enhancing the system's survivability and reliability in complex environments.
[0095] It should be understood that the embodiments of this application use the first range as an example for illustrative purposes. The same technical effect can be achieved for other ranges in the storage space. To avoid repetition, it will not be described again here.
[0096] In one example, such as Figure 6 As shown, the data reading operation process is as follows:
[0097] ① Respond to the data query request sent by the upper-level executor;
[0098] ②Request the corresponding data from the file system and check whether the CRC confirmation page is corrupted;
[0099] ③ Repair damaged pages. Check other pages within the scope of the damaged page, including verification pages. If other pages are not damaged, perform quick repair through the verification pages. If more pages are damaged and exist in the backup database, access the backup database and pull the corresponding pages for regular repair.
[0100] ④ Write the repaired page back to persistent storage;
[0101] ⑤ After repairing the damaged page, there is no need to validate the page anymore; just load the data page into the data buffer.
[0102] ⑥ Return the target data and respond to the access request.
[0103] This application incorporates a CRC checksum mechanism to automatically repair damaged pages. After reading a page from persistent storage, the CRC of that page is recalculated and compared with the CRC recorded in the page log. If they are inconsistent, it indicates data corruption. Then, other pages within the same range are read based on the page number, and recovery attempts are made. Even if multiple pages are corrupted, data can be recovered through a backup database mechanism, thus improving the reliability of data retrieval.
[0104] Optionally, when the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, repairing the damaged page according to the verification page corresponding to the first range to obtain the repaired data pages of the first range includes:
[0105] If the page detection results of the first range indicate that there is a damaged page in the data pages corresponding to the first range, then detect the consistency status of each page in the first range.
[0106] If the detection result of the consistency state indicates that at least one page in the first range has a log sequence number (LSN) that is inconsistent with the LSN of other pages, the pages in the first range are restored to the state when they have the same LSN, thus obtaining the first range in a unified state.
[0107] The damaged page is repaired according to the verification page corresponding to the first range of the unified state to obtain the data page after the first range is repaired. In the data page after the first range is repaired, the LSN of the repaired page is the same as the LSN of other pages.
[0108] In this embodiment, the database's Write-Ahead Log (WAL) mechanism can be used to perform transaction control on updates and modifications to data pages and validation pages, ensuring that modifications are either fully completed or fully rolled back, thus maintaining a consistent state. This solves the write vulnerability problem caused by system crashes, abnormal power outages, and other failures.
[0109] If the page detection result in the first range indicates that there is a damaged page among the data pages corresponding to the first range, the consistency status of each page in the first range is checked. If the consistency status detection result indicates that at least one page in the first range has a Log Sequence Number (LSN) that is inconsistent with the LSN of other pages, the WAL logs can be used to first restore each page in the first range to a state with the same LSN, resulting in a unified state for the first range. This unified state can be a unified restoration to the latest version or a unified restoration to the initial version. Then, based on the verification page corresponding to the unified state of the first range, the damaged page is repaired to obtain the repaired data pages for the first range.
[0110] Specifically, such as Figure 7 As shown, ① During the database recovery phase, page corruption is detected, recorded, and recovery of other pages within the relevant extent continues; ② Branch 1: The transaction is successfully committed, and other pages are uniformly restored to the latest version using the WAL log; ③ The corrupted page is restored using consistent verification pages and data pages; ④ The repaired data page is rewritten back to persistent storage; ⑤ Branch 2: The transaction is abnormally rolled back, and other pages are uniformly restored to the initial version using the WAL log; ⑥ The corrupted page is restored using consistent verification pages and data pages; ⑦ The repaired data page is rewritten back to persistent storage.
[0111] This ensures LSN consistency across all pages within the same scope, thereby improving data consistency and reliability. It effectively repairs damaged pages, ensuring their consistency with other pages, reducing data errors or inconsistencies that could arise from LSN discrepancies. Even in the event of a single page corruption, the database can recover to a consistent state, allowing subsequent data access and operations to proceed normally, thus improving database availability and reliability. Furthermore, LSN-based consistency detection and repair also helps to more accurately locate and resolve problems during database recovery or fault handling.
[0112] Further reference Figure 8 As a response to the above Figure 8 To implement the method shown, this application provides an embodiment of a data processing apparatus 800, which is similar to... Figure 8 Corresponding to the method embodiment shown, the device can be specifically applied to a server.
[0113] This application provides a data processing apparatus, the data processing apparatus 800 comprising:
[0114] The application module 810 is used to apply for storage space for the target data when a write request for the target data is received. The storage space includes at least one range, which is a basic unit for database storage allocation. The range includes 7 data pages and 1 verification page, and the verification page is located after the data pages within the same range.
[0115] The first writing module 820 is used to write the verification information of the first range into the verification page corresponding to the first range. The verification information of the first range is the verification information calculated when the target data is written into the data page corresponding to the first range. The verification information is used for redundancy verification. The first range is any range among the at least one range.
[0116] The second write module 830 is used to write the first range to persistent storage.
[0117] Optionally, the device further includes:
[0118] The processing module is used to merge the input and output I / O of the multiple dirty pages when multiple dirty pages are detected in the first range, and generate an I / O request for the first range. The multiple dirty pages are the data pages corresponding to the first range that have been modified.
[0119] The update module is used to respond to the IO request in the data page corresponding to the first range, and update the verification information in the verification page corresponding to the first range according to the IO request.
[0120] Optionally, the device further includes:
[0121] The detection module is used to perform page detection on at least one range stored in the persistent storage when a read request for the target data is received, and to obtain the page detection result for each range.
[0122] The first repair module is used to repair the damaged page according to the verification page corresponding to the first range when the page detection result of the first range indicates that there is a damaged page in the data page corresponding to the first range, so as to obtain the data page after the first range is repaired. The data page after the first range is repaired includes the repaired page of the damaged page and other pages in the data page corresponding to the first range.
[0123] The response module is used to respond to the read request of the target data according to the data page after repair in the first range, and replace the damaged page with the repaired page in the persistent storage.
[0124] Optionally, the device further includes:
[0125] The second repair module is used to repair the multiple damaged pages according to the backup first range when the page detection result of the first range indicates that there are multiple damaged pages in the data pages corresponding to the first range, so as to obtain the data pages after the first range is repaired. The data pages after the first range is repaired pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The backup first range is the storage space in the backup database.
[0126] Optionally, the first repair module is specifically used for:
[0127] If the page detection results of the first range indicate that there is a damaged page in the data pages corresponding to the first range, then detect the consistency status of each page in the first range.
[0128] If the detection result of the consistency state indicates that at least one page in the first range has a log sequence number (LSN) that is inconsistent with the LSN of other pages, the pages in the first range are restored to the state when they have the same LSN, thus obtaining the first range in a unified state.
[0129] The damaged page is repaired according to the verification page corresponding to the first range of the unified state to obtain the data page after the first range is repaired. In the data page after the first range is repaired, the LSN of the repaired page is the same as the LSN of other pages.
[0130] Optionally, the device further includes:
[0131] The acquisition module is used to acquire the stored data of each of the seven data pages corresponding to the first range after writing the target data into the seven data pages corresponding to the first range.
[0132] The calculation module is used to perform an XOR operation on the stored data of each of the seven data pages to obtain the verification information of the first range.
[0133] To address the aforementioned technical problems, this application also provides an electronic device. Please refer to the following for details. Figure 9 , Figure 9 This is a basic structural block diagram of the electronic device in this embodiment.
[0134] The electronic device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected via a system bus. It should be noted that only electronic device 9 with components 91-93 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0135] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.
[0136] The memory 91 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the electronic device 9, such as the hard disk or memory of the electronic device 9. In other embodiments, the memory 91 may also be an external storage device of the electronic device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 9. Of course, the memory 91 may include both the internal storage unit and its external storage device of the electronic device 9. In this embodiment, the memory 91 is typically used to store the operating system and various application software installed on the electronic device 9, such as program code for data processing methods. In addition, the memory 91 may also be used to temporarily store various types of data that have been output or will be output.
[0137] In some embodiments, the processor 92 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 92 is typically used to control the overall operation of the electronic device 9. In this embodiment, the processor 92 is used to run program code stored in the memory 91 or process data, for example, to run program code for the data processing method.
[0138] The network interface 93 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the electronic device 9 and other electronic devices.
[0139] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a program of the data processing method, the program of the data processing method being executable by at least one processor to cause the at least one processor to perform the steps of the data processing method as described above.
[0140] This application also provides another implementation, namely, a computer program product including computer instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware online platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0142] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0143] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. A data processing method, characterized by, The method includes: Upon receiving a write request for target data, storage space is allocated for the target data. The storage space includes at least one range, which is the basic unit for database storage allocation. The range includes 7 data pages and 1 verification page, and the verification page is located after the data pages within the same range. Upon receiving a write request sent by the upper-layer executor, the storage system allocates storage space for the target data in 64KB ranges as basic units, ensuring that data within the same range will only come from write requests of the same transaction, i.e., all writes are full-range writes. The verification information of the first range is written into the verification page corresponding to the first range. The verification information of the first range is the verification information calculated when the target data is written into the data page corresponding to the first range. The verification information is used for redundancy verification. The first range is any range among the at least one range. According to the actual storage space required by the target data, the target data is written through one or more ranges. If the storage space required by the target data is less than or equal to 56KB, the target data is written into the data page corresponding to one range. If the storage space required by the target data is greater than 56KB, the target data is written into the data pages corresponding to multiple ranges. For ranges that are not filled, 0 is filled into the unfilled pages of that range. The process involves writing the target data into the first range. Before writing the target data into the corresponding data page of the first range, an 8KB page-sized memory space is allocated to temporarily record the verification information. This ensures sufficient space for calculating and storing the verification information during subsequent data writing. Furthermore, since memory read / write speeds are much faster than disk speeds, the efficiency of verification information calculation and processing is improved. Then, the target data is sequentially written into the seven data pages corresponding to the first range. As each data page is written, the verification information in memory is continuously updated until all seven data pages are written. Finally, the verification information for the first range is obtained by performing an XOR operation on the data stored in the seven data pages. Write the first range to persistent storage.
2. The method according to claim 1, characterized in that, Before writing the first range to persistent storage, and after writing the verification information of the first range to the verification page corresponding to the first range, the method further includes: If multiple dirty pages are detected in the first range, the input and output I / O of the multiple dirty pages are merged to generate an I / O request for the first range. The multiple dirty pages are the data pages corresponding to the first range that have been modified. The system responds to the IO request in the data page corresponding to the first range and updates the verification information in the verification page corresponding to the first range according to the IO request.
3. The method according to claim 1, characterized in that, After writing the first range to persistent storage, the method further includes: Upon receiving a read request for the target data, page detection is performed on at least one range stored in the persistent storage to obtain a page detection result for each range; If the page detection result of the first range indicates that there is a damaged page in the data page corresponding to the first range, the damaged page is repaired according to the verification page corresponding to the first range to obtain the data page after repair of the first range. The data page after repair of the first range includes the repaired page of the damaged page and other pages in the data page corresponding to the first range. The system responds to the target data read request based on the data page after repair in the first range, and replaces the damaged page with the repaired page in the persistent storage.
4. The method according to claim 3, characterized in that, After receiving a read request for the target data and performing page detection on at least one range stored in the persistent storage to obtain a page detection result for each range, the method further includes: If the page detection results of the first range indicate that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages are repaired according to the data pages corresponding to the backup first range to obtain the data pages after the first range is repaired. The data pages after the first range is repaired include the repaired pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The backup first range is the storage space in the backup database.
5. The method according to claim 3, characterized in that, When the page detection result within the first range indicates the presence of a damaged page within the corresponding data page range, the damaged page is repaired based on the verification page corresponding to the first range to obtain the repaired data page within the first range, including: If the page detection results of the first range indicate that there is a damaged page in the data pages corresponding to the first range, then detect the consistency status of each page in the first range. If the detection result of the consistency state indicates that at least one page in the first range has a log sequence number (LSN) that is inconsistent with the LSN of other pages, the pages in the first range are restored to the state when they have the same LSN, thus obtaining the first range in a unified state. The damaged page is repaired according to the verification page corresponding to the first range of the unified state to obtain the data page after the first range is repaired. In the data page after the first range is repaired, the LSN of the repaired page is the same as the LSN of other pages.
6. The method according to claim 1, characterized in that, Before writing the verification information of the first range into the verification page corresponding to the first range, the method includes: After writing the target data into the seven data pages corresponding to the first range, the stored data of each data page in the seven data pages corresponding to the first range is obtained; The stored data of each of the seven data pages is XORed to obtain the verification information for the first range.
7. A data processing apparatus, characterized in that, The device includes: The application module is used to apply for storage space for the target data upon receiving a write request for the target data. The storage space includes at least one range, which is the basic unit of database storage allocation. The range includes 7 data pages and 1 verification page, and the verification page is located after the data pages within the same range. Upon receiving a write request sent by the upper-layer executor, the storage system applies for storage space for the target data in 64KB ranges as basic units, ensuring that data within the same range will only come from write requests of the same transaction, i.e., all writes are full-range writes. The first writing module is used to write the verification information of a first range into the verification page corresponding to the first range. The verification information of the first range is the verification information calculated when the target data is written into the data page corresponding to the first range. The verification information is used for redundancy verification. The first range is any range among the at least one range. According to the actual storage space required by the target data, the target data is written through one or more ranges. If the storage space required by the target data is less than or equal to 56KB, the target data is written into the data page corresponding to one range. If the storage space required by the target data is greater than 56KB, the target data is written into the data pages corresponding to multiple ranges. For ranges that are not filled, 0s are filled into the unfilled pages of that range. The process involves writing the target data into the first range. Before writing the target data into the corresponding data page of the first range, an 8KB page-sized memory space is allocated to temporarily record the verification information. This ensures sufficient space for calculating and storing the verification information during subsequent data writing. Furthermore, since memory read / write speeds are much faster than disk speeds, the efficiency of verification information calculation and processing is improved. Then, the target data is sequentially written into the seven data pages corresponding to the first range. As each data page is written, the verification information in memory is continuously updated until all seven data pages are written. Finally, the verification information for the first range is obtained by performing an XOR operation on the data stored in the seven data pages. The second write module is used to write the first range to persistent storage.
8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data processing method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
PM-based database page caching method and system
CN113655949A
Data protection method, solid state disk controller and solid state disk
CN114297737A
