Data processing method and related equipment

Abstract

The invention provides a data processing method and related equipment, and relates to the field of database application, the method comprises the following steps: when a write request of target data is received, a storage space is applied for the target data, the storage space comprises at least one range, and the range comprises a data page and a verification page; the verification information of the first range is written into a verification page corresponding to the first range, the verification information of the first range is the verification information obtained through calculation when the target data is written into a data page corresponding to the first range, the verification information is used for conducting redundancy verification, and the first range is any range in the at least one range; and writing the first range into persistent storage. Whether the data is damaged or not can be detected by using the verification information, so that the reliability of data storage is improved, and the risk of data damage caused by a silent error and the like is reduced; and meanwhile, the problem of write amplification is reduced, the storage space occupied by configuration of additional copies is reduced, and the storage overhead and the data reliability can be effectively balanced.

Description

Technical Field

[0001] The present application relates to the technical field of database application, and in particular to a data processing method and related equipment. Background Art

[0002] In today's digital age, databases carry critical business data, and data security is of vital importance. Any data loss or damage may lead to serious business interruptions and economic losses. Therefore, stable and reliable storage is crucial to databases, which ensures the security and integrity of data.

[0003] There are many types of storage available. Among them, Redundant Array of Independent Disks (RAID) has relatively low construction and maintenance requirements and is widely used in databases. For example, RAID 5 achieves performance improvement and data redundancy through striping and check disks. However, this type of storage system using redundant check has problems such as write amplification, write holes, and long rebuild time. In addition, no matter what type of storage system, the final storage carrier is a hard disk, either a mechanical hard disk or a solid-state hard disk. These storage carriers may cause bit flipping of storage units due to wear and tear, unstable voltage, electromagnetic interference, etc., resulting in silent errors (Silent Data Corruption). This is especially obvious in large-scale storage scenarios. When the database is upgraded from a centralized architecture to a distributed architecture, the total amount of data can even reach the PB level, which poses a huge challenge to the reliability of the storage system during data processing.

[0004] It can be seen that the existing data processing methods have the problem of poor reliability. Summary of the invention

[0005] The main purpose of this application is to provide a data processing method and related equipment, aiming to solve the problem of poor reliability of existing data processing methods.

[0006] To achieve the above objectives, the present application provides a data processing method, the method comprising:

[0007] In case of receiving a write request for target data, applying for storage space for the target data, the storage space includes at least one range, the range is a basic unit of database storage allocation, the range includes 7 data pages and 1 check page, and the check page is located after the data page in the same range;

[0008] Writing the check information of a first range into a check page corresponding to the first range, wherein the check information of the first range is check information calculated when the target data is written into a data page corresponding to the first range, and the check information is used for performing redundancy check, and the first range is any range of the at least one range;

[0009] The first range is written to persistent storage.

[0010] Optionally, before writing the first range into persistent storage and after writing the verification information of the first range into a verification page corresponding to the first range, the method further includes:

[0011] In the case where it is detected that there are multiple dirty pages in the first range, merging the input and output IOs of the multiple dirty pages to generate an IO request for the first range, wherein the multiple dirty pages are modified pages in the data pages corresponding to the first range;

[0012] The IO request is responded to in a data page corresponding to the first range, and verification information in a verification page corresponding to the first range is updated according to the IO request.

[0013] Optionally, after writing the first range to persistent storage, the method further includes:

[0014] When receiving a read request for the target data, performing page detection on the at least one range stored in the persistent storage to obtain a page detection result for each range;

[0015] When the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, the damaged page is repaired according to the check page corresponding to the first range to obtain the repaired data pages of the first range, and the repaired data pages of the first range include the repair page of the damaged page and other pages in the data pages corresponding to the first range;

[0016] A read request for the target data is responded to according to the data page repaired in the first range, and the damaged page is replaced with the repaired page in the persistent storage.

[0017] Optionally, in the case of receiving a read request for the target data, after performing page detection on the at least one range stored in the persistent storage and obtaining a page detection result for each range, the method further includes:

[0018] When the page detection result of the first range indicates that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages are repaired according to the data pages corresponding to the spare first range to obtain the repaired data pages of the first range, and the repaired data pages of the first range include the repair pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The spare first range is the storage space in the spare database.

[0019] Optionally, when the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, repairing the damaged page according to the check page corresponding to the first range to obtain the repaired data page of the first range includes:

[0020] If the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, detecting the consistency status of each page of the first range;

[0021] When the detection result of the consistency state indicates that the log sequence number LSN of at least one page among the pages of the first range is inconsistent with the LSNs of other pages, the pages of the first range are restored to the state when they have the same LSN, so as to obtain the first range in a unified state;

[0022] The damaged page is repaired according to the check page corresponding to the first range of the unified state to obtain the data page after the repair of the first range. In the data page after the repair of the first range, the LSN of the repaired page is the same as the LSN of other pages.

[0023] Optionally, before writing the verification information of the first range into the verification page corresponding to the first range, the method includes:

[0024] After writing the target data into the seven data pages corresponding to the first range, acquiring storage data of each of the seven data pages corresponding to the first range;

[0025] An XOR operation is performed on the storage data of each of the seven data pages to obtain the verification information of the first range.

[0026] In order to solve the above technical problems, an embodiment of the present application further provides a data processing device, the device comprising:

[0027] an application module, configured to apply for storage space for the target data upon receiving a write request for the target data, wherein the storage space includes at least one range, the range being a basic unit of database storage allocation, the range including 7 data pages and 1 check page, and the check page is located after the data page in the same range;

[0028] A first writing module, configured to write verification information of a first range into a verification page corresponding to the first range, wherein the verification information of the first range is verification information calculated when the target data is written into a data page corresponding to the first range, and the verification information is used for redundancy verification, and the first range is any range of the at least one range;

[0029] The second writing module is used to write the first range into persistent storage.

[0030] In order to solve the above technical problems, the embodiment of the present application further provides an electronic device, which adopts the following technical solution:

[0031] The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any data processing method proposed in the embodiments of the present application are implemented.

[0032] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0033] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of any data processing method proposed in the embodiments of the present application are implemented.

[0034] In order to solve the above technical problems, the embodiment of the present application also provides a computer program product, which adopts the following technical solution:

[0035] The computer program product includes computer instructions, which, when executed by a processor, implement the steps of any data processing method proposed in the embodiments of the present application.

[0036] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0037] The present application provides a data processing method, which takes a range as a basic unit of database storage allocation. When target data is written to a data page corresponding to the range, verification information is calculated in real time and stored in the verification page corresponding to the range. During subsequent data reading or other operations, the verification information can be used to detect whether the data is damaged, thereby improving the reliability of data storage, reducing the risk of data damage caused by silent errors and other reasons, and ensuring that the written data can remain complete and correct during the storage process; at the same time, it reduces the write amplification problem and the storage space occupied by configuring additional copies, avoiding excessive waste of storage resources, especially in scenarios such as analytical databases that are sensitive to storage costs and have large data volumes, and can effectively balance storage overhead and data reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is one of the flow charts of the data processing method provided in the embodiment of the present application;

[0040] Figure 2 is a structural diagram of a storage structure provided in an embodiment of the present application;

[0041] Figure 3 It is a schematic diagram of the structure of the range in the storage structure provided in the embodiment of the present application;

[0042] Figure 4 This is the second flowchart of the data processing method provided in the embodiment of the present application;

[0043] Figure 5 This is the third flowchart of the data processing method provided in the embodiment of the present application;

[0044] Figure 6 This is the fourth flow chart of the data processing method provided in the embodiment of the present application;

[0045] Figure 7 This is the fifth flow chart of the data processing method provided in the embodiment of the present application;

[0046] Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0047] Fig. 9 This is a basic structural block diagram of the electronic device of this embodiment. DETAILED DESCRIPTION

[0048] The data processing method provided in the embodiment of the present application is applied to a data processing device. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" in the specification and claims of the present application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned figures are used to distinguish different objects, not to describe a specific order.

[0049] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0050] refer to Figure 1 , shows one of the flow charts of a data processing method proposed in the present application. A data processing method provided in an embodiment of the present application can be applied to a storage system, such as Figure 1 As shown, a data processing method provided in an embodiment of the present application includes the following steps:

[0051] S110, in the case of receiving a write request for target data, applying for storage space for the target data, the storage space including at least one range, the range being a basic unit of database storage allocation, the range including 7 data pages and 1 check page, and the check page is located after the data page in the same range;

[0052] The database of the storage system can adopt a hierarchical management storage structure, and its storage structure can include table space, segment, extent and page. Figure 2 As shown. Tablespace is the highest level logical storage unit, segment is the storage unit in tablespace, range is the component of segment, and range is composed of a group of continuous pages. For example, a page can be fixed to 8KB in size; a range can be composed of 8 pages, and the range is fixed to 64KB in size.

[0053] In this step, when the storage system receives a write request from the upper executor, it applies for storage space for the target data with a range of 64KB as a basic unit, ensuring that data in the same range only comes from write requests of the same transaction, that is, all writes are full-range writes, reducing the write amplification caused by multiple transactions concurrently writing to different pages of the same range. When the storage space includes multiple ranges, any range includes 8 pages, such as Figure 3 As shown in the figure, the first 7 pages in the same range are data pages, and the last page is a check page. The data page is used to store the actual user data, that is, the target data, and the check page is used to store the check data, or check information. As the basic unit of database storage allocation, the range has continuous storage space and ensures 64KB alignment, so that during the data access process, any page can be used to quickly locate other related pages, including the check page, laying the foundation for subsequent data verification and repair operations.

[0054] S120, writing verification information of a first range into a verification page corresponding to the first range, where the verification information of the first range is verification information calculated when the target data is written into a data page corresponding to the first range, and the verification information is used for redundancy verification, and the first range is any range of the at least one range;

[0055] In this step, the target data can be written into one or more ranges according to the actual storage space required by the target data. For example, if the storage space required by the target data is less than or equal to 56KB, the target data can be written into a data page corresponding to a range; if the storage space required by the target data is greater than 56KB, the target data can be written into data pages corresponding to multiple ranges. Exemplarily, if the storage space required by the target data is 80KB, the target data can be written into data pages corresponding to 2 ranges, and for writing into an unfilled range, the unfilled pages of the range can be filled with 0.

[0056] Taking writing the target data into the first range as an example, before writing the target data into the data page corresponding to the first range, first apply for a memory space of 8KB (page size) for temporarily recording the verification information, so that there is enough space to gradually calculate and store the verification information during the subsequent data writing process, and because the reading and writing speed of the memory is much faster than the disk, the efficiency of the calculation and processing of the verification information can be improved; then the target data is sequentially written into the 7 data pages corresponding to the first range, and as the data pages are written one by one, the verification information in the memory is continuously updated until the first 7 data pages are all written, and the verification information of the first range is obtained by performing an XOR operation on the data stored in the 7 data pages;

[0057] When the verification page is ready to be written, that is, after the target data is written to the 7 data pages corresponding to the first range, the verification information finally obtained in the memory is written to the verification page corresponding to the first range, completing the process of writing the data of the entire first range and generating the verification information. If an abnormality occurs during this process (such as system failure, power failure, etc.), since the verification information is updated in real time and temporarily stored in the memory, it can be processed accordingly according to the written data page and the verification information status in the memory during recovery (such as data recovery through write-ahead logging (WAL) and verification information), further ensuring the reliability and consistency of data writing. This method of synchronously calculating the verification information ensures that the verification information can reflect the integrity and accuracy of the currently written data page in real time, and does not require additional steps to recalculate the verification information after all data pages are written, saving computing resources and time.

[0058] S130. Write the first range into persistent storage.

[0059] In this step, by writing the first range storing the target data into persistent storage, the data will not be lost even if the system fails (such as power outage, software crash, etc.), the server restarts or other abnormal situations occur, because the persistent storage device (such as a hard disk) can retain data in a non-volatile storage medium. This improves the long-term stability and reliability of data in the storage system and meets the business needs for long-term data preservation and anytime access.

[0060] In addition, the data written to the persistent storage can be used by subsequent data query, update, delete and other operations. When there is a data read request, the system can quickly read the data from the persistent storage and return it to the user; when performing data update operations, the original data in the persistent storage can be used as a basis for modification; and for data deletion operations, the system also needs to mark or actually delete the corresponding data in the persistent storage. Writing ranges to persistent storage provides a data foundation for the normal operation and data management of the entire database, enabling the database system to efficiently handle various business operations.

[0061] Compared with the related art that is equipped with a large amount of redundant information or adopts a complex distributed redundant storage scheme, in the embodiment of the present application, the range is used as the basic unit of database storage allocation. When the target data is written to the data page corresponding to the range, the verification information is calculated in real time and stored in the verification page corresponding to the range. During subsequent data reading or other operations, the verification information can be used to detect whether the data is damaged, thereby improving the reliability of data storage, reducing the risk of data corruption caused by silent errors and other reasons, and ensuring that the written data can remain complete and correct during the storage process; at the same time, it reduces the write amplification problem and the storage space occupied by configuring additional copies, avoiding excessive waste of storage resources, especially in scenarios such as analytical databases that are sensitive to storage costs and have large data volumes, and can effectively balance storage overhead and data reliability.

[0062] Optionally, before writing the verification information of the first range into the verification page corresponding to the first range, the method includes:

[0063] After writing the target data into the seven data pages corresponding to the first range, acquiring storage data of each of the seven data pages corresponding to the first range;

[0064] An XOR operation is performed on the storage data of each of the seven data pages to obtain the verification information of the first range.

[0065] In this embodiment, the target data is written into the 7 data pages corresponding to the first range. For example, the storage data of the first data page is AAA, the storage data of the second data page is BBB, the storage data of the third data page is CCC, and similarly, the storage data of the seventh data page is GGG. Then the check information of the first range can be expressed as HHH=AAA⊕BBB⊕CCC⊕…⊕GGG.

[0066] Before writing the target data into the first range, apply for 8KB of memory space for temporarily recording verification information. For example, when writing CCC into the third data page, the temporary verification information between the first data page and the second data page (i.e., AAA⊕BBB) can be calculated at the same time, and the result can be temporarily stored in the memory space. In this way, when writing CCC into the third data page, the temporary verification information can be quickly read to further calculate the temporary verification information between the first data page, the second data page, and the third data page (i.e., AAA⊕BBB⊕CCC). Similarly, as the data pages are written one by one, the temporary verification information in the memory is continuously updated until all 7 data pages are written. The verification information of the first range is obtained by performing an XOR operation on the data stored in the 7 data pages.

[0067] The corresponding CRC value in each page can be determined based on a cyclic redundancy check (CRC) algorithm. The CRC algorithm regards the data to be checked as a polynomial coefficient sequence, divides it with a predefined generator polynomial, and uses the remainder as the check code. For example, assuming that the CRC-32 algorithm is used, its generator polynomial can be expressed as:

[0068] G(x)=x 32 +x 26 +x 23 +x 22 +x 16 +x 12 +x 11 +x 10 +x 8 +x 7 +x 5 +x 4 +x 2 +x+2;

[0069] In this way, during the data reading process, after reading a page from persistent storage, the checksum information of the page is recalculated first. The page data is again calculated according to the above polynomial processing and division method to obtain new checksum information, which is set as CRC new , and then compare it with the verification information recorded on the verification page. new Not equal to CRC recorded If the data page is damaged, the data can be repaired by using the checksum information of other undamaged data pages and the checksum page.

[0070] Through the above series of operations based on the CRC algorithm, reliable verification information is generated when data is written, and whether the page is damaged can be detected when data is read, and damaged pages can be repaired to a certain extent, thereby improving the reliability of database storage and data integrity. In actual applications, the specific implementation and optimization of the CRC algorithm will be adjusted according to the performance requirements and hardware characteristics of the database system to ensure that the verification and repair tasks are completed efficiently and accurately.

[0071] Optionally, before writing the first range into persistent storage and after writing the verification information of the first range into a verification page corresponding to the first range, the method further includes:

[0072] In the case where it is detected that there are multiple dirty pages in the first range, merging the input and output IOs of the multiple dirty pages to generate an IO request for the first range, wherein the multiple dirty pages are modified pages in the data pages corresponding to the first range;

[0073] The IO request is responded to in a data page corresponding to the first range, and verification information in a verification page corresponding to the first range is updated according to the IO request.

[0074] Among them, the database system can identify dirty pages through a marking mechanism during operation. For example, when a data page is modified in the memory buffer, it will be marked as a "dirty page". The background flushing thread will regularly scan the memory buffer to find these marked "dirty pages" and collect them for flushing operations. In traditional storage systems, if each dirty page performs an IO operation separately, the disk head needs to move frequently to different locations for reading and writing, which will cause a lot of disk seek time and seriously affect IO performance.

[0075] In this embodiment, when it is detected that there are multiple dirty pages in the first range, the IO of these dirty pages are merged to generate an IO request. Through IO merging, the read and write operations of multiple dirty pages in consecutive or similar positions are merged into one request, so that the disk head can complete the operation of these pages in a shorter time, reducing the number and time of disk seeks, thereby significantly improving the efficiency of IO operations, especially in the scenario of processing large amounts of data writing or updating, this performance improvement is more obvious.

[0076] In the data page corresponding to the first range, the IO request is responded to and the verification information in the verification page is updated, ensuring that the verification information is always consistent with the actual state of the data page. With the modification of the data page and the progress of the IO operation, the timely update of the verification information can accurately reflect the latest state of the data, ensuring the timeliness and accuracy of the verification information. In the data update process, by timely updating the verification information, it is more reliable in terms of transaction processing and data consistency assurance. For example, when multiple data pages are modified in one transaction, IO merging and verification information updating ensure that the atomicity and consistency of these modification operations are reflected at the storage level. If the system fails before the transaction is committed, the recovery process can determine the correct state of the data based on accurate verification information and transaction logs, preventing data inconsistency caused by the inconsistency between the verification information and the actual state of the data, further enhancing the data consistency assurance capability of the database system in a complex operating environment. In addition, for each collected dirty page, the background flushing thread will calculate its CRC checksum, enhancing the reliability and detectability of the data and ensuring the stable operation of the storage system.

[0077] In one example, if Figure 4 As shown, the data writing operation process is as follows:

[0078] ① Apply for storage space for the data to be written in ranges to ensure that data in the same range only comes from write requests of the same transaction, that is, all writes are full-range writes to avoid write amplification caused by multiple transactions concurrently writing to different pages in the same range;

[0079] ② Write the actual data. Before writing the range, apply for a memory space of 8KB (page size) and temporarily record the verification information. Each time a data page is written, the verification information is calculated synchronously.

[0080] ③Write the verification data. After writing the first 7 data pages, write the generated verification information to the verification page. If a range is not full, fill the remaining space with 0 and then calculate the verification information;

[0081] ④ The background flushing thread collects "dirty pages" and tries to flush them. This process will merge IO and calculate the CRC checksum for each page;

[0082] ⑤Write to persistent storage.

[0083] In this way, the range is used as the basic unit of database storage allocation. When the target data is written to the data page corresponding to the range, the verification information is calculated in real time and stored in the verification page corresponding to the range. During subsequent data reading or other operations, the verification information can be used to detect whether the data is damaged, thereby improving the reliability of data storage and reducing the risk of data corruption due to silent errors and other reasons, ensuring that the written data can remain complete and correct during the storage process; at the same time, it reduces the write amplification problem and the storage space occupied by configuring additional copies, avoiding excessive waste of storage resources. Especially in scenarios such as analytical databases that are sensitive to storage costs and have large data volumes, it can effectively balance storage overhead and data reliability.

[0084] Optionally, after writing the first range to persistent storage, the method further includes:

[0085] When receiving a read request for the target data, performing page detection on the at least one range stored in the persistent storage to obtain a page detection result for each range;

[0086] When the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, the damaged page is repaired according to the check page corresponding to the first range to obtain the repaired data pages of the first range, and the repaired data pages of the first range include the repair page of the damaged page and other pages in the data pages corresponding to the first range;

[0087] A read request for the target data is responded to according to the data page repaired in the first range, and the damaged page is replaced with the repaired page in the persistent storage.

[0088] In this embodiment, when reading the target data, the page detection is performed on the range stored in the persistent storage to obtain the page detection results of each range. If a damaged page is found in the page, the system can quickly locate the verification page in the range and try to repair it according to the verification information. Take the first range as an example:

[0089] When the page detection result of the first range indicates that there is a damaged page in the data page corresponding to the first range, the damaged page can be repaired according to the check page corresponding to the first range to obtain the repaired data page of the first range. In this way, the damaged page is repaired using the check page in the same range, which reduces the overhead of obtaining data from remote backup or other storage locations for repair. The check page and the data page are closely related in storage structure and are located in the same range. The delay in reading the check page is relatively low, and the information required for repair can be obtained quickly. Therefore, the repair process can be completed quickly locally, reducing the delay in data reading and improving the response speed of read requests. Especially for frequent data reading operations, this local fast repair mechanism can significantly improve the performance of the system and meet application scenarios with high requirements for data reading performance.

[0090] After the damaged page is repaired, the read request is responded to directly based on the repaired data page, and the damaged page is replaced in the persistent storage, so that subsequent reads of the data in the range do not need to be repaired again. This optimized reading process reduces repeated detection and repair work and improves the efficiency of data reading. Over time, the system is able to maintain the good state of the data, further improving the overall performance of data reading and providing users with a faster and smoother data access experience.

[0091] Optionally, in the case of receiving a read request for the target data, after performing page detection on the at least one range stored in the persistent storage and obtaining a page detection result for each range, the method further includes:

[0092] When the page detection result of the first range indicates that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages are repaired according to the data pages corresponding to the spare first range to obtain the repaired data pages of the first range, and the repaired data pages of the first range include the repair pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The spare first range is the storage space in the spare database.

[0093] In this embodiment, Figure 5 As shown, the data in the standby database (i.e., standby database) is usually a backup or copy of the main database (i.e., main database), and its data status reflects the correct status of the main database at a certain point in time to a certain extent. By using the data pages of the standby first range for repair, a relatively complete and accurate repair source can be obtained, thereby improving the success rate of repairing multiple damaged pages. Even if some data in the main database has serious errors due to various reasons, the existence of the standby database provides a reliable alternative for data recovery, increases the possibility that data can be correctly recovered, and ensures the stability and reliability of data in the database system.

[0094] In the case where the page detection result of the first range indicates that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages can be repaired according to the data pages corresponding to the standby first range to obtain the repaired data pages of the first range. The data in the standby database can be more directly used to repair the damaged pages, reducing the time and complexity of data recovery. In this way, even if the main database faces more serious storage problems, the system can still switch to the standby database resources to repair the data, maintain the normal operation of the database service, and avoid complete system crash or service interruption due to data corruption. It has stronger resilience in the face of various abnormal situations such as hardware failures, software errors or external interference, can continue to provide support for the business, reduce the risk of business losses caused by system failures, and enhance the survivability and reliability of the system in complex environments.

[0095] It should be understood that the embodiment of the present application takes the first range as an example for exemplary description, and the same technical effect can be achieved for other ranges in the storage space. To avoid repetition, it will not be described here.

[0096] In one example, if Figure 6 As shown, the data reading operation process is as follows:

[0097] ① Respond to data query requests sent by the upper-level executor;

[0098] ② Apply to the file system to obtain the corresponding data and check the CRC to confirm whether the page is damaged;

[0099] ③ Repair the damaged page and check other pages in the range where the damaged page is located, including the verification page. If other pages are not damaged, quickly repair them through the verification page. If more pages are damaged and there is a backup database, access the backup database and pull the corresponding pages for regular repairs.

[0100] ④Write the repaired page back to persistent storage;

[0101] ⑤After repairing the damaged page, there is no need to verify the page anymore, only load the data page into the data buffer;

[0102] ⑥Return target data and respond to access requests.

[0103] This application combines the CRC checksum mechanism of the page to realize automatic repair of damaged pages. After reading the page from the persistent storage, first recalculate the CRC of the page and compare it with the CRC recorded in the page. If the two are inconsistent, it means that the data is damaged. Then read other pages in the range of the page number and try to recover it. Even if multiple pages are damaged, the data can be restored through the backup mechanism. The reliability of data reading is improved.

[0104] Optionally, when the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, repairing the damaged page according to the check page corresponding to the first range to obtain the repaired data page of the first range includes:

[0105] If the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, detecting the consistency status of each page of the first range;

[0106] When the detection result of the consistency state indicates that the log sequence number LSN of at least one page among the pages of the first range is inconsistent with the LSNs of other pages, the pages of the first range are restored to the state when they have the same LSN, so as to obtain the first range in a unified state;

[0107] The damaged page is repaired according to the check page corresponding to the first range of the unified state to obtain the data page after the repair of the first range. In the data page after the repair of the first range, the LSN of the repaired page is the same as the LSN of other pages.

[0108] In this embodiment, the WAL log mechanism of the database can be used to perform transaction control on the update and modification of the data page and the verification page to ensure that the modification is either completed or rolled back, so as to always maintain a consistent state. This solves the write hole problem caused by system crashes, abnormal power outages, and other failures.

[0109] When the page detection result of the first range indicates that there is a damaged page in the data page corresponding to the first range, the consistency status of each page of the first range is detected; when the consistency status detection result indicates that there is at least one page in the first range whose log sequence number (LSN) is inconsistent with the LSN of other pages, the pages of the first range can be restored to the state with the same LSN with the help of WAL log to obtain the first range of unified state, wherein the unified state can be unified restoration to the latest version or unified restoration to the initial version. Then, the damaged page is repaired according to the check page corresponding to the first range of unified state to obtain the repaired data page of the first range.

[0110] Specifically, Figure 7 As shown in the figure, ① During the database recovery phase, it is found that a page is damaged, and the other pages in the range Extent are recorded and restored; ② Branch 1: The transaction is successfully committed, and other pages are uniformly restored to the latest version through the WAL log; ③ The damaged page is restored using the recovery-consistent check page and data page; ④ The repaired data page is written back to the persistent storage; ⑤ Branch 2: The transaction is abnormally rolled back, and other pages are uniformly restored to the initial version through the WAL log; ⑥ The damaged page is restored using the recovery-consistent check page and data page; ⑦ The repaired data page is written back to the persistent storage.

[0111] In this way, the LSN consistency of all pages in the same range can be ensured, thereby improving the consistency and reliability of data. The damaged page can be effectively repaired to keep it consistent with the status of other pages, reducing data errors or inconsistencies that may be caused by LSN inconsistencies. In the event of a single page being damaged, the database can be restored to a consistent state so that subsequent data access and operations can proceed normally, improving the availability and reliability of the database. At the same time, consistency detection and repair based on LSN also help to more accurately locate and solve problems during database recovery or troubleshooting.

[0112] Further references Figure 8 , as a response to the above Figure 8 In order to realize the method shown in the figure, the present application provides an embodiment of a data processing device 800, which is similar to Figure 8 Corresponding to the method embodiment shown, the device can be specifically applied in a server.

[0113] An embodiment of the present application provides a data processing device, the data processing device 800 comprising:

[0114] The application module 810 is used to apply for storage space for the target data when receiving a write request for the target data, wherein the storage space includes at least one range, the range is a basic unit of database storage allocation, the range includes 7 data pages and 1 check page, and the check page is located after the data page in the same range;

[0115] A first writing module 820 is configured to write verification information of a first range into a verification page corresponding to the first range, where the verification information of the first range is verification information calculated when the target data is written into a data page corresponding to the first range, and the verification information is used for redundancy verification, and the first range is any range of the at least one range;

[0116] The second writing module 830 is used to write the first range into persistent storage.

[0117] Optionally, the device further comprises:

[0118] a processing module, configured to merge input and output IOs of the multiple dirty pages to generate an IO request for the first range when it is detected that there are multiple dirty pages in the first range, wherein the multiple dirty pages are modified pages in the data pages corresponding to the first range;

[0119] An update module is used to respond to the IO request in the data page corresponding to the first range, and update the verification information in the verification page corresponding to the first range according to the IO request.

[0120] Optionally, the device further comprises:

[0121] A detection module, configured to, upon receiving a read request for the target data, perform page detection on the at least one range stored in the persistent storage to obtain a page detection result for each range;

[0122] a first repair module, configured to repair a damaged page according to a check page corresponding to the first range to obtain a repaired data page of the first range when a page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, wherein the repaired data page of the first range includes a repair page of the damaged page and other pages in the data pages corresponding to the first range;

[0123] A response module is used to respond to the read request of the target data according to the data page repaired in the first range, and replace the damaged page with the repaired page in the persistent storage.

[0124] Optionally, the device further comprises:

[0125] A second repair module is used to repair the multiple damaged pages according to the data pages corresponding to the spare first range to obtain the repaired data pages of the first range when the page detection result of the first range indicates that there are multiple damaged pages in the data pages corresponding to the first range. The repaired data pages of the first range include the repair pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The spare first range is the storage space in the spare database.

[0126] Optionally, the first repair module is specifically used to:

[0127] If the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, detecting the consistency status of each page of the first range;

[0128] When the detection result of the consistency state indicates that the log sequence number LSN of at least one page among the pages of the first range is inconsistent with the LSNs of other pages, the pages of the first range are restored to the state when they have the same LSN, so as to obtain the first range in a unified state;

[0129] The damaged page is repaired according to the check page corresponding to the first range of the unified state to obtain the data page after the repair of the first range. In the data page after the repair of the first range, the LSN of the repaired page is the same as the LSN of other pages.

[0130] Optionally, the device further comprises:

[0131] an acquisition module, configured to acquire storage data of each of the seven data pages corresponding to the first range after writing the target data into the seven data pages corresponding to the first range;

[0132] The calculation module is used to perform an XOR operation on the storage data of each of the seven data pages to obtain the verification information of the first range.

[0133] To solve the above technical problems, the present application also provides an electronic device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the electronic device of this embodiment.

[0134] The electronic device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected and communicated through a system bus. It should be noted that the figure only shows an electronic device 9 with components 91-93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0135] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0136] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. In other embodiments, the memory 91 can also be an external storage device of the electronic device 9, such as a plug-in hard disk equipped on the electronic device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 91 can also include both the internal storage unit of the electronic device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the electronic device 9, such as the program code of the data processing method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0137] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the electronic device 9. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of the data processing method.

[0138] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the electronic device 9 and other electronic devices.

[0139] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a program of the data processing method, and the program of the data processing method can be executed by at least one processor so that the at least one processor performs the steps of the data processing method as described above.

[0140] The present application also provides another implementation, namely, a computer program product, wherein the computer program product includes computer instructions, and when the computer instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware online platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0142] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0143] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data processing method, characterized in that: The method comprises: In case of receiving a write request for target data, applying for storage space for the target data, the storage space includes at least one range, the range is a basic unit of database storage allocation, the range includes 7 data pages and 1 check page, and the check page is located after the data page in the same range; Writing the check information of a first range into a check page corresponding to the first range, wherein the check information of the first range is check information calculated when the target data is written into a data page corresponding to the first range, and the check information is used for performing redundancy check, and the first range is any range of the at least one range; The first range is written to persistent storage.

2. The method according to claim 1, characterized in that Before writing the first range into persistent storage, and after writing the verification information of the first range into a verification page corresponding to the first range, the method further includes: In the case where it is detected that there are multiple dirty pages in the first range, merging the input and output IOs of the multiple dirty pages to generate an IO request for the first range, wherein the multiple dirty pages are modified pages in the data pages corresponding to the first range; The IO request is responded to in a data page corresponding to the first range, and verification information in a verification page corresponding to the first range is updated according to the IO request.

3. The method according to claim 1, characterized in that After writing the first range to persistent storage, the method further includes: When receiving a read request for the target data, performing page detection on the at least one range stored in the persistent storage to obtain a page detection result for each range; When the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, the damaged page is repaired according to the check page corresponding to the first range to obtain the repaired data pages of the first range, and the repaired data pages of the first range include the repair page of the damaged page and other pages in the data pages corresponding to the first range; A read request for the target data is responded to according to the data page repaired in the first range, and the damaged page is replaced with the repaired page in the persistent storage.

4. The method according to claim 3, characterized in that In the case where a read request for the target data is received, after performing page detection on the at least one range stored in the persistent storage and obtaining a page detection result for each range, the method further includes: When the page detection result of the first range indicates that there are multiple damaged pages in the data pages corresponding to the first range, the multiple damaged pages are repaired according to the data pages corresponding to the spare first range to obtain the repaired data pages of the first range, and the repaired data pages of the first range include the repair pages of the multiple damaged pages and other pages in the data pages corresponding to the first range. The spare first range is the storage space in the spare database.

5. The method according to claim 3, characterized in that: When the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, repairing the damaged page according to the check page corresponding to the first range to obtain the repaired data page of the first range includes: If the page detection result of the first range indicates that there is a damaged page in the data pages corresponding to the first range, detecting the consistency status of each page of the first range; When the detection result of the consistency state indicates that the log sequence number LSN of at least one page among the pages of the first range is inconsistent with the LSNs of other pages, the pages of the first range are restored to the state when they have the same LSN, so as to obtain the first range in a unified state; The damaged page is repaired according to the check page corresponding to the first range of the unified state to obtain the data page after the repair of the first range. In the data page after the repair of the first range, the LSN of the repaired page is the same as the LSN of other pages.

6. The method according to claim 1, characterized in that Before writing the verification information of the first range into the verification page corresponding to the first range, the method includes: After writing the target data into the seven data pages corresponding to the first range, acquiring storage data of each of the seven data pages corresponding to the first range; An XOR operation is performed on the storage data of each of the seven data pages to obtain the verification information of the first range.

7. A data processing device, characterized in that: The device comprises: an application module, configured to apply for storage space for the target data upon receiving a write request for the target data, wherein the storage space includes at least one range, the range being a basic unit of database storage allocation, the range including 7 data pages and 1 check page, and the check page is located after the data page in the same range; A first writing module, configured to write verification information of a first range into a verification page corresponding to the first range, wherein the verification information of the first range is verification information calculated when the target data is written into a data page corresponding to the first range, and the verification information is used for redundancy verification, and the first range is any range of the at least one range; The second writing module is used to write the first range into persistent storage.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 6.

Images