Snapshot-based data corruption detection

Abstract

The embodiments described herein detect data corruption in a distributed dataset system. For example, one system includes multiple nodes for processing queries about a distributed dataset comprising multiple storage segments. Write transactions resulting from queries about specific storage segments are logged in a log describing modifications to the storage segments. A log service provides the log records to a data server that manages a portion of the distributed dataset, including the storage segments, and executes write transactions about the storage segments. For redundancy, the data server has multiple replicas, each managing a corresponding replica of the portion of the distributed dataset it manages. For backup purposes, snapshots of the replicas are periodically generated. To determine data corruption, a snapshot of one replica is cross-validated with snapshots of another replica.

Description

Background Technology

[0001] A distributed database is a database consisting of an interconnected collection of storage and computing units distributed across one or more computer networks and / or the Internet. A Distributed Database Management System (DDBMS) manages distributed databases and provides mechanisms to make the distributed nature of the system transparent to users. In these systems, data is intentionally distributed across multiple nodes to allow for the elasticity of computing or storage resources for easy scaling. Distributed databases offer numerous advantages, such as, but not limited to, modular development, higher throughput, high concurrency, and high scalability. However, distributed databases also require increased coordination between different components. Therefore, the probability of failure increases due to the increased number of components required to implement the system. This increases the likelihood of data corruption. Although the entire system is designed to prevent data corruption and / or recover to a consistent state after random failures, in reality, data corruption can still occur due to various software errors, faults, etc. If data corruption is not detected, its root cause addressed, and mitigated in a timely manner, it can severely impact the integrity of query results and the availability of the database. Summary of the Invention

[0002] This summary is provided to introduce a set of concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] The methods, systems, apparatuses, and computer-readable storage media described herein are used to detect data corruption in a distributed dataset system. For example, a system according to an embodiment includes one or more computing nodes for processing queries about a distributed dataset (e.g., a database) comprising multiple storage segments (e.g., pages). Write transactions resulting from queries about specific storage segments are logged in log records describing modifications to the storage segments. Each log record is identified by a log sequence number associated with it. A log service provides the log records to a data server that manages a portion of the distributed dataset that includes the storage segments, and the data server executes write transactions about the storage segments specified by the log records. For redundancy purposes, the data server has one or more replicas that manage corresponding copies of the portions of the distributed dataset it manages. For backup purposes, snapshots of each replica are periodically generated. To determine data corruption, a snapshot of one replica is automatically cross-validated with a snapshot of another replica. For example, the log sequence number of a storage segment of one replica is compared with the log sequence number of the corresponding storage segment of another replica. If a log sequence mismatch is detected, this means that there is an inconsistency between the log records applied by one replica data server and the log records applied by the other replica data server. Therefore, the two page server replicas applied different sets of write transactions, resulting in inconsistent snapshots. Upon detecting such data corruption, an alert will be automatically generated to notify the administrator of the corruption.

[0004] Other features and advantages, as well as the structure and operation of various exemplary embodiments, will now be described in detail with reference to the accompanying drawings. Note that the exemplary implementations are not limited to the specific embodiments described herein. Such exemplary embodiments are presented herein for illustrative purposes only. Additional implementations based on the teachings contained herein will be readily apparent to those skilled in the art. Attached Figure Description

[0005] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate exemplary embodiments of the present application and, together with the specification, further serve to explain the principles of the exemplary embodiments and enable those skilled in the art to make and use the exemplary embodiments.

[0006] Figure 1 A block diagram of a system for detecting data corruption in a distributed dataset system, according to an example embodiment, is shown.

[0007] Figure 2 The logging is described according to an example embodiment.

[0008] Figure 3 A logging block according to an example embodiment is depicted.

[0009] Figure 4A system for determining snapshot pairs according to an example embodiment is described.

[0010] Figure 5 A system for cross-validating snapshot pairs to detect data corruption, according to an example embodiment, is described.

[0011] Figure 6 A flowchart is shown of a method for detecting data corruption based on cross-validation snapshots according to an example embodiment.

[0012] Figure 7 A flowchart of a method for determining snapshot pairs according to an example embodiment is shown.

[0013] Figure 8 A flowchart illustrating a method for determining data corruption based on a storage segment identifier mismatch, according to an example embodiment, is shown.

[0014] Figure 9 This is a block diagram of an example processor-based computer system that can be used to implement various embodiments.

[0015] The features and advantages of the implementation described herein will become more apparent when viewed in conjunction with the accompanying drawings, in which the same reference numerals identify corresponding elements from beginning to end. In the drawings, similar reference numerals generally denote identical, functionally similar, and / or structurally similar elements. The first appearance of an element in the drawing is indicated by the leftmost (or more) numerals in the corresponding reference numerals. Detailed Implementation

[0016] I. Introduction

[0017] This specification and accompanying drawings disclose numerous exemplary implementations. The scope of this application is not limited to the disclosed implementations, but also includes combinations of disclosed implementations and modifications thereof. In this specification, references to "an implementation," "implementation," "exemplary embodiment," "exemplary implementation," etc., indicate that the described implementation may include a specific feature, structure, or characteristic, but not every implementation necessarily includes that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same implementation. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an implementation, it is proposed that, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other implementations is within the knowledge of those skilled in the art.

[0018] In this discussion, unless otherwise stated, any modification of the conditional or relational adjectives of one or more features of embodiments of this disclosure, such as “substantially” and “approximately”, should be understood as defining the condition or feature as being within an operationally acceptable tolerance for the intended application of the implementation.

[0019] Furthermore, it should be understood that the spatial descriptions used in this article (e.g., "above", "below", "on top", "left", "right", "down", "top", "bottom", "vertical", "horizontal", etc.) are for illustrative purposes only, and the actual implementation of the structures described in this article can be arranged in space in any direction or manner.

[0020] Many example embodiments are described below. Please note that any section / subsection headings provided herein are not limiting. Implementations are described throughout this document, and any type of implementation may be included in any section / subsection. Furthermore, implementations disclosed in any part / subsection may be combined in any way with any other implementations described in the same part / subsection and / or different parts / subsections.

[0021] II. Example Implementation

[0022] The embodiments described herein relate to detecting data corruption in a distributed dataset system. For example, one system according to an embodiment includes one or more compute nodes for processing queries about a distributed dataset (e.g., a database) comprising multiple storage segments (e.g., pages). Write transactions resulting from queries about specific storage segments are logged in log records describing modifications to the storage segments. Each log record is identified by a log sequence number associated with it. A log service provides the log records to a data server that manages a portion of the distributed dataset including the storage segments, and the data server executes write transactions about the storage segments specified by the log records. For redundancy purposes, the data server has one or more replicas, each managing a corresponding replica of the portion of the distributed dataset it manages. For backup purposes, snapshots of each replica are periodically generated. To determine data corruption, a snapshot of one replica needs to be cross-validated with a snapshot of another replica. For example, the log sequence number of a storage segment of one replica is compared with the log sequence number of the corresponding storage segment of another replica. If a log sequence mismatch is detected, it means that there is an inconsistency between the log records applied by one replica data server and the log records applied by the other replica data server. Therefore, the two page server replicas applied different sets of write transactions, resulting in inconsistent snapshots. In the event of such data corruption, an alert will be automatically generated to notify the administrator of the existence of the corruption.

[0023] Traditionally, corruption is detected when a data server attempts to apply logging to execute a write transaction about a segment of storage it manages. During the write transaction, the data server analyzes fields in the log record (also known as the "previous page log sequence number field," which indicates the log sequence number that should be included in the header of the segment to be modified). If this log sequence number is not included in the header, data corruption is determined. One problem with this approach is that the data server may take some time to apply the logging, especially if the segment is only read. In some cases, it may take weeks for the data server to update the segment. Therefore, if a segment is not modified for a long time, corruption remains undetected for a relatively long period. During this time, applications will operate on the corrupted data, thus returning invalid results.

[0024] By performing cross-validation between replica snapshots, the embodiments described herein advantageously detect data corruption faster than conventional techniques, enabling quicker action to be taken against corruption (since one no longer needs to wait for the data to be modified to detect corruption). This limits the time window for applications to operate on corrupted data. Therefore, the integrity of the distributed dataset system is advantageously improved, and applications accessing the dataset are more likely to operate on the correct data. Furthermore, because such data corruption is often the result of software errors, the aforementioned techniques can be implemented during the development and testing of new features of the distributed dataset system. This allows data corruption caused by these new features to be detected, and the code causing this data corruption can be fixed before these features are rolled out to end users.

[0025] For example, Figure 1 A block diagram of a system 100 for detecting data corruption in a distributed dataset system, according to an example embodiment, is shown. Figure 1As shown, system 100 includes one or more primary compute nodes 102, secondary compute nodes 104A-104N, a persistent log 106, a log service 108, data servers 110A-110N, replica data servers 112A-112N, datasets 114A-114N, replica datasets 116A-116N, dataset snapshots 118A-118N, replica dataset snapshots 120A-120N, and a snapshot service 124. System 100 can be implemented in a cloud-based environment, although the embodiments described herein are not limited thereto. Each of the (multiple) primary compute nodes and secondary compute nodes 104A-104N may include one or more physical computing devices or virtual machines. Examples of physical computing devices include, but are not limited to, server computers, server systems, etc. Data sets 114A-114N, replica data sets 116A-116N, dataset snapshots 118A-118N, and replica dataset snapshots 120A-120N can be stored via storage nodes, each of which includes multiple physical storage disks (e.g., hard disk drives, solid-state drives, etc.), which can be accessed by the corresponding data servers of data servers 110A-110N and replica data servers 112A-112N, for example, via a network.

[0026] like Figure 1 As shown, (multiple) primary compute nodes 102 are configured to execute one or more applications, such as database application 122. Database application 122 can be any database server application, including but not limited to those published by Microsoft Corporation in Redmond, Washington. Azure SQL Database TM Database application 122 is configured to execute statements that create, modify, and delete (multiple) data files based on incoming queries. Queries can be initiated by the user or automatically generated by one or more background processes. Such queries can be configured to add (multiple) data files, merge (multiple) data files into a larger data file, reorganize (or re-cluster) (multiple) data files within a specific data file set (e.g., based on commonalities among (multiple) data files), delete (multiple) data files (e.g., via a garbage collection process that periodically removes unwanted or outdated data), etc.

[0027] Each of the secondary compute nodes 104A-104N can also be configured to execute an instance of database application 122. If one or more of the primary compute nodes(s) 102 fail and cannot be effectively recovered, one or more secondary compute nodes 104A-104N can be utilized. In this case, one or more of the secondary compute nodes 104A-104N are promoted to primary compute nodes and / or new secondary compute nodes can be added to replace the promoted secondary compute nodes. Note that secondary compute nodes can be added or removed regardless of whether a primary compute node fails.

[0028] In each embodiment, each of the datasets 114A-114N includes a database, which can be a very large dataset, for example, for "big data" analytics and / or data warehousing. In the embodiments, it is anticipated that one or more of the datasets 114A-114N are on the order of petabytes or larger. The datasets 114A-114N can be logically represented as structured relational data, organized as rows of a table with columns for the data. The data in the datasets 114A-114N can be stored in one or more data files. Each data file can be logically divided into storage segments, such as pages. The size of a page can be 8 kilobytes, although the embodiments described herein are not limited to this.

[0029] Data servers 110A-110N are configured to provide storage segments of datasets 114A-114N to (multiple) primary compute nodes 102 and update the storage segments of datasets 114A-114N when transactions update data. Each of data servers 110A-110N is responsible for a subset of storage segments in the database. For example, data server 110A is responsible for the storage segment of dataset 114A, data server 110B is responsible for the storage segment of dataset 114B, data server 110C is responsible for the storage segment of dataset 114C, and data server 110N is responsible for the storage segment of dataset 114N. Depending on the embodiment where storage segments correspond to pages, each of data servers 110A-110N may be referred to as a page server.

[0030] System 100 also includes one or more copies for each of the data servers 110A-110N, which are maintained for redundancy and availability. For example, as Figure 1As shown, system 100 includes one or more replica data servers 112A (which are replicas of data server 110A), one or more replica data servers 112B (which are replicas of data server 110B), one or more replica data servers 112C (which are replicas of data server 110C), and one or more replica data servers 112N (which are replicas of data server 110N). Each of the replica data servers 112A-112N is associated with a replica dataset. For example, replica data servers 112A are associated with replica dataset 116A (which is a replica of dataset 114A), replica data servers 112B are associated with replica dataset 116B (which is a replica of dataset 114B), replica data servers 112C are associated with replica dataset 116C (which is a replica of dataset 114C), and replica data servers 112D are associated with replica dataset 116D (which is a replica of dataset 114D).

[0031] When a write transaction, such as creating or modifying a data file, is executed, the primary compute nodes 102 log the operation (shown as 103) to a permanent log 106. The permanent log 106 is used to recover data in the event of a system failure. For example, the database application 122 of the primary compute nodes 102 may first read the storage segment from the data servers 110A-110N responsible for it. This ensures that a copy of the storage segment is brought into the cache of the primary compute nodes 102. The storage segment may have been previously read by the primary compute nodes 102; in this case, the storage segment will already be in the cache of the primary compute nodes 102. To obtain the storage segment, the database application 122 of the primary compute nodes 102 may issue a read operation to the responsible data server (e.g., data server 110B), and the data server provides data (shown as data 101) to the primary compute nodes 102.

[0032] Then, the database application 116 on (multiple) primary compute nodes 102 can write read data stored in the cache. The database application 116 also creates a log record for this write operation. The log record includes incremental changes made or to be made as part of the write operation. The log record also includes a log sequence number and a storage segment identifier. For example, Figure 2 A log recording 200 according to an example embodiment is depicted. For example... Figure 2As shown, log record 200 includes a log sequence number 202, a description 204 of incremental changes made via a write transaction, and a segment identifier 206. The log sequence number 202 uniquely identifies a log record (e.g., log record 200) and indicates its position within a log (e.g., permanent log 106). The log sequence number 202 is typically a monotonically increasing number, such that a higher log sequence number indicates a more recent time the log record was placed in the log. The segment identifier 206 identifies the location of a segment within one of the datasets 114A-114N to which the write transaction is applied.

[0033] In one embodiment, log records are organized into blocks. A block is a unit of storage that can be written to and read from atomically (i.e., once—so that a write or read is completed for the entire block, or the write is abandoned and not performed at all). The typical block size will be constant in a computing system, but example block sizes can be any multiple of the disk sector size (e.g., multiples of 512 bytes or 4096 bytes, depending on the disk type). Because log records can be much smaller than blocks, a block can include multiple log records. For example, Figure 3 A block 300, filled with multiple log records 302, 304, 306, and 308 according to an example embodiment, is depicted. As an example, Figure 2 The log record 200 can be Figure 3 Log record 302. Each of log records 302, 304, 306, and 308 can occupy a slot within block 300. When a log record is written to permanent log 106 (e.g.) Figure 1 As shown), it can be a block of log records (e.g., block 300) written to the permanent log 106. (Multiple) primary compute nodes 102 can wait for a certain number of log records to be generated before writing the log records as blocks to the permanent log 106.

[0034] Log service 108 may execute on one or more server computers or computing devices, or alternatively, on a virtual machine. Log service 108 facilitates the propagation of write operations performed and recorded by the primary compute node(s) 102 to the appropriate data servers 110A-110N and their corresponding replicas 112A-112N. For example, when it is determined that a particular log record will be applied to one of the datasets 114A-114N, log service 108 identifies a specific data server among the data servers 110A-110N that is assigned to the dataset 114A-114N, which includes a specific storage segment to be written via storage segment identifier 206. Log service 108 may also identify specific replica data servers 112A-110N corresponding to specific storage segments in the corresponding replica datasets responsible for replica datasets 116A-116N. Where the storage segment set is contiguous in the address space, the allocation of the storage segment set to each data server may be aligned with the most significant bit of the storage segment identifier. For example, suppose the range of storage segment identifiers is from binary 00000 00000 to binary 01101 01111. In this case, there might be seven data servers: one for a storage segment with an identifier having four most significant bits of 0000, another for a storage segment with an identifier having four most significant bits of 0001, and so on, up to a seventh storage segment with an identifier having four most significant bits of 0110. Therefore, determining which data server is allocated to store a particular storage segment becomes a valid computation. Thus, log service 108 can determine the appropriate data server for log records (e.g., log record 200) based on the storage segment identifier included in the log record (e.g., storage segment identifier 206). Log service 108 can maintain a mapping between data servers 110A-110N and replicated data servers 112A-112N. Therefore, when log service 108 determines which data server provides the log record, log service 108 can easily determine which replicated data server corresponds to it.

[0035] Then, the data server and the corresponding replica data server execute specific write transactions on specific storage segments (specified in specific log records) of the corresponding datasets of datasets 114A-114N (and the corresponding datasets of replica datasets 116A-116N). For example, suppose log service 108 determines that data server 110C is responsible for the storage segment identified by storage segment identifier 206. In this example, log service 108 provides a request 105 including a log record to data server 110C and replica data server 112C. Data server 110C and replica data server 116 execute the write transaction (called a redo log record) specified in the received log record relative to the storage segment identified by the received log record. This signifies the completion of a write transaction that began when the primary compute node(s) 102 reads the storage segment (e.g., data 101) from data server 110B. In another example, data servers 110A-110N and 112A-112N can query (e.g., periodically) log services 108 for appropriate logging instead of logging services 108 by providing request 105. Therefore, providing appropriate logging to the data servers can be done in response to a request from the data servers.

[0036] When data servers 110A-110N (and replica data servers 112A-112N) modify the storage segments applied to their respective datasets, data servers 110A-110N (and replica data servers 112A-112N) also update the storage segment headers. The headers may include various fields describing the various attributes of the storage segment. For example, the headers may include a storage segment identifier (e.g., ...). Figure 2 The storage segment identifier 206 shown and the log sequence number of the storage segment (e.g., such as...) Figure 2 (See log sequence number 202). In embodiments where the storage segment is a page, the storage segment identifier and log sequence number may be referred to as the page identifier and page log sequence number. According to such an embodiment, each page identifier is a multiple of 8 kilobytes, where the first page is at offset 0, the next page at offset 8 kilobytes, the third page at offset 16 kilobytes, and so on. The log sequence number, stored in the header of a specific storage segment, identifies the corresponding log record (e.g., log record 200) for the most recent modification to the storage segment. The log sequence number is updated each time the storage segment is modified.

[0037] Log service 108 may provide a log record at a time, or alternatively, a block at a time. For example, if log service 108 determines that a log record exists (e.g., any log record block with a storage segment identifier within that set of storage segments allocated to a particular storage segment server), log service 108 may send the entire block to that particular data server.

[0038] Furthermore, log service 108 ensures that log records are not delivered to any entity until log service 108 has been notified (e.g., by database application 122) that the log record has been securely written to permanent log 106. This contributes to the consistency of the recovery process. During recovery, the recovery process uses permanent log 106 to determine which operations have been completed. If other components in the system (e.g., any of the secondary compute nodes 104A-104N or data servers 110A-110N) have performed operations unknown to permanent log 106, recovery will fail to reach a reliable state. The datasets 114A-114N then become uncertain and thus corrupted.

[0039] Log service 108 can provide log records to secondary compute nodes 104A-104N, which update their respective caches based on the incremental changes described in the log records. For example, log service 108 can provide corresponding log records to secondary compute nodes 104A-104N via requests 107A-107N. Although log records can be pushed to secondary compute nodes 104A-104N, log service 108 can similarly handle requests for log records (e.g., from secondary compute nodes 104A-104N). As described above, log service 108 can ensure that log records are not transmitted to secondary computing systems 104A-104N unless the log record has been confirmed to be written to the permanent log 106. According to one embodiment, secondary compute nodes 104A-104N can be configured to ignore log records if the log record is used to write to a storage segment that is not yet cached (and thus would result in a read from the data server). In this scenario, if the secondary compute nodes 104A-104N need to use the storage segment later, they can read it from the data server (which has already applied skipped log writes).

[0040] Although secondary compute nodes 104A-104N primarily serve as hot standby nodes for failover purposes, according to an embodiment, secondary compute nodes 104A-104N can also act as read-only compute nodes for offloading read workloads. That is, secondary compute nodes 104A-104N can be used to read datasets 114A-114N by issuing read commands to data servers 110A-110N, which then retrieve data from datasets 114A-114N and return the retrieved data to secondary compute nodes 104A-104N.

[0041] Snapshot service 124 can be executed on one or more server computers or computing devices, or alternatively on a virtual machine. Snapshot service 124 is configured to generate snapshots of datasets 114A-114N and replica datasets 116A-116N. For example, snapshot service 124 can generate dataset snapshot 118A for dataset 114A, dataset snapshot 118B for dataset 114B, dataset snapshot 118C for dataset 114C, dataset snapshot 118D for dataset 114D, and replica dataset snapshots 120A, 120B, 120C, and 120D for replica datasets 116A and 116B, 120C, and 120D, respectively. Dataset snapshots 118A-118N and replica dataset snapshots 120A-120N are used as backups.

[0042] Snapshot service 124 can generate any number of snapshots for a given dataset 114A-114N and a given replica dataset 116A-116N. For example, snapshot service 124 can generate snapshots periodically (hourly, daily, weekly, etc.), thereby enabling data recovery at various points in time based on the backup retention period implemented for dataset 114A-114N and / or replica dataset 116A-116N.

[0043] When a snapshot is generated for a specific dataset, the snapshot service 124 records the log sequence number of the first log record (in the permanent log 106) that has not yet been redone at the time the snapshot is generated. That is, the snapshot service 124 determines the first log record that has not yet been written to the dataset by its corresponding data server. The snapshot service 124 associates the log sequence number of this log record with the snapshot at the time of its generation. For example, the snapshot service 124 may store the log sequence number as metadata of the generated snapshot. Such a log sequence number is referred to herein as the start log sequence number of the snapshot. The snapshot service 124 may also store a timestamp indicating the time when the snapshot was generated in the snapshot's metadata.

[0044] In some cases, data servers and / or one or more replicas may inadvertently skip logging applications (e.g., due to software errors, hardware failures, etc.). In such cases, data corruption occurs because the dataset and / or one or more replicas have inconsistent data versions. Such data corruption should be detected as soon as possible to prevent applications from using incorrect data. The technique described in this paper detects data corruption by cross-validating snapshots of a given dataset.

[0045] To cross-validate snapshots, snapshot pairs are first identified. For example, Figure 4 A system 400 for determining snapshot pairs according to an example embodiment is described. For example... Figure 4 As shown, system 400 includes a verification manager 402, a first replica data server 404A, a second replica data server 404B, a first replica dataset 406A, a second replica dataset 406B, first replica snapshots 408, 410, 412, and 414, second replica snapshots 416, 418, 420, and 422, and a snapshot service 424. Replica data servers 404A and 404B are replicas of page servers. For example, replica data servers 404A and 404B are examples of replica data servers(s) 112A, which are replicas of data server 110A. Replica datasets 406A and 406B are replicas of datasets managed by page servers associated with replica data servers 404A and 404B. For example, replica datasets 406A and 406B are examples of replica datasets(s) 116A, which are replicas of dataset 114A. Figure 4 As shown, snapshot service 424 has generated four snapshots of replica dataset 406A (replica snapshots 408, 410, 412, and 414), and has generated four snapshots of replica datasets 406B and 406A (replica snapshots 416, 418, 420, and 422). Each of replica snapshots 408, 410, 412, and 414 represents a snapshot of replica dataset 406A generated at a different time. Similarly, each of replica snapshots 416, 418, 420, and 422 represents a snapshot of replica dataset 406B generated at a different time.

[0046] Verification Manager 402 is configured to determine snapshot pairs between a first set of snapshots (i.e., snapshots 408, 410, 412, and 414) generated for replica dataset 406A and a second set of snapshots (i.e., snapshots 416, 418, 420, and 222) generated for replica dataset 406B. To determine a snapshot pair, Verification Manager 402 determines the latest snapshot from either one of the replica snapshots generated for replica dataset 406A or one of the replica snapshots generated for replica dataset 406B. The latest snapshot can be determined based on the timestamp associated with the replica snapshot. For example, Verification Manager 402 can designate replica snapshots 408, 410, 412, and 414 as the primary list of replica snapshots and determine the latest snapshot generated from them based on their respective timestamps. The replica snapshot 408, 410, 412, and 414 with the latest timestamp can be designated as the primary snapshot.

[0047] After determining the primary snapshot, the verification manager 402 can determine the replica snapshots from replica snapshots 416, 418, 420, and 422 that are temporally closest to the primary snapshot. For example, the verification manager 402 can compare the timestamp of each of replica snapshots 416, 418, 420, and 422 with the timestamp of the primary snapshot and determine which timestamps are temporally closest to the timestamp of the primary snapshot. The replica snapshot with the timestamp closest to the timestamp of the primary snapshot is designated as the secondary snapshot. The verification manager 402 attempts to select the snapshot closest to the primary snapshot to minimize the number of unverifiable pages, because newer snapshot pairs will have experienced more write transactions and have higher segment log sequence numbers for the corresponding segment, making them unverifiable (as described below). The determined primary and secondary snapshots are designated as the snapshot pairs to be cross-verified. Note that in other embodiments, replica snapshots 416, 418, 420, and 422 can be designated as a primary list from which the primary snapshot can be determined, and secondary snapshots can be determined from replica snapshots 408, 410, 412, and 414.

[0048] Once the snapshot pair is identified, the verification manager 402 transitions to the cross-validation phase. In this phase, the verification manager 402 compares various characteristics of the primary snapshot with those of the secondary snapshot to determine if data corruption has occurred. Figure 5 A system 500 for cross-validating snapshot pairs to detect data corruption, according to an example embodiment, is described. For example... Figure 5 As shown, system 500 includes a verification manager 502, a primary replica snapshot 504, and a secondary replica snapshot 506. Verification manager 502 is as described above (reference provided). Figure 4 The example of the verification engine 402 described above. The primary replica snapshot 504 represents the identified primary snapshot (e.g., from replica snapshots 408, 410, 412, and 414, as referenced above). Figure 4 (As described). Secondary copy snapshot 506 represents the identified secondary snapshot (e.g., from copy snapshots 416, 418, 420, and 422, as referenced above). Figure 4 (As described). The verification manager 502 includes an input / output (IO) reader 508, a storage segment checker 510, a cross-validator 512, and an alarm generator 518.

[0049] IO cell reader 508 is configured to read a predetermined amount (or "IO cell") of data from each of the primary replica snapshot 504 and the secondary replica snapshot 506 (shown as IO cells 514 and 516, respectively, where each IO cell comprises multiple storage segments (e.g., each is 8 kilobytes)). According to one embodiment, the IO cell size is 4 megabytes. IO cell reader 508 provides IO cells 514 and 516 to storage segment checker 510.

[0050] The storage segment checker 510 is configured to analyze each storage segment in each of I / O units 514 and 516 and determine which storage segments should undergo cross-validation. For example, for each storage segment in I / O units 514 and 516, the storage segment checker 510 can determine whether the storage segment contains all zeros. Such a storage segment is designated as corrupted by the storage segment checker 510.

[0051] According to one embodiment, the storage segments can be encrypted for security purposes. Storage segment checker 510 can be configured to decrypt each storage segment in I / O units 514 and 516. If decryption of some storage segments fails, storage segment checker 510 determines that these storage segments are corrupt and designates them as corrupt.

[0052] The storage segment checker 510 can also be configured to perform one or more logical consistency checks on each storage segment of I / O units 514 and 516. If any one or more logical consistency checks of a particular storage segment fail, the storage segment checker 510 designates that storage segment as corrupt. Examples of logical consistency checks include, but are not limited to, checksum-based checks, tear-based checks, short-transfer-based checks, stale-read-based checks, page audit failure-based checks, etc.

[0053] The storage segment checker 510 can also be configured to determine whether a specific storage segment of I / O unit 514 and / or I / O unit 516 is designated as free. If a specific storage segment of an I / O unit is designated as free, verification of that storage segment and its corresponding storage segment in another I / O unit is skipped. The storage segment checker 510 specifies that such storage segments cannot be used for cross-validation.

[0054] Any memory segment not designated as corrupted or unusable may be left undesignated, or alternatively, may be designated as verifiable. After the memory segment checker 510 completes its various checks, it provides I / O units 514 and 516 to the cross-validator 512.

[0055] Cross-validator 512 is configured to cross-validate the memory segment of I / O unit 514 with the corresponding memory segment of I / O unit 516 located at the same logical location. Cross-validator 512 only validates memory segments that are not designated as corrupt or unusable. Any memory segments designated as corrupt or unusable are skipped because they cannot be validated definitively. For example, cross-validator 512 may initially compare the memory segment identifier of the first memory segment from I / O unit 514 with the memory segment identifier of the corresponding first memory segment from I / O unit 516 (i.e., the memory segment located at the same offset) (assuming neither is designated as corrupt or unusable). If the memory segment identifiers do not match, cross-validator 512 determines that data corruption has been detected between the two memory segments. Cross-validator 512 performs this check on each memory segment of I / O units 514 and 516 that is not designated as corrupt or unusable.

[0056] If the segment identifiers match, cross-validator 512 compares the segment log sequence number of the segment in I / O unit 516 (not designated as corrupted or unavailable) with the segment log sequence number of the corresponding segment in I / O unit 516 located at the same logical location (not designated as corrupted or unavailable). However, for deterministic comparison, cross-validator 512 only compares the segment log sequence number that is less than the minimum value between the start log sequence number of the primary replica snapshot 504 and the start log sequence number of the secondary replica snapshot 506 (referred to herein as the minimum log sequence number). If both segment log sequence numbers are less than the minimum log sequence number, the two segment log sequence numbers should match. Otherwise, one of the primary replica snapshot 504 or the secondary replica snapshot 56 has lost logging application.

[0057] Therefore, cross-validator 512 compares the storage segment log sequence numbers to determine if they match. If they match, cross-validator 512 determines that there is no data corruption between the two storage segments and performs the aforementioned validation on the storage segments located at the next offset IO units 514 and 516. If they do not match, cross-validator 512 determines that there is data corruption between the two storage segments.

[0058] If cross-validator 512 detects data corruption, it provides notification 520 to alarm generator 518. Alarm generator 518 can automatically issue alarm 122 to the administrator's computing device, indicating that data corruption has been detected regarding these two storage segments. This allows the administrator to quickly determine the cause of the data corruption and resolve the problem. Examples of alarm 122 include Short Message Service (SMS) messages, telephone calls, emails, notifications presented via incident management services, etc.

[0059] Note that the verification manager 502 performs the aforementioned operations for each IO unit included in the primary replica snapshot 504 and the secondary replica snapshot 506. For example, after reading and cross-validating IO units 514 and 516, the IO unit reader 508 can read the next IO unit from the primary replica snapshot 506 and the secondary replica snapshot 506, respectively, and cross-validate these IO units in a similar manner as described above. In another example, after providing IO units 514 and 516 to the storage segment checker 510, the IO unit reader 508 can read the next IO unit from the primary replica snapshot 506 and the secondary replica snapshot 506, respectively.

[0060] Therefore, data corruption can be detected in several ways. For example, Figure 6 A flowchart 600 illustrating a method for detecting data corruption based on cross-validation snapshots according to an example embodiment is shown. In one embodiment, flowchart 600 may be provided by Figure 4 and Figure 5 The system implementations 400 and 500 are shown, although the method is not limited to these implementations. Therefore, reference will continue to be made to... Figure 4 and 5 To describe flowchart 600. Based on flowchart 600 and Figure 4 and Figure 5 The discussion of systems 400 and 500, other structures and operational embodiments will be obvious to those skilled in the art.

[0061] Flowchart 600 begins at step 602. In step 602, a snapshot pair is determined, comprising a first snapshot of a first copy of the dataset and a second snapshot of a second copy of the dataset. The first copy comprises multiple first storage segments of the dataset, and the second copy comprises multiple second storage segments of the dataset. For example, refer to... Figure 5 The verification manager 502 determines a snapshot pair, which includes a first snapshot of a first copy of the dataset (e.g., a primary copy snapshot 504 as a snapshot of a copy dataset 406A) and a second snapshot of a second copy of the data (e.g., a secondary copy snapshot 506 as a snapshot of a copy dataset 406B). The primary copy snapshot 504 includes multiple first storage segments, while the secondary copy snapshot 506 includes multiple second storage segments.

[0062] According to one or more embodiments, a plurality of first storage segments correspond to a plurality of first pages of a dataset, and a plurality of second storage segments correspond to a plurality of second pages of the dataset. For example, refer to Figure 5 The storage segment of IO unit 514 corresponds to a page of the dataset, and the storage segment of IO unit 516 also corresponds to a page of the dataset.

[0063] According to one or more embodiments, a third log sequence number identifies a corresponding first log record, which identifies the latest modification made to the first storage segment, and a fourth log sequence number identifies a corresponding second log record, which identifies the latest modification made to the second storage segment. For example, refer to... Figure 5 The third log sequence number identifies the corresponding first log record, which indicates the latest modification made to the first storage segment of IO unit 514, and the fourth log sequence number identifies the corresponding second log record, which indicates the latest modification made to the second storage segment of IO unit 516.

[0064] According to one or more embodiments, as described below Figure 7 The flowchart 700 described is used to determine the snapshot pairs.

[0065] In step 604, the smaller log sequence number is determined between the first log sequence number associated with the first snapshot and the second log sequence number associated with the second snapshot. The first log sequence number identifies the corresponding first log record, indicating an unmerged change to the dataset in the first snapshot, while the second log sequence number identifies the corresponding second log record, indicating an unmerged change to the dataset in the second snapshot. For example, refer to... Figure 5 Cross-validator 512 determines the minimum log sequence number from the first log sequence number associated with the primary replica snapshot 504 and the second log sequence number associated with the secondary replica snapshot 506.

[0066] In step 606, for each of the plurality of first storage segments, it is determined whether a third log sequence number less than the minimum log sequence number and associated with the first storage segment matches a fourth log sequence number less than the minimum log sequence number and associated with a second storage segment corresponding to the first storage segment. If a mismatch is determined between the third and fourth log sequence numbers, the process continues to step 608. Otherwise, the process continues to step 610. For example, refer to... Figure 5 Cross-validator 512 determines whether a third log sequence number, which is less than the minimum log sequence number and associated with the first storage segment of IO unit 514, matches a fourth log sequence number, which is less than the minimum log sequence number and associated with the second storage segment of the plurality of second storage segments of IO unit 516 corresponding to the first storage segment.

[0067] In step 608, in response to determining a mismatch between the third log sequence number and the fourth log sequence number, it is determined that data corruption exists regarding the dataset. For example, refer to... Figure 5 Cross-validator 512 determines that there is data corruption in the dataset (e.g., the copy datasets 406A and / or 406B).

[0068] According to one or more embodiments, in response to determining that data corruption exists in a dataset, an alert indicating that data corruption exists in the dataset is automatically generated. For example, refer to... Figure 5 If cross-validator 512 detects data corruption, it provides a notification 520 to alarm generator 518. Alarm generator 518 can automatically issue an alarm 522 to the administrator's computing device, indicating that data corruption has been detected regarding these two storage segments. This allows the administrator to quickly determine the cause of the data corruption and resolve the problem. Examples of alarms 522 include Short Message Service (SMS) messages, telephone calls, emails, notifications presented via incident management services, etc.

[0069] In step 610, in response to determining that the third log sequence number and the fourth log sequence number match, it is determined that there is no data corruption in the dataset. For example, refer to Figure 5 Cross-validator 512 determines that there is no data corruption with respect to the dataset (e.g., the copy datasets 406A and / or 406B).

[0070] Figure 7 A flowchart 700 illustrating a method for determining snapshot pairs according to an example embodiment is shown. In one embodiment, flowchart 700 may be provided by Figure 4 The system 400 implementation shown is not limited to this implementation. Therefore, reference will continue to be made to... Figure 4 To describe flowchart 700. Based on flowchart 700 and system 400 and Figure 4 The discussion will reveal other structural and operational embodiments to those skilled in the art.

[0071] Flowchart 700 begins at step 702. In step 702, the most recently generated snapshot is determined from the multiple snapshots generated for the first copy. The most recently generated snapshot includes a first timestamp and is the first snapshot of the snapshot pair. For example, refer to... Figure 4 Verification manager 402 determines the most recently generated snapshot from replica snapshots 408, 410, 412, and 414. For example, verification manager 402 can determine the snapshots of replica snapshots 408, 410, 412, and 414 with the latest timestamp.

[0072] In step 704, a snapshot including the second timestamp closest to the first timestamp is determined from among the multiple snapshots generated for the second copy; the determined snapshot is the second snapshot of the snapshot pair. For example, refer to... Figure 4 The verification manager 402 determines from the copy snapshots 416, 418, 420 and 422 the snapshot that includes the second timestamp closest to the first timestamp, and the determined snapshot is the second snapshot.

[0073] According to one or more embodiments, if the storage segment identifier of the first storage segment of the first replica snapshot does not match the storage segment identifier of the second storage segment of the second replica snapshot corresponding to the first storage segment, data corruption can also be detected. Figure 8 A flowchart 800 illustrating a method for determining data corruption based on a storage segment identifier mismatch, according to an example embodiment, is shown. In one embodiment, flowchart 800 may be provided by... Figure 5 The system 500 implementation shown is not limited to this implementation. Therefore, reference will continue to be made to... Figure 5 To describe flowchart 800. Based on flowchart 800 and system 500 and Figure 5 The discussion will reveal other structural and operational embodiments to those skilled in the art.

[0074] Flowchart 800 begins at step 802. In step 802, for each of the plurality of first storage segments, it is determined whether the first storage segment identifier associated with the first storage segment matches a second storage segment associated with the corresponding first storage segment among the plurality of second storage segments. In response to determining a mismatch between the first and second storage segment identifiers, the process continues to step 804. Otherwise, the process continues to step 806. For example, refer to... Figure 5 Cross-validator 512 determines whether the first storage segment identifier associated with the first storage segment of IO unit 514 matches the second storage segment identifier associated with the second storage segment of IO unit 516 corresponding to the first storage segment (e.g., the first storage segment and the second storage segment have the same logical offset).

[0075] In step 804, it is determined that data corruption exists in the dataset. For example, refer to... Figure 5 Cross-validator 512 determines that there is data corruption in the dataset (e.g., the copy datasets 406A and / or 406B).

[0076] In step 806, in response to determining that the first segment identifier and the second segment identifier match, it is determined that there is no data corruption regarding the dataset. For example, refer to Figure 5 Cross-validator 512 determines that there is no data corruption with respect to the dataset (e.g., the copy datasets 406A and / or 406B).

[0077] III. Example Computer System Implementation

[0078] The above reference Figures 1 to 8 The described system and method can be implemented in hardware or in combination with one or both of software and / or firmware. For example, system 900 can be used to implement any of the following: Figure 1The following are included: nodes 102 and 104A-104N, persistent log 106, log service 108, data servers 110A-110N, replica data servers 112A-112N, datasets 114A-114N, replica datasets 116A-116N, dataset snapshots 118A-118N, replica dataset snapshots 120A-120N, database application 122, and / or snapshot service 124. Figure 4 The replica data servers 404A and 404B, replica datasets 406A and 406B, snapshot service 424, verification manager 402, replica snapshots 408, 410, 412 and 414 and / or replica snapshots 416, 418, 420, 420, Figure 5 The verification manager 502, primary copy snapshots 504 and 506, I / O cell reader 508, storage segment checker 510, cross-validator 512, and / or alarm generator 518, and / or any components described therein, as well as flowcharts 600, 700, and / or 800, may each be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, any of the following may be implemented as hardware logic / circuit: Figure 1 The following are included: nodes 102 and 104A-104N, persistent log 106, log service 108, data servers 110A-110N, replica data servers 112A-112N, datasets 114A-114N, replica datasets 116A-116N, dataset snapshots 118A-118N, replica dataset snapshots 120A-120N, database application 122, and / or snapshot service 124. Figure 4 The replica data servers 404A and 404B, replica datasets 406A and 406B, snapshot service 424, verification manager 402, replica snapshots 408, 410, 412 and 414 and / or replica snapshots 416, 418, 420, 420, Figure 5 The verification manager 502, primary copy snapshots 504 and 506, I / O cell reader 508, storage segment checker 510, cross-validator 512 and / or alarm generator 518, and / or any components described therein, as well as flowcharts 600, 700 and / or 800. In one embodiment, any of the following may be implemented in one or more SoCs (System-on-Chip): Figure 1 The following are included: nodes 102 and 104A-104N, persistent log 106, log service 108, data servers 110A-110N, replica data servers 112A-112N, datasets 114A-114N, replica datasets 116A-116N, dataset snapshots 118A-118N, replica dataset snapshots 120A-120N, database application 122, and / or snapshot service 124. Figure 4 The replica data servers 404A and 404B, replica datasets 406A and 406B, snapshot service 424, verification manager 402, replica snapshots 408, 410, 412 and 414 and / or replica snapshots 416, 418, 420, 420, Figure 5 The SoC may include a verification manager 502, primary copy snapshots 504 and 506, an I / O cell reader 508, a storage segment checker 510, a cross-validator 512, and / or an alarm generator 518, and / or any components described therein, as well as flowcharts 600, 700, and / or 800. The SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry, and may optionally execute received program code and / or include embedded firmware for performing functions.

[0079] Figure 9 An exemplary implementation of computing device 900 is depicted, wherein embodiments may be implemented, including any of the following: Figure 1 The following are included: nodes 102 and 104A-104N, persistent log 106, log service 108, data servers 110A-110N, replica data servers 112A-112N, datasets 114A-114N, replica datasets 116A-116N, dataset snapshots 118A-118N, replica dataset snapshots 120A-120N, database application 122, and / or snapshot service 124. Figure 4 The replica data servers 404A and 404B, replica datasets 406A and 406B, snapshot service 424, verification manager 402, replica snapshots 408, 410, 412 and 414 and / or replica snapshots 416, 418, 420, 420, Figure 5 The verification manager 502, primary copy snapshots 504 and 506, I / O cell reader 508, storage segment checker 510, cross-validator 512 and / or alarm generator 518, and / or any components described therein, as well as flowcharts 600, 700 and / or 800. The description of the computing device 900 provided herein is for illustrative purposes and is not intended to be limiting. Embodiments may be implemented in other types of computer systems, as are well known to those skilled in the art.

[0080] like Figure 9As shown, computing device 900 includes one or more processors, referred to as processor circuitry 902, system memory 904, and a bus 906 coupling various system components, including system memory 904, to processor circuitry 902. Processor circuitry 902 is electrical and / or optical circuitry implemented in one or more physical hardware circuitry device elements and / or integrated circuit devices (semiconductor material chips or chips) as a central processing unit (CPU), microcontroller, microprocessor, and / or other physical hardware processor circuitry. Processor circuitry 902 can execute program code stored in a computer-readable medium, such as operating system 930, application program 932, other programs 934, etc. Bus 906 represents any one or more of several types of bus architectures, including memory bus or memory controller, peripheral bus, accelerated graphics port, and processor or local bus using any of various bus architectures. System memory 904 includes read-only memory (ROM) 908 and random access memory (RAM) 910. Basic input / output system 912 (BIOS) is stored in ROM 908.

[0081] The computing device 900 also includes one or more of the following drives: a hard disk drive 914 for reading and writing to the hard disk; a disk drive 916 for reading or writing to a removable disk 918; and an optical disc drive 920 for reading or writing to a removable optical disc 922, such as a CD-ROM, DVD-ROM, or other optical media. The hard disk drive 914, disk drive 916, and optical disc drive 920 are connected to the bus 906 via a hard disk drive interface 924, a disk drive interface 926, and an optical disc drive interface 928, respectively. The drives and their associated computer-readable media provide the computer with non-volatile storage of computer-readable instructions, data structures, program modules, and other data. While hard disks, removable disks, and removable optical discs have been described, other types of hardware-based computer-readable storage media may also be used to store data, such as flash memory cards, digital video discs, RAM, ROM, and other hardware storage media.

[0082] Multiple program modules may be stored on a hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include an operating system 930, one or more application programs 932, other programs 934, and program data 936. Application programs 932 or other programs 934 may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the system described above, including the references above. Figures 1 to 8 The described embodiments.

[0083] Users can input commands and information into computing device 900 through input devices such as keyboard 938 and pointing device 940. Other input devices (not shown) may include microphone, joystick, gamepad, satellite antenna, scanner, touch screen and / or touchpad, voice identification system for receiving voice input, gesture identification system for receiving gesture input, etc. These and other input devices are typically connected to processor circuitry 902 via serial port interface 942 coupled to bus 906, but may also be connected via other interfaces such as parallel port, game port, or Universal Serial Bus (USB).

[0084] Display screen 944 is also connected to bus 906 via an interface such as video adapter 946. Display screen 944 can be external to or incorporated into computing device 900. Display screen 944 can display information and can also be a user interface for receiving user commands and / or other information (e.g., via touch, finger gestures, virtual keyboard, by providing tactile input (where the user lightly presses and quickly releases display screen 944), by providing "touch and hold" input (where the user touches and holds his finger (or touch instrument) on display screen 944 for a predetermined time), by providing touch input exceeding a predetermined pressure threshold, etc.). In addition to display screen 944, computing device 900 may also include other peripheral output devices (not shown), such as speakers and printers.

[0085] Computing device 900 is connected to network 948 (e.g., the Internet) via an adapter or network interface 950, modem 952, or other means for establishing communication on the network. Figure 9 As shown, the modem 952, which can be built-in or external, can be connected to the bus 906 via the serial port interface 942, or it can be connected to the bus 906 using another interface type that includes a parallel interface.

[0086] As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium” generally refer to physical hardware media, such as hard disks associated with hard disk drives 914, removable disks 918, and removable optical disks 922, such as RAM, ROM, flash memory cards, digital video disks, zip disks, MEM, nanotechnology-based storage devices, and other types of physical / tangible hardware storage media (including…). Figure 9The system memory 904). This computer-readable storage medium is different from and does not overlap with communication media (excluding communication media). Communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves. The term "modulated data signal" refers to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media include wireless media, such as acoustic, RF, infrared, and other wireless media, as well as wired media. Embodiments also relate to such communication media.

[0087] As described above, computer programs and modules (including application program 932 and other programs 934) can be stored on a hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage media. Such computer programs can also be received via a network interface 950, a serial port interface 952, or any other interface type. When executed or loaded by an application, such computer programs enable the computing device 900 to implement the features of the embodiments discussed herein. Therefore, such computer programs represent the controller of the computing device 900.

[0088] The embodiments also relate to computer program products that include computer code or instructions stored on any computer-readable medium. Such computer program products include hard disk drives, optical disk drives, storage device packages, memory sticks, memory cards, and other types of physical storage hardware.

[0089] IV. Other Example Implementations

[0090] This document describes a method. The method includes: identifying a snapshot pair, comprising a first snapshot of a first copy of a dataset and a second snapshot of a second copy of the dataset, the first copy comprising a plurality of first storage segments of the dataset, and the second copy comprising a plurality of second storage segments of the dataset; determining a minimum log sequence number from a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that identifies a change to the dataset that was not incorporated into the first snapshot, and the second log sequence number identifying a corresponding second log record that identifies a change to the dataset that was not incorporated into the second snapshot; and for each of the plurality of first storage segments: determining whether a third log sequence number less than the minimum log sequence number and associated with the first storage segment matches a fourth log sequence number less than the minimum log sequence number and associated with a second storage segment corresponding to the first storage segment; and determining, in response to determining a mismatch between the third and fourth log sequence numbers, that data corruption exists regarding the dataset.

[0091] In one implementation of the aforementioned method, the method further includes determining, in response to determining that the third log sequence number and the fourth log sequence number match, that there is no data corruption in the dataset.

[0092] In one implementation of the aforementioned method, determining the snapshot pair includes determining the most recently generated snapshot from a plurality of snapshots generated for the first copy, the most recently generated snapshot including a first timestamp, the most recently generated snapshot being the first snapshot of the snapshot pair; and determining the snapshot from a plurality of snapshots generated for the second copy, including a second timestamp closest to the first timestamp, the determined snapshot being the second snapshot of the snapshot pair.

[0093] In one implementation of the aforementioned method, the method further includes: for each of a plurality of first storage segments: determining whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with a second storage segment corresponding to the first storage segment in a plurality of second storage segments; determining, in response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, that there is data corruption in the dataset; and determining, in response to determining that the first storage segment identifier and the second storage segment identifier match, that there is no data corruption in the dataset.

[0094] In one implementation of the aforementioned method, the third log sequence number identifies the corresponding first log record, which indicates the latest modification made to the first storage segment, while the fourth log sequence number identifies the corresponding second log record, which indicates the latest modification made to the second storage segment.

[0095] In one implementation of the aforementioned method, a plurality of first storage segments correspond to a plurality of first pages of the dataset, and a plurality of second storage segments correspond to a plurality of second pages of the dataset.

[0096] In one implementation of the aforementioned method, the method further includes automatically generating an alert indicating that data corruption exists in the dataset in response to determining that data corruption exists in the dataset.

[0097] Systems according to any of the embodiments described herein are also disclosed. The system includes: at least one processor circuit; and at least one memory storing program code configured to be executed by the at least one processor circuit, the program code including: a verification manager configured to: determine a snapshot pair comprising a first snapshot of a first copy of a dataset and a second snapshot of a second copy of the dataset, the first copy comprising a plurality of first storage segments of the dataset, and the second copy comprising a plurality of second storage segments of the dataset; determine a minimum log sequence number from a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that has not been incorporated into the first snapshot, and the second log sequence number identifying a corresponding second log record that has not been incorporated into the second snapshot; and for each of the plurality of first storage segments: determine whether a third log sequence number less than the minimum log sequence number and associated with the first storage segment matches a fourth log sequence number less than the minimum log sequence number and associated with a second storage segment corresponding to the first storage segment; and in response to determining a mismatch between the third and fourth log sequence numbers, determine that data corruption exists regarding the dataset.

[0098] In one implementation of the aforementioned system, the verification manager is also configured to: determine that there is no data corruption in the dataset in response to determining that the third log sequence number and the fourth log sequence number match.

[0099] In one implementation of the aforementioned system, the verification manager is configured to determine snapshot pairs by: determining the latest generated snapshot from a plurality of snapshots generated for a first replica, the latest generated snapshot including a first timestamp, the latest generated snapshot being the first snapshot of the snapshot pair; and determining a snapshot from a plurality of snapshots generated for a second replica, the snapshot including a second timestamp closest to the first timestamp, the determined snapshot being the second snapshot of the snapshot pair.

[0100] In one implementation of the aforementioned system, the verification manager is further configured to: for each of a plurality of first storage segments: determine whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with a second storage segment corresponding to the first storage segment in a plurality of second storage segments; in response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, determine that there is data corruption in the dataset; and in response to determining that the first storage segment identifier and the second storage segment identifier match, determine that there is no data corruption in the dataset.

[0101] In one implementation of the aforementioned system, the third log sequence number identifies the corresponding first log record, which indicates the latest modification made to the first storage segment, while the fourth log sequence number identifies the corresponding second log record, which indicates the latest modification made to the second storage segment.

[0102] In one implementation of the aforementioned system, multiple first storage segments correspond to multiple first pages of the dataset, and multiple second storage segments correspond to multiple second pages of the dataset.

[0103] In one implementation of the aforementioned system, the verification manager is also configured to automatically generate an alert indicating that data corruption exists in the dataset in response to determining that data corruption exists in the dataset.

[0104] A computer-readable storage medium having program instructions thereon, the program instructions, when executed by at least one processor, performing a method. The method includes: determining a snapshot pair, comprising a first snapshot of a first copy of a dataset and a second snapshot of a second copy of the dataset, the first copy comprising a plurality of first storage segments of the dataset, and the second copy comprising a plurality of second storage segments of the dataset; determining a minimum log sequence number from a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that indicates a change to the dataset not incorporated into the first snapshot, and the second log sequence number identifying a corresponding second log record that indicates a change to the dataset not incorporated into the second snapshot; and for each of the plurality of first storage segments: determining whether a third log sequence number less than the minimum log sequence number and associated with the first storage segment matches a fourth log sequence number less than the minimum log sequence number and associated with a second storage segment corresponding to the first storage segment; and determining, in response to determining a mismatch between the third and fourth log sequence numbers, that data corruption exists regarding the dataset.

[0105] In one implementation of the aforementioned computer-readable storage medium, the method further includes: determining, in response to determining that the third log sequence number and the fourth log sequence number match, that there is no data corruption regarding the dataset.

[0106] In one implementation of the aforementioned computer-readable storage medium, determining a snapshot pair includes: determining a newly generated snapshot from a plurality of snapshots generated for a first copy, the newly generated snapshot including a first timestamp, the newly generated snapshot being the first snapshot of the snapshot pair; and determining a snapshot from a plurality of snapshots generated for a second copy, the snapshot including a second timestamp closest to the first timestamp, the determined snapshot being the second snapshot of the snapshot pair.

[0107] In one implementation of the aforementioned computer-readable storage medium, the method further includes: for each of a plurality of first storage segments: determining whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with a second storage segment corresponding to the first storage segment in a plurality of second storage segments; determining, in response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, that there is data corruption with respect to the dataset; and determining, in response to determining that the first storage segment identifier and the second storage segment identifier match, that there is no data corruption with respect to the dataset.

[0108] In one implementation of the aforementioned computer-readable storage medium, a third log sequence number identifies a corresponding first log record that indicates the latest modification to the first storage segment, and a fourth log sequence number identifies a corresponding second log record that indicates the latest modification to the second storage segment.

[0109] In one implementation of the aforementioned computer-readable storage medium, a plurality of first storage segments correspond to a plurality of first pages of a dataset, and a plurality of second storage segments correspond to a plurality of second pages of a dataset.

[0110] V. Conclusion

[0111] While various exemplary embodiments have been described above, it should be understood that they are presented as examples only and not as limitations. Those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the embodiments as defined in the appended claims. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above, but should be defined solely by the following claims and their equivalents.

Claims

1. A method comprising: A snapshot pair is determined, the snapshot pair including a first snapshot of a first copy of the dataset and a second snapshot of a second copy of the dataset, the first copy including a plurality of first storage segments of the dataset, and the second copy including a plurality of second storage segments of the dataset; The minimum log sequence number is determined from the first log sequence number associated with the first snapshot and the second log sequence number associated with the second snapshot, wherein the first log sequence number identifies the corresponding first log record, the corresponding first log record identifies a change to the dataset that was not incorporated into the first snapshot, and the second log sequence number identifies the corresponding second log record, the corresponding second log record identifies a change to the dataset that was not incorporated into the second snapshot; as well as For each of the plurality of first storage segments: Determine whether a third log sequence number, which is less than the minimum log sequence number and associated with the first storage segment, matches a fourth log sequence number, which is less than the minimum log sequence number and associated with a second storage segment among the plurality of second storage segments corresponding to the first storage segment; as well as In response to the determination of a mismatch between the third log sequence number and the fourth log sequence number, it is determined that there is data corruption in the dataset.

2. The method according to claim 1, further comprising: In response to determining that the third log sequence number and the fourth log sequence number match, it is determined that there is no data corruption regarding the dataset.

3. The method of claim 1, wherein determining the snapshot pair comprises: The latest generated snapshot is determined from a plurality of snapshots generated for the first copy, the latest generated snapshot including a first timestamp, and the latest generated snapshot is the first snapshot of the snapshot pair; as well as From a plurality of snapshots generated for the second copy, a snapshot including a second timestamp that is closest to the first timestamp is determined, and the determined snapshot is the second snapshot of the snapshot pair.

4. The method according to claim 1, further comprising: For each of the plurality of first storage segments: Determine whether the first storage segment identifier associated with the first storage segment matches the second storage segment identifier associated with the second storage segment corresponding to the first storage segment among the plurality of second storage segments; In response to determining a mismatch between the first segment identifier and the second segment identifier, it is determined that data corruption exists regarding the dataset; as well as In response to determining that the first segment identifier and the second segment identifier match, it is determined that there is no data corruption with respect to the dataset.

5. The method according to claim 1, wherein the third log sequence number identifies the corresponding first log record, the corresponding first log record identifies the latest modification made to the first storage segment, wherein the fourth log sequence number identifies the corresponding second log record, the corresponding second log record identifies the latest modification made to the second storage segment.

6. The method of claim 1, wherein the plurality of first storage segments correspond to a plurality of first pages of the dataset, and wherein the plurality of second storage segments correspond to a plurality of second pages of the dataset.

7. The method according to claim 1, further comprising: In response to the determination that data corruption exists in the dataset, an alert indicating the existence of data corruption in the dataset is automatically generated.

8. A system comprising: At least one processor circuit; as well as At least one memory stores program code configured to be executed by the at least one processor circuit, the program code comprising: The verification manager is configured as follows: A snapshot pair is determined, the snapshot pair including a first snapshot of a first copy of the dataset and a second snapshot of a second copy of the dataset, the first copy including a plurality of first storage segments of the dataset, and the second copy including a plurality of second storage segments of the dataset; The minimum log sequence number is determined from the first log sequence number associated with the first snapshot and the second log sequence number associated with the second snapshot, wherein the first log sequence number identifies the corresponding first log record, the corresponding first log record identifies a change to the dataset that was not incorporated into the first snapshot, and the second log sequence number identifies the corresponding second log record, the corresponding second log record identifies a change to the dataset that was not incorporated into the second snapshot; For each of the plurality of first storage segments: Determine whether a third log sequence number, which is less than the minimum log sequence number and associated with the first storage segment, matches a fourth log sequence number, which is less than the minimum log sequence number and associated with a second storage segment among the plurality of second storage segments corresponding to the first storage segment; and In response to the determination of a mismatch between the third log sequence number and the fourth log sequence number, it is determined that there is data corruption in the dataset.

9. The system of claim 8, wherein the verification manager is further configured to: In response to determining that the third log sequence number and the fourth log sequence number match, it is determined that there is no data corruption regarding the dataset.

10. The system of claim 8, wherein the verification manager is configured to determine the snapshot pair by: The latest generated snapshot is determined from a plurality of snapshots generated for the first copy, the latest generated snapshot including a first timestamp, the latest generated snapshot being the first snapshot of the snapshot pair; and From a plurality of snapshots generated for the second copy, a snapshot including a second timestamp that is closest to the first timestamp is determined, and the determined snapshot is the second snapshot of the snapshot pair.

11. The system according to claim 8, wherein the verification manager is configured as follows: For each of the plurality of first storage segments: Determine whether the first storage segment identifier associated with the first storage segment matches the second storage segment identifier associated with the second storage segment corresponding to the first storage segment among the plurality of second storage segments; In response to determining a mismatch between the first segment identifier and the second segment identifier, it is determined that data corruption exists regarding the dataset; as well as In response to determining that the first segment identifier and the second segment identifier match, it is determined that there is no data corruption with respect to the dataset.

12. The system according to claim 8, wherein the third log sequence number identifies the corresponding first log record, the corresponding first log record identifies the latest modification made to the first storage segment, wherein the fourth log sequence number identifies the corresponding second log record, the corresponding second log record identifies the latest modification made to the second storage segment.

13. The system of claim 8, wherein the plurality of first storage segments correspond to a plurality of first pages of the dataset, and wherein the plurality of second storage segments correspond to a plurality of second pages of the dataset.

14. The system of claim 8, wherein the program code further comprises an alarm generator configured to: In response to the determination that data corruption exists in the dataset, an alert indicating the existence of data corruption in the dataset is automatically generated.

15. A computer-readable storage medium having program instructions recorded thereon, the program instructions performing a method when executed by at least one processor of a computing device, the method comprising: A snapshot pair is determined, the snapshot pair including a first snapshot of a first copy of the dataset and a second snapshot of a second copy of the dataset, the first copy including a plurality of first storage segments of the dataset, and the second copy including a plurality of second storage segments of the dataset; The minimum log sequence number is determined from the first log sequence number associated with the first snapshot and the second log sequence number associated with the second snapshot, wherein the first log sequence number identifies the corresponding first log record, the corresponding first log record identifies a change to the dataset that was not incorporated into the first snapshot, and the second log sequence number identifies the corresponding second log record, the corresponding second log record identifies a change to the dataset that was not incorporated into the second snapshot; as well as For each of the plurality of first storage segments: Determine whether a third log sequence number, which is less than the minimum log sequence number and associated with the first storage segment, matches a fourth log sequence number, which is less than the minimum log sequence number and associated with a second storage segment among the plurality of second storage segments corresponding to the first storage segment; as well as In response to the determination of a mismatch between the third log sequence number and the fourth log sequence number, it is determined that there is data corruption in the dataset.

16. The computer-readable storage medium of claim 15, further comprising: In response to determining that the third log sequence number and the fourth log sequence number match, it is determined that there is no data corruption regarding the dataset.

17. The computer-readable storage medium of claim 15, wherein determining the snapshot pair comprises: The latest generated snapshot is determined from a plurality of snapshots generated for the first copy, the latest generated snapshot including a first timestamp, and the latest generated snapshot is the first snapshot of the snapshot pair; as well as From a plurality of snapshots generated for the second copy, a snapshot including a second timestamp that is closest to the first timestamp is determined, and the determined snapshot is the second snapshot of the snapshot pair.

18. The computer-readable storage medium of claim 15, further comprising: For each of the plurality of first storage segments: Determine whether the first storage segment identifier associated with the first storage segment matches the second storage segment identifier associated with the second storage segment corresponding to the first storage segment among the plurality of second storage segments; In response to determining a mismatch between the first segment identifier and the second segment identifier, it is determined that data corruption exists regarding the dataset; as well as In response to determining that the first segment identifier and the second segment identifier match, it is determined that there is no data corruption with respect to the dataset.

19. The computer-readable storage medium of claim 15, wherein the third log sequence number identifies a corresponding first log record, the corresponding first log record identifies a latest modification to the first storage segment, wherein the fourth log sequence number identifies a corresponding second log record, the corresponding second log record identifies a latest modification to the second storage segment.

20. The computer-readable storage medium of claim 15, wherein the plurality of first storage segments correspond to a plurality of first pages of the dataset, and wherein the plurality of second storage segments correspond to a plurality of second pages of the dataset.

Images