Shuffle data consistency checking method and device, and distributed computing system

Abstract

The application relates to a Shuffle data consistency checking method and device and a distributed computing system. The method is applied to a Shuffle reader end in a distributed computing system and comprises the following steps: in response to a data reading request for a target data partition, at least one first metadata corresponding to the target data partition is acquired from a remote Shuffle service; wherein the at least one first metadata is generated by at least one Shuffle writer end when data of the target data partition is written into the remote Shuffle service and is stored in the remote Shuffle service; data of the target data partition is read from the remote Shuffle service, and second metadata corresponding to the read data is collected; and based on the at least one first metadata and the second metadata, a data consistency checking result of the target data partition is determined. Therefore, data damage, loss or tampering introduced by any intermediate link occurring after data writing and before reading can be efficiently detected, so that the credibility of data consumed by a downstream computing task is fundamentally guaranteed.

Description

Technical Field

[0001] This application relates to the field of distributed computing, and in particular to a Shuffle data consistency verification method and apparatus, and a distributed computing system. Background Technology

[0002] With the development of the internet, in distributed computing frameworks such as Spark, Shuffle is a crucial operation connecting different computation stages. It is responsible for redistributing the output data of upstream tasks according to partitioning rules for downstream tasks to process. Its correctness directly affects the accuracy of the final computation result. Traditional Shuffle mechanisms rely on local disk storage on compute nodes, which suffers from problems such as tight coupling between computation and storage, data loss due to node failures, and disk I / O performance bottlenecks. To address these issues, Remote Shuffle Service (RSS) has been proposed and widely adopted. RSS adopts a storage-compute separation architecture, with an independent service cluster uniformly storing and managing Shuffle data, improving system reliability, resource utilization, and I / O performance.

[0003] However, the introduction of RSS also brings new data consistency issues. Data needs to go through multiple stages, including generation on the writer side, network transmission, RSS storage, and retrieval on the reader side. Hardware failure, network anomalies, or software defects at any stage can lead to data loss, duplication, or corruption during transmission and storage. Furthermore, in the existing framework, the reader side unconditionally trusts the received data, causing errors to be propagated covertly, ultimately leading to incorrect calculation results. Summary of the Invention

[0004] This application provides a Shuffle data consistency verification method and apparatus, and a distributed computing system, to solve the technical problem in the prior art that data transmission errors and unreliable final calculation results are caused by the reader end's unconditional trust in the data received from the remote Shuffle service.

[0005] Firstly, this application provides a Shuffle data consistency verification method, which is applied to the Shuffle reader in a distributed computing system. The method includes: In response to a data read request for a target data partition, at least one first metadata corresponding to the target data partition is obtained from a remote Shuffle service; wherein, the at least one first metadata is generated by at least one Shufflewriter when writing data of the target data partition to the remote Shuffle service and is stored in the remote Shuffle service. Read data from the target data partition from the remote Shuffle service, and collect the second metadata corresponding to the read data; Based on the at least one first metadata and the second metadata, the data consistency verification result of the target data partition is determined.

[0006] In one possible implementation, both the first metadata and the second metadata include at least one of the following verification elements: The number of data entries in the target data partition; The sum of the cyclic redundancy check (CRC) values ​​of all data within the target data partition; The number of data blocks aggregated within the target data partition.

[0007] In one possible implementation, determining the data consistency verification result of the target data partition based on the at least one first metadata and the second metadata includes: When the data of the target data partition comes from multiple Shuffle writer ends, the verification benchmark metadata is obtained by summing the verification elements of the same type in the first metadata of the target data partition from the multiple Shuffle writer ends. The second metadata is compared with the verification benchmark metadata, and the data consistency verification result of the target data partition is determined based on the comparison result.

[0008] In one possible implementation, the actual metadata corresponding to the collected and read data includes: During the process of reading data from the target data partition, at least one of the following operations is performed on each piece of data read: The total number of data entries; Calculate the CRC value of the data to update the accumulated CRC value; The number of data blocks is accumulated based on the completion status of the data block reading; The algorithm used to collect the actual metadata is the same as the algorithm used by the Shuffle writer to generate the first metadata.

[0009] In one possible implementation, determining the data consistency verification result of the target data partition based on the comparison result includes: If the comparison result shows that the second metadata is inconsistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to have failed. If the comparison result shows that the second metadata is consistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to be successful.

[0010] In one possible implementation, the method further includes: In response to the verification failure, the distributed computing task where the Shuffle reader is located is triggered to report an error or be terminated.

[0011] Secondly, this application provides a distributed computing system, the system comprising: a remote Shuffle service, at least one Shuffle writer terminal and at least one Shuffle reader terminal; The Shuffle writer is configured to: after processing upstream data, write the data belonging to the target data partition to the remote Shuffle service, and synchronously generate the first metadata of the target data partition and send it to the remote Shuffle service for storage. The remote Shuffle service is configured to store the data written by the Shuffle writer and the corresponding first metadata. The Shuffle reader is configured to perform the method as described in any one of claims 1-6.

[0012] In one possible implementation, the remote Shuffle service stores the first metadata in a highly available in-memory storage.

[0013] Thirdly, this application provides a Shuffle data consistency verification device, characterized in that the device is applied to a Shuffle reader in a distributed computing system, and the device includes: The metadata acquisition module is used to obtain at least one first metadata corresponding to the target data partition from the remote Shuffle service in response to a data read request for the target data partition; wherein the at least one first metadata is generated by at least one Shuffle writer when writing data of the target data partition to the remote Shuffle service and stored in the remote Shuffle service. The data reading module is used to read data from the target data partition from the remote Shuffle service and collect the second metadata corresponding to the read data; The verification module is used to determine the data consistency verification result of the target data partition based on the at least one first metadata and the second metadata.

[0014] In one possible implementation, both the first metadata and the second metadata include at least one of the following verification elements: The number of data entries in the target data partition; The sum of the cyclic redundancy check (CRC) values ​​of all data within the target data partition; The number of data blocks aggregated within the target data partition.

[0015] In one possible implementation, the verification module includes: An aggregation unit is used to perform a summation operation on each of the same type of verification elements in the first metadata of the target data partition from the multiple Shuffle writer ends when the data of the target data partition comes from multiple Shuffle writer ends, to obtain verification benchmark metadata. The comparison unit is used to compare the second metadata with the verification benchmark metadata, and determine the data consistency verification result of the target data partition based on the comparison result.

[0016] In one possible implementation, the data reading module includes: The metadata calculation unit is used to perform at least one of the following operations on each piece of data read during the process of reading data from the target data partition: The total number of data entries; Calculate the CRC value of the data to update the accumulated CRC value; The number of data blocks is accumulated based on the completion status of the data block reading; The algorithm used to collect the actual metadata is the same as the algorithm used by the Shuffle writer to generate the first metadata.

[0017] In one possible implementation, the comparison unit is specifically used for: If the comparison result shows that the second metadata is inconsistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to have failed. If the comparison result shows that the second metadata is consistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to be successful.

[0018] In one possible implementation, the device further includes: An exception response module is used to trigger an error or abort the distributed computing task where the Shuffle reader is located in response to the verification failure.

[0019] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the Shuffle data consistency verification method described in any one aspect.

[0020] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application responds to a data read request for a target data partition by obtaining at least one first metadata corresponding to the target data partition from a remote Shuffle service. The at least one first metadata is generated and stored in the remote Shuffle service by at least one Shuffle writer when writing data to the target data partition. Data of the target data partition is read from the remote Shuffle service, and second metadata corresponding to the read data is collected. Based on at least one first metadata and the second metadata, the data consistency verification result of the target data partition is determined. This provides a metadata comparison mechanism based on collaboration between the producer (Shuffle writer) and the consumer (Shuffle reader), which can proactively and accurately diagnose data integrity problems during the data shuffle stage of distributed computing. Its core lies in generating and solidifying its state baseline (first metadata) at the beginning of the data lifecycle (during writing), and generating a corresponding state snapshot (second metadata) based on the actually acquired data at the consumption point of the data lifecycle (during reading), and determining data consistency through comparison of the two. This design enables the efficient detection of any silent data corruption, loss, or tampering introduced during intermediate stages (including network transmission, remote storage persistence, server-side internal processing, and retransmission) that occur after data is written and before it is read. Therefore, this method fundamentally guarantees the reliability of data consumed by downstream computing tasks, prevents the spread of erroneous results and waste of computing resources due to dirty data injection, and significantly improves the reliability and inherent robustness of the final results of the distributed processing system. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0024] Figure 1 This application provides a schematic diagram of the architecture of a distributed computing system. Figure 2 A flowchart illustrating a Shuffle data consistency verification method provided in an embodiment of this application; Figure 3 A block diagram illustrating an embodiment of a Shuffle data consistency verification device provided in this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0027] To address the technical problem in existing technologies where data transmission errors and unreliable final calculation results are caused by the reader's unconditional trust in data received from a remote Shuffle service, this application provides a Shuffle data consistency verification method and apparatus, as well as a distributed computing system. This system can efficiently detect any data corruption, loss, or tampering introduced in the intermediate stage after data writing and before reading, thereby fundamentally ensuring the reliability of data consumed by downstream computing tasks.

[0028] Figure 1 This is a schematic diagram of the architecture of a distributed computing system provided in an embodiment of this application. Figure 1As shown, the distributed computing system includes: a remote Shuffle service 11, at least one Shuffle writer terminal 12, and at least one Shuffle reader terminal 13.

[0029] exist Figure 1 In the illustrated distributed computing system, by decoupling the data storage and transmission functions of Shuffle from the computing nodes executing computational tasks, an independent, centralized remote Shuffle service is formed. This service is not a simple storage pool, but a comprehensive data exchange layer integrating high-performance data reception, intelligent storage management, and efficient data retrieval and distribution capabilities. Built on a highly available and scalable distributed infrastructure, its core responsibility is to act as a globally trusted data intermediary, persistently storing Shuffle data from all upstream tasks and responding to data retrieval requests from downstream tasks in an efficient and reliable manner.

[0030] In this architecture, the Shuffle writer instance 12 refers to the upstream computing task instance (e.g., the Map phase task) responsible for producing Shuffle data in distributed computing. Its core function is to organize and sort the intermediate data (usually key-value pairs) that needs to be repartitioned by Shuffle after processing the allocated data shards, according to predetermined partitioning rules (e.g., modulo the hash value of the key), and directly push and persist it to the remote Shuffle service 11 via a high-speed network, rather than writing it to the local storage medium of the computing task (such as a local disk or SSD). After each Shuffle writer instance 12 completes its data writing, its output data lifecycle management responsibilities (including data integrity, availability, and final cleanup) are completely transferred to the remote Shuffle service 11. This allows computing nodes to release resources or execute new tasks statelessly, improving resource utilization.

[0031] Accordingly, Shuffle reader 13 refers to the downstream computing task instance responsible for consuming Shuffle data in distributed computing (e.g., Reduce tasks in the MapReduce framework, or Reduce or aggregation operation tasks in Spark). Its core function is to determine the target data partition to be processed based on its task allocation, and locate and pull all data fragments belonging to that partition from the remote Shuffle service 11. These data fragments originate from all related Shufflewriter 12. After acquiring the data locally, Shuffle reader 13 typically performs further merging, sorting, or aggregation operations to provide input for subsequent computing logic.

[0032] Based on the above architecture, this system establishes a remote Shuffle service 11, which, together with the Shuffle writer end 12 and the Shuffle reader end 13, forms a collaborative data pipeline. The basic workflow is as follows: multiple Shuffle writer ends 12 push partitioned data to the remote Shuffle service 11 for centralized and reliable storage; subsequently, multiple Shuffle reader ends 13 actively pull their respective required partitioned data from the remote Shuffle service 11 in parallel. This architecture separates computational and Shuffle storage resources, significantly improving the system's robustness against node failures, resource management flexibility, and scalability under large-scale deployments.

[0033] Furthermore, in the technical solution provided in the embodiments of this application, each component in the above architecture is assigned the following specific configurations to implement the Shuffle data consistency verification method provided in the embodiments of this application: Shuffle writer 12 is configured to: after processing upstream data, write data belonging to the target data partition to the remote Shuffle service 11, and synchronously generate the first metadata of the target data partition and send it to the remote Shuffle service 11 for storage. The remote Shuffle service 11 is configured to store the data written by the Shuffle writer 12 and the corresponding first metadata. Shuffle reader 13 is configured to execute the Shuffle data consistency verification method as provided in the embodiments of this application.

[0034] The principle behind this mechanism is that an immutable first metadata layer is generated and submitted by the Shuffle writer 12 when data is generated, stored reliably on a remote server, and simultaneously calculated and executed with strict comparison and verification when the Shuffle reader 13 pulls data. This constructs a proactive data integrity assurance system that spans the entire data production, storage, and consumption chain. The specific details of the first metadata layer, the second metadata layer involved in the verification process, and the Shuffle data consistency verification method provided in this application embodiment will be elaborated and explained in detail below.

[0035] Figure 2 This is a flowchart illustrating a Shuffle data consistency verification method provided in an embodiment of this application. In one embodiment, the method is applied to... Figure 1 The example shows the Shuffle reader in the distributed computing system. (e.g., ...) Figure 2 As shown, the method includes the following steps: Step 201: In response to a data read request for the target data partition, obtain at least one first metadata corresponding to the target data partition from the remote Shuffle service; wherein, the at least one first metadata is generated by at least one Shufflewriter end when writing data of the target data partition to the remote Shuffle service and stored in the remote Shuffle service.

[0036] During the execution of a distributed computing task, after the upstream Shuffle writer completes data writing, the downstream Shuffle reader is launched according to the scheduling logic of the computing task to execute the subsequent data processing stage. After launch, the Shuffle reader determines the partition identifier corresponding to the data it needs to process based on its computing logic (e.g., being assigned a Reduce task to process a specific key range or hash partition). These partitions are defined as the target data partitions for the Shuffle reader. After identifying the target data partitions, the Shuffle reader, acting as a data consumer and verification initiator, actively generates one or more data read requests for the target data partitions. In response to these data read requests, the Shuffle reader retrieves the data content and corresponding first metadata within the target data partitions from the remote Shuffle service.

[0037] Therefore, in this embodiment, when the Shuffle reader initiates a data read request, it does not only request the original data content, but also synchronously requests and obtains at least one first metadata associated with the target data partition. This first metadata is generated and committed to storage at the beginning of the data lifecycle, i.e., when all relevant Shuffle writers write data to the remote Shuffle service for the target data partition. Each Shuffle writer is independently responsible for generating the first metadata corresponding to its own written data portion. This metadata, as a true and authoritative record of the data writing state, is centrally stored in the high-availability memory storage of the remote Shuffle service, collectively forming a globally unified and tamper-proof benchmark reference system for subsequent consistency verification.

[0038] In one embodiment, the first metadata includes at least one of the following verification elements: The number of data entries in the target data partition; the sum of the CRC (Cyclic Redundancy Check) values ​​of all data in the target data partition; the number of data blocks formed by aggregation within the target data partition.

[0039] The number of data entries in the target data partition is used to record the total number of data entries written to the target data partition by the shuffle writer, which can be used to quickly verify whether data has been read repeatedly or missed.

[0040] The cumulative CRC value of all data within the target data partition is the result of performing cyclic redundancy check (CRC) calculations on the binary content of all written data (which can be byte-wise or block-wise) and accumulating the results according to certain rules. This can be used to detect with high probability whether bit-level corruption has occurred during data storage or transmission. The CRC checksum accumulation operation satisfies the commutative and associative laws algebraically. This means that regardless of the order in which data is written, stored, or read within the partition, as long as the data content itself remains unchanged, the final calculated CRC sum is unique. Specifically, if the data is divided into multiple blocks, and the CRC value of each block is CRC(Block_i), then the total sum ΣCRC(Block_i) is independent of the calculation order. This mathematical property is the theoretical basis for this method's ability to support parallel writing, out-of-order storage, and complete verification through a single final comparison at the reading end.

[0041] The number of data blocks refers to the number of persistent storage units (blocks) into which the written data is organized. It can be used to verify the integrity of the physical storage structure of the data. When shuffled data is persisted, it is usually aggregated and packaged into fixed-size data blocks to improve I / O efficiency.

[0042] See Figure 1 In the example, the first metadata recorded on the remote Shuffle service 11 for partition 1 indicates that both Shuffle writer_1 and Shuffle writer_2 have written data to partition 1. Each Shuffle writer independently generates and submits the first metadata for the corresponding partition upon completing its data write. This first metadata is received by the remote Shuffle service 11 and organized and persistently stored according to its partition. Taking partition 1 as an example, the first metadata stored in the remote Shuffle service 11 will contain independent verification information submitted separately by Shuffle writer_1 and Shuffle writer_2.

[0043] Step 202: Read data from the target data partition from the remote Shuffle service and collect the second metadata corresponding to the read data.

[0044] In step 202, while fetching the actual data, the Shuffle reader generates a new set of verification information, namely the second metadata, based on the data actually read from the target data partition using the exact same calculation rules and algorithms as the first metadata. Therefore, the second metadata should theoretically be a "mirror" or "reproduction" of the first metadata on the reading side, and the two should theoretically be completely identical.

[0045] In one embodiment, to ensure efficiency and real-time performance, the Shuffle reader synchronously (or "online") executes a metadata acquisition algorithm during the reading and decoding of the data stream. Instead of calculating after reading all the data, it performs calculations as it reads, efficiently generating the second metadata. Specifically, acquiring the actual metadata corresponding to the read data includes: during the reading of data from the target data partition, performing at least one of the following operations on each piece of data read: accumulating the number of data entries; calculating the CRC value of the data to update the CRC value sum; and accumulating the number of data blocks based on the completion status of the data block reading. The algorithm used to acquire the actual metadata is the same as the algorithm used by the Shuffle writer to generate the first metadata.

[0046] Therefore, the second metadata item collected by the Shuffle reader must correspond one-to-one with the first metadata item obtained in step 201. For example, if the first metadata includes the CRC sum and the number of entries, then the second metadata must also include the CRC sum and the number of entries calculated based on the data read in this instance. This is a crucial prerequisite for achieving reliable data consistency verification.

[0047] Step 203: Determine the data consistency verification result of the target data partition based on at least one first metadata and second metadata.

[0048] Step 203 aims to scientifically determine whether the data has undergone a complete and consistent lifecycle by accurately comparing the first metadata representing the "original state" with the second metadata representing the "actual state".

[0049] Furthermore, since the data of a target data partition may originate from multiple shuffle writers, the acquired first metadata is a set (at least one). Therefore, during verification, the data consistency verification result of the target data partition is jointly determined based on each first and second metadata.

[0050] Specifically, in one embodiment, determining the data consistency verification result of the target data partition based on at least one first metadata and a second metadata includes: when the data of the target data partition comes from multiple Shuffle writer ends, performing a summation operation on each verification element of the same type in the first metadata of the target data partition from multiple Shuffle writer ends to obtain verification benchmark metadata; comparing the second metadata with the verification benchmark metadata, and determining the data consistency verification result of the target data partition based on the comparison result.

[0051] In this embodiment, after obtaining the first metadata from all relevant Shuffle writer endpoints, an aggregation calculation is performed: for each verification element of the same type, a summation operation is performed to obtain the verification baseline metadata. This verification baseline metadata represents the complete, consistent, and globally correct state that the target data partition should have from the perspective of all writers.

[0052] For example, suppose the first metadata for Shuffle writer_1 is (recordNumber: 10, crcSum: 10, blockNumber: 10), and the first metadata for Shuffle writer_2 is (recordNumber: 20, crcSum: 20, blockNumber: 20). Then, after the above processing, the verification baseline metadata is (recordNumber: 30, crcSum: 30, blockNumber: 30). This result will be used as the baseline for subsequent consistency comparison with the second metadata generated by the Shuffle reader.

[0053] The effectiveness of the above aggregation operation depends on the mathematical properties of the verification element algorithm. As mentioned earlier, the summation of the number of data entries clearly satisfies the commutative and associative laws. More importantly, the CRC summation, due to the characteristics of the CRC algorithm itself and the definition of the summation operation, also fully satisfies the commutative and associative laws. This means that regardless of the order in which the data of each Shuffle writer is aggregated or when it is aggregated, as long as the data content remains unchanged, the final global CRC summation is definite and unique. The summation of the number of data blocks also has this property. Therefore, the aggregated verification benchmark metadata is mathematically equivalent to the metadata calculated using the same algorithm after treating all Shuffle writer data as a whole. It represents the complete, consistent, and globally correct state that the target data partition should have from the perspective of all writers.

[0054] Subsequently, the second metadata collected by the Shuffle reader based on the actual data read is compared item by item with the verification benchmark metadata calculated above. Based on the comparison results, the data consistency verification result of the target data partition is determined: if the comparison result shows that the second metadata is inconsistent with the verification benchmark metadata (any inconsistency exists), the data consistency verification of the target data partition fails; if the comparison result shows that the second metadata is consistent with the verification benchmark metadata (every item is consistent), the data consistency verification of the target data partition passes. Passing the verification indicates that the data has maintained integrity throughout the entire chain from writing, remote storage to reading, and is trustworthy and safe for subsequent computation. Conversely, failing the verification indicates that the data may have been damaged, lost, or tampered with in one or more of the above stages, and its integrity has been compromised. For example, bit errors occurred during data transmission, silent corruption occurred in the storage medium, or unexpected anomalies were triggered during the read / write process. If such inconsistent data is used for subsequent computation, it will directly lead to unreliable computation results, incorrect task output, and even cascading system problems. Therefore, verification failure is a critical risk signal that must trigger predefined fault tolerance or anomaly handling procedures. For example, in response to a verification failure, the distributed computing task hosting the Shuffle reader can be triggered to report an error or abort. This could involve causing the distributed computing task hosting the Shuffle reader to report an error, attempt to retry reading, or, in severe cases, abort the task to prevent erroneous data from contaminating the computation results.

[0055] The above-described embodiment, which performs an overall comparison by aggregating the first metadata, is an efficient verification method. By aggregating the first metadata from each Shuffle writer, it simplifies the comparison logic on the Shuffle reader side, making it particularly suitable for scenarios with high verification efficiency requirements and where precise location of the specific erroneous writer is not necessary.

[0056] In addition, the data consistency verification method of this application can also employ other comparison strategies. For example, it can use item-by-item pairing comparison logic: that is, independently pairing and comparing the first metadata corresponding to each Shuffle writer with the second metadata fragment read and calculated from the data written by that Shuffle writer. Its comparison logic includes: Data entry count comparison: If the number of entries for the corresponding Shuffle writer is inconsistent, it directly indicates that the data of that Shuffle writer has been lost or duplicated.

[0057] CRC summation and comparison: If the CRC values ​​of the corresponding Shuffle writer are inconsistent, regardless of whether the number of entries is the same, it proves that the binary content of the data written by the Shuffle writer is corrupted.

[0058] Data block count comparison: If they are inconsistent, it may indicate that the underlying storage index of the Shuffle writer data is abnormal or the data blocks are incomplete.

[0059] While this item-by-item pairing comparison method may introduce more comparison overhead when there are a large number of Shuffle writers, its advantage lies in its ability to accurately pinpoint which one or more Shuffle writers have inconsistent data, thus providing clearer information for fault diagnosis and rapid recovery. It is suitable for scenarios with high requirements for problem localization.

[0060] In summary, whether it is a comprehensive comparison through aggregation or a pairwise comparison, the core is to ensure the integrity and consistency of Shuffle data during distributed transmission and storage through a rigorous metadata calculation and comparison mechanism, thereby laying a solid foundation for the correctness of upper-layer calculations.

[0061] The Shuffle data consistency verification method provided in this application involves the Shuffle reader responding to a data read request for a target data partition by obtaining at least one first metadata corresponding to the target data partition from a remote Shuffle service. This first metadata is generated and stored in the remote Shuffle service by at least one Shuffle writer when writing data to the target data partition. The method reads data from the target data partition from the remote Shuffle service and collects the corresponding second metadata. Based on the first and second metadata, the data consistency verification result of the target data partition is determined. This provides a metadata comparison mechanism based on collaboration between the producer (Shuffle writer) and consumer (Shuffle reader), enabling proactive and accurate diagnosis of data integrity issues during the data shuffle phase of distributed computing. Its core principle is to generate and solidify the state baseline (first metadata) at the beginning of the data lifecycle (during writing), and to generate a corresponding state snapshot (second metadata) based on the actually acquired data at the consumption point (during reading), using the comparison between the two to determine data consistency. This design enables the efficient detection of any silent data corruption, loss, or tampering introduced during intermediate stages (including network transmission, remote storage persistence, server-side internal processing, and retransmission) that occur after data is written and before it is read. Therefore, this method fundamentally guarantees the reliability of data consumed by downstream computing tasks, prevents the spread of erroneous results and waste of computing resources due to dirty data injection, and significantly improves the reliability and inherent robustness of the final results of the distributed processing system.

[0062] Figure 3 This is a block diagram illustrating an embodiment of a Shuffle data consistency verification device provided in this application. Figure 3 As shown, this device is used in the Shuffle reader of a distributed computing system, and the device includes: Metadata acquisition module 31 is used to obtain at least one first metadata corresponding to the target data partition from the remote Shuffle service in response to a data read request for the target data partition; wherein the at least one first metadata is generated by at least one Shuffle writer end when writing data of the target data partition to the remote Shuffle service and stored in the remote Shuffle service. The data reading module 32 is used to read data from the target data partition from the remote Shuffle service and collect the second metadata corresponding to the read data; The verification module 33 is used to determine the data consistency verification result of the target data partition based on the at least one first metadata and the second metadata.

[0063] In one possible implementation, both the first metadata and the second metadata include at least one of the following verification elements: The number of data entries in the target data partition; The sum of the cyclic redundancy check (CRC) values ​​of all data within the target data partition; The number of data blocks aggregated within the target data partition.

[0064] In one possible implementation, the verification module 33 includes: An aggregation unit is used to perform a summation operation on each of the same type of verification elements in the first metadata of the target data partition from the multiple Shuffle writer ends when the data of the target data partition comes from multiple Shuffle writer ends, to obtain verification benchmark metadata. The comparison unit is used to compare the second metadata with the verification benchmark metadata, and determine the data consistency verification result of the target data partition based on the comparison result.

[0065] In one possible implementation, the data reading module 32 includes: The metadata calculation unit is used to perform at least one of the following operations on each piece of data read during the process of reading data from the target data partition: The total number of data entries; Calculate the CRC value of the data to update the accumulated CRC value; The number of data blocks is accumulated based on the completion status of the data block reading; The algorithm used to collect the actual metadata is the same as the algorithm used by the Shuffle writer to generate the first metadata.

[0066] In one possible implementation, the comparison unit is specifically used for: If the comparison result shows that the second metadata is inconsistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to have failed. If the comparison result shows that the second metadata is consistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to be successful.

[0067] In one possible implementation, the device further includes: An exception response module is used to trigger an error or abort the distributed computing task where the Shuffle reader is located in response to the verification failure.

[0068] like Figure 4 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the Shuffle data consistency verification method provided in any of the foregoing method embodiments, including: In response to a data read request for a target data partition, at least one first metadata corresponding to the target data partition is obtained from a remote Shuffle service; wherein, the at least one first metadata is generated by at least one Shufflewriter when writing data of the target data partition to the remote Shuffle service and is stored in the remote Shuffle service. Read data from the target data partition from the remote Shuffle service, and collect the second metadata corresponding to the read data; Based on the at least one first metadata and the second metadata, the data consistency verification result of the target data partition is determined.

[0069] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the Shuffle data consistency verification method provided in any of the foregoing method embodiments.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0073] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for verifying the consistency of Shuffle data, characterized in that, The method is applied to the Shuffle reader in a distributed computing system, and the method includes: In response to a data read request for a target data partition, at least one first metadata corresponding to the target data partition is obtained from a remote Shuffle service; wherein, the at least one first metadata is generated by at least one Shufflewriter when writing data of the target data partition to the remote Shuffle service and is stored in the remote Shuffle service. Read data from the target data partition from the remote Shuffle service, and collect the second metadata corresponding to the read data; Based on the at least one first metadata and the second metadata, the data consistency verification result of the target data partition is determined.

2. The method according to claim 1, characterized in that, Both the first metadata and the second metadata include at least one of the following verification elements: The number of data entries in the target data partition; The sum of the cyclic redundancy check (CRC) values ​​of all data within the target data partition; The number of data blocks aggregated within the target data partition.

3. The method according to claim 2, characterized in that, The step of determining the data consistency verification result of the target data partition based on the at least one first metadata and the second metadata includes: When the data of the target data partition comes from multiple Shuffle writer ends, the verification benchmark metadata is obtained by summing the verification elements of the same type in the first metadata of the target data partition from the multiple Shuffle writer ends. The second metadata is compared with the verification benchmark metadata, and the data consistency verification result of the target data partition is determined based on the comparison result.

4. The method according to claim 2, characterized in that, The actual metadata corresponding to the data collected and read includes: During the process of reading data from the target data partition, at least one of the following operations is performed on each piece of data read: The total number of data entries; Calculate the CRC value of the data to update the accumulated CRC value; The number of data blocks is accumulated based on the completion status of the data block reading; The algorithm used to collect the actual metadata is the same as the algorithm used by the Shuffle writer to generate the first metadata.

5. The method according to claim 3, characterized in that, The step of determining the data consistency verification result of the target data partition based on the comparison result includes: If the comparison result shows that the second metadata is inconsistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to have failed. If the comparison result shows that the second metadata is consistent with the verification benchmark metadata, then the data consistency verification of the target data partition is determined to be successful.

6. The method according to claim 5, characterized in that, The method further includes: In response to the verification failure, the distributed computing task where the Shuffle reader is located is triggered to report an error or be terminated.

7. A distributed computing system, characterized in that, The system includes: a remote Shuffle service, at least one Shuffle writer client, and at least one Shuffle reader client; The Shuffle writer is configured to: after processing upstream data, write the data belonging to the target data partition to the remote Shuffle service, and synchronously generate the first metadata of the target data partition and send it to the remote Shuffle service for storage. The remote Shuffle service is configured to store the data written by the Shuffle writer and the corresponding first metadata. The Shuffle reader is configured to perform the method as described in any one of claims 1-6.

8. The system according to claim 7, characterized in that, The remote Shuffle service stores the first metadata in a highly available in-memory storage.

9. A Shuffle data consistency verification device, characterized in that, The device is used in a Shuffle reader of a distributed computing system, and the device includes: The metadata acquisition module is used to obtain at least one first metadata corresponding to the target data partition from the remote Shuffle service in response to a data read request for the target data partition; wherein the at least one first metadata is generated by at least one Shuffle writer when writing data of the target data partition to the remote Shuffle service and stored in the remote Shuffle service. The data reading module is used to read data from the target data partition from the remote Shuffle service and collect the second metadata corresponding to the read data; The verification module is used to determine the data consistency verification result of the target data partition based on the at least one first metadata and the second metadata.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the Shuffle data consistency verification method according to any one of claims 1-6.

Images