Data checking method, device, equipment, storage medium and program product

Abstract

This application discloses a data verification method, apparatus, device, storage medium, and program product. The method includes: dividing a continuous data stream to be synchronized into multiple data segments, each data segment including multiple data records; during the transmission of a first data segment from a data source end through multiple data synchronization services to a target end, receiving the first data segment sent by a second data synchronization node through a first data synchronization node; obtaining a first feature value and a second feature value of the first data segment, where the first feature value is the received data feature value corresponding to a first synchronization stage where the first data synchronization node is located, and the second feature value is the transmitted data feature value corresponding to a second synchronization stage where the second data synchronization node is located; and performing data verification based on the first and second feature values. According to the embodiments of this application, the accuracy of streaming data synchronization verification can be improved, and abnormal links can be quickly located when verification fails.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data verification method, apparatus, device, storage medium and program product. Background Technology

[0002] With the development of big data technology, the construction of next-generation data query platforms increasingly relies on the collection and synchronization of massive amounts of real-time streaming data. Compared to batch data synchronization, streaming data synchronization usually requires multiple data synchronization stages, resulting in longer data transmission links and more complex structures. Furthermore, streaming data itself has the dynamic characteristics of continuous generation and no clear endpoint.

[0003] Data consistency verification is a key technical means to ensure the accuracy of data synchronization. Currently, verification for batch data synchronization mainly relies on direct comparison between source and target data. However, in streaming data synchronization scenarios, due to the continuous generation of data, the dynamic updating of source data, and the long synchronization link, it is difficult for source and target data to remain consistent at any given time.

[0004] Therefore, data verification that relies on direct comparison between source and target data is difficult to adapt to streaming data synchronization scenarios, resulting in low verification accuracy and an inability to quickly locate abnormal links when verification fails. Summary of the Invention

[0005] This application provides a data verification method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of streaming data synchronization verification and quickly locate abnormal links when verification fails.

[0006] In a first aspect, embodiments of this application provide a data verification method, the method comprising: The continuous data stream to be synchronized is divided into multiple data segments, each of which includes multiple data records; During the process of transmitting the first data segment from the data source end to the target end through multiple data synchronization services in sequence, the first data synchronization node receives the first data segment sent by the second data synchronization node. The first data segment is any one of the multiple data segments, and the first data synchronization node is any one of the target end and the multiple data synchronization services. Obtain the first feature value and the second feature value of the first data segment. The first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage. The first synchronization stage is the synchronization stage in which the first data synchronization node is located, and the second synchronization stage is the synchronization stage in which the second data synchronization node is located. Based on the first feature value and the second feature value, the consistency of the first data segment during the data synchronization process is verified.

[0007] Secondly, embodiments of this application provide a data verification device, which includes: The segmentation module is used to segment the continuous data stream to be synchronized into multiple data segments, each of which includes multiple data records; The receiving module is used to receive a first data segment sent by a second data synchronization node through a first data synchronization node during the process of transmitting a first data segment from the data source end to the target end through multiple data synchronization services in sequence. The first data segment is any one of the multiple data segments, and the first data synchronization node is any one of the target end and the multiple data synchronization services. The acquisition module is used to acquire a first feature value and a second feature value of the first data segment. The first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage. The first synchronization stage is the synchronization stage in which the first data synchronization node is located, and the second synchronization stage is the synchronization stage in which the second data synchronization node is located. The verification module is used to verify the consistency of the first data segment during the data synchronization process based on the first feature value and the second feature value.

[0008] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements any of the possible implementations of the first aspect described above.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method in any of the possible implementations of the first aspect described above.

[0010] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a method as described in any of the possible implementations of the first aspect above.

[0011] This application's embodiments dynamically segment continuous, unbounded streaming data into data segments with clearly defined boundaries, transforming the streaming data verification problem into a comparison problem of discrete data units. This overcomes the technical obstacle of being unable to directly compare data due to continuous data generation and inconsistencies between the source and target states. Since the first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage, and the first synchronization stage is the synchronization stage where the first data synchronization node is located, and the second synchronization stage is the synchronization stage where the second data synchronization node is located, and the second data synchronization node is the upstream node of the first data synchronization node, by acquiring the first and second feature values ​​of the first data segment during the transmission of the first data segment from the data source end through multiple data synchronization services to the target end, and verifying the consistency of the first data segment during the data synchronization process based on the first and second feature values, a dual feature value comparison mechanism between upstream and downstream nodes is established for the first data segment. This improves the verification accuracy of the first data segment, thereby improving the accuracy of streaming data synchronization verification. Furthermore, if the consistency check fails, the abnormal process can be quickly located based on the first and second data synchronization nodes. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of streaming data synchronization provided in one embodiment of this application; Figure 2 This is a schematic diagram of streaming data synchronization provided in another embodiment of this application; Figure 3 This is a flowchart illustrating a data verification method provided in one embodiment of this application; Figure 4 This is a schematic diagram of a data segment provided in one embodiment of this application; Figure 5 This is a schematic diagram of data verification based on centralized verification provided in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a data verification device provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0017] Furthermore, the acquisition, storage, use, and processing of data in this application's technical solution all comply with relevant national laws and regulations.

[0018] With the development of big data technology, the construction of next-generation data query platforms increasingly relies on the collection and synchronization of massive amounts of real-time streaming data. Traditional big data platforms primarily handle batch data tasks, typically processing fixed batches of data. However, next-generation data query platforms involve the collection of large amounts of real-time streaming data. Streaming data synchronization usually requires multiple transmission stages, being processed by multiple service processes before being written to a distributed database at the end of the data synchronization chain. Compared to batch data synchronization, streaming data synchronization presents its own specific complexities and challenges: 1) Streaming data synchronization is endless: Batch data synchronization processes relatively fixed data objects; while streaming data synchronization processes an endless stream of data. 2) Streaming data transmission links are usually long and complex: Batch data synchronization is usually end-to-end, such as writing directly from the source system library to the big data cluster through batch extraction, transformation, loading (ETL) tools, and its transmission link is simple; while streaming data synchronization may involve source data systems / data source libraries, enterprise-level real-time data buses, distributed message middleware, real-time computing engines, etc., and its data transmission link is long and complex. 3) Streaming data processing typically requires fault tolerance: Batch data synchronization can ensure no data loss through atomic batch tasks. If a single or multiple data entries fail to be written, the batch task can fail entirely, and the data can be re-imported by redoing the task. Streaming data synchronization cannot interrupt the processing flow due to individual data issues. It often needs to skip abnormal data and continue with subsequent data synchronization to ensure the Service Level Agreement (SLA) requirements of real-time data services are met.

[0019] Data consistency verification is a key technical means to ensure the accuracy of data synchronization. Data synchronization accuracy verification is generally completed by comparing data information at the data source and data target ends. Using a scheme that compares source and target data can effectively solve the accuracy verification problem of batch data synchronization. Because the data objects synchronized in batch tasks have a fixed beginning and end, the comparison between source and target data can be directly obtained after the batch synchronization task is completed. However, compared to batch synchronization scenarios, streaming data synchronization cannot guarantee data consistency between the data source and target ends at any given time. The main reasons include: 1) Streaming data synchronization is continuous, with no endpoint; during the synchronization process, new data is constantly being inserted and historical data is being updated on the source end; 2) The long synchronization chain and extended synchronization time of streaming data further exacerbate the possibility of inconsistency between source and target data; 3) Streaming data synchronization often requires fault-tolerant handling when data anomalies occur, resulting in some abnormal data being temporarily removed from the target database. In summary, for streaming data synchronization, it is not possible to directly compare the source data with the target data.

[0020] In addition, comparing data between the data source and the target has the following problems and shortcomings: 1) Data comparison is applied to the data storage of the source and target ends, which will cause additional resource / performance consumption to the source and target ends. For example, running statistical data count or reading details for comparison on the source and target databases will consume certain database resources, thereby affecting performance and impacting normal business operations. 2) Data comparison, performed on the source and target data storage, can directly identify inconsistencies. However, in the long chain of streaming data synchronization, it's impossible to determine which link experienced data loss or tampering. This makes it difficult to pinpoint the problematic link and initiate further remediation efforts. Furthermore, if multiple parties (such as system developers) are involved in the streaming data synchronization process, inconsistencies in small amounts of data can lead to situations where responsibility cannot be determined, resulting in finger-pointing and blame-shifting.

[0021] To address the related technical issues, embodiments of this application provide a data verification method, apparatus, electronic device, computer-readable storage medium, and computer program product. The data verification method can be applied to scenarios where data accuracy needs to be verified during real-time streaming synchronization of massive amounts of data.

[0022] Figure 1 A schematic diagram of streaming data synchronization provided in one embodiment of this application is shown.

[0023] like Figure 1 As shown, data synchronization from the source to the target can be achieved through layered data transfer along a synchronization link. The data synchronization link can include the data source, data synchronization services / tools, and the target data source. The data source can include the data source system and its underlying data source library. Data synchronization services / tools can include data publishing services, enterprise-level data buses, data subscription services, Change Data Capture (CDC) synchronization tools, message queues, and database persistence services. The target data source can include the data-using system and its underlying target library.

[0024] In addition, streaming data synchronization can include multiple data publishing scenarios, such as data sources publishing data directly, data sources publishing data through a data bus, and data sources publishing data through CDC synchronization tools.

[0025] In scenarios where data is directly published from the data source, the data source system bypasses an intermediate database and directly publishes data events to a message queue via a data publishing service. The message queue then processes the data. Subsequently, the data processing results are written to a specified target database (e.g., target database 1) via a database write service 1. Target database 1 can belong to data center 1.

[0026] In scenarios where data sources publish data via a data bus, the data source system, acting as a standardized data producer, sends the generated business data to the enterprise-level data bus through a unified data publishing service. Simultaneously, it persists the generated business data to its underlying data source database. The enterprise-level data bus is responsible for data persistence, routing, and broadcasting. Subsequently, based on business needs, data subscription services 1 and 2 can simultaneously subscribe to and consume the message, and publish the message to a message queue for data processing. Then, through write-to-database services 2 and 3, respectively, the data processing results are independently written to the corresponding target databases (such as target database 1 and target database 2). Target database 2 can belong to data center 2.

[0027] In scenarios where the data source publishes data via the CDC tool, the data source system first persists the data changes generated by business operations to its underlying data source database. Subsequently, the CDC synchronization tool monitors the database logs in real time, captures this update / change event, and publishes it as a structured data change message to a message queue for data processing. The message queue then writes the data processing results to the specified target database (e.g., target database 2) via the write-to-database service 4.

[0028] This application takes into account that, if the above data synchronization process is refined and abstracted, streaming data synchronization can be divided into, for example: Figure 2 The diagram shows several (e.g., N, where N is a positive integer greater than 1) synchronization stages. Each synchronization stage has a set of several service processes that simultaneously receive data from upstream and send data downstream. If we assume that the S-th synchronization stage has M (M is a positive integer greater than 1) service processes, then ST can be used to identify the T-th (0 < T <= M) synchronization service in the S-th stage.

[0029] Based on the abstraction of multiple data synchronization processes into multiple data synchronization stages, the data verification method provided in the embodiments of this application is described below.

[0030] Figure 3 A flowchart illustrating a data verification method according to an embodiment of this application is shown. This data verification method can be executed by a data verification system. The data verification system may include a data source (hereinafter referred to as the source end), a data target (hereinafter referred to as the target end), and multiple data synchronization services. The data verification system can perform data verification through both centralized and distributed verification methods.

[0031] Distributed verification can be performed by various data synchronization nodes along the data transmission link, which is simple to deploy and requires no additional application services. However, this distributed verification has certain requirements for the data transmission method. Each synchronization stage must ensure that the data segment is an atomic unit of data transmission: that is, all data records of the same data segment sent by the data publishing service must be centrally received and processed by the same data consumer service. In other words, each synchronization stage can correspond to one data synchronization node. For example, Figure 2 Data source system → Synchronization service 1.3 → Synchronization service 2.2 → ... → Synchronization service SM → ... → Synchronization service N.3 → Data usage system data transmission method can use distributed verification method to verify data consistency.

[0032] Furthermore, centralized data verification has a wider range of applications. Data in the same segment can be processed by multiple data consumer services without needing to be centralized within a single service. That is, each synchronization stage can correspond to one or more data synchronization nodes. If centralized verification is used, the data verification system can also include a data feature storage service, a data consistency verification service, and a data storage module. The data feature storage service collects data feature values ​​from all synchronization services, globally aggregates them, and centrally stores them in the data storage module. The data consistency verification service periodically retrieves the aggregated data feature values ​​from the data storage module for comparison and verification.

[0033] like Figure 3 As shown, the data verification method provided in this application includes the following steps: S310. Divide the continuous data stream to be synchronized into multiple data segments, each data segment including multiple data records; S320. During the process of transmitting the first data segment from the data source end to the target end through multiple data synchronization services in sequence, the first data synchronization node receives the first data segment sent by the second data synchronization node. The first data segment is any one of the multiple data segments, and the first data synchronization node is any one of the target end and multiple data synchronization services. S330. Obtain the first feature value and the second feature value of the first data segment. The first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage. The first synchronization stage is the synchronization stage in which the first data synchronization node is located, and the second synchronization stage is the synchronization stage in which the second data synchronization node is located. S340. Based on the first feature value and the second feature value, verify the consistency of the first data segment during the data synchronization process.

[0034] This application's embodiments dynamically segment continuous, unbounded streaming data into data segments with clearly defined boundaries, transforming the streaming data verification problem into a comparison problem of discrete data units. This overcomes the technical obstacle of being unable to directly compare data due to continuous data generation and inconsistencies between the source and target states. Since the first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage, and the first synchronization stage is the synchronization stage where the first data synchronization node is located, and the second synchronization stage is the synchronization stage where the second data synchronization node is located, and the second data synchronization node is the upstream node of the first data synchronization node, by acquiring the first and second feature values ​​of the first data segment during the transmission of the first data segment from the data source end through multiple data synchronization services to the target end, and verifying the consistency of the first data segment during the data synchronization process based on the first and second feature values, a dual feature value comparison mechanism between upstream and downstream nodes is established for the first data segment. This improves the verification accuracy of the first data segment, thereby improving the accuracy of streaming data synchronization verification. Furthermore, if the consistency check fails, the abnormal process can be quickly located based on the first and second data synchronization nodes.

[0035] The specific implementation methods for each of the above steps are described below.

[0036] In some embodiments, in S310, before sending data, the data source system may first divide the continuous data stream to be synchronized into multiple data segments and set a unique segment identifier for each data segment. Each data segment may include multiple data records.

[0037] As an example, different data stream segmentation methods can be used for different data validation methods.

[0038] In cases where the data verification method is distributed verification, in order to improve data verification efficiency, in some embodiments, the above-mentioned S310 may specifically include: Based on the preset data record volume, the continuous data stream to be synchronized is divided into multiple data segments.

[0039] Here, the preset data record count can be the number of continuously collected data records set in advance. The preset data record count can be denoted as L.

[0040] As an example, a data source system can divide a data stream into multiple data segments based on a preset number of data records. For instance, each L data records can generate a data segment, and a segment identifier can be randomly generated for that data segment. This segment identifier can uniquely identify the data segment.

[0041] In this embodiment of the application, when the data verification method is distributed verification, the continuous data stream to be synchronized is divided into multiple data segments based on a preset data record volume. This allows for the establishment of clear logical boundaries for each data segment, making it an atomic unit that can be independently transmitted and verified. Furthermore, this segmentation method enables data verification to be embedded within the flow of data segments, thereby improving data verification efficiency.

[0042] In cases where the data verification method is centralized verification, to improve the reliability of data verification, in some embodiments, the above-mentioned S310 may specifically include: Based on a preset time slice, the continuous data stream to be synchronized is divided into multiple data segments.

[0043] Here, the preset time slice can be a pre-set time slice used to determine the data segment to which a data record belongs. If multiple data records belong to the same time slice, then these multiple data records belong to the same data segment and have the same segment identifier. In addition, since the data records are transmitted in real time streaming, and the data segments are obtained by dividing the streaming data based on the time slice, multiple data records within the same data segment can be transmitted in multiple sessions.

[0044] As an example, when sending data records, the data source system can generate a unique segment identifier for each data record based on the time slice to which the data record belongs. The segment identifier can be determined based on the time slice to which the data record belongs. For example, the start point of the time slice to which the data record belongs can be used as the segment identifier. For instance, if every 15 minutes can be considered a time slice, then the segment identifier 202511051600 can represent the unique identifier of the data segment corresponding to the time slice from 16:00 to 16:15 on November 5, 2025.

[0045] This application embodiment, in the case of centralized data verification, divides the continuous data stream to be synchronized into multiple data segments based on a preset time slice. This establishes clear logical boundaries for each data segment, making it an atomic unit that can be independently transmitted and verified. Furthermore, this segmentation method provides a unified statistical aggregation window for all data synchronization nodes across the entire link, thereby enabling global data verification through a unified data consistency verification service and improving the reliability of data verification.

[0046] In some embodiments, in S320, the first data synchronization node can be any one of the data target end and multiple data synchronization services, and the second data synchronization node can be the data source end and multiple data synchronization services, located upstream of the first data synchronization node and adjacent to the first data synchronization node.

[0047] During the process of transmitting any data segment (i.e., the first data segment) from the data source end to the target end through multiple data synchronization services, the first data synchronization node can receive the first data segment sent by the second data synchronization node.

[0048] If the data verification method is distributed verification, sending the first data segment from the second data synchronization node to the first data synchronization node can involve first sending a first segment start marker with a data table identifier and a first segment identifier, then sequentially sending multiple first data records within the first data segment. Simultaneously, the data feature value (i.e., the data segment feature value) of the segment is calculated and generated at the moment the first data record is sent, and finally, a first segment end marker with the data feature value is sent. The data table identifier can be a unique identifier of the data table to which the first data segment belongs. The data table identifier can be abbreviated as data id. The first segment identifier can be the segment identifier corresponding to the first data segment when the data verification method is distributed verification. The segment identifier can be abbreviated as segment id. A schematic diagram of a data segment provided in one embodiment of this application can be shown as follows: Figure 4 As shown.

[0049] In addition, data characteristics can include data fingerprints and data statistics. A data fingerprint can be a fixed-length string generated by hashing a data segment to calculate a summary. Data statistics can be the number of data records in the data segment (count), or a summary value calculated based on certain important column values ​​(such as sum / min / max).

[0050] If the data verification method is centralized verification, the first data segment sent from the second data synchronization node to the first data synchronization node can be multiple data records within the streaming time slice, and the second segment identifiers corresponding to the multiple data records. The second segment identifier can be the segment identifier corresponding to the first data segment when the data verification method is centralized verification.

[0051] In some embodiments, in S330, the first feature value may be a data feature value of a first data segment calculated by a first data synchronization node. The second feature value may be a data feature value of a second data segment calculated by a second data synchronization node.

[0052] Based on this, in order to improve data verification efficiency and accurately locate abnormal steps when verification fails, in some embodiments, the above-mentioned S330 may specifically include: Perform the following operations through the first data synchronization node: Based on multiple first data records in the first data segment, determine the first feature value; Obtain the second feature value from the end marker of the first segment.

[0053] Here, the first synchronization stage can correspond to a first data synchronization node. Upon receiving the first data segment, the first data synchronization node can, on the one hand, determine a first feature value based on multiple first data records in the first data segment according to the feature value calculation rules; on the other hand, it can obtain a second feature value from the end marker of the first segment. The second feature value can be calculated by the second data synchronization node based on multiple first data records in the first data segment according to the feature value calculation rules. Furthermore, the feature value calculation rules can be data fingerprint calculation rules or data statistical information calculation rules.

[0054] This application embodiment uses distributed verification, that is, the first feature value and the second feature value of the first data segment are obtained by the first data synchronization node respectively, and the data consistency verification is performed by the first data synchronization node based on the first feature value and the second feature value. This enables real-time verification of the first data segment in single-hop transmission, which not only improves the data verification efficiency, but also accurately locates the abnormal link when the verification fails.

[0055] In addition, to ensure the feasibility and reliability of data consistency verification in complex streaming links, in some embodiments, the above-mentioned S330 may specifically include: The first feature value and the second feature value of the first data segment are obtained from the data storage module. The data storage module is used to store the data feature values ​​corresponding to each data segment at the data source end, the target end and multiple synchronization stages respectively. The data feature values ​​include at least one of the transmitted data feature value and the received data feature value.

[0056] Here, the data storage module can be a relational database or a time-series database. The data storage module can summarize and calculate data characteristic values ​​based on dimensions such as data ID, synchronization stage, and segment ID. Each synchronization stage can correspond to one or more data synchronization services. If the data characteristic value is the number of data records in a data segment, then a storage example can be shown in Table 1 below: Table 1 Since each synchronization stage can correspond to one or more data synchronization services, and multiple data records corresponding to a data segment can be sent in multiple batches, the data feature storage service can periodically receive the data feature values ​​sent by each data synchronization node, and summarize the data feature values ​​based on the synchronization stage corresponding to each data synchronization node to obtain the data feature values ​​of each data segment in each synchronization stage as shown in Table 1.

[0057] This application embodiment addresses the challenge of asynchronous reporting and aggregation of data feature values ​​in a distributed architecture by first receiving the data feature values ​​sent by each data synchronization node separately, then summarizing the data feature values ​​based on the synchronization stage corresponding to each data synchronization node, and storing them in the data storage module. This effectively solves the problem of asynchronous reporting and aggregation of data feature values ​​in a distributed architecture. Thus, even in scenarios where data records arrive in batches and feature values ​​are reported in a dispersed manner, a complete verification view can still be reconstructed according to the synchronization stage dimension, providing an accurate and complete cross-stage data foundation for subsequent centralized comparison, thereby ensuring the feasibility and reliability of data consistency verification in complex streaming links.

[0058] In cases where the first data segment is transmitted multiple times, to ensure the feasibility and reliability of data consistency verification in complex streaming links, in some embodiments, before obtaining the first and second feature values ​​of the first data segment from the data storage module, the method may further include: Before sending multiple sub-first data records in the first data segment to the first data synchronization node, the second sub-second feature value is determined based on the multiple sub-first data records through the second data synchronization node. The second data synchronization node sends the second segment identifier and sub-second feature value corresponding to the first data segment to the data feature storage service, so that the data feature storage service can determine the second feature value corresponding to the first data segment based on the multiple sub-second feature values ​​corresponding to the second synchronization stage, and store it in the data storage module. In the case of receiving multiple sub-first data records sent by the second data synchronization node through the first data synchronization node, the sub-first feature value is determined based on the multiple sub-first data records; The first data synchronization node sends the second segment identifier and sub-first feature value corresponding to the first data segment to the data feature storage service, so that the data feature storage service determines the first feature value corresponding to the first data segment based on the multiple sub-first feature values ​​corresponding to the first synchronization stage, and stores it in the data storage module.

[0059] Here, after receiving multiple sub-first data records from the first data segment sent by its upstream and adjacent fourth data synchronization node, the second data synchronization node can first determine the receiving feature value corresponding to the second data synchronization node based on these multiple sub-first data records according to the data feature value calculation rules, obtain the sub-fourth feature value, and cache the sub-fourth feature value. If the data feature value calculation rules are used to calculate the number of data records, the caching format can be as follows: {number of received data records: [segment id, data feature value]}. For example, {“number of received segment data records”: [202511051600,10000]} indicates that the data synchronization node received 10,000 data records during the time segment from 16:00 to 16:15 on November 5, 2025.

[0060] Then, the second data synchronization node can process the multiple sub-first data records to obtain updated sub-first data records, and send the updated sub-first data records to the first data synchronization node. Alternatively, the second data synchronization node may choose not to process the multiple sub-first data records and directly send them to the first data synchronization node.

[0061] Before sending multiple sub-first data records to the first data synchronization node, the second data synchronization node can first determine the corresponding sending characteristic value of the second data synchronization node based on these multiple sub-first data records according to the data characteristic value calculation rules, obtain the sub-second characteristic value, and cache the sub-second characteristic value. If the data characteristic value calculation rules are used to calculate the number of data records, the caching format can be as follows: {number of data records sent: [segment id, data characteristic value]}. For example, {“number of segment data records sent”: [202511051600, 8000]} indicates that the data synchronization node sent 8000 data records during the time segment from 16:00 to 16:15 on November 5, 2025.

[0062] During the caching of the sub-second feature value, the second data synchronization node may have generated multiple sub-second feature values ​​based on multiple sub-first data records received separately. In this case, the second data synchronization node can also locally summarize the multiple sub-second feature values ​​and periodically (e.g., every 5 seconds) report the locally cached data feature values ​​and their corresponding second segment identifiers to the data feature storage service. The data feature storage service collects the feature values ​​sent by all data synchronization nodes corresponding to the second synchronization stage, and performs a global summary of the data feature values ​​based on dimensions such as data ID, synchronization stage, and segment ID to obtain the final feature value (i.e., the second feature value) corresponding to the second synchronization stage, and centrally stores it in the data storage module.

[0063] In addition, after receiving multiple sub-first data records sent by the second data synchronization node, the first data synchronization node can first determine the receiving feature value corresponding to the first data synchronization node based on the multiple sub-first data records according to the data feature value calculation rules, obtain the sub-first feature value, and cache the sub-first feature value.

[0064] Then, the first data synchronization node can process the multiple sub-first data records to obtain updated sub-first data records, and send the updated sub-first data records to the data synchronization node in the next synchronization stage. Alternatively, the first data synchronization node may not process the multiple sub-first data records and directly send them to the data synchronization node in the next synchronization stage.

[0065] Before sending multiple sub-first data records to the data synchronization node in the next synchronization stage, the first data synchronization node can first determine the sending feature value corresponding to the first data synchronization node based on the multiple sub-first data records according to the data feature value calculation rules, obtain the sub-first feature value, and cache the sub-first feature value.

[0066] During the caching of the first sub-feature value, the first data synchronization node may have generated multiple first sub-feature values ​​based on multiple separately received first sub-data records. In this case, the first data synchronization node can also locally summarize the multiple first sub-feature values ​​and periodically (e.g., every 5 seconds) report the locally cached data feature values ​​and their corresponding second segment identifiers to the data feature storage service. The data feature storage service collects the feature values ​​sent by all data synchronization nodes corresponding to the first synchronization stage, and performs a global summary of the data feature values ​​based on dimensions such as data ID, synchronization stage, and segment ID to obtain the final feature value (i.e., the first feature value) corresponding to the first synchronization stage, and centrally stores it in the data storage module.

[0067] This application embodiment periodically sends sub-feature values ​​determined based on partial sub-data records in a data segment to the data feature storage service by each data synchronization node. The data feature storage service collects the feature values ​​sent by all data synchronization nodes and performs a global summary of the data feature values ​​based on dimensions such as data ID, synchronization stage, and segment ID to obtain the final feature value corresponding to the synchronization stage. This value is then centrally stored in the data storage module. This effectively solves the problem of asynchronous reporting and aggregation of data feature values ​​in a distributed architecture. Thus, even in scenarios where data records arrive in batches and feature values ​​are reported in a distributed manner, a complete verification view can still be reconstructed according to the synchronization stage dimension. This provides an accurate and complete cross-stage data foundation for subsequent centralized comparison, thereby ensuring the feasibility and reliability of data consistency verification in complex streaming links.

[0068] Based on this, in order to provide a structured data foundation for centralized data consistency comparison across nodes and stages, in some embodiments, for any one of the first and second data synchronization nodes, a target data synchronization node is sent to the data feature storage service through the target data synchronization node, along with the second segment identifier corresponding to the first data segment and the sub-target feature value determined by the target data synchronization node. Specifically, this may include: To synchronize data with the target node, perform the following operations: The third segment identifier is determined based on the node identifier corresponding to the target data synchronization node, the synchronization stage corresponding to the target data synchronization node, the data table identifier to which the first data segment belongs, and the second segment identifier corresponding to the first data segment. According to the target format, based on the data feature name corresponding to the target feature value, the third segment identifier, and the sub-target feature value, determine the sub-data feature information corresponding to the first data segment; Send the sub-data feature information corresponding to the first data segment to the data feature storage service.

[0069] Here, if the node identifier is denoted as node id and the second segment identifier is denoted as segment id, then the target format can be as follows: {data feature name (number of segment data records sent / number of segment data records received): {node id, synchronization stage, data id, segment id}, data feature value}. Sub-data feature information can represent the data feature value information of multiple sub-first data records of a synchronization node within the first data segment of a certain data.

[0070] This application embodiment assembles data feature names, node IDs, synchronization stages, data IDs, segment IDs, and data feature values ​​according to the above target format to obtain sub-data feature information corresponding to the first data segment, and sends the sub-data feature information corresponding to the first data segment to the data feature storage service. This enables the data feature storage service to globally summarize data feature values ​​based on dimensions such as data ID, synchronization stage, and segment ID, providing a structured data foundation for centralized data consistency comparison across nodes and stages.

[0071] In some embodiments, in S340, after obtaining the first feature value and the second feature value corresponding to the first data synchronization segment, the first feature value and the second feature value can be compared to verify the consistency of the first data segment during the data synchronization process.

[0072] If the data verification method is distributed verification, the first data synchronization node can compare the first feature value and the second feature value. If the first feature value and the second feature value are the same, the verification result can be determined as successful. If the first feature value and the second feature value are different, the verification result can be determined as failed.

[0073] Therefore, in order to improve the efficiency of problem investigation, in some embodiments, after S340 above, the method may further include: If the verification result is a failure, the first feature value and the first data segment are stored to investigate and analyze the reasons for the verification failure.

[0074] Here, the first data segment may include a first segment start marker with a data table identifier and a first segment identifier, multiple first data records, and a first segment end marker with a second feature value.

[0075] If the verification result is a failure, that is, if the data comparison is found to be inconsistent, the first data synchronization node can save the inconsistent data feature values ​​(i.e., the first feature value and the second feature value) and the data details of the abnormal data segment (i.e., multiple first data records) for subsequent troubleshooting and analysis.

[0076] This application embodiment stores a first feature value and a first data segment when the verification result is a verification failure. This not only enables rapid location of abnormal links, but also provides a complete snapshot of the situation for the abnormality. This supports accurate diagnosis of specific problems such as data loss, tampering, or duplication, and greatly improves the efficiency of problem investigation.

[0077] In addition, an error alarm can be issued when the verification fails, in order to further improve the efficiency of troubleshooting.

[0078] Furthermore, to ensure the structural integrity and verification continuity of the data segments, in some embodiments, after S340 above, the method may further include: Perform the following operations through the first data synchronization node: Multiple first data records are processed to obtain multiple second data records; The third feature value is determined based on multiple second data records; Based on the third feature value, determine the updated end marker of the first segment; Assemble the first segment start marker, multiple second data records, and the updated first segment end marker to obtain the updated first data segment; Send the updated first data segment to the third data synchronization node.

[0079] Here, after comparing and verifying the first and second feature values, the first data synchronization node can perform data processing on multiple first data records for this stage, regardless of whether the verification result is successful or unsuccessful, to obtain multiple second data records. Based on the feature value calculation rules and the multiple second data records, it determines a third feature value, determines an updated first segment end marker based on the third feature value, and assembles the first segment start marker, multiple second data records, and the updated first segment end marker to obtain an updated first data segment. The updated first data segment is then sent downstream to a third data synchronization node adjacent to the first data synchronization node, so that the third data synchronization node can continue to repeat the above distributed verification process. This assembly can include logical assembly and physical assembly, which are not limited here. Specifically, when sending the updated first data segment, it supports sending it in the form of continuous streaming messages or aggregated messages. That is, if the assembly is logical assembly, this embodiment can continuously send the first segment start marker with data table identifier and first segment identifier, multiple second data records, and the updated first segment end marker. If the assembly is a physical assembly, then in this embodiment of the application, the message can be assembled according to the target message protocol format, based on the first segment start marker, multiple second data records, and the updated first segment end marker, to obtain an aggregated message, which is then sent to the third data node. The target message protocol format can be the same as the message protocol format of the first data segment received by the first data synchronization node.

[0080] Of course, the first data node may not perform any data processing, so the first data node can directly send the received first data segment to the third data synchronization node.

[0081] The embodiments of this application perform the above operations through the first data synchronization node, which can ensure the structural integrity and verification continuity of data segments while performing necessary processing (such as cleaning and transformation) on data records.

[0082] Furthermore, if the data verification method is centralized, the data consistency verification service can periodically scan the data storage module, determine multiple data feature values ​​corresponding to each data segment within the data storage module, and compare the transmitted data feature value corresponding to the previous synchronization stage with the received data feature value corresponding to the next synchronization stage as a checksum. For example, for the first data segment, the received data feature value corresponding to the first synchronization stage (i.e., the first feature value) can be compared with the transmitted data feature value corresponding to the second synchronization stage (i.e., the second feature value). If all checksums corresponding to the first data segment are successfully verified, the verification result is considered successful. If any checksum fails to verify, the verification result is considered unsuccessful.

[0083] Therefore, in order to improve the efficiency of problem investigation, in some embodiments, after S340 above, the method may further include: If the verification result is a verification failure, record the second segment identifier, the first synchronization stage, and the second synchronization stage to investigate and analyze the reasons for the verification failure.

[0084] This application's embodiments, by recording the abnormal data segments and inconsistent synchronization stages when verification fails, can quickly locate the abnormal link and improve troubleshooting efficiency. Furthermore, an anomaly alarm can be issued when verification fails to further enhance troubleshooting efficiency.

[0085] To better understand the overall solution, some specific examples are given based on the above embodiments.

[0086] For example, in the case of Figure 1 The data synchronization process shown is extracted and abstracted to obtain the following: Figure 2 Following the data synchronization process shown, this application embodiment can perform data verification using both centralized and distributed verification methods.

[0087] For centralized verification, a data feature storage service and a data consistency verification service can be built for centralized feature value storage and comparison. When a synchronization service at each stage receives a data record, it can calculate the record's data feature value, cache it locally and aggregate it partially, and periodically report the aggregated data segment feature values ​​to the data feature storage service. The data feature storage service can collect the data feature values ​​from all synchronization services, aggregate them globally, and store them centrally. The data consistency verification service can periodically retrieve the aggregated data feature values ​​from the data feature storage service for comparison and verification. A schematic diagram of data verification based on centralized verification provided in one embodiment of this application is shown below. Figure 5 As shown.

[0088] The centralized verification method can be implemented as follows: First, when the data source system sends data records, it can generate a unique segment ID for each data record based on the data transmission time slice. Specifically, the start point of the time slice can be used as the segment ID. For example, if each 15 minutes is considered a time slice, then segment ID 202511051600 represents the time slice from 16:00 to 16:15 on November 5, 2025. The data source includes this segment ID in the metadata of the data record it sends.

[0089] Secondly, after receiving the data records sent in the previous stage, each data synchronization service can calculate the data feature value based on the data record information and summarize it locally to calculate {number of received data records: [segment ID, data feature value]}. For example, {“number of received segment data records”: [202511051600, 10000]} indicates that the data synchronization node received 10,000 data records during the time segment from 16:00 to 16:15 on November 5, 2025.

[0090] Furthermore, when sending data records to the next stage, each data synchronization service can calculate data feature values ​​based on the data record information and locally summarize and calculate {number of data records sent: [segment ID, data feature value]}. For example, {"number of segment data records sent": [202511051600, 8000]} indicates that the data synchronization node sent 8000 data records during the time segment from 16:00 to 16:15 on November 5, 2025.

[0091] In addition, each data synchronization service can periodically (e.g., every 5 seconds) report the locally cached data feature values ​​to the data feature storage service. The data feature values ​​can be in the format {data feature name (number of data segments sent / number of data segments received): {node id, synchronization stage, data id, segment id}, data feature value}, which represents the data feature value information related to a certain data segment of a certain synchronization service.

[0092] In addition, after receiving the data feature value information reported by the synchronization service, the data feature storage service can summarize and calculate the data feature values ​​in the external data storage module based on dimensions such as data ID, synchronization stage, and segment ID. A storage example is shown in Table 1.

[0093] In addition, the data consistency verification service can periodically scan the data feature storage service to compare various indicators of the same data segment to see if they are consistent. If they are inconsistent, an anomaly alarm will be issued, and the relevant abnormal data segment and the synchronization stage of inconsistency will be recorded to facilitate subsequent troubleshooting and resolution.

[0094] For distributed verification, during data synchronization, the data synchronization service can perform distributed comparison and verification of the feature values ​​of segmented data. When the data source sends data records for a data segment, it can generate data feature values ​​based on the data of that segment and send them along with the data through the transmission channel. When data is received at each stage of the transmission link, the synchronization service can recalculate the data feature values ​​based on the received data segment records and compare them with the data feature values ​​received on the link to verify the accuracy of the data at that stage. By running this process at each stage of the streaming transmission link, each step is interconnected, ensuring the reliability of the data throughout the entire link.

[0095] The distributed verification method can be specifically described as follows: First, the data source system can divide the data pipeline into data segments based on the number of data records, such as generating a data segment for every L records. The segment ID is randomly generated to ensure global uniqueness. Before sending a data segment, the data source system can first send a segment start marker (with both data ID and segment ID), and then send the data records in the data segment sequentially. While sending data records, the system can simultaneously calculate and generate the data characteristic value of the segment, such as a data fingerprint or the number of data records. Finally, a segment end marker (with the characteristic value of the data segment) is sent at the end of the data segment.

[0096] Secondly, for each synchronization service, upon receiving the segment start marker, the calculation of the data segment feature value is reset. The data feature value is calculated according to the rules based on the subsequently received data segment records. Finally, it is compared with the data feature value in the received segment end marker to verify the accuracy of the received data in this step.

[0097] Furthermore, when data discrepancies are detected, the synchronization service can save the inconsistent data characteristics along with the data details of the abnormal segments for subsequent troubleshooting and analysis.

[0098] In addition, after completing the data processing in this stage, the synchronization service assembles the data packet according to the message protocol format of the received data segment (if no data processing is performed in this stage, the received data segment packet can be used directly), and sends the data to the synchronization service in the next stage, where the received data is compared.

[0099] In summary, this application embodiment implements segmented data pipeline comparison in a streaming data transmission link. The data comparison process is completed during transmission, rather than being carried out in the data storage at the source and target ends. According to preset rules, the continuous data stream is divided into multiple fixed data segments, and the data comparison is performed on these data segments. At this time, the object of data comparison is a fixed, immutable data pipeline segment, rather than the constantly changing data source and target storage, avoiding the problem of inconsistent data states at the source and target in streaming data. Furthermore, the data comparison scheme in this application embodiment does not need to operate on the data storage at the source and target ends, does not occupy the computing resources of the source and target storage, and will not negatively impact their performance or service operations. In addition, by comparing each link in the data transmission link, if an anomaly is found in the transmitted data, the problematic link can be quickly located, responsibility is clearly defined, and further troubleshooting is convenient. Thus, through this application embodiment, anomalies can be quickly detected in long streaming data transmission links, ensuring the accuracy and reliability of the data.

[0100] Based on the data verification method provided in the above embodiments, this application also provides specific implementation methods of the data verification device. Please refer to the following embodiments.

[0101] like Figure 6 As shown, a data verification device 600 provided in one embodiment of this application includes the following modules: The segmentation module 610 is used to segment the continuous data stream to be synchronized into multiple data segments, each data segment including multiple data records; The receiving module 620 is used to receive the first data segment sent by the second data synchronization node through the first data synchronization node during the process of transmitting the first data segment from the data source end through multiple data synchronization services to the target end. The first data segment is any one of the multiple data segments, and the first data synchronization node is any one of the target end and multiple data synchronization services. The acquisition module 630 is used to acquire a first feature value and a second feature value of a first data segment. The first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage. The first synchronization stage is the synchronization stage in which the first data synchronization node is located, and the second synchronization stage is the synchronization stage in which the second data synchronization node is located. The verification module 640 is used to verify the consistency of the first data segment during the data synchronization process based on the first feature value and the second feature value.

[0102] The data verification device 600 described above will be explained in detail below: In some embodiments, the acquisition module 630 may specifically include: The acquisition submodule is used to acquire the first feature value and the second feature value of the first data segment from the data storage module. The data storage module is used to store the data feature values ​​corresponding to each data segment at the data source end, the target end and multiple synchronization stages respectively. The data feature values ​​include at least one of the sent data feature value and the received data feature value.

[0103] In some embodiments, the first data segment is transmitted in multiple parts. Based on this, the acquisition module 630 may further include: The determination submodule is used to determine the sub-second feature value based on multiple sub-first data records before obtaining the first feature value and the second feature value of the first data segment from the data storage module, and before sending multiple sub-first data records in the first data segment to the first data synchronization node through the second data synchronization node. The sending submodule is used to send the second segment identifier and sub-second feature value corresponding to the first data segment to the data feature storage service through the second data synchronization node, so that the data feature storage service can determine the second feature value corresponding to the first data segment based on the multiple sub-second feature values ​​corresponding to the second synchronization stage, and store it in the data storage module; The determination submodule is also used to determine a sub-first feature value based on multiple sub-first data records when multiple sub-first data records are received from the second data synchronization node via the first data synchronization node. The sending submodule is also used to send the second segment identifier and sub-first feature value corresponding to the first data segment to the data feature storage service through the first data synchronization node, so that the data feature storage service can determine the first feature value corresponding to the first data segment based on the multiple sub-first feature values ​​corresponding to the first synchronization stage, and store it in the data storage module.

[0104] In some embodiments, the sending submodule can specifically be used for: For either the first data synchronization node or the second data synchronization node, perform the following operations through the target data synchronization node: The third segment identifier is determined based on the node identifier corresponding to the target data synchronization node, the synchronization stage corresponding to the target data synchronization node, the data table identifier to which the first data segment belongs, and the second segment identifier corresponding to the first data segment. According to the target format, based on the data feature name corresponding to the target feature value, the third segment identifier, and the sub-target feature value, determine the sub-data feature information corresponding to the first data segment; Send the sub-data feature information corresponding to the first data segment to the feature storage service.

[0105] In some embodiments, the data verification device 600 may further include: The recording module is used to verify the consistency of the first data segment during the data synchronization process based on the first feature value and the second feature value. If the verification result is a failure, the module records the second segment identifier, the first synchronization stage, and the second synchronization stage to analyze the reasons for the data inconsistency.

[0106] In some embodiments, the segmentation module 610 may specifically include: The segmentation module is used to divide the continuous data stream to be synchronized into multiple data segments based on a preset time slice.

[0107] In some embodiments, the first data segment further includes a first segment end marker, which includes a second feature value. Based on this, the acquisition module 630 may specifically include: The execution submodule is used to perform the following operations through the first data synchronization node: Based on multiple first data records in the first data segment, determine the first feature value; Obtain the second feature value from the end marker of the first segment.

[0108] In some embodiments, the first data segment further includes a first segment start marker, which includes an identifier of the data table to which the first data segment belongs and a first segment identifier corresponding to the first data segment. Based on this, the data verification device 600 may further include: The storage module is used to verify the consistency of the first data segment during the data synchronization process based on the first feature value and the second feature value. If the verification result is a failure, the module stores the first feature value and the first data segment to analyze the reasons for the data inconsistency.

[0109] In some embodiments, the data verification device 600 may further include: The execution module, after verifying the consistency of the first data segment during the data synchronization process based on the first and second feature values, performs the following operations through the first data synchronization node: Multiple first data records are processed to obtain multiple second data records; The third feature value is determined based on multiple second data records; Based on the third feature value, determine the updated end marker of the first segment; Assemble the first segment start marker, multiple second data records, and the updated first segment end marker to obtain the updated first data segment; Send the updated first data segment to the third data synchronization node.

[0110] In some embodiments, the segmentation module 610 may specifically include: The segmentation module is also used to divide the continuous data stream to be synchronized into multiple data segments based on a preset data record volume.

[0111] This application's embodiments dynamically segment continuous, unbounded streaming data into data segments with clearly defined boundaries, transforming the streaming data verification problem into a comparison problem of discrete data units. This overcomes the technical obstacle of being unable to directly compare data due to continuous data generation and inconsistencies between the source and target states. Since the first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage, and the first synchronization stage is the synchronization stage where the first data synchronization node is located, and the second synchronization stage is the synchronization stage where the second data synchronization node is located, and the second data synchronization node is the upstream node of the first data synchronization node, by acquiring the first and second feature values ​​of the first data segment during the transmission of the first data segment from the data source end through multiple data synchronization services to the target end, and verifying the consistency of the first data segment during the data synchronization process based on the first and second feature values, a dual feature value comparison mechanism between upstream and downstream nodes is established for the first data segment. This improves the verification accuracy of the first data segment, thereby improving the accuracy of streaming data synchronization verification. Furthermore, if the consistency check fails, the abnormal process can be quickly located based on the first and second data synchronization nodes.

[0112] Based on the data verification method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of this application is shown.

[0113] like Figure 7 As shown, the electronic device 700 may include a processor 710 and a memory 720 storing computer program instructions.

[0114] Specifically, the processor 710 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0115] Memory 720 may include mass storage for data or instructions. For example, and not limitingly, memory 720 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 720 may include removable or non-removable (or fixed) media. Where suitable, memory 720 may be internal or external to electronic device 700. In a particular embodiment, memory 720 is a non-volatile solid-state memory.

[0116] In specific embodiments, the memory 720 may be implemented as a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 720 may store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 720 and executed by the processor 710. The processor 710 implements any of the data verification methods in the above embodiments by reading and executing the computer program instructions stored in the memory 720.

[0117] The processor 710 implements any of the data verification methods described in the above embodiments by reading and executing computer program instructions stored in the memory 720.

[0118] In one example, the electronic device 700 may further include a communication interface 730 and a bus 740. For example, Figure 7 As shown, the processor 710, memory 720, and communication interface 730 are connected via bus 740 and communicate with each other.

[0119] The communication interface 730 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0120] Bus 740 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 740 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0121] For example, the electronic device 700 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0122] The electronic device can execute the data verification method in the embodiments of this application, thereby achieving the combination Figures 1 to 5 The data verification method described herein, and the beneficial effects of the corresponding method embodiments, will not be elaborated further here.

[0123] Furthermore, in conjunction with the data verification methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data verification methods in the above embodiments. Examples of such computer-readable storage media include non-transitory computer-readable storage media, such as read-only memory (ROM).

[0124] The computer program instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the data verification method as shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0125] Based on the data verification methods described in the above embodiments, this application provides a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, they implement any of the data verification methods described in the above embodiments.

[0126] The computer program products of the above embodiments are used to implement the data verification method shown in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0128] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0129] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0130] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0131] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data verification method, characterized in that, include: The continuous data stream to be synchronized is divided into multiple data segments, each of which includes multiple data records; During the process of transmitting the first data segment from the data source end to the target end through multiple data synchronization services in sequence, the first data synchronization node receives the first data segment sent by the second data synchronization node. The first data segment is any one of the multiple data segments, and the first data synchronization node is any one of the target end and the multiple data synchronization services. Obtain the first feature value and the second feature value of the first data segment. The first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage. The first synchronization stage is the synchronization stage in which the first data synchronization node is located, and the second synchronization stage is the synchronization stage in which the second data synchronization node is located. Based on the first feature value and the second feature value, the consistency of the first data segment during the data synchronization process is verified.

2. The method according to claim 1, characterized in that, The step of obtaining the first feature value and the second feature value of the first data segment includes: The data storage module obtains the first feature value and the second feature value of the first data segment. The data storage module is used to store the data feature values ​​corresponding to each data segment at the data source end, the target end and multiple synchronization stages, respectively. The data feature values ​​include at least one of the transmitted data feature value and the received data feature value.

3. The method according to claim 2, characterized in that, The method further includes the following steps before the first data segment is transmitted in multiple transmissions and before obtaining the first feature value and the second feature value of the first data segment from the data storage module: Before sending multiple sub-first data records in the first data segment to the first data synchronization node, the second sub-second feature value is determined based on the multiple sub-first data records through the second data synchronization node. The second data synchronization node sends the second segment identifier corresponding to the first data segment and the sub-second feature value to the data feature storage service, so that the data feature storage service determines the second feature value corresponding to the first data segment based on the multiple sub-second feature values ​​corresponding to the second synchronization stage, and stores it in the data storage module; Upon receiving the plurality of sub-first data records sent by the second data synchronization node, the first data synchronization node determines a sub-first feature value based on the plurality of sub-first data records. The first data synchronization node sends the second segment identifier corresponding to the first data segment and the sub-first feature value to the data feature storage service, so that the data feature storage service determines the first feature value corresponding to the first data segment based on the multiple sub-first feature values ​​corresponding to the first synchronization stage, and stores it in the data storage module.

4. The method according to claim 3, characterized in that, For any one of the first data synchronization node and the second data synchronization node, the target data synchronization node sends the second segment identifier corresponding to the first data segment and the sub-target feature value determined by the target data synchronization node to the data feature storage service, including: Through the target data synchronization node, perform the following operations: Based on the node identifier corresponding to the target data synchronization node, the synchronization stage corresponding to the target data synchronization node, the data table identifier to which the first data segment belongs, and the second segment identifier corresponding to the first data segment, the third segment identifier is determined. According to the target format, based on the data feature name corresponding to the target feature value, the third segment identifier, and the sub-target feature value, the sub-data feature information corresponding to the first data segment is determined; Send the sub-data feature information corresponding to the first data segment to the feature storage service.

5. The method according to claim 3, characterized in that, After verifying the consistency of the first data segment during data synchronization based on the first feature value and the second feature value, the method further includes: If the verification result is a failure, the second segment identifier, the first synchronization stage, and the second synchronization stage are recorded to analyze the reasons for the data inconsistency.

6. The method according to any one of claims 1-5, characterized in that, The process of dividing the continuous data stream to be synchronized into multiple data segments includes: Based on a preset time slice, the continuous data stream to be synchronized is divided into multiple data segments.

7. The method according to claim 1, characterized in that, The first data segment also includes a first segment end marker, which includes the second feature value; The step of obtaining the first feature value and the second feature value of the first data segment includes: Perform the following operations through the first data synchronization node: Based on multiple first data records in the first data segment, a first feature value is determined; Obtain the second feature value from the end marker of the first segment.

8. The method according to claim 7, characterized in that, The first data segment further includes a first segment start marker, which includes the identifier of the data table to which the first data segment belongs and the first segment identifier corresponding to the first data segment; After verifying the consistency of the first data segment during data synchronization based on the first feature value and the second feature value, the method further includes: If the verification result is a verification failure, the first feature value and the first data segment are stored to analyze the reasons for the data inconsistency.

9. The method according to claim 8, characterized in that, After verifying the consistency of the first data segment during data synchronization based on the first feature value and the second feature value, the method further includes: Perform the following operations through the first data synchronization node: Data processing is performed on the plurality of first data records to obtain a plurality of second data records; Based on the plurality of second data records, a third feature value is determined; Based on the third feature value, the updated first segment end marker is determined; Assemble the first segment start marker, the plurality of second data records, and the updated first segment end marker to obtain the updated first data segment; The updated first data segment is sent to the third data synchronization node.

10. The method according to any one of claims 1, 7-9, characterized in that, The process of dividing the continuous data stream to be synchronized into multiple data segments includes: Based on a preset data recording volume, the continuous data stream to be synchronized is divided into multiple data segments.

11. A data verification device, characterized in that, The device includes: The segmentation module is used to segment the continuous data stream to be synchronized into multiple data segments, each of which includes multiple data records; The receiving module is used to receive a first data segment sent by a second data synchronization node through a first data synchronization node during the process of transmitting a first data segment from the data source end to the target end through multiple data synchronization services in sequence. The first data segment is any one of the multiple data segments, and the first data synchronization node is any one of the target end and the multiple data synchronization services. The acquisition module is used to acquire a first feature value and a second feature value of the first data segment. The first feature value is the received data feature value corresponding to the first synchronization stage, and the second feature value is the transmitted data feature value corresponding to the second synchronization stage. The first synchronization stage is the synchronization stage in which the first data synchronization node is located, and the second synchronization stage is the synchronization stage in which the second data synchronization node is located. The verification module is used to verify the consistency of the first data segment during the data synchronization process based on the first feature value and the second feature value.

12. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data verification method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data verification method as described in any one of claims 1-10.

14. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data verification method as described in any one of claims 1-10.

Images