A data consistency checking method and system based on cursor pushing and an electronic device

Abstract

The present application belongs to the technical field of data consistency verification, and particularly relates to a data consistency verification method and system based on cursor promotion and electronic equipment. The present application effectively adapts to data verification requirements under different load scenarios by dynamically adjusting the cursor interval length and the verification window sliding step. Meanwhile, the present application reduces resource consumption in the verification process and improves the verification efficiency of the data consistency verification process by generating and comparing lightweight verification factors. In addition, the present application uses the cursor reverse backtracking mechanism and the method of recursively reducing the dynamic cursor interval to accurately locate the difference points, further improves the accuracy and reliability of the data consistency verification, and can trace back to the data source end to ensure data integrity and consistency. When repairing the difference data block, the present application also considers resource utilization and repair efficiency, and can improve the execution efficiency of the repair task through priority sorting and distributed repair.

Description

Technical Field

[0001] This invention belongs to the field of data consistency verification technology, specifically relating to a data consistency verification method, system, and electronic device based on cursor advancement. Background Technology

[0002] In scenarios such as distributed databases, blockchain node synchronization, and cross-cloud storage systems, the fragmented storage of massive amounts of data makes consistency verification among multiple replicas a core challenge. Traditional verification methods require comparison of the entire dataset, which is inefficient and consumes significant bandwidth resources. For example, financial transaction systems need to verify the consistency of primary and backup databases in real time, while IoT device clusters generate terabytes of data every hour, urgently requiring efficient verification technologies to ensure data integrity.

[0003] Existing consistency verification methods often require traversing all data nodes to achieve data consistency verification. This process is time-consuming and increases linearly with the amount of data, resulting in low verification efficiency with large datasets. Other methods, such as timestamp-based incremental verification, struggle to capture intermediate state changes, leading to missed checks and compromising data integrity. Therefore, this solution provides a cursor-based data consistency verification method to address these issues. Summary of the Invention

[0004] The purpose of this invention is to provide a data consistency verification method, system, and electronic device based on cursor propagation, which can reduce verification time and bandwidth consumption, ensure data integrity, and is applicable to scenarios such as distributed databases and blockchain synchronization, thereby improving system stability and reliability.

[0005] The specific technical solution adopted by this invention is as follows:

[0006] A data consistency verification method based on cursor advancement includes:

[0007] The data stream to be verified is acquired and divided into dynamic cursor intervals based on the system load. Each dynamic cursor interval corresponds to a data block.

[0008] A lightweight check factor is generated for each data block under each dynamic cursor interval and stored in the metadata index;

[0009] Multiple verification windows are generated by the truncation order of the dynamic cursor interval. By advancing the verification windows, the lightweight verification factor of the data block is compared with the lightweight verification factor in the metadata index in real time.

[0010] When the lightweight check factor is inconsistent with the lightweight check factor in the metadata index, the cursor reverse backtracking mechanism is triggered. By recursively narrowing the dynamic cursor interval, the difference is located, the difference point is identified, and the difference data block is obtained.

[0011] The difference is quantified based on the size, hierarchical attributes, and timestamp of the difference data block to obtain the data difference quantity. Based on the data difference quantity, the repair priority of multiple difference points is sorted to generate a distributed repair task queue. The difference repair operation is executed sequentially on the distributed repair task queue according to the repair task priority.

[0012] In a preferred embodiment, the step of dividing the data stream to be verified into dynamic cursor intervals based on system load includes:

[0013] The arrival interval of data packets in the data stream to be verified is captured in real time, and the stability qualitative index of the data stream to be verified is determined based on the arrival interval.

[0014] Establish a mapping relationship between stability indices and dynamic cursor interval lengths, and dynamically adjust the cursor interval lengths based on the mapping relationship;

[0015] Embed structural feature markers in the header of the data stream to be verified, nest transaction boundaries using predefined transaction feature markers, and force the cursor split point to align with the transaction boundary;

[0016] The data blocks are divided based on the aligned cursor split points, and the start and end positions of each block are recorded to generate dynamic cursor intervals.

[0017] In a preferred embodiment, the step of generating a lightweight check factor for each data block under each dynamic cursor interval includes:

[0018] By scanning the data blocks byte by byte, a unique and fixed-length feature code is generated and recorded as the basic verification identifier;

[0019] The starting position number of the data block is combined with the basic verification identifier according to a preset rule to generate a composite verification factor containing position information.

[0020] Extract the core identifier fields from the data block. The core identifier fields include the operation sequence number, timestamp, and transaction ID. Generate a core identifier feature code through feature marking and fuse it with the composite check factor to form a lightweight check factor.

[0021] In a preferred embodiment, the step of generating multiple verification windows based on the truncation order of the dynamic cursor interval includes:

[0022] Based on the length of the dynamic cursor interval, multiple consecutive dynamic cursor intervals are extracted from the data stream to be verified according to the time series to form the initial verification window;

[0023] The actual window sliding step size is calculated based on the current resource idle ratio, available memory capacity, and preset baseline window length.

[0024] Starting from the initial verification window, the window is moved backward step by step according to the actual window sliding size to generate multiple verification windows until the entire range of the data stream to be verified is covered.

[0025] Specifically, when the verification window advances to the end of the data stream to be verified, the verification window at the end is overlapped with the verification window at the beginning of the data stream to be verified, forming a cyclic verification process.

[0026] In a preferred embodiment, the step of comparing the lightweight check factor of the data block with the metadata index in real time by advancing the check window includes:

[0027] Within each verification window, multiple verification threads are started in parallel, and each verification thread extracts the lightweight verification factor of the data block from the corresponding verification window.

[0028] Based on the location number and timestamp recorded in the metadata index, a verification logic tree is constructed, and the consistency between the lightweight verification factor and the metadata is compared layer by layer.

[0029] If the lightweight check factor successfully matches the metadata, a check pass flag is recorded, and the check status in the metadata index is updated.

[0030] If the lightweight check factor language data does not match, the exception handling mechanism is triggered, and the exception location and timestamp are recorded to generate an error log.

[0031] In a preferred embodiment, when dividing the verification window, a region that overlaps with the tail of the preceding verification window is divided in each verification window and recorded as the overlapping verification area.

[0032] Within the overlapping verification area, the lightweight verification factor of the preceding verification window is reused for a second comparison with the metadata. When the lightweight verification factor in the overlapping verification area is inconsistent with the metadata index, a local backtracking mechanism is triggered.

[0033] In a preferred embodiment, the step of recursively narrowing the dynamic cursor interval, locating differences, identifying difference points, and outputting difference data blocks includes:

[0034] Record the dynamic cursor interval under inconsistent lightweight check factors as the interval to be identified;

[0035] Obtain the start and end position labels of the interval to be identified, generate the initial difference location interval, and use the bisection method to classify the initial difference location interval into a left check sub-interval and a right check sub-interval;

[0036] A lightweight check factor is regenerated for the left check sub-interval and compared with the metadata index a second time. If the comparison results are inconsistent, the left check sub-interval is recorded as a suspicious difference interval. If the comparison results are consistent, the right check sub-interval is marked as a suspicious difference interval.

[0037] When the amount of data within the suspicious difference interval exceeds the preset segmentation threshold, the binary search operation continues to be executed recursively until the smallest difference unit is located.

[0038] Within the smallest difference unit, transaction boundary verification is performed. By cross-validating transaction codes and operation sequence numbers, difference points are located, and the difference data blocks corresponding to the difference transactions are extracted.

[0039] In a preferred embodiment, the step of quantifying the amount of data difference based on the size, hierarchical attributes, and timestamps of the difference data blocks, and prioritizing the repair of multiple difference points based on the amount of data difference to generate a distributed repair task queue includes:

[0040] Retrieve the predefined difference size mapping table, transaction level table, and time decay table;

[0041] By using the difference size mapping table, transaction hierarchy table, and time decay table, the difference data block size is converted into the first feature parameter, the transaction type in the hierarchy attribute is converted into the second feature parameter, and the interval between the timestamp and the current time is converted into the third feature parameter.

[0042] The first feature parameter, the second feature parameter, and the third feature parameter are fused and calculated to generate a comprehensive difference score;

[0043] The discrepancies are sorted in descending order based on the comprehensive discrepancy score, and a unique repair identifier is assigned to each discrepancy and associated with the corresponding data block.

[0044] Based on the priority order of the repair identifiers, a distributed repair task queue is constructed, and the resource utilization rate under the distributed nodes is collected in real time. When the resource utilization rate is lower than the preset utilization threshold, adjacent low-scoring repair tasks in the repair task queue are merged until all differences are repaired.

[0045] The present invention also provides a data consistency verification system based on cursor advancement, comprising:

[0046] The initialization module is used to acquire the data stream to be verified and divide the data stream to be verified into dynamic cursor intervals according to the system load. Each dynamic cursor interval corresponds to a data block.

[0047] The check factor output module is used to generate a lightweight check factor for each data block under each dynamic cursor interval and store it in the metadata index;

[0048] The dynamic cursor advance module is used to generate multiple verification windows based on the truncation order of the dynamic cursor interval. By advancing the verification windows, the lightweight verification factor and metadata index of the data block are compared in real time.

[0049] The difference localization module is used to trigger the cursor reverse backtracking mechanism when there is inconsistency in the lightweight check factor. It performs difference localization by recursively narrowing the dynamic cursor interval, identifies the difference points, and outputs the difference data block.

[0050] The distributed repair module is used to quantify the amount of data difference based on the size, hierarchical attributes and timestamps of the difference data blocks, sort the repair priorities of multiple difference points based on the amount of data difference, generate a distributed repair task queue, and then execute the difference repair operations in sequence according to the priority of the repair tasks.

[0051] And, an electronic device, the electronic device comprising:

[0052] At least one processor;

[0053] and a memory communicatively connected to the at least one processor;

[0054] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the aforementioned cursor-driven data consistency verification method.

[0055] The technical effects achieved by this invention are as follows:

[0056] This invention effectively adapts to data verification needs under different load scenarios by dynamically adjusting the cursor interval length and the verification window sliding step size. Simultaneously, it reduces resource consumption during the verification process and improves the verification efficiency by generating and comparing lightweight verification factors. Furthermore, this invention utilizes a cursor backtracking mechanism and a method of recursively narrowing the dynamic cursor interval to achieve precise location of discrepancies, further improving the accuracy and reliability of data consistency verification. It can also trace back to the data source, ensuring data integrity and consistency. When repairing discrepancy data blocks, it considers resource utilization and repair efficiency. Through priority sorting and distributed repair, it improves the execution efficiency of repair tasks, achieving a balance between resource utilization and repair efficiency, effectively shortening repair time, and ensuring the stable operation of the data verification process. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0058] Figure 2 This is a schematic diagram of the system modules of the present invention;

[0059] Figure 3 This is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0061] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0062] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0063] Please see Figure 1 As shown, the present invention provides a data consistency verification method based on cursor advancement, comprising:

[0064] S1. Obtain the data stream to be verified and divide it into dynamic cursor intervals according to the system load. Each dynamic cursor interval corresponds to a data block.

[0065] In step S1, during the data verification process, the data stream to be verified is first collected. This data stream can originate from scenarios such as distributed databases, blockchain node synchronization, or cross-cloud storage systems. Since fluctuations in system load directly affect data processing capabilities, the data stream to be verified is dynamically divided into multiple dynamic cursor intervals based on the current system load. Each dynamic cursor interval corresponds to an independent data block to facilitate subsequent data verification and processing. The step of dividing the data stream to be verified into dynamic cursor intervals based on system load includes:

[0066] The arrival interval of data packets in the data stream to be verified is captured in real time, and the stability qualitative index of the data stream to be verified is determined based on the arrival interval.

[0067] Establish a mapping relationship between stability indices and dynamic cursor interval lengths, and dynamically adjust the cursor interval lengths based on the mapping relationship;

[0068] Embed structural feature markers in the header of the data stream to be verified, nest transaction boundaries using predefined transaction feature markers, and force the cursor split point to align with the transaction boundary;

[0069] The data blocks are divided based on the aligned cursor split points, and the start and end positions of each block are recorded to generate dynamic cursor intervals.

[0070] Specifically, when dividing the dynamic cursor interval, the arrival status of data packets in the data stream to be verified is captured in real time. The stability of the data stream is determined by analyzing the arrival interval of the data packets. The stability index takes into account the changes in the arrival interval of the data packets, thus reflecting the fluctuation of the data stream to be verified. In order to dynamically adjust the length of the cursor interval according to the stability of the data stream, a mapping relationship is established, which associates the stability index with the length of the cursor interval. When the data stream is relatively stable, the cursor interval can be relatively long to reduce the frequency of verification operations and improve efficiency. When the data stream fluctuates greatly, the cursor interval is shortened to enable more precise monitoring and processing of data. In addition, some specific structural feature markers are embedded in the header of the data stream. The feature markers are used to identify the start and end of data packets and the transaction boundaries to which they belong. By using predefined transaction feature markers, transaction boundaries can be nested, and the cursor splitting point can be aligned with the transaction boundary. This makes it easier to divide and process data according to the transaction boundary in subsequent data processing and verification, avoiding the splitting of data within a transaction, thus ensuring data consistency and integrity. Then, based on the aligned cursor splitting point, data blocks are divided, and the start and end positions of each data block are recorded and integrated into a dynamic cursor range. This allows for quick location and access to specific data blocks in subsequent verification. The division of the dynamic cursor range provides a basis for data consistency verification, making the verification process more efficient and accurate.

[0071] S2. Generate a lightweight check factor for each data block under each dynamic cursor interval and store it in the metadata index;

[0072] In step S2, after the dynamic cursor interval is generated, features are extracted from each data block to generate a corresponding lightweight check factor. This ensures that the lightweight check factor can represent the characteristics of the block data. The lightweight check factor is then stored in the metadata index for subsequent fast retrieval and verification. The step of generating a lightweight check factor for each data block under the dynamic cursor interval includes:

[0073] By scanning the data blocks byte by byte, a unique and fixed-length feature code is generated and recorded as the basic verification identifier;

[0074] The starting position number of the data block is combined with the basic verification identifier according to a preset rule to generate a composite verification factor containing position information.

[0075] Extract the core identifier fields from the data block. The core identifier fields include the operation sequence number, timestamp, and transaction ID. Generate a core identifier feature code through feature marking and fuse it with the composite verification factor to form a lightweight verification factor.

[0076] Specifically, when generating a lightweight check factor, the data block is scanned byte by byte. For example, a unique and fixed-length 32-bit signature is generated using the CRC32 cyclic redundancy check algorithm. This generated signature is recorded as the basic check identifier. Then, the starting position number of the data block in the data stream to be checked (in the form of byte offset) is combined with the basic check identifier using a hash concatenation rule to generate a multi-bit composite check factor containing spatial location information, such as 128 bits. Predefined core identifier fields within the data block are extracted, including operation sequence number, timestamp, and transaction ID. Then, the core identifier fields are processed using an algorithm such as SHA-1 to generate a multi-bit core identifier signature, which can be 160 bits or more. Finally, the core identifier signature is fused with the composite check factor through a bitwise XOR operation to form the corresponding lightweight check factor. The lightweight check factor simultaneously contains three verification dimensions: data content characteristics, spatial location, and transaction metadata, ensuring multi-dimensional verification during data verification and thus improving verification efficiency.

[0077] S3. Multiple verification windows are generated by the truncation order of the dynamic cursor interval. By advancing the verification windows, the lightweight verification factor of the data block is compared with the lightweight verification factor in the metadata index in real time.

[0078] In step S3, after the dynamic cursor interval is truncated, multiple verification windows are generated according to the truncating order. Each verification window corresponds to one or more data blocks. By advancing the verification windows one by one, real-time verification of the data blocks can be achieved. The lightweight verification factor of each block in the current window is compared with the records in the metadata index to determine the consistency and integrity of the data. The step of generating multiple verification windows according to the truncating order of the dynamic cursor interval includes:

[0079] Based on the length of the dynamic cursor interval, multiple consecutive dynamic cursor intervals are extracted from the data stream to be verified according to the time series to form the initial verification window;

[0080] The actual window sliding step size is calculated based on the current resource idle ratio, available memory capacity, and preset baseline window length.

[0081] Starting from the initial verification window, the window is moved backward step by step according to the actual window sliding size to generate multiple verification windows until the entire range of the data stream to be verified is covered.

[0082] Specifically, when the verification window advances to the end of the data stream to be verified, the verification window at the end is overlapped with the verification window at the beginning of the data stream to be verified, forming a loop verification process.

[0083] When dividing the verification window, a region that overlaps with the tail of the preceding verification window is divided in each verification window and recorded as the overlapping verification area.

[0084] Within the overlapping verification area, the lightweight verification factor of the preceding verification window is reused for a second comparison. When the lightweight verification factor in the overlapping verification area is inconsistent with the metadata index, a local backtracking mechanism is triggered.

[0085] Specifically, when generating the verification window, it successively truncates and moves forward according to the length of the dynamic cursor interval and the preset step size, thus forming multiple consecutive initial verification windows. Then, combined with the current resource idle ratio, available memory capacity, and the preset baseline window length, a sliding window mechanism is used to calculate the actual sliding step size, ensuring that the window sliding step size can meet the load changes. The formula for calculating the actual sliding step size is: Actual sliding step size = Baseline window length × (1 + Resource idle ratio - Preset load threshold) × Available memory capacity / Preset baseline memory capacity. This formula is used to adjust the window sliding step size, ensuring that the verification window adapts to different loads. It should be noted that in When the resource idle ratio is lower than the preset load threshold, the sliding step size is forcibly set to the minimum step size unit. When advancing the verification window, an area overlapping with the tail of the previous verification window is divided in each verification window and recorded as the overlapping verification area. Within the overlapping verification area, the lightweight verification factor of the previous verification window is reused and compared with the metadata a second time. If there is a discrepancy, a local backtracking mechanism is triggered to optimize resource utilization and improve verification efficiency. At the same time, during the window generation process, the cursor split point is also forcibly aligned with the transaction boundary to avoid data splitting within the transaction. Specifically, boundary recognition is enhanced by embedding structural feature markers to ensure the continuity and integrity of the verification process.

[0086] In addition, the steps of comparing the lightweight check factor of data blocks with the metadata index in real time by advancing the check window include:

[0087] Within each verification window, multiple verification threads are started in parallel, and each verification thread extracts the lightweight verification factor of the data block from the corresponding verification window.

[0088] Based on the location number and timestamp recorded in the metadata index, a verification logic tree is constructed, and the consistency between the lightweight verification factor and the metadata is compared layer by layer.

[0089] If the lightweight check factor successfully matches the metadata, a check pass flag is recorded, and the check status in the metadata index is updated.

[0090] If the lightweight check factor does not match the metadata, the exception handling mechanism is triggered, and the exception location and timestamp are recorded to generate an error log.

[0091] In the above process, multiple verification threads are activated in parallel within each verification window to extract lightweight verification factors from data blocks, thus ensuring the efficiency of the verification process. Then, based on the position number and timestamp recorded in the metadata index, a multi-layered verification logic tree is constructed. In the verification logic tree, the first layer compares the consistency of the starting position number, the second layer compares the hash value of the basic verification identifier, and the third layer verifies the timeliness of the core identifier feature code. If all three layers of comparison in the verification logic tree pass, a verification pass mark is recorded and the last verification timestamp in the metadata index is updated. If any level of comparison fails in the verification logic tree, an exception handling mechanism is immediately triggered, and the specific exception location, timestamp, and failure level information are recorded synchronously, generating a detailed error log to facilitate subsequent problem localization.

[0092] S4. When there is an inconsistency between the lightweight check factor and the lightweight check factor in the metadata index, the cursor reverse backtracking mechanism is triggered. By recursively narrowing the dynamic cursor interval, the difference is located, the difference point is identified, and the difference data block is obtained.

[0093] In step S4, when a lightweight check factor inconsistency occurs during the verification process, a cursor reverse backtracking mechanism is immediately activated to recursively narrow the dynamic cursor interval to locate the difference point and output the difference data block, ensuring the accuracy of the verification result. The steps of recursively narrowing the dynamic cursor interval to locate the difference, identify the difference point, and output the difference data block include:

[0094] Record the dynamic cursor interval under inconsistent lightweight check factors as the interval to be identified;

[0095] Obtain the start and end position labels of the interval to be identified, generate the initial difference location interval, and use the bisection method to classify the initial difference location interval into a left check sub-interval and a right check sub-interval;

[0096] A lightweight check factor is regenerated for the left check sub-interval and compared with the metadata index a second time. If the comparison results are inconsistent, the left check sub-interval is recorded as a suspicious difference interval. If the comparison results are consistent, the right check sub-interval is marked as a suspicious difference interval.

[0097] When the amount of data within the suspicious difference interval exceeds the preset segmentation threshold, the binary search operation continues to be executed recursively until the smallest difference unit is located.

[0098] Perform transaction-by-transaction boundary verification within the smallest difference unit. Locate the difference point by cross-validation of transaction code and operation sequence number, and extract the difference data block corresponding to the difference transaction.

[0099] Specifically, when lightweight check factors are inconsistent, the inconsistent dynamic cursor intervals are recorded as intervals to be identified. Then, the start and end position labels of the intervals to be identified are obtained to generate initial difference location intervals. These initial difference location intervals are then divided equally into left and right check sub-intervals using a bisection method. For the left check sub-interval, a new lightweight check factor is generated and compared a second time with the original lightweight check factor of the corresponding interval recorded in the metadata index. If the comparison results are inconsistent, the left check sub-interval is recorded as a suspicious difference interval. If the comparison results are consistent, the difference is determined to exist in the right check sub-interval, and the right check sub-interval is marked as a suspicious difference interval. The amount of data within the suspicious difference interval exceeds... If the number of bytes or transaction blocks exceeds the preset segmentation threshold, the above binary search operation will continue to be recursively executed. That is, the suspicious difference interval will be divided into left and right sub-intervals and checked separately. In this way, the suspicious range can be gradually narrowed down until the smallest difference unit with a data volume less than or equal to the segmentation threshold is located. Within the smallest difference unit, transaction boundary verification will be performed according to the transaction boundary markers embedded in step S1. By comparing the transaction code, operation sequence number and timestamp of each transaction in the smallest difference unit with the cross-validation results of the corresponding records in the metadata index, the specific transaction point where the difference occurred can be located. Finally, the complete data content corresponding to the difference transaction is extracted, the difference data block is generated and output for subsequent diagnosis or repair.

[0100] S5. Quantify the data difference based on the size, hierarchical attributes and timestamp of the difference data block to obtain the data difference amount, sort the repair priority of multiple difference points based on the data difference amount, generate a distributed repair task queue, and perform difference repair operations on the distributed repair task queue in sequence according to the repair task priority.

[0101] In step S5, after the difference data blocks are output, the amount of data difference is quantified based on the size, hierarchical attributes, and timestamps of the difference data blocks. Then, a comprehensive evaluation is performed based on the amount of difference, hierarchical attributes, and timestamps to determine the repair priority of each difference point, generating an ordered repair task queue to ensure that high-priority tasks are processed first. The steps of quantifying the amount of data difference based on the size, hierarchical attributes, and timestamps of the difference data blocks, and sorting the repair priorities of multiple difference points based on the amount of data difference to generate a distributed repair task queue include:

[0102] Predefined difference size mapping table, transaction hierarchy table, and time decay table;

[0103] By using the difference size mapping table, transaction hierarchy table, and time decay table, the difference data block size is converted into the first feature parameter, the transaction type in the hierarchy attribute is converted into the second feature parameter, and the interval between the timestamp and the current time is converted into the third feature parameter.

[0104] The first feature parameter, the second feature parameter, and the third feature parameter are fused and calculated to generate a comprehensive difference score;

[0105] The discrepancies are sorted in descending order based on the comprehensive discrepancy score, and a unique repair identifier is assigned to each discrepancy and associated with the corresponding data block.

[0106] Based on the priority order of the repair identifiers, a distributed repair task queue is constructed, and the resource utilization rate under the distributed nodes is collected in real time. When the resource utilization rate is lower than the preset utilization threshold, adjacent low-scoring repair tasks in the repair task queue are merged until all differences are repaired.

[0107] In this implementation, when the distributed repair task queue outputs, it first performs feature transformation on the size of the difference data block, transaction type, and time interval according to a predefined difference size mapping table, transaction hierarchy table, and time decay table. This transforms the size of the difference data block, transaction type, and time interval into quantifiable first feature parameters, second feature parameters, and third feature parameters. Subsequently, a comprehensive difference score is generated through fusion calculation. The fusion calculation can employ a conventional weighted average method or a machine learning algorithm. This process ensures the accuracy and objectivity of the score. Then, the difference points are sorted in descending order according to the comprehensive difference score, and a unique repair identifier is assigned to each difference point to ensure that high-priority tasks are processed first. In addition, during the specific repair process, resource utilization is monitored in real time. When the resource utilization is lower than a preset utilization threshold, adjacent low-scoring repair tasks are merged. For example, when the resource utilization is lower than 20%, adjacent tasks with repair scores below 50 are merged into one task for execution. This optimizes resource allocation, reduces resource waste, and ensures that critical data is repaired first, thereby improving repair efficiency.

[0108] Please see Figure 2 A data consistency verification system based on cursor propagation includes:

[0109] The initialization module is used to acquire the data stream to be verified and divide the data stream to be verified into dynamic cursor intervals according to the system load. Each dynamic cursor interval corresponds to a data block.

[0110] The check factor output module is used to generate a lightweight check factor for each data block under each dynamic cursor interval and store it in the metadata index;

[0111] The dynamic cursor advance module is used to generate multiple verification windows based on the truncation order of the dynamic cursor interval. By advancing the verification windows, the lightweight verification factor and metadata index of the data block are compared in real time.

[0112] The difference localization module is used to trigger the cursor reverse backtracking mechanism when there is inconsistency in the lightweight check factor. It performs difference localization by recursively narrowing the dynamic cursor interval, identifies the difference points, and outputs the difference data block.

[0113] The distributed repair module is used to quantify the amount of data difference based on the size, hierarchical attributes and timestamps of the difference data blocks, sort the repair priorities of multiple difference points based on the amount of data difference, generate a distributed repair task queue, and then execute the difference repair operations in sequence according to the priority of the repair tasks.

[0114] In the above, the initialization module is responsible for the preliminary processing of the data stream to be verified. It acquires the entire data stream and then intelligently divides it into several dynamic cursor intervals based on the current system load. Each dynamic cursor interval corresponds to an independent data block, providing a structured data foundation for subsequent data processing and verification. The verification factor output module is responsible for generating lightweight verification factors for each data block under each divided dynamic cursor interval. The lightweight verification factor is stored in the metadata index as an identifier for the data block for subsequent verification and comparison. The dynamic cursor advancement module is used to advance the verification window, generating verification factors based on the truncation order of the dynamic cursor interval. Multiple verification windows are used to perform real-time verification of data blocks in a progressive manner, ensuring data consistency and integrity. When the difference location module detects inconsistencies in lightweight verification factors, it initiates a cursor reverse backtracking mechanism. By recursively narrowing the dynamic cursor interval, it locates the difference point and outputs the difference data block, providing crucial information for subsequent repair. The distributed repair module quantifies the amount of data difference based on the size, hierarchical attributes, and timestamp of the difference data block. Based on the amount of data difference, it prioritizes multiple difference points for repair, generating an ordered distributed repair task queue. This ensures that high-priority tasks are processed first, thereby effectively improving the efficiency and accuracy of data repair.

[0115] Please see Figure 3 An electronic device, comprising:

[0116] At least one processor;

[0117] and memory that is communicatively connected to at least one processor;

[0118] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the aforementioned cursor-based data consistency verification method.

[0119] The processor of the aforementioned electronic device can be a high-performance central processing unit (CPU), or a chip with data processing capabilities such as a graphics processing unit (GPU), field-programmable gate array (FPGA), digital signal processor (DSP), or application-specific integrated circuit (ASIC), or any combination of multiple such chips. The memory can include high-speed random access memory (RAM) and non-volatile memory (such as ROM, flash memory, etc.). High-speed RAM is used to temporarily store data and instructions during processor operation to improve data access speed, while non-volatile memory is used to store computer programs and various types of data for a long time to ensure that data is not lost after the device is powered off. The processor reads and executes the computer program in the memory to realize all the functions of the aforementioned cursor-based data consistency verification method. The electronic device may also include an arithmetic logic unit (ALU), input devices, and output devices. The ALU can be a logic unit used to perform arithmetic and logical operations to assist the processor in data processing. Input devices, such as a keyboard, mouse, or touch screen, are used to receive user input instructions and data. Output devices, such as a monitor or printer, are used to display processing results or print output.

[0120] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0121] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A data consistency verification method based on cursor advancement, characterized in that: include: The data stream to be verified is acquired and divided into dynamic cursor intervals based on the system load. Each dynamic cursor interval corresponds to a data block. A lightweight check factor is generated for each data block under each dynamic cursor interval and stored in the metadata index; Multiple verification windows are generated by the truncation order of the dynamic cursor interval. By advancing the verification windows, the lightweight verification factor of the data block is compared with the lightweight verification factor in the metadata index in real time. When the lightweight check factor is inconsistent with the lightweight check factor in the metadata index, the cursor reverse backtracking mechanism is triggered. By recursively narrowing the dynamic cursor interval, the difference is located, the difference point is identified, and the difference data block is obtained. The difference is quantified based on the size, hierarchical attributes and timestamp of the difference data block to obtain the amount of data difference. Based on the amount of data difference, the repair priority of multiple difference points is sorted to generate a distributed repair task queue. The difference repair operation is executed sequentially on the distributed repair task queue according to the repair task priority. The step of generating a lightweight check factor for each data block under each dynamic cursor interval includes: By scanning the data blocks byte by byte, a unique and fixed-length feature code is generated and recorded as the basic verification identifier; The starting position number of the data block is combined with the basic verification identifier according to a preset rule to generate a composite verification factor containing position information. Extract the core identifier fields from the data block. The core identifier fields include the operation sequence number, timestamp, and transaction ID. Generate a core identifier feature code through feature marking and fuse it with the composite verification factor to form a lightweight verification factor. The step of generating multiple verification windows based on the truncation order of the dynamic cursor range includes: Based on the length of the dynamic cursor interval, multiple consecutive dynamic cursor intervals are extracted from the data stream to be verified according to the time series to form the initial verification window; The actual window sliding step size is calculated based on the current resource idle ratio, available memory capacity, and preset baseline window length. Starting from the initial verification window, the window is moved backward step by step according to the actual window sliding size to generate multiple verification windows until the entire range of the data stream to be verified is covered. Specifically, when the verification window advances to the end of the data stream to be verified, the verification window at the end is overlapped with the verification window at the beginning of the data stream to be verified, forming a cyclic verification process.

2. The data consistency verification method based on cursor advancement according to claim 1, characterized in that: The step of dividing the data stream to be verified into dynamic cursor intervals based on system load includes: The arrival interval of data packets in the data stream to be verified is captured in real time, and the stability qualitative index of the data stream to be verified is determined based on the arrival interval. Establish a mapping relationship between stability indices and dynamic cursor interval lengths, and dynamically adjust the cursor interval lengths based on the mapping relationship; Embed structural feature markers in the header of the data stream to be verified, nest transaction boundaries using predefined transaction feature markers, and force the cursor split point to align with the transaction boundary; The data blocks are divided based on the aligned cursor split points, and the start and end positions of each block are recorded to generate dynamic cursor intervals.

3. The data consistency verification method based on cursor advancement according to claim 1, characterized in that: The step of comparing the lightweight check factor of the data block with the metadata index in real time by advancing the check window includes: Within each verification window, multiple verification threads are started in parallel, and each verification thread extracts the lightweight verification factor of the data block from the corresponding verification window. Based on the location number and timestamp recorded in the metadata index, a verification logic tree is constructed, and the consistency between the lightweight verification factor and the metadata is compared layer by layer. If the lightweight check factor successfully matches the metadata, a check pass flag is recorded, and the check status in the metadata index is updated. If the lightweight check factor does not match the metadata, the exception handling mechanism is triggered, the exception location and timestamp are recorded, and an error log is generated.

4. The data consistency verification method based on cursor advancement according to claim 3, characterized in that: When dividing the verification window, a region that overlaps with the tail of the preceding verification window is divided in each verification window and recorded as the overlapping verification area. Within the overlapping verification area, the lightweight verification factor of the preceding verification window is reused for a second comparison with the metadata. When the lightweight verification factor in the overlapping verification area is inconsistent with the metadata index, a local backtracking mechanism is triggered.

5. The data consistency verification method based on cursor advancement according to claim 1, characterized in that: The steps of recursively narrowing the dynamic cursor interval, locating differences, identifying difference points, and outputting difference data blocks include: Record the dynamic cursor interval under inconsistent lightweight check factors as the interval to be identified; Obtain the start and end position labels of the interval to be identified, generate the initial difference location interval, and use the bisection method to classify the initial difference location interval into a left check sub-interval and a right check sub-interval; A lightweight check factor is regenerated for the left check sub-interval and compared with the metadata index a second time. If the comparison results are inconsistent, the left check sub-interval is recorded as a suspicious difference interval. If the comparison results are consistent, the right check sub-interval is marked as a suspicious difference interval. When the amount of data within the suspicious difference interval exceeds the preset segmentation threshold, the binary search operation continues to be executed recursively until the smallest difference unit is located. Within the smallest difference unit, transaction boundary verification is performed. By cross-validating transaction codes and operation sequence numbers, difference points are located, and the difference data blocks corresponding to the difference transactions are extracted.

6. The data consistency verification method based on cursor advancement according to claim 1, characterized in that: The steps of quantifying the amount of data difference based on the size, hierarchical attributes, and timestamps of the difference data blocks, prioritizing the repair of multiple difference points based on the amount of data difference, and generating a distributed repair task queue include: Retrieve the predefined difference size mapping table, transaction level table, and time decay table; By using the difference size mapping table, transaction hierarchy table, and time decay table, the difference data block size is converted into the first feature parameter, the transaction type in the hierarchy attribute is converted into the second feature parameter, and the interval between the timestamp and the current time is converted into the third feature parameter. The first feature parameter, the second feature parameter, and the third feature parameter are fused and calculated to generate a comprehensive difference score; The discrepancies are sorted in descending order based on the comprehensive discrepancy score, and a unique repair identifier is assigned to each discrepancy and associated with the corresponding data block. Based on the priority order of the repair identifiers, a distributed repair task queue is constructed, and the resource utilization rate under the distributed nodes is collected in real time. When the resource utilization rate is lower than the preset utilization threshold, adjacent low-scoring repair tasks in the repair task queue are merged until all differences are repaired.

7. A data consistency verification system based on cursor advancement, characterized in that: include: The initialization module is used to acquire the data stream to be verified and divide the data stream to be verified into dynamic cursor intervals according to the system load. Each dynamic cursor interval corresponds to a data block. The check factor output module is used to generate a lightweight check factor for each data block under each dynamic cursor interval and store it in the metadata index; The dynamic cursor advance module is used to generate multiple verification windows based on the truncation order of the dynamic cursor interval. By advancing the verification windows, the lightweight verification factor and metadata index of the data block are compared in real time. The difference localization module is used to trigger the cursor reverse backtracking mechanism when there is inconsistency in the lightweight check factor. It performs difference localization by recursively narrowing the dynamic cursor interval, identifies the difference points, and outputs the difference data block. The distributed repair module is used to quantify the amount of data difference based on the size, hierarchical attributes and timestamp of the difference data block, sort the repair priorities of multiple difference points based on the amount of data difference, generate a distributed repair task queue, and then execute the difference repair operation in sequence according to the priority of the repair task. The step of generating a lightweight check factor for each data block under each dynamic cursor interval includes: By scanning the data blocks byte by byte, a unique and fixed-length feature code is generated and recorded as the basic verification identifier; The starting position number of the data block is combined with the basic verification identifier according to a preset rule to generate a composite verification factor containing position information. Extract the core identifier fields from the data block. The core identifier fields include the operation sequence number, timestamp, and transaction ID. Generate a core identifier feature code through feature marking and fuse it with the composite verification factor to form a lightweight verification factor. The step of generating multiple verification windows based on the truncation order of the dynamic cursor range includes: Based on the length of the dynamic cursor interval, multiple consecutive dynamic cursor intervals are extracted from the data stream to be verified according to the time series to form the initial verification window; The actual window sliding step size is calculated based on the current resource idle ratio, available memory capacity, and preset baseline window length. Starting from the initial verification window, the window is moved backward step by step according to the actual window sliding size to generate multiple verification windows until the entire range of the data stream to be verified is covered. Specifically, when the verification window advances to the end of the data stream to be verified, the verification window at the end is overlapped with the verification window at the beginning of the data stream to be verified, forming a cyclic verification process.

8. An electronic device, characterized in that: The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the data consistency verification method based on cursor propagation as described in any one of claims 1 to 6.

Images