Data synchronization testing methods, devices, equipment, storage media, and software products

Abstract

This application provides a data synchronization testing method, apparatus, device, storage medium, and program product, relating to the field of data synchronization technology. The method includes: periodically recording the offset information of the target-end message queue during the source-end data import process; the target-end message queue is a distributed message system that receives and stores synchronized data; the offset information is a unique location identifier for each message in the target-end message queue; in the event of an abnormal synchronization interruption, obtaining the mapping relationship between the target-end message queue offset information and the source-end data parsing location; generating a new untested data file based on the mapping relationship, and resuming synchronization testing from the data location corresponding to the mapping relationship. This method aims to improve synchronization efficiency and reduce resource waste.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to a data synchronization testing method, apparatus, equipment, storage medium, and program product. Background Technology

[0002] In scenarios such as finance, IoT device data aggregation, and enterprise-level log analysis, it is necessary to synchronize massive amounts of data from a source database to a target database. During the synchronization process, it is essential to test the consistency of the data between the source and target databases. Any missing or duplicate data can lead to business logic anomalies; therefore, data synchronization testing is crucial.

[0003] Currently, existing data synchronization testing methods mainly involve writing all the original data to be tested into the source database, waiting for all data processing to complete, starting the target consumer program, consuming the data in the target database in sequence, and verifying whether the synchronization is successful by comparing the data content, order, and quantity between the source and target ends one by one.

[0004] However, existing data synchronization testing methods compare data from the source and target end one by one. If a network interruption, abnormal database restart, or operating system failure occurs during the synchronization process, the full import and synchronization process must be re-executed, resulting in a waste of resources and low synchronization efficiency. Summary of the Invention

[0005] This application provides a data synchronization testing method, apparatus, device, storage medium, and program product to improve synchronization efficiency and reduce resource waste.

[0006] Firstly, this application provides a data synchronization testing method, including:

[0007] During the data import process from the source end, the offset information of the target end message queue is recorded periodically; the target end message queue is a distributed message system that receives and stores synchronized data; the offset information is a unique location identifier for each message in the target end message queue.

[0008] In the event of a synchronization failure, obtain the mapping relationship between the target end message queue offset information and the source end data parsing position;

[0009] A new untested data file is generated based on the mapping relationship, and synchronization testing is resumed from the data location corresponding to the mapping relationship.

[0010] In one possible implementation, generating a new untested data file based on the mapping relationship includes: obtaining the source data parsing position based on the mapping relationship; extracting incremental data from the original data file after the source data parsing position at the time of interruption to generate a new untested data file, wherein the incremental data is the data to be processed in the original data file.

[0011] In one possible implementation, before extracting incremental data from the source data parsing position after the interruption in the original data file, the process includes: verifying the integrity of the original data file; and if the original data file is intact, recording the total amount of data in the original data file.

[0012] In one possible implementation, the periodic recording of the offset information of the target message queue includes: recording the current maximum offset of the target message queue according to a preset time period, and calculating the offset increment within the current period.

[0013] In one possible implementation, after calculating the offset increment within the current period, the process includes: comparing the offset increment with the original position of the source data, and adjusting the parsing position of the source data.

[0014] In one possible implementation, after adjusting the source data parsing position, the consistency between the source data and the target data is verified by at least one of content hash verification, serialization order verification, and timestamp verification.

[0015] In one possible implementation, the method further includes: obtaining the verification result of the verification source data and the target data;

[0016] The verification results are stored locally to form a synchronization status record.

[0017] In one possible implementation, recording the offset information of the target message queue according to a preset time period includes: dynamically adjusting the length of the time period according to the synchronization rate of the target message queue.

[0018] Secondly, this application provides a data synchronization testing device, comprising:

[0019] The recording module is used to periodically record the offset information of the message queue on the target side during the data import process from the source side.

[0020] The acquisition module is used to acquire the mapping relationship between the target end message queue offset information and the source end data parsing position in the event of a synchronization abnormal interruption.

[0021] The testing module is used to generate new untested data files based on the mapping relationship and to resume synchronization testing from the data location corresponding to the mapping relationship.

[0022] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0023] The memory stores computer-executed instructions;

[0024] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0026] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0027] This application provides a data synchronization testing method, apparatus, device, storage medium, and program product. During the source data import process, the offset information of the target message queue is periodically recorded. This lays the foundation for resuming synchronization testing from the breakpoint, avoiding repeated testing or data loss. The target message queue is a distributed message system that receives and stores synchronized data, and the offset information is a unique location identifier for each message in the target message queue. In the event of an abnormal synchronization interruption, the mapping relationship between the target message queue offset information and the source data parsing position is obtained. This divides the synchronization test process into the smallest independently recoverable unit, laying the foundation for subsequent breakpoint resumption. A new untested data file is generated based on this mapping relationship, and the synchronization test is resumed from the data position corresponding to the mapping relationship, achieving breakpoint resumption of the synchronization test. This reduces resource waste caused by repeated processing of full data and improves data synchronization efficiency. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0029] Figure 1 A flowchart illustrating a data synchronization testing method provided in this application;

[0030] Figure 2 A flowchart illustrating another data synchronization testing method provided in this application;

[0031] Figure 3 This application provides a schematic diagram of the structure of a data synchronization testing device.

[0032] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application.

[0033] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0037] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0038] In scenarios such as finance, IoT device data aggregation, and enterprise-level log analysis, it is necessary to synchronize massive amounts of data from a source database to a target database. During the synchronization process, it is essential to test the consistency of the data between the source and target databases. Any missing or duplicate data can lead to business logic anomalies; therefore, data synchronization testing is crucial.

[0039] Currently, existing data synchronization testing methods mainly involve writing all the original data to be tested into the source database, waiting for all data processing to complete, starting the target consumer program, consuming the data in the target database in sequence, and verifying whether the synchronization is successful by comparing the data content, order, and quantity between the source and target ends one by one.

[0040] However, existing data synchronization testing methods compare data from the source and target end one by one. If a network interruption, abnormal database restart, or operating system failure occurs during the synchronization process, the full import and synchronization process must be re-executed, resulting in a waste of resources and low synchronization efficiency.

[0041] Considering the aforementioned problems with existing data synchronization testing methods, this application proposes a method for dynamically detecting the synchronization process and resuming interrupted data synchronization after abnormal interruptions by periodically recording the mapping relationship between the target offset and the source data parsing position. This method improves synchronization efficiency and reduces resource waste.

[0042] The entity executing this data synchronization testing method can be, for example, a data synchronization testing system. Optionally, the data synchronization testing system can be any existing electronic device with processing capabilities, such as a terminal or a server. In some embodiments, the data synchronization testing system can also be deployed in a server cluster or cloud environment. This application does not limit the deployment environment of the data synchronization testing system.

[0043] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0044] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0045] Figure 1 This is a flowchart illustrating a data synchronization testing method provided in this application. Figure 1 As shown, the method includes:

[0046] S101, During the data import process from the source end, the offset information of the target end message queue is periodically recorded; the target end message queue is a distributed message system that receives and stores synchronized data; the offset information is a unique location identifier for each message in the target end message queue.

[0047] Optionally, in the source data import program, the data synchronization test system can integrate the target message queue's software development kit (SDK) to trigger offset recording at fixed time intervals via a scheduled task framework. At the trigger time, the message queue's application programming interface (API) is called to retrieve the offset information corresponding to the synchronized data.

[0048] Optionally, the data synchronization test system can associate the offset information with the corresponding queue name and record timestamp, and store it in a database or local file.

[0049] In some embodiments, the data synchronization test system can record the current maximum offset of the target message queue according to a preset time period, and calculate the offset increment within the current period.

[0050] For example, the preset time period can be a fixed period set according to the testing requirements.

[0051] For example, the current maximum offset can be the maximum location identifier of messages stored in the target message queue within a preset time period.

[0052] For example, the offset increment can be the difference between the maximum offset in the current period and the maximum offset in the previous period. For instance, the offset increment value could correspond to the number of new messages added to the target message queue within the current period.

[0053] Optionally, the data synchronization test system can bind to the target message queue based on its connection information, and record the periodic timestamp of the target message queue, as well as information such as the queue identifier.

[0054] Optionally, the data synchronization test system can obtain the maximum offset by calling the SDK interface of the target message queue within a preset time period. If the target message queue is a multi-partition queue, the global maximum offset of the queue can be generated by summing or taking the maximum value, and then stored in the database or log module of the data synchronization test system after being associated with the timestamp.

[0055] Optionally, after recording the maximum offset for the current period, the data synchronization test system can automatically retrieve historical offset data from the previous period. By calculating the difference between the maximum offset starting this week and the maximum offset from the previous period, the offset increment can be obtained. If the target message queue contains multiple partitioned queues, the data synchronization test system can calculate the offset difference for each partition separately, and summarize the increments of each partition to obtain the total offset increment of the target message queue within the current period.

[0056] Using the above method, the current maximum offset of the target message queue is recorded according to the preset time period, and the offset increment within the current period is calculated to ensure the periodicity of the synchronization test and provide data basis for improving the synchronization efficiency in the future.

[0057] As one possible implementation, after calculating the offset increment within the current period, the data synchronization test system can compare the offset increment with the original position of the source data and adjust the parsing position of the source data.

[0058] Optionally, the data synchronization testing system can obtain the original data volume, start position, and end position of the original data from the source end through the source data acquisition interface when configuring a preset time period. For example, the data synchronization testing system can calculate the number of events generated within the period by reading the start and end positions of the log file, using this as the original data volume; it can also count the number of newly added log lines within the period by detecting the file's read / write pointer offset, using this as the original data volume; and it can also obtain the total original data volume by summing the differences in the data packet sequence numbers of each acquisition node.

[0059] Optionally, the data synchronization test system can retrieve the offset increment of the target message queue within the same period, as well as the original data volume of the source data. The synchronization status is determined by calculating the difference between the offset increment of the target message queue and the original data volume of the source data within the same period.

[0060] Optionally, when the difference between the offset increment and the original data volume is 0, the data synchronization test system can maintain the current source data parsing position unchanged and continue to collect the next batch of data according to the preset cycle. When the difference between the offset increment and the original data volume is greater than 0, the data synchronization test system can determine that the source parsing is lagging or the target end is losing data, and perform backtracking adjustment. For example, the data synchronization test system recalculates the parsing position offset that needs to be backtracked, and locates the starting position of the source original data in the current cycle according to the source original data position ledger. Then, the reading starting point of the source data parsing program is reset to this starting position, the source data is re-parsed and pushed, and the offset recording interval for the next cycle is shortened.

[0061] Optionally, when the offset increment is less than 0 compared to the original data volume, the data synchronization test system can determine that the source parsing position has not been updated, repeatedly read historical data, and perform a forward adjustment. For example, the data synchronization test system can use the target offset increment to deduce the maximum data position that the source has successfully synchronized, and directly adjust the starting point of the source data parsing program to the next position after that maximum position.

[0062] By using the above method, after calculating the offset increment within the current period, the offset increment is compared with the original position of the source data, and the parsing position of the source data is adjusted, reducing manual intervention and improving the accuracy of data synchronization.

[0063] As one possible implementation, after adjusting the source data parsing location, the data synchronization test system can verify the consistency between the source data and the target data through at least one of the following methods: content hash verification, serialization order verification, and timestamp verification.

[0064] For example, content hash verification can be performed by calculating and comparing hash values ​​of source and target data.

[0065] For example, serialization order verification can be used to verify whether the consumption order of the target data is consistent with the insertion order of the source data.

[0066] For example, timestamp verification can be a comparison of the insertion timestamp of the source data with the consumption timestamp of the target data.

[0067] Optionally, the data synchronization testing system can, based on the source-side data parsing program rereading the data at the adjusted position, calculate a unique hash value for each data item (or batch of data blocks), and synchronously push the data content, hash value, and unique data identifier to the target-side message queue. After reading the data from the message queue, the target end uses the same hash algorithm as the source end to recalculate the hash value for the received data or data block. The data synchronization testing system compares the hash value calculated by the target end with the hash value passed from the source end. If the two are completely consistent, the data content is determined to be consistent; if they are inconsistent, the data is determined to have been tampered with or transmitted incorrectly, triggering a data retransmission command.

[0068] Optionally, the data synchronization testing system can add a serialization sequence number to each data item based on the adjusted data pushed from the source end. For batch synchronization, it can also record the start and end sequence numbers of the batch. After the target end reads the data, it extracts the serialization sequence numbers in the receiving order, checks whether the sequence numbers of adjacent data items are continuously increasing without jumps or repetitions, and also checks whether the start and end sequence numbers of the batch cover all data in the batch without any missing items. If a broken or repeated sequence number is detected, the data transmission is determined to be out of order. The data synchronization testing system then readjusts the reading granularity of the source end's parsing position based on the correct serialization order and resynchronizes the out-of-order data.

[0069] Optionally, the data synchronization testing system can use the original timestamp inherent in the source data, binding this original timestamp to the data and synchronizing it to the target end. Based on the timestamp and primary key identifier of the data already stored on the target end, if the timestamp, primary key, and identifier of the received data completely match existing data, it is determined to be duplicate data. The data synchronization testing system can also set a maximum tolerable latency threshold for data synchronization; if the difference between the original timestamp of the received data and the receiving time at the target end exceeds the threshold, it is determined to be timeout data.

[0070] Optionally, if timeout delays are frequently detected, the data synchronization test system can determine that the current parsing position adjustment has too large a backtracking range, resulting in the reading of old data. In this case, the source parsing position can be slightly moved forward to reduce the time range of data reading. If a large amount of duplicate data is detected, the data synchronization test system can adjust the parsing position to the position corresponding to the maximum timestamp of the synchronized data.

[0071] By adjusting the source data parsing location using the methods described above, the data synchronization testing system can verify the consistency between the source and target data through at least one of the following methods: content hash verification, serialization order verification, and timestamp verification. This ensures the comprehensiveness of data integrity verification and improves the accuracy and synchronization efficiency of the test.

[0072] As one possible implementation, the data synchronization testing system can acquire the verification results of the source data and the target data. Then, the verification results are stored locally to form a synchronization status record.

[0073] Optionally, the data synchronization testing system can integrate the results of content hash verification, serialization order verification, or timestamp verification to generate a verification result data packet. For example, this verification result data packet may include the synchronization task name, periodic timestamp, source end parsing location, target end offset information, consistency verification result, and details of abnormal data.

[0074] Optionally, the data synchronization test system can store the above verification result data packets in binary format to a local file based on time sequence, and summarize the structured single verification result records into a synchronization status record set according to the switching batch. This record set contains verification data from multiple mode switching in the same batch.

[0075] By using the above method, the verification results of the source and target data are obtained, laying the foundation for subsequent on-chain evidence storage operations. The verification results are stored locally to form a synchronized status record, improving the credibility and traceability of the test results.

[0076] As one possible implementation, the data synchronization test system can also dynamically adjust the length of the time period based on the synchronization rate of the target message queue.

[0077] For example, the synchronization rate can be the offset increment of the target message queue per unit time.

[0078] Optionally, the data synchronization testing system can automatically calculate the synchronization rate after each cycle ends, based on the statistically calculated cycle offset increment and the current cycle duration. For multi-partition message queues, the data synchronization testing system can calculate the rate of each partition and obtain the global synchronization rate of the queue through a weighted average.

[0079] Optionally, the data synchronization testing system can predefine a low-speed threshold, a standard threshold, and a high-speed threshold. For example, the low-speed threshold can be a critical value indicating an excessively low synchronization rate; below this threshold, the data synchronization testing system can determine that synchronization is lagging. The standard threshold can be a stable synchronization rate range that does not require period adjustment. The high-speed threshold can be a critical value indicating an excessively high synchronization rate; above this value, the synchronization testing system can determine that synchronization is busy.

[0080] Optionally, if the synchronization rate is less than the low-speed threshold, the data synchronization test system can determine that there is a delay or packet loss in the data synchronization link, or that the amount of data generated at the source end is insufficient, shorten the cycle time, and increase the frequency of offset recording and incremental calculation.

[0081] Optionally, if the synchronization rate is less than or equal to the high-speed threshold and greater than or equal to the low-speed threshold, the data synchronization test system can determine that the synchronization link is in a stable state and maintain the current cycle duration unchanged.

[0082] Optionally, if the synchronization rate exceeds the high-speed threshold, the data synchronization test system can determine that the source data is experiencing explosive growth and the target message queue is under high load. This allows for extending the cycle duration, reducing the pressure on the message queue's interface calls due to high-frequency cycle recording, and preventing frequent offset readings from consuming system resources and affecting data synchronization efficiency. Optionally, the data synchronization test system can also preset a minimum (e.g., 1 minute) and a maximum (e.g., 15 minutes) cycle duration, with the adjustment increment set to a fixed step size (e.g., ±10%). If the synchronization rate returns to the standard range for three consecutive cycles, the data synchronization test system can roll back the cycle duration to the initial preset value, ensuring the stability of the detection strategy.

[0083] By dynamically adjusting the length of the time period according to the synchronization rate of the target message queue using the above method, resource waste can be reduced and the verification frequency can be increased. S102, in the event of an abnormal synchronization interruption, the mapping relationship between the target message queue offset information and the source data parsing position is obtained.

[0084] For example, the mapping relationship can be a correspondence between the source data parsing location and the target message queue offset.

[0085] Optionally, the data synchronization testing system can establish a heartbeat interaction between the source parser and the target message queue, with preset heartbeat timeout thresholds (e.g., 5 seconds) and packet loss rate thresholds (e.g., a packet loss rate greater than 30% for 10 consecutive heartbeats). If the data synchronization testing system fails to receive a heartbeat response from either the source or target for three consecutive times, it determines that the network connection is interrupted. If the heartbeat packet loss rate continues to exceed the threshold, accompanied by a sharp increase in data transmission latency, the data synchronization testing system determines that the network is unstable and triggers a synchronization interruption warning.

[0086] Optionally, the data synchronization test system can also compare the amount of data sent by the source end with the amount of data received by the target end. If the difference between the two is stable within 5% when the network is normal, but the difference suddenly spikes to more than 20% in a certain period, and there is no data tampering or duplicate verification anomaly, it can be determined that the network packet loss caused the synchronization interruption.

[0087] Optionally, the data synchronization testing system can monitor the database service's running status in real time through the database client or operation and maintenance interface. If the database process state switching time and the synchronization pause time match perfectly, it can be determined that the database has restarted abnormally; if the database returns a restart-type error code, and the source parsing program cannot read the log position and data table data, it can be determined that a database failure caused the synchronization interruption.

[0088] Optionally, the data synchronization test system can collect operating system metrics of the source and target servers through operation and maintenance tools. If the server's system uptime suddenly drops to zero and the synchronization program starts later than the system restart time, it can be determined that the synchronization is interrupted due to the operating system restart. If the server's central processing unit (CPU) utilization is consistently 100% and memory usage exceeds 95%, and the synchronization program generates error logs, it can be determined that insufficient system resources are causing the synchronization process to be interrupted.

[0089] Optionally, if network connection interruption, network instability, abnormal database restart, or operating system malfunction occurs, the data synchronization test system can determine that synchronization has been abnormally interrupted.

[0090] Optionally, in the event of a synchronization interruption, the data synchronization test system can call the message queue status query interface to obtain the current offset of each partition, exclude temporary offsets that have not been persisted, perform integrity verification on the messages already stored in the target message queue, determine the effective maximum offset of each partition, integrate the effective offsets of all partitions, and determine the global offset range in which the target has successfully received data.

[0091] Optionally, in the event of a synchronization interruption, the data synchronization test system can trace back based on a business identifier or a timestamp. The system can extract a unique business identifier or data timestamp from the target offset range, and then query the data storage medium based on this identifier to locate the corresponding source data record. Based on the storage location of the source data record, the corresponding parsing location is determined, establishing a mapping relationship between the target offset range and the source parsing location range.

[0092] Optionally, in the event of a synchronization interruption, the data synchronization test system can also retrieve the offset-resolution position lookup table periodically generated during normal synchronization. From the lookup table, it can filter out the period record containing the target offset and obtain the corresponding source-end resolution position interval for that period. If the target offset is the endpoint of a certain period's offset, it directly matches the endpoint of the corresponding source-end resolution position for that period. If the target offset is within the offset interval of a certain period, it calculates the corresponding source-end resolution position by combining the ratio of the offset increment to the source-end resolution position increment for that period.

[0093] Optionally, the data synchronization testing system can read data based on the parsing location calculated at the source end, generate a hash value, and compare it with the hash value of the corresponding message at the target end to verify the accuracy of the mapping relationship. If the verification finds a mapping deviation, it is determined that there is data packet loss or tampering during the synchronization process, triggering an anomaly alarm, narrowing the tracing scope, and re-matching.

[0094] S103, Generate a new untested data file based on the mapping relationship, and resume synchronization testing from the data location corresponding to the mapping relationship.

[0095] For example, the untested data file can be a data file to be tested that is extracted and regenerated after the interruption point. In some embodiments, the data synchronization test system can obtain the source data parsing position according to the mapping relationship. Then, incremental data after the source data parsing position at the time of interruption is extracted from the original data file to generate a new untested data file, where the incremental data is the data to be processed in the original data file.

[0096] Optionally, the data synchronization testing system can extract the maximum synchronized parsing position of the source end corresponding to the maximum effective offset of the target end from the established mapping relationship. If the mapping relationship is the range between the target end offset and the source end parsing position, then the maximum synchronized parsing position of the source end is used as the breakpoint parsing position; if there is a slight deviation in the mapping relationship, then the source end parsing position corresponding to the last data that passed the verification is used as the corrected breakpoint parsing position.

[0097] Optionally, the data synchronization testing system can match the index identifier of the original data file based on the type of breakpoint resolution location. If the breakpoint resolution location is a log position, it will locate the byte position after the offset in the original log file; if the breakpoint resolution location is a log file line number, it will locate the log file line number and the content after it in the original log file; if the breakpoint resolution location is a data acquisition sequence number, it will filter all data in the original data file whose sequence number is greater than the data acquisition sequence number.

[0098] Optionally, the data synchronization testing system can call the file reading interface to extract incremental data to be processed from the original data files in batches, starting from the located breakpoint index. The data synchronization testing system can filter the extracted incremental data, encapsulate the filtered incremental data in the format of the original data files, and generate independent untested data files.

[0099] Optionally, the data synchronization testing system can also generate metadata information for the untested data file. This metadata information includes the breakpoint parsing location, the number of incremental data entries, the data volume, and the generation time, among other things.

[0100] Using the method described above, the source data parsing position is obtained based on the mapping relationship. Incremental data following this source data parsing position at the time of interruption is extracted from the original data file to generate a new untested data file, avoiding redundant processing of the entire dataset and reducing resource waste. This incremental data is the data to be processed from the original data file.

[0101] As one possible implementation, before extracting incremental data from the source data parsing position at the time of interruption from the original data file, the data synchronization test system can verify the integrity of the original data file. Then, assuming the original data file is intact, the total amount of data in the original data file is recorded.

[0102] For example, the total data volume can be the total number of data entries contained in the original data file.

[0103] Optionally, the data synchronization testing system can detect whether the storage path of the original data file exists, and call the file reading interface to test whether the file can be opened and read normally. If the file does not exist or cannot be accessed due to insufficient permissions, it can be determined that the file is incomplete, triggering an alarm and terminating incremental data capture.

[0104] Optionally, the data synchronization testing system can calculate and store a unique hash value for the original data file, and associate the hash value with the file identifier for recording. Integrity verification is performed by comparing the hash value of the current file. Alternatively, the file can be divided into blocks of fixed size, and the hash value of each block can be calculated separately for verification by comparing the hash values ​​block by block.

[0105] Optionally, the data synchronization testing system can verify whether the file header identifier and event length of log files conform to the database log format specifications; for text data files, it can verify whether the number of fields and field types of data rows are consistent with the preset template. If there are a large number of invalid data rows with incorrect formats, it can be determined that the file structure is corrupted.

[0106] Optionally, the data synchronization testing system can extract data from multiple random positions at the beginning, end, and middle of the file to verify whether the unique identifier of the data is continuously increasing without jumps or repetitions, and to determine whether the file data is complete.

[0107] Optionally, once the original data file passes all the above integrity checks and is determined to be complete and valid, the data synchronization test system can obtain the total byte size of the file by calling the operating system's file attribute interface, and can also obtain the accurate number of entries by parsing the file content line by line or event by event.

[0108] Optionally, the data synchronization testing system can associate the total amount of data with the file's unique identifier, integrity verification results, breakpoint parsing location, and record timestamps, and write them to the blockchain for evidence storage. It can also back up the original records in the local database.

[0109] By employing the above method, before extracting incremental data from the source data parsing position at the time of interruption, the integrity of the original data file is verified to avoid test failures due to file corruption. If the original data file is intact, the total data volume of the original data file is recorded to ensure the accuracy and reliability of subsequent incremental data extraction.

[0110] In this embodiment, during the source data import process, the offset information of the target message queue is periodically recorded. This lays the foundation for resuming synchronization testing from the breakpoint, avoiding repeated testing or data loss. The target message queue is a distributed message system that receives and stores synchronization data, and the offset information is a unique location identifier for each message in the target message queue. In the event of an abnormal synchronization interruption, the mapping relationship between the target message queue offset information and the source data parsing location is obtained. This divides the synchronization test process into the smallest independently recoverable unit, laying the foundation for subsequent breakpoint resumption. A new untested data file is generated based on this mapping relationship, and the synchronization test resumes from the data location corresponding to the mapping relationship, achieving breakpoint resumption of the synchronization test. This reduces resource waste caused by repeated processing of all data and improves data synchronization efficiency.

[0111] Figure 2 A flowchart illustrating another data synchronization testing method provided in this application. Figure 2 As shown, the method includes:

[0112] (1) Environment preparation completed: Complete system initialization and environment setup to prepare for data synchronization.

[0113] (2) Importing raw data from the source database: Importing the initial raw data from the source database into the system as the basis for synchronization.

[0114] (3) Environmental fault check: Determine if the environment is faulty:

[0115] If so: Record the maximum offset of the target end and the original data position of the previous cycle when the anomaly occurs, truncate the original data at that position, regenerate a new original data file, and return to "Import original data from source database" to re-execute.

[0116] If not: proceed to (4).

[0117] (4) Wait for a cycle: Enter the cycle waiting period and collect incremental data within this cycle.

[0118] (5) Environmental fault check: Determine if the environment is faulty:

[0119] If so: Execute the above fault recovery logic and return to the initial import step.

[0120] If not: proceed to (6).

[0121] (7) Record offset and original data position: Record the current maximum offset of the target end and the original data reading position.

[0122] (8) Environmental fault check: Determine if the environment is faulty:

[0123] If so: Execute the fault recovery logic and return to the initial import step.

[0124] If not: Enter (9).

[0125] (9) Compare the data within the period and adjust the original data position according to the results: Compare the source data and the synchronized data within the period and adjust the reading position of the original data according to the data consistency results.

[0126] (10) Environmental fault check: Determine if the environment is faulty:

[0127] If so: Execute the fault recovery logic and return to the initial import step.

[0128] If not: The process of "waiting for one cycle → recording the offset → adjusting the position" is executed repeatedly until all data is synchronized. Optionally, the data synchronization test system can, based on the source database and target link built with the synchronization tool, reset and clear the message queue used for output results on the target side, record the offset as q0, and set the number of executed Structured Query Language (SQL) statements to p0.

[0129] Optionally, the data synchronization testing system can set a time period according to the scenario to be tested, import the raw data to be tested into the source database, wait for a time period, and record the offset q1. Based on the difference Δq between the current offset and the offset of the previous period, where Δq = q1 - q0, the number of raw data rows that have been executed, p1, is calculated using this difference, where p1 = Δq + p0, and the p1th SQL statement is found in the raw data.

[0130] Optionally, the data synchronization test system can compare the consumer's offset difference Δq with the original data Δp. If the data corresponding to the consumer's offset q1 is the same as the data corresponding to the original data p1, it indicates that the data volume is synchronized within this period, and only the data consistency needs to be compared; if the data corresponding to the consumer's offset q1 is inconsistent with the data corresponding to the original data p1, both the original and the original data need to be compared.

[0131] Optionally, if the data corresponding to q1 is located before the data corresponding to p1 in the original data, record the difference between this position and the number of rows in p1 as △k, indicating that the amount of data output by the target end is △k more than the amount of data input by the source database. Test and record the error, and reset p1 to p1-△k. If the data corresponding to q1 is located after the data corresponding to p1 in the original data, record the difference between this position and the number of rows in p1 as △k, indicating that the amount of data output by the target end is △k less than the amount of data input by the source database. Test and record the error, and reset p1 to p1+△k. Repeat the above steps until the original data synchronization is complete.

[0132] Optionally, if the process encounters faults such as network instability, abnormal database restart, or operating system malfunction, resulting in synchronization failure, the data synchronization test system can record the maximum offset qn of the target end in the previous cycle (denoted as cycle n) and pn in the original statement at the time of the failure. The original data will be truncated at the pn position, and a new original data file will be regenerated. After the synchronization environment is restored, the new original data will be used to resynchronize until the process is completed.

[0133] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0134] Figure 3 This is a schematic diagram of the structure of a data synchronization testing device provided in this application, as shown below. Figure 3 As shown, the data synchronization testing device 300 provided in this embodiment includes: a recording module 301, an acquisition module 302, and a testing module 303. Wherein,

[0135] The recording module 301 is used to periodically record the offset information of the target message queue during the data import process. The target message queue is a distributed message system that receives and stores synchronized data. The offset information is a unique location identifier for each message in the target message queue.

[0136] The acquisition module 302 is used to acquire the mapping relationship between the message queue offset information of the target end and the data parsing position of the source end in the event of a synchronization abnormal interruption.

[0137] Test module 303 is used to generate a new untested data file based on the mapping relationship and to resume synchronization testing from the data location corresponding to the mapping relationship.

[0138] Optionally, the recording module 301 is also used to record the current maximum offset of the target message queue according to a preset time period, and calculate the offset increment within the current period.

[0139] Optionally, the recording module 301 is also used to compare the offset increment with the original position of the source data after calculating the offset increment in the current period, and adjust the parsing position of the source data.

[0140] Optionally, the recording module 301 is further configured to, after adjusting the source data parsing position, verify the consistency between the source data and the target data through at least one of the following methods: content hash verification, serialization order verification, and timestamp verification.

[0141] Optionally, the recording module 301 is also used to obtain the verification result of the verification source data and the target data; and to store the verification result locally to form a synchronization status record.

[0142] Optionally, the recording module 301 is also used to dynamically adjust the length of the time period according to the synchronization rate of the target message queue.

[0143] Optionally, the test module 303 is further configured to obtain the source data parsing position according to the mapping relationship; extract incremental data after the source data parsing position when the interruption occurs from the original data file, and generate a new untested data file, wherein the incremental data is the data to be processed in the original data file.

[0144] Optionally, the test module 303 is also used to verify the integrity of the original data file before extracting incremental data after the source data parsing position when the interruption occurs; and if the original data file is intact, to record the total amount of data in the original data file.

[0145] The data synchronization testing device provided in this embodiment can execute the methods provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0146] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0147] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0148] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0149] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0150] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0151] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0152] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0153] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0154] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0155] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0156] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0157] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0158] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0159] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0160] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0161] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0162] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data synchronization testing method, characterized in that, The method includes: During the data import process from the source end, the offset information of the target end message queue is recorded periodically; the target end message queue is a distributed message system that receives and stores synchronized data; the offset information is a unique location identifier for each message in the target end message queue. In the event of a synchronization failure, obtain the mapping relationship between the target message queue offset information and the source data parsing position; A new untested data file is generated based on the mapping relationship, and synchronization testing is resumed from the data location corresponding to the mapping relationship.

2. The method according to claim 1, characterized in that, The step of generating a new untested data file based on the mapping relationship includes: The source data parsing location is obtained based on the mapping relationship; Extract incremental data from the original data file after the source data parsing position at the time of interruption, and generate a new untested data file. The incremental data is the data to be processed in the original data file.

3. The method according to claim 2, characterized in that, The step of extracting incremental data from the original data file after the interrupted source data parsing position includes: Verify the integrity of the original data file; If the original data file is complete, record the total amount of data in the original data file.

4. The method according to claim 1, characterized in that, The periodically recorded offset information of the target message queue includes: Record the current maximum offset of the target message queue according to the preset time period, and calculate the offset increment within the current period.

5. The method according to claim 4, characterized in that, The step of recording the current maximum offset of the target message queue according to a preset time period, and calculating the offset increment within the current period, includes: The offset increment is compared with the original position of the source data, and the parsing position of the source data is adjusted.

6. The method according to claim 5, characterized in that, After adjusting the parsing position of the source data, the consistency between the source data and the target data is verified by at least one of the following methods: content hash verification, serialization order verification, and timestamp verification.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the verification results of the source data and the target data; The verification results are stored locally to form a synchronization status record.

8. The method according to any one of claims 4-7, characterized in that, The step of recording the offset information of the target message queue according to a preset time period includes: The length of the time period is dynamically adjusted based on the synchronization rate of the target message queue.

9. A data synchronization testing device, characterized in that, include: The recording module is used to periodically record the offset information of the target message queue during the data import process; the target message queue is a distributed message system that receives and stores synchronized data; the offset information is a unique position identifier for each message in the target message queue. The acquisition module is used to acquire the mapping relationship between the target end message queue offset information and the source end data parsing position in the event of a synchronization abnormal interruption. The testing module is used to generate new untested data files based on the mapping relationship and to resume synchronization testing from the data location corresponding to the mapping relationship.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images