Data comparison method

Abstract

This application provides a data comparison method, which includes: acquiring source data and target data, and iteratively executing the following steps until all source and target data are compared: acquiring the target split value and the target data start position corresponding to the current iteration; based on the target split value and the target data start position, acquiring the initial data segment corresponding to the current iteration from the source data, and acquiring the target data segment corresponding to the current iteration from the target data; comparing the initial data segment with the target data segment to determine the comparison result, which includes resource consumption indicators; and based on the target split value and resource consumption indicators corresponding to the current iteration, determining the target split value corresponding to the next iteration, and determining the target data start position corresponding to the next iteration. This solves the technical problem in existing technologies where using a fixed split granularity easily leads to memory overflow or process crashes, thus interfering with the normal business operation of the database.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data comparison method. Background Technology

[0002] With the explosive growth of enterprise data volume, the need for data migration and integration between heterogeneous databases is increasing. In database migration projects, data consistency comparison between the source and target databases is a critical step to ensure successful migration. Data snapshot comparison is currently the mainstream comparison method, which involves obtaining snapshots of the source and target databases at specific points in time and verifying data consistency by comparing the two sets of snapshot data.

[0003] Currently, data snapshot comparison of heterogeneous databases mainly involves pulling full snapshot data from both the source and target databases, splitting the full snapshot data according to a preset fixed splitting granularity, and then starting multi-threaded parallel comparison after all the full snapshot data has been split, finally outputting the consistency comparison result. However, this method uses a fixed splitting granularity, which is prone to memory overflow or process crashes, thereby interfering with the normal operation of the database. Summary of the Invention

[0004] The purpose of this application is to provide a data comparison method to solve the technical problem that existing technologies using fixed splitting granularity are prone to memory overflow or process crashes, thereby interfering with the normal operation of the database. The specific technical solution is as follows: In a first aspect of this application, a data comparison method is provided, the method comprising: Acquire source data and target data, and iteratively execute the following steps until all source data and target data have been compared: Get the target split value and the starting position of the target data for the current iteration; Based on the target split value and the target data start position, obtain the initial data segment corresponding to the current iteration from the source data, and obtain the target data segment corresponding to the current iteration from the target data; The initial data segment is compared with the target data segment to determine the comparison result, which includes resource consumption indicators. Based on the target split value corresponding to the current iteration and the resource consumption index, determine the target split value corresponding to the next iteration, and determine the target data starting position corresponding to the next iteration.

[0005] In an optional implementation, obtaining the target split value and the target data start position corresponding to the current iteration includes: If the current iteration is the first iteration, obtain the attribute information corresponding to the source data; Obtain the initial split value that matches the attribute information corresponding to the source data from the preset segmentation relationship table; Based on the attribute information corresponding to the source data, the initial data location is determined; The initial split value is used as the target split value for the current iteration, and the initial data position is used as the target data starting position for the current iteration.

[0006] In an optional implementation, obtaining the initial data segment corresponding to the current iteration from the source data based on the target split value and the target data start position, and obtaining the target data segment corresponding to the current iteration from the target data, includes: The starting position of the target data is taken as the starting point of the interval, and the sum of the target split value and the starting position of the target data is taken as the ending point of the interval; Based on the start point and end point of the interval, an initial interval segment is constructed; Obtain the data corresponding to the initial interval segment from the source data, and use it as the initial data segment for the current iteration; The data corresponding to the initial interval segment is obtained from the target end data and used as the target data segment for the current iteration.

[0007] In an optional implementation, the initial data segment is compared with the target data segment to determine the comparison result, wherein the comparison result includes resource consumption indicators, including: The data in the initial data segment is compared with the data in the target data segment; The average comparison time for comparing the data in the initial data segment with the data in the target data segment is obtained, and the average comparison time is used as the resource consumption indicator. And / or, The resource parameters are obtained during the process of comparing the data in the initial data segment with the data in the target data segment, and the resource parameters are used as the resource consumption index.

[0008] In an optional implementation, determining the target split value for the next iteration based on the target split value corresponding to the current iteration and the resource consumption index includes: Determine whether the resource consumption index exceeds a preset resource threshold; If the resource consumption index exceeds the preset resource threshold, the target split value corresponding to the current iteration is adjusted to obtain the target split value corresponding to the next iteration. If the resource consumption index does not exceed the preset resource threshold, the target split value corresponding to the current iteration is used as the target split value corresponding to the next iteration.

[0009] In an optional implementation, determining whether the resource consumption index exceeds a preset resource threshold includes: If the resource consumption indicator is the average comparison time, determine whether the average comparison time exceeds a preset time threshold. When the resource consumption indicator is a resource parameter, determine whether the resource parameter exceeds a preset resource parameter threshold; When the resource consumption indicators include the average comparison time and the resource parameters, it is determined whether the average comparison time exceeds a preset time threshold, and / or whether the resource parameters exceed a preset resource parameter threshold.

[0010] In an optional implementation, adjusting the target split value corresponding to the current iteration to obtain the target split value corresponding to the next iteration includes: Get the preset step size; The difference between the target split value corresponding to the current iteration and the preset step size is used as the target split value corresponding to the next iteration.

[0011] In an optional implementation, obtaining the preset step size includes: The degree of resource consumption exceeding the limit is determined based on the resource consumption index and the preset resource threshold. Find the step size value corresponding to the degree of resource consumption exceeding the preset mapping relationship, and use it as the preset step size.

[0012] In an optional implementation, determining the degree of excessive resource consumption based on the resource consumption index and the preset resource threshold includes: Determine the ratio of the resource consumption index to the preset resource threshold, and use the ratio as the degree of resource consumption exceeding the limit; or, The difference between the resource consumption index and the preset resource threshold is determined, and the difference is used as the degree of resource consumption exceeding the limit.

[0013] In an optional implementation, determining the starting position of the target data for the next iteration includes: The end position of the initial data segment corresponding to the current iteration is taken as the starting position of the target data segment corresponding to the next iteration.

[0014] In a second aspect of this application, a data comparison apparatus is also provided, the apparatus comprising: The data acquisition module is used to acquire data from both the source and target ends. The comparison module is used to iteratively execute the following steps until the source data and the target data are fully compared: obtaining the target split value and the target data start position corresponding to the current iteration; based on the target split value and the target data start position, obtaining the initial data segment corresponding to the current iteration from the source data and the target data segment corresponding to the current iteration from the target data; comparing the initial data segment with the target data segment to determine the comparison result, the comparison result including resource consumption indicators; based on the target split value and the resource consumption indicators corresponding to the current iteration, determining the target split value corresponding to the next iteration and determining the target data start position corresponding to the next iteration.

[0015] In a third aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the data comparison method described in any one of the first aspects above.

[0016] In a fourth aspect of the embodiments of this application, a storage medium is also provided, wherein the storage medium stores instructions that, when run on a computer, cause the computer to execute any of the data comparison methods described in the first aspect above.

[0017] In a fifth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data comparison methods described in the first aspect above.

[0018] The technical solution provided in this application involves acquiring source data and target data, and iteratively executing the following steps until all source and target data are compared: obtaining the target split value and the starting position of the target data for the current iteration; obtaining the initial data segment corresponding to the current iteration from the source data and the target data segment corresponding to the current iteration from the target data based on the target split value and the starting position of the target data; comparing the initial data segment with the target data segment to determine the comparison result, which includes resource consumption indicators; and determining the target split value and the starting position of the target data for the next iteration based on the target split value and resource consumption indicators for the current iteration. This improves the overall stability and execution efficiency of the comparison task, and solves the technical problem in existing technologies where fixed split granularity can easily lead to memory overflow or process crashes, thus interfering with the normal operation of the database. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0022] Figure 1 A schematic diagram illustrating the implementation process of a data comparison method provided in this application embodiment; Figure 2 A schematic diagram illustrating the implementation process of a method for obtaining target split values ​​and target data starting positions provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the implementation process of a method for obtaining an initial data segment and a target data segment, provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the implementation process of a method for determining comparison results provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the implementation process of a method for determining a target splitting value provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a data comparison device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0025] The data comparison method provided in this application will be further explained and described below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0026] To address the technical problem that existing technologies, which use fixed granularity for data splitting, are prone to memory overflow or process crashes, thus interfering with normal database operations, this application provides a data comparison method. This method involves acquiring source and target data and iteratively executing the following steps until the source and target data are fully compared: obtaining the target split value and the starting position of the target data for the current iteration; based on the target split value and the starting position, obtaining the initial data segment for the current iteration from the source data and the target data segment for the current iteration from the target data; comparing the initial data segment with the target data segment to determine the comparison result, which includes resource consumption indicators; and based on the target split value and resource consumption indicators for the current iteration, determining the target split value for the next iteration and the starting position of the target data for the next iteration. This method can dynamically adjust the split granularity according to the actual resource consumption during the comparison process, improving the overall stability and execution efficiency of the comparison task.

[0027] like Figure 1 The diagram shown illustrates the implementation flow of a data comparison method provided in this application embodiment, which may specifically include the following steps: S101, Obtain source data and target data, and iteratively execute the following steps until all source data and target data are compared.

[0028] The aforementioned source data refers to the data to be compared in the source database to be verified for consistency. It can be a data table. The source database can be the source database system in a heterogeneous database migration project. For example, the source data is table data in relational databases such as Oracle, MySQL, PostgreSQL, or GoldenDB.

[0029] The aforementioned target data refers to the data to be compared in the target database to be verified for consistency. In other words, it is the data in the target database that corresponds to the source data. This target database can be the target database system for source data migration or synchronization, and it can be of the same or different type as the source database. For example, in a database migration scenario, the target data is the corresponding data in the target database after the source data has been migrated.

[0030] The above-mentioned comparison of source data and target data is complete, meaning that all data records to be compared in the source data and target data have been divided into the corresponding data segments, and the comparison operation of each data segment has been completed, with no omissions or unprocessed record intervals.

[0031] In this embodiment, the execution monitor can acquire source data and target data, and iteratively execute steps S102 to S105 until all source and target data are compared. This step is used to initiate the data comparison task, determine the overall data range for this comparison, and provide a data foundation for subsequent iterative splitting and comparison.

[0032] It should be noted that the acquisition of source and target data can be carried out in parallel to make full use of network and database I / O (Input / Output) resources and shorten data preparation time.

[0033] S102, obtain the target split value and the starting position of the target data corresponding to the current iteration.

[0034] The aforementioned target split value refers to the numerical value used to determine the size of the data segment (initial data segment / target data segment) in the current iteration, that is, the amount of data contained in the data segment (initial data segment / target data segment) obtained in this iteration. This target split value can be represented by the number of records (e.g., 1000 records) or by the span of primary key values ​​(e.g., the primary key value range length is 1000). The specific representation can depend on the data type and distribution characteristics of the source data.

[0035] The aforementioned target data starting position refers to the starting boundary of the initial / target data segment to be acquired in the current iteration within the data table. It indicates from which record or primary key value in the data table the data segment of this iteration begins. For numeric primary keys, this target data starting position can be a specific primary key value; for non-numeric primary keys, this starting position can be the sequence number or offset of the corresponding record in the sorted sequence.

[0036] In this embodiment, the target split value and the target data start position corresponding to the current iteration are obtained. This step is used to determine the data interval boundary targeted by the current comparison operation, providing splitting parameters for subsequent acquisition of corresponding data segments from the source and target ends. For the first iteration, the target split value can be a pre-set initial value or an empirical value matched from a preset segmentation relationship table based on the attribute information of the source and target data. The target data start position is the starting boundary of the initial data segment / target data segment. For subsequent iterations, the target split value and the target data start position are dynamically determined based on the comparison results and resource consumption indicators of the previous iteration.

[0037] As for how the target split value and the starting position of the target data for the current iteration are obtained, this will be explained below. Figure 2 The process shown will be explained in detail here.

[0038] S103, based on the target split value and the starting position of the target data, obtain the initial data segment corresponding to the current iteration from the source data, and obtain the target data segment corresponding to the current iteration from the target data.

[0039] The aforementioned initial data segment refers to the set of data records obtained from the source data based on the target split value and the starting position of the target data in the current iteration. This set contains all records in the source data that are within the specified interval range.

[0040] The aforementioned target data segment refers to the set of data records obtained from the target data based on the target split value and the starting position of the target data in the current iteration. This set contains all records in the target data that fall within the specified interval. The initial data segment and the target data segment logically correspond to the same data interval, meaning they have the same primary key value range, in order to perform subsequent row-by-row consistency comparisons.

[0041] In this embodiment, the initial data segment corresponding to the current iteration can be obtained from the source data and the target data segment corresponding to the current iteration can be obtained from the target data based on the target splitting value and the target data start position. This step is used to extract the data records to be compared from the source and target databases respectively according to the determined splitting parameters (i.e., the target splitting value and the target data start position), preparing data input for subsequent comparison operations. By constructing the initial interval segment by using the target data start position as the interval start point and the sum of the target splitting value and the target data start position as the interval end point, it can be ensured that the initial interval segments obtained in each iteration are logically continuous and non-overlapping, thereby achieving complete coverage of the entire data table.

[0042] The specific methods for obtaining the initial data segment corresponding to the current iteration from the source data and the target data segment corresponding to the current iteration from the target data, based on the target split value and the starting position of the target data, will be explained below. Figure 3 The process shown will be explained in detail here.

[0043] S104. Compare the initial data segment with the target data segment to determine the comparison result, which includes resource consumption indicators.

[0044] The aforementioned comparison refers to the operation of verifying the consistency of each record in the initial data segment with the corresponding record in the target data segment. Specifically, the comparison may include comparing the values ​​of each field, comparing the hash values ​​of key fields, and verifying the number of record rows, to determine whether the source data and the target data are completely consistent within the same data range.

[0045] The aforementioned resource consumption metrics refer to a quantitative measure of system resource usage during the comparison operation between the initial data segment and the target data segment. These metrics can include the average time spent on the comparison operation (e.g., the time spent processing each record), or hardware resource parameters (e.g., CPU (Central Processing Unit) utilization, memory usage, disk I / O overhead, network bandwidth usage, etc.). These resource consumption metrics can serve as feedback signals for subsequent adjustments to the splitting granularity, reflecting the load on system resources imposed by the current splitting strategy.

[0046] The comparison results mentioned above refer to the conclusions obtained after comparing the initial data segment with the target data segment. These results may include resource consumption indicators, specific information about inconsistent records (such as inconsistent row numbers, field names, and value differences), the number of data rows that passed the comparison, and the number of data rows that failed the comparison.

[0047] In this embodiment, the initial data segment and the target data segment can be compared to determine the comparison result, which includes resource consumption indicators. This step is used to perform consistency verification on the two data segments obtained in the current iteration and to simultaneously monitor resource consumption during the comparison process.

[0048] As for the specific process of comparing the initial data segment with the target data segment and determining the comparison result, it will be explained below. Figure 4 The process shown will be explained in detail here.

[0049] S105. Based on the target split value and resource consumption index corresponding to the current iteration, determine the target split value corresponding to the next iteration, and determine the starting position of the target data corresponding to the next iteration.

[0050] In this embodiment of the application, the target split value for the next iteration and the starting position of the target data for the next iteration can be determined based on the target split value and resource consumption index corresponding to the current iteration.

[0051] Specifically, the end position of the initial data segment corresponding to the current iteration (i.e., the end point of the interval) can be used as the starting position of the target data for the next iteration.

[0052] For example, if the target data starting position of the current iteration is 5000 and the target split value is 1000, then the initial data segment of the current iteration covers the primary key value range [5000, 6000]. The end position of this interval is 6000. After taking this as the target data starting position of the next iteration, the next iteration will process the subsequent data with the primary key value starting from 6000, until all data has been processed.

[0053] As to how the target split value for the next iteration is determined based on the target split value and resource consumption indicators of the current iteration, this will be explained below. Figure 5 The process shown will be explained in detail here.

[0054] Based on the above description of the technical solution provided in the embodiments of this application, by acquiring source data and target data, the following steps are iteratively executed until all source data and target data are compared: The target split value and target data start position corresponding to the current iteration are obtained; based on the target split value and target data start position, the initial data segment corresponding to the current iteration is obtained from the source data, and the target data segment corresponding to the current iteration is obtained from the target data; the initial data segment and target data segment are compared to determine the comparison result, which includes resource consumption indicators; based on the target split value and resource consumption indicators corresponding to the current iteration, the target split value corresponding to the next iteration is determined, and the target data start position corresponding to the next iteration is determined.

[0055] This technical solution obtains the initial and target data segments in each iteration based on the target splitting value and the starting position of the target data for the current iteration. Then, based on the resource consumption indicators of comparing the initial and target data segments and the target splitting value for the current iteration, it determines the target splitting value for the next iteration and the starting position of the target data for the next iteration. This ensures that the splitting granularity of each iteration is adapted to the resource consumption indicators, thus avoiding memory resource over-allocation and abnormal process exits caused by excessively large fixed splitting granularity, and also avoiding frequent scheduling overhead and decreased comparison efficiency caused by excessively small fixed splitting granularity. This effectively mitigates interference with normal database operations during the comparison process. Simultaneously, because the splitting granularity for the next iteration is dynamically determined based on the actual execution results of the previous iteration, the system can adaptively handle differences between different data tables without relying on manual parameter tuning. This improves the overall stability and execution efficiency of the comparison task, solving the technical problem that existing technologies using fixed splitting granularity are prone to memory overflow or process crashes, thus interfering with normal database operations.

[0056] like Figure 2 The diagram shown is a schematic representation of the implementation process of a method for obtaining target split values ​​and target data starting positions according to an embodiment of this application. Figure 2 exist Figure 1 Building upon this, the paper details how to obtain the target split value and the starting position of the target data for the current iteration, which may include the following: S201, if the current iteration is the first iteration, obtain the attribute information corresponding to the source data.

[0057] The aforementioned first iteration refers to the first round of iterations after the data comparison task is started. In the first iteration, no feedback information has been obtained regarding the comparison process between the source and target data, so other methods are needed to determine the initial target split value and the starting position of the target data.

[0058] The aforementioned attribute information refers to metadata information used to describe the structural and storage characteristics of the source data, providing a basis for matching appropriate initial splitting parameters from the preset segmentation relationship table. The attribute information may include at least one of the following: the total number of records in the source data (i.e., data size), the data type of the primary key column (e.g., integer, character, date, etc.), the distribution characteristics of the primary key values ​​(e.g., whether they are continuous, whether there is sparse distribution or hotspot areas, etc.), the storage size of the source data (in MB or GB), and the business type identifier to which the source data belongs.

[0059] In this embodiment of the application, when the current iteration is the first iteration, it means that no experience has been accumulated in the process of comparing source data and target data. At this time, it is necessary to determine a reasonable initial split granularity (i.e. target split value) to start the comparison process. Therefore, the attribute information corresponding to the source data is obtained to determine the initial split granularity, so as to avoid the problem of the initial split granularity being too large or too small due to blindly using fixed default values ​​in the first iteration.

[0060] For example, in the case that the current iteration is the first iteration (i.e. the first iteration), in order to match the source data with the optimal initial splitting granularity that is similar to the data in the historical comparison task, it is necessary to first obtain the total number of records in the source data, the data type of the primary key column and the distribution characteristics of the primary key value, and the storage size of the source data.

[0061] S202, retrieve the initial split value from the preset segmentation relationship table that matches the attribute information corresponding to the source data.

[0062] The aforementioned pre-built segmentation relationship table refers to a data structure pre-built and stored in the configuration manager, used to record the optimal initial splitting granularity (i.e., target splitting value) corresponding to data with different attribute characteristics in historical comparison tasks. This pre-built segmentation relationship table stores at least one historical record, and each historical record is associated with and stores the attribute characteristic information of the data and the optimal initial splitting granularity (i.e., target splitting value) that the data finally converged in each comparison.

[0063] In this embodiment, after obtaining the attribute information corresponding to the source data, the attribute information can be used as a query index to retrieve the historical record with the highest matching degree from the preset segmentation relationship table, and the target split value and the starting position of the target data stored in the record can be read. This step is used to utilize the experience and knowledge accumulated in historical comparison tasks to provide a reasonable initial split granularity for the current first iteration, so that the comparison can be started with a near-optimal split granularity even before the feedback information of the current comparison task is obtained, reducing the number of subsequent iteration adjustments and improving the overall comparison efficiency.

[0064] For example, if the source data A to be compared has a total of 5 million records, and the primary key is an auto-incrementing integer with a continuous distribution, then the optimal initial splitting granularity (i.e., the target splitting value) used in the past when processing similar characteristic data tables can be found from the preset segmentation relationship table using the query condition "data volume of 5 million, primary key type continuous integer". If the query finds that a similar data table achieved the best balance between comparison efficiency and resource consumption when the splitting value is 2000, then this splitting value of 2000 can be used as the target splitting value for source data A in the first iteration.

[0065] S203, determine the initial data location by combining the attribute information corresponding to the source data.

[0066] In this embodiment of the application, the initial data location can be determined by combining the attribute information corresponding to the source data.

[0067] As an optional implementation, the initial data position can be determined based on the data type of the primary key column corresponding to the attribute information of the source data. When the data type of the primary key column is numeric (such as integer, long integer, floating-point, etc.), the minimum value of the primary key column can be obtained as the initial data position. In this case, the first iteration will start to retrieve the data segment from the record with the smallest primary key value. When the data type of the primary key column is non-numeric (such as character, date, etc.), the source data can be sorted in ascending order according to the primary key column, and then the position of the first record after sorting (such as sequence number 0 or offset 0) can be used as the initial data position. In this case, the first iteration will start to retrieve the data segment from the first record after sorting.

[0068] As another optional embodiment, the physical storage start address of the tablespace or data file corresponding to the source data can be obtained, and the physical storage start address can be mapped to the logical record sequence number as the initial data position.

[0069] As another optional implementation, when the source data is large and there is partitioned storage, the position of the first record in the first non-empty partition can be determined as the initial data position based on the partition information.

[0070] S204, take the initial split value as the target split value for the current iteration, and take the initial data position as the target data starting position for the current iteration.

[0071] In this embodiment of the application, the initial splitting value can be used as the target splitting value corresponding to the current iteration, and the preset initial data position can be used as the target data starting position corresponding to the current iteration.

[0072] For example, if the primary key of the source data is an integer and its minimum value is 1, and the initial split value matched from the preset segmentation relationship table is 2000, and the initial data position is determined to be the minimum primary key value 1 in S203, then the target split value can be set to 2000 and the target data starting position can be set to 1, indicating that the first iteration will compare the data segment with 2000 records starting from the primary key value 1.

[0073] For example, if the source data is large and there is partitioned storage, and the position of the first record in the first non-empty partition is 10001, then the starting position of the target data is set to 10001, and the first iteration will start to obtain the data segment from this position.

[0074] In another embodiment of this application, when the current iteration is not the first iteration, the target split value determined in the previous iteration can be used as the target split value for the current iteration, and the target data start position determined in the previous iteration can be used as the target data start position for the current iteration. That is, in a non-first iteration, it is not necessary to re-execute the above steps S201 to S204 to determine the target split value and the target data start position, but the split parameters determined and passed down after the end of the previous iteration are directly used as the input for the current iteration.

[0075] like Figure 3 The diagram shown is a schematic representation of the implementation flow of a method for obtaining an initial data segment and a target data segment according to an embodiment of this application. Figure 3 exist Figure 1 Based on this, it describes in detail how to obtain the initial data segment corresponding to the current iteration from the source data and the target data segment corresponding to the current iteration from the target data, according to the target split value and the target data starting position. Specifically, it may include the following: S301, take the starting position of the target data as the starting point of the interval, and take the sum of the target split value and the starting position of the target data as the ending point of the interval.

[0076] In this embodiment of the application, the starting position of the target data can be used as the starting point of the interval, and the sum of the target split value and the starting position of the target data can be used as the ending point of the interval.

[0077] For example, if the starting position of the target data in the current iteration is 5000 and the target split value is 1000, then the starting point of the interval is 5000 and the ending point of the interval is 5000+1000=6000.

[0078] S302, construct the initial interval segment based on the interval start point and interval end point.

[0079] In this embodiment, an initial interval segment can be constructed based on the interval start point and interval end point. The initial interval segment defines the range of data to be obtained from the source and target data in this iteration, and its representation depends on the data type of the primary key in the data table: when the primary key is numeric, the interval segment can be represented as a left-closed, right-open primary key value interval [interval start point, interval end point); when the primary key is non-numeric, the interval segment can be represented as a sorted record sequence number interval [interval start point, interval end point].

[0080] For example, if the starting point of the interval is 5000 and the ending point is 6000, then the initial interval segment constructed is [5000, 6000).

[0081] S303: Obtain the data corresponding to the initial interval segment from the source data and use it as the initial data segment for the current iteration.

[0082] In this embodiment of the application, data corresponding to the initial interval segment can be obtained from the source data and used as the initial data segment for the current iteration.

[0083] Specifically, a data query statement can be constructed based on the start and end points of the initial interval segment, and then sent to the source database to retrieve the corresponding data records. For example, when the primary key is a numeric type, a structured query statement of the form "SELECT * FROM source_data WHERE primary_key_column>= interval_start_point AND primary_key_column< interval_end_point" can be constructed, sent to the source database for execution, and the returned result set can be used as the initial data segment for the current iteration. When the primary key is a non-numeric type, a query statement of the form "SELECT * FROM source_data ORDER BY primary_key_column LIMIT interval_size OFFSET interval_start_point" can be constructed to retrieve several records starting from the specified offset, which will serve as the initial data segment.

[0084] S304: Obtain the data corresponding to the initial interval segment from the target data and use it as the target data segment for the current iteration.

[0085] In this embodiment of the application, data corresponding to the initial interval segment can be obtained from the target end data and used as the target data segment corresponding to the current iteration.

[0086] Specifically, a data query statement can be constructed based on the start and end points of the initial interval segment, and then sent to the target database to retrieve the corresponding data records. For example, when the primary key is a numeric type, a structured query statement of the form "SELECT * FROM target_data WHERE primary_key_column>= interval_start_point AND primary_key_column< interval_end_point" can be constructed, sent to the target database for execution, and the returned result set can be used as the target data segment for the current iteration. When the primary key is a non-numeric type, a query statement of the form "SELECT * FROM target_data ORDER BY primary_key_column LIMIT interval_size OFFSET interval_start_point" can be constructed to retrieve several records starting from a specified offset, which will be used as the target data segment.

[0087] It should be noted that there is no strict order of execution between steps S303 and S304. Query requests can be sent to both the source and target databases in parallel to make full use of network and I / O resources and shorten the total time for data acquisition. Alternatively, the data acquisition operation on one end can be executed first, followed by the data acquisition operation on the other end.

[0088] like Figure 4The diagram shown is a schematic representation of the implementation process of a method for determining comparison results provided in an embodiment of this application. Figure 4 exist Figure 1 Based on this, a detailed description is provided on how to compare the initial data segment with the target data segment and determine the comparison result, which may include the following: S401, compare the data in the initial data segment with the data in the target data segment.

[0089] In this embodiment of the application, the data in the initial data segment can be compared with the data in the target data segment.

[0090] Specifically, each record in the initial data segment can be matched one-to-one with each record in the target data segment, and then the values ​​of corresponding columns in the two records can be compared field by field to see if they are completely identical. During the comparison process, various comparison strategies can be employed: For data segments with a small amount of data, a full field-by-field comparison can be used, where the values ​​of every field in the two records are evaluated for equality, and the two records are considered identical if all fields are the same; for data segments with a large amount of data, a hash comparison method can be used, where the hash values ​​of corresponding records in the initial and target data segments are calculated separately, and the data is judged to be identical by comparing whether the hash values ​​are consistent, thus reducing the overhead of field-by-field comparison. Furthermore, a combination of sampling comparison and full comparison can be used, prioritizing the comparison of key fields when resources are limited.

[0091] It should be noted that if inconsistencies are found when comparing data in the initial data segment with data in the target data segment, specific information about the inconsistency must be recorded, including but not limited to: the row number (or primary key value) of the inconsistent record, the name of the inconsistent field, the values ​​of the source and target fields, and the offset of the inconsistent record within the data segment. This inconsistency information will be included in the final comparison result output to facilitate subsequent data repair and problem localization for the user.

[0092] S402, obtain the average comparison time for comparing the data in the initial data segment with the data in the target data segment, and use the average comparison time as a resource consumption indicator; and / or, obtain the resource parameters during the process of comparing the data in the initial data segment with the data in the target data segment, and use the resource parameters as a resource consumption indicator.

[0093] The average comparison time mentioned above refers to the average time spent processing each data record during the current iteration when performing a record-by-record comparison between the initial data segment and the target data segment. This average comparison time can be calculated by dividing the total comparison time of the current data segment by the total number of records in that data segment. The average comparison time reflects the execution efficiency of the comparison operation at the current splitting granularity. When this value is too large, it indicates that the current splitting granularity (i.e., the target splitting value corresponding to this iteration) is too large, resulting in an excessively heavy load for a single comparison and a decrease in processing speed.

[0094] The aforementioned resource parameters refer to a quantitative measure of the system hardware resource usage during the comparison operation between the initial data segment and the target data segment. These resource parameters may include at least one of the following: CPU utilization (the percentage of CPU time used during the comparison process, reflecting the degree of CPU resource consumption), memory usage (the amount of memory used by system processes during the comparison process (in MB or GB), reflecting the degree of memory resource consumption), disk I / O overhead (the frequency and throughput of disk read / write operations during the comparison process), and network bandwidth usage (the amount of network bandwidth used to acquire data from the source and target ends and transmit the comparison results). Resource parameters can be obtained in real time through resource monitoring interfaces provided by the operating system (such as the / proc file system, performance counters, etc.) or performance views provided by the database management system.

[0095] In this embodiment, the average comparison time for comparing data in the initial data segment with data in the target data segment can be obtained, and this average comparison time can be used as a resource consumption indicator; and / or, resource parameters during the comparison process can be obtained, and these resource parameters can be used as resource consumption indicators. This step is used to quantitatively evaluate the actual load on system resources generated by the comparison operation at the current splitting granularity (target splitting value), transforming abstract resource consumption into quantifiable indicator data. The resource consumption indicator can be used as input parameters for the subsequent step S105 (determining the target splitting value for the next iteration based on the target splitting value and resource consumption indicator corresponding to the current iteration), to determine whether the current splitting granularity is appropriate and whether it needs to be adjusted in the next iteration. By using the average comparison time and / or resource parameters as feedback signals, the resource consumption status of the current comparison operation can be accurately perceived, thereby providing a data basis for realizing dynamic adaptive adjustment of the splitting granularity.

[0096] like Figure 5 The diagram shown is a schematic representation of the implementation process of a method for determining a target splitting value provided in an embodiment of this application. Figure 5 exist Figure 1Building upon this, the paper details how to determine the target split value for the next iteration based on the target split value and resource consumption metrics corresponding to the current iteration. Specifically, this may include the following: S501, determine whether the resource consumption index exceeds the preset resource threshold.

[0097] The aforementioned preset resource thresholds refer to pre-set critical values ​​used to measure whether resource consumption is within a reasonable range, serving as the basis for determining whether the current splitting granularity (i.e., the target splitting value) needs adjustment. These preset resource thresholds may include at least one of the following: a preset time consumption threshold (used to measure whether the average comparison time is within an acceptable range, such as 10ms), a preset CPU utilization threshold (used to measure whether CPU resource consumption is too high, such as 70% or 80%), a preset memory usage threshold (used to measure whether memory resource consumption is close to the system limit, such as 80%), a preset disk I / O threshold (used to measure whether disk read / write load is too heavy), and a preset network bandwidth threshold (used to measure whether network transmission has become a bottleneck).

[0098] In this embodiment of the application, it can be determined whether the resource consumption index exceeds the preset resource threshold in order to determine whether the splitting granularity of the current iteration is appropriate, and thus decide whether the splitting value needs to be adjusted in the next iteration.

[0099] As an optional embodiment, determining whether the resource consumption index exceeds a preset resource threshold may specifically include the following steps: Step 11: If the resource consumption indicator is the average comparison time, determine whether the average comparison time exceeds the preset time threshold.

[0100] In this embodiment of the application, when the resource consumption indicator is the average comparison time, it can be determined whether the average comparison time exceeds a preset time threshold. When the average comparison time exceeds the preset time threshold, it indicates that the current splitting granularity is too large (i.e., the target splitting value), resulting in too much data processed in a single comparison, increasing the average processing time per record, and reducing the comparison efficiency. At this time, it is necessary to reduce the amount of data in a single comparison by adjusting the target splitting value.

[0101] Step 12: When the resource consumption indicator is a resource parameter, determine whether the resource parameter exceeds the preset resource parameter threshold.

[0102] In this embodiment, when the resource consumption indicator is a resource parameter, it can be determined whether the resource parameter exceeds a preset resource parameter threshold. When the resource parameter (such as CPU utilization, memory usage, etc.) exceeds the corresponding preset resource parameter threshold, it indicates that the current comparison operation has approached or exceeded the safe operating boundary for system resources. If the splitting granularity is not adjusted in time, it may cause system failures such as resource exhaustion and process crashes, and may even affect the normal operation of other services on the same database instance.

[0103] Step 13: If the resource consumption indicators include average comparison time and resource parameters, determine whether the average comparison time exceeds the preset time threshold, and / or determine whether the resource parameters exceed the preset resource parameter threshold.

[0104] In this embodiment of the application, when the resource consumption indicators include average comparison time and resource parameters, it is determined whether the average comparison time exceeds a preset time threshold, and / or whether the resource parameters exceed a preset resource parameter threshold.

[0105] As an optional embodiment, when the resource consumption indicators include average comparison time and resource parameters, it can be determined whether the average comparison time exceeds a preset time threshold.

[0106] As an optional embodiment, when the resource consumption indicators include average comparison time and resource parameters, it can be determined whether the resource parameters exceed a preset resource parameter threshold.

[0107] As an optional embodiment, when the resource consumption indicators include average comparison time and resource parameters, it can be determined whether the average comparison time exceeds a preset time threshold and whether the resource parameters exceed a preset resource parameter threshold.

[0108] S502, if the resource consumption index exceeds the preset resource threshold, adjust the target split value corresponding to the current iteration to obtain the target split value corresponding to the next iteration.

[0109] In this embodiment of the application, if the resource consumption index exceeds the preset resource threshold, it indicates that the splitting granularity of the current iteration is too large, resulting in a decrease in the processing efficiency of the comparison operation or an excessively high resource load, which has exceeded the preset safety or efficiency boundary. At this time, it is necessary to adjust the target splitting value corresponding to the current iteration to obtain the target splitting value corresponding to the next iteration, so as to reduce the amount of single comparison data in subsequent iterations, reduce resource consumption, and bring the system operation back to a reasonable range.

[0110] As an optional implementation, adjusting the target split value corresponding to the current iteration to obtain the target split value corresponding to the next iteration can specifically include the following steps: Step 21: Obtain the preset step size.

[0111] In this embodiment, a preset step size can be obtained. The preset step size refers to the magnitude of the decrease in value each time the target split value is adjusted, used to control the speed and magnitude of the split granularity adjustment. The preset step size can be a fixed value (e.g., a fixed decrease of 100 each time) or a variable value dynamically determined based on the degree of resource consumption exceeding the limit.

[0112] As an optional embodiment, obtaining the preset step size may include the following steps: Step 211: Determine the extent of excessive resource consumption based on resource consumption indicators and preset resource thresholds.

[0113] In this embodiment, the degree of resource consumption exceeding the preset resource threshold can be determined based on resource consumption indicators and preset resource thresholds. The degree of resource consumption exceeding the preset resource threshold is used to quantify the deviation of the current resource consumption from the preset resource threshold.

[0114] Specifically, determining the degree of resource consumption exceeding the limit based on resource consumption indicators and preset resource thresholds can include: determining the ratio of the resource consumption indicator to the preset resource threshold and using the ratio as the degree of resource consumption exceeding the limit; or determining the difference between the resource consumption indicator and the preset resource threshold and using the difference as the degree of resource consumption exceeding the limit. When resource consumption indicators include multiple types (such as simultaneously including average comparison time and CPU utilization), the degree of exceeding the limit for each type can be calculated separately, and the maximum value among them can be taken as the final degree of resource consumption exceeding the limit; alternatively, a weighted average of the degree of exceeding the limit for each type can be taken as the final degree of resource consumption exceeding the limit.

[0115] For example, if the average comparison time is 5 milliseconds per record and the preset time threshold is 3 milliseconds per record, then the resource consumption exceeds the limit by approximately 1.67 (5 / 3).

[0116] Step 212: Find the step size value corresponding to the degree of resource consumption excess from the preset mapping relationship, and use it as the preset step size.

[0117] The aforementioned preset mapping relationship refers to a data structure that is pre-established and stored to describe the correspondence between the degree of resource consumption exceeding the limit and the step size value.

[0118] In this embodiment of the application, a step value corresponding to the degree of resource consumption exceeding the limit can be found from a preset mapping relationship and used as the preset step value.

[0119] Step 22: Use the difference between the target split value corresponding to the current iteration and the preset step size as the target split value corresponding to the next iteration.

[0120] In this embodiment, the difference between the target split value corresponding to the current iteration and the preset step size is used as the target split value for the next iteration. This step reduces the granularity of the split by subtracting the preset step size from the current target split value, thereby reducing the amount of data compared in a single iteration and lowering resource consumption in the next iteration.

[0121] For example, if the target split value for the current iteration is 1000, and the preset step size obtained through step 21 above is 200, then the target split value for the next iteration is 1000 - 200 = 800. This means that in the next iteration, each data segment will contain 800 records, a reduction of 200 records compared to the current iteration, in order to reduce the load of a single comparison.

[0122] S503: If the resource consumption index does not exceed the preset resource threshold, the target split value corresponding to the current iteration will be used as the target split value corresponding to the next iteration.

[0123] In this embodiment of the application, if the resource consumption index does not exceed the preset resource threshold, it indicates that the splitting granularity (i.e. the target splitting value) of the current iteration is within a reasonable range, and the efficiency and resource consumption of the comparison operation are within an acceptable level. Therefore, no adjustment is required. Thus, the target splitting value corresponding to the current iteration can be used as the target splitting value corresponding to the next iteration.

[0124] In another embodiment of this application, if the resource consumption index does not exceed the preset lower limit resource threshold, it indicates that the splitting granularity of the current iteration is too small, resulting in too little data for a single comparison. The system frequently performs data acquisition and comparison scheduling operations, introducing excessive task switching overhead and network I / O interaction times, failing to fully utilize system resources, and the overall comparison efficiency is low. At this time, it is necessary to adjust the target splitting value corresponding to the current iteration to obtain the target splitting value corresponding to the next iteration, so as to appropriately increase the splitting granularity, increase the data processing volume of a single comparison, reduce the total number of iterations, and thus optimize the overall comparison efficiency.

[0125] As an optional embodiment, if the resource consumption index does not exceed a preset lower limit resource threshold, the target split value corresponding to the current iteration is adjusted, which may specifically include the following steps: Step 31: Obtain the preset growth step size.

[0126] In this embodiment, a preset growth step size can be obtained. The preset growth step size refers to the increment of the target split value each time it is increased, used to control the speed and magnitude of the increase in split granularity. The preset growth step size can be a fixed value (e.g., a fixed increase of 100 each time), or it can be dynamically determined based on the deviation between the resource consumption index and a preset lower limit resource threshold.

[0127] As an optional embodiment, obtaining a preset growth step size may include the following steps: Step 311: Determine the degree of insufficient resource consumption based on resource consumption indicators and preset lower limit resource thresholds.

[0128] In this embodiment, the ratio of a preset lower limit resource threshold to a resource consumption index can be determined, and this ratio can be used as the degree of insufficient resource consumption; alternatively, the difference between the preset lower limit resource threshold and the resource consumption index can be determined, and this difference can be used as the degree of insufficient resource consumption. The degree of insufficient resource consumption is used to quantify the degree of deviation of the current resource consumption from the preset lower limit threshold. The greater the degree of insufficiency, the smaller the current granularity of the split, and the more significant the adjustment is required.

[0129] Step 312: Find the step size value corresponding to the degree of insufficient resource consumption from the preset growth mapping relationship, and use it as the preset growth step size.

[0130] The aforementioned preset growth mapping relationship refers to a data structure that is pre-established and stored to describe the correspondence between the degree of insufficient resource consumption and the growth step size value.

[0131] Step 32: Use the sum of the target split value corresponding to the current iteration and the preset growth step size as the target split value corresponding to the next iteration.

[0132] In this embodiment, the sum of the target split value corresponding to the current iteration and the preset growth step size can be used as the target split value for the next iteration. By adding the preset growth step size to the current target split value, the split granularity can be amplified, thereby increasing the amount of data compared in a single iteration in the next iteration, reducing the total number of iterations, and improving the utilization rate of system resources and the overall comparison throughput.

[0133] For example, if the target split value for the current iteration is 500, and the preset growth step size obtained through step 31 above is 100, then the target split value for the next iteration is 500 + 100 = 600. This means that in the next iteration, each data segment will contain 600 records, an increase of 100 records compared to the current iteration, in order to reduce the number of iterations and lower task scheduling overhead.

[0134] It should be noted that the preset lower limit resource threshold and the preset resource threshold (i.e., the preset resource threshold used in S501 to determine whether resource consumption is too high, which can be regarded as the upper limit resource threshold) together constitute a reasonable resource consumption range. When the resource consumption index is between the preset lower limit resource threshold and the preset resource threshold, it indicates that the current splitting granularity is appropriate, the system is running well, and there is no need to adjust the target splitting value. When the resource consumption index exceeds the preset resource threshold, the splitting granularity can be reduced to decrease resource consumption in accordance with the method in step S502 above. When the resource consumption index does not exceed the preset lower limit resource threshold, the splitting granularity can be increased in accordance with the method in this embodiment to improve resource utilization efficiency. Through the setting of the above upper and lower limit dual thresholds and the corresponding two-way adjustment mechanism, resource consumption can always be maintained within a reasonable range, ensuring stable system operation while optimizing comparison efficiency.

[0135] In addition, the reasonableness of each determined target split value can be verified. Specifically, the upper and lower limits of the preset split value can be obtained. It is then determined whether the target split value for the next iteration is between the lower and upper limits of the preset split value. If the target split value for the next iteration is between these limits, the verification is successful. If the target split value for the next iteration is less than the lower limit, the lower limit is used as the target split value. If the target split value for the next iteration is greater than the upper limit, the upper limit is used as the target split value. The upper limit refers to the maximum allowed target split value (e.g., 10000), used to prevent the split value from being adjusted too large. When the target split value is too large, the number of records in a single comparison segment may increase dramatically, exceeding the system's available memory capacity, causing memory overflow or frequent garbage collection (GC) pauses, severely affecting the stability of the comparison task and the normal operation of the database. The preset lower limit of the split value refers to the minimum value of the target split value that can be set (e.g., 100), which is used to prevent the split value from being adjusted too small. When the target split value is too small, the number of records contained in the data segment of a single comparison is too small. The system needs to perform more iterations to complete the comparison of all data, which leads to a significant increase in task scheduling overhead, network I / O interaction count, and database connection overhead, and the overall comparison efficiency actually decreases.

[0136] Corresponding to the above method embodiments, this application also provides a data comparison device, such as... Figure 6 As shown, the device may include a data acquisition module 601 and a comparison module 602.

[0137] Data acquisition module 601 is used to acquire source data and target data; The comparison module 602 is used to iteratively execute the following steps until all source and target data are compared: obtain the target split value and target data start position corresponding to the current iteration; based on the target split value and target data start position, obtain the initial data segment corresponding to the current iteration from the source data and the target data segment corresponding to the current iteration from the target data; compare the initial data segment with the target data segment to determine the comparison result, which includes resource consumption indicators; based on the target split value and resource consumption indicators corresponding to the current iteration, determine the target split value corresponding to the next iteration and determine the target data start position corresponding to the next iteration.

[0138] This application also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. Memory 703 is used to store computer programs; In one embodiment of this application, when the processor 701 executes a program stored in the memory 703, it performs the following steps: Acquire source and target data, and iteratively execute the following steps until all source and target data are compared: Obtain the target split value and target data start position corresponding to the current iteration; Based on the target split value and target data start position, obtain the initial data segment corresponding to the current iteration from the source data and the target data segment corresponding to the current iteration from the target data; Compare the initial data segment with the target data segment to determine the comparison result, which includes resource consumption indicators; Based on the target split value and resource consumption indicators corresponding to the current iteration, determine the target split value corresponding to the next iteration and determine the target data start position corresponding to the next iteration.

[0139] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0140] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0141] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0142] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0143] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute any of the data comparison methods described in the above embodiments.

[0144] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the data comparison methods described in the above embodiments.

[0145] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0146] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0147] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0148] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the protection scope of this application.

Claims

1. A data comparison method, characterized in that, The method includes: Acquire source data and target data, and iteratively execute the following steps until all source data and target data have been compared: Get the target split value and the starting position of the target data for the current iteration; Based on the target split value and the target data start position, obtain the initial data segment corresponding to the current iteration from the source data, and obtain the target data segment corresponding to the current iteration from the target data; The initial data segment is compared with the target data segment to determine the comparison result, which includes resource consumption indicators. Based on the target split value corresponding to the current iteration and the resource consumption index, determine the target split value corresponding to the next iteration, and determine the target data starting position corresponding to the next iteration.

2. The method according to claim 1, characterized in that, The step of obtaining the target split value and the starting position of the target data corresponding to the current iteration includes: If the current iteration is the first iteration, obtain the attribute information corresponding to the source data; Obtain the initial split value that matches the attribute information corresponding to the source data from the preset segmentation relationship table; Based on the attribute information corresponding to the source data, the initial data location is determined; The initial split value is used as the target split value for the current iteration, and the initial data position is used as the target data starting position for the current iteration.

3. The method according to claim 1, characterized in that, The step of obtaining the initial data segment corresponding to the current iteration from the source data based on the target split value and the target data start position, and obtaining the target data segment corresponding to the current iteration from the target data, includes: The starting position of the target data is taken as the starting point of the interval, and the sum of the target split value and the starting position of the target data is taken as the ending point of the interval; Based on the start point and end point of the interval, an initial interval segment is constructed; Obtain the data corresponding to the initial interval segment from the source data, and use it as the initial data segment for the current iteration; The data corresponding to the initial interval segment is obtained from the target end data and used as the target data segment for the current iteration.

4. The method according to claim 1, characterized in that, The initial data segment is compared with the target data segment to determine the comparison result. The comparison result includes resource consumption indicators, including: The data in the initial data segment is compared with the data in the target data segment; The average comparison time for comparing the data in the initial data segment with the data in the target data segment is obtained, and the average comparison time is used as the resource consumption indicator. And / or, The resource parameters are obtained during the process of comparing the data in the initial data segment with the data in the target data segment, and the resource parameters are used as the resource consumption index.

5. The method according to claim 1, characterized in that, Determining the target split value for the next iteration based on the target split value corresponding to the current iteration and the resource consumption index includes: Determine whether the resource consumption index exceeds a preset resource threshold; If the resource consumption index exceeds the preset resource threshold, the target split value corresponding to the current iteration is adjusted to obtain the target split value corresponding to the next iteration. If the resource consumption index does not exceed the preset resource threshold, the target split value corresponding to the current iteration is used as the target split value corresponding to the next iteration.

6. The method according to claim 5, characterized in that, The step of determining whether the resource consumption index exceeds a preset resource threshold includes: If the resource consumption indicator is the average comparison time, determine whether the average comparison time exceeds a preset time threshold. When the resource consumption indicator is a resource parameter, determine whether the resource parameter exceeds a preset resource parameter threshold; When the resource consumption indicators include the average comparison time and the resource parameters, it is determined whether the average comparison time exceeds a preset time threshold, and / or whether the resource parameters exceed a preset resource parameter threshold.

7. The method according to claim 5, characterized in that, The step of adjusting the target split value corresponding to the current iteration to obtain the target split value corresponding to the next iteration includes: Get the preset step size; The difference between the target split value corresponding to the current iteration and the preset step size is used as the target split value corresponding to the next iteration.

8. The method according to claim 7, characterized in that, The process of obtaining the preset step size includes: The degree of resource consumption exceeding the limit is determined based on the resource consumption index and the preset resource threshold. Find the step size value corresponding to the degree of resource consumption exceeding the preset mapping relationship, and use it as the preset step size.

9. The method according to claim 8, characterized in that, The step of determining the degree of excessive resource consumption based on the resource consumption index and the preset resource threshold includes: Determine the ratio of the resource consumption index to the preset resource threshold, and use the ratio as the degree of resource consumption exceeding the limit; or, The difference between the resource consumption index and the preset resource threshold is determined, and the difference is used as the degree of resource consumption exceeding the limit.

10. The method according to claim 1, characterized in that, Determining the starting position of the target data for the next iteration includes: The end position of the initial data segment corresponding to the current iteration is taken as the starting position of the target data segment corresponding to the next iteration.

Images