A multi-scale progressive difference data localization method for the education industry

Abstract

The present invention discloses a multi-scale progressive difference data location method for the education industry, comprising the following steps: S1, data sorting: sorting the running table TR and the backup table TB according to the size of the primary key value; S2, calculating the starting primary key value KStart and the ending primary key value KEnd; S3, dividing the primary key value data range [Kttart, KSnd] into scales; S4, judging the grouping: S4.1, KStart i <KEnd i , calculate the data records in TR and TB whose primary key values ​​are within the grouping range; S4.2, KStart i =KEnd i , let K = KStart i =KEnd i , extract the data records corresponding to K for comparison; S5, perform the operation of S4 on all groups to obtain the difference data items. The present invention can quickly and efficiently locate the data records with differences in the running table and the backup table, and output the data comparison results, while also placing less pressure on database performance.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a multi-scale progressive difference data positioning method for the education industry. Background Art

[0002] With the continued advancement and widespread application of educational informatization, the education industry generates massive amounts of new data daily. Compared to other industries, the education industry is characterized by complex data structures, a huge number of users, and rapid data growth. For example, a typical university has tens of thousands of students and thousands of teachers, with various departments running a variety of application systems, such as finance, personnel, student affairs, and academic affairs. These systems generate massive amounts of data constantly. To store this data, a database is required. Relational databases are the most commonly used and mature database systems, storing and retrieving data in rows.

[0003] Educational databases contain massive amounts of data and are subject to constant changes, such as additions, modifications, and deletions. To manage data more securely and efficiently, regular database backups are often necessary. Backups provide essential support for maintenance operations like data cleansing, error correction, and recovery. These operations often require tracing data changes, essentially comparing backup data with current data to identify differences.

[0004] Traditional data comparison methods require comparing each row in the backup database with the runtime database. This operation is extremely time-consuming, and the time required increases exponentially as the amount of data increases. Therefore, a more efficient and rapid data comparison method is needed that can quickly locate differences between two databases and identify the specific types of changes (additions, modifications, deletions).

[0005] Relational databases store data in the form of tables. A table is a two-dimensional table consisting of several columns and several rows, where columns represent data fields and rows represent data records. For example, Table 1 shows the table student used to store student information:

[0006] Table 1: Example of student information table (student)

[0007] Student ID Name age gender College type …… 1 Zhang San 20 male School of Computer Science master …… 2 Li Si 19 female College of Humanities PhD …… 3 Wang Wu 21 male School of Foreign Languages Bachelor …… …… …… …… …… …… …… ……

[0008] Every table has one or more columns that uniquely identify rows of data, called a primary key. For example, the student ID in Table 1 is the primary key of the student table. Although a primary key can be composed of multiple fields, in most cases, a unique column is used as the primary key because it is easier to manage and maintain, and data retrieval is more efficient. A primary key generally has the following characteristics:

[0009] (1) The primary key cannot be empty and cannot be repeated: The primary key must have a value and cannot be repeated with the primary key value of an existing record. This is the basic requirement of relational databases for primary keys.

[0010] (2) Primary key values ​​are comparable, so they can be sorted by size. The most common primary key values ​​are recorded in ascending order using positive integers. Although strings can be used as primary keys, they are rarely used because they cannot be added or subtracted and have many limitations.

[0011] (3) Primary key values ​​cannot be modified. The primary key uniquely identifies a data record and represents the sole retrieval basis for a particular record. Modifying it may lead to data inconsistencies, so most databases do not recommend or even allow modification of primary key values. In fact, we can always avoid modifying its value by adding an additional column with unique values ​​but no actual physical meaning as the primary key.

[0012] (4) Primary key values ​​cannot be reused. If a record is deleted, the primary key value corresponding to the record will not be used in the newly inserted data record. For example, in Table 1, after the student with ID 3 is deleted, the newly inserted data no longer uses 3 as the primary key value. The reasons for not reusing the primary key value are: first, because the primary key value exists in both the running table and the backup table, when comparing the data in the backup table and the running table, it is impossible to distinguish whether the record was deleted and then added or directly modified; second, because the primary key is only used as a data retrieval identifier and has no actual physical meaning, there is no need to reuse the primary key value.

[0013] The meaning of data volume is: in most cases, table columns (i.e., fields) do not change. If a column changes, it is considered a qualitative change in the database structure, which will significantly impact all data usage, operation, and maintenance programs related to the data table. Therefore, in actual database operation, once the database is officially launched, column definitions will never change. Therefore, the metric for data volume in a data table is determined by the number of rows. Currently, there is no universal standard for database volume. It is generally considered that millions of rows are large amounts of data, tens of millions are very large amounts of data, and hundreds of millions and above are huge amounts of data.

[0014] The data tables actually used in educational information systems are called running tables. For example, the student information table in the student affairs system records all student information in real time. By copying and saving the structure and records of a running table at a specific moment, the backup table contains exactly the same data as the current running table at the time the backup is completed. However, over time, the running table may undergo many changes. For example, if the type of student ID 3 changes from "Bachelor" to "Master", the data in the backup table will not change.

[0015] As mentioned above, the operation table will undergo many changes as the business operations are executed. These changes mainly include the following:

[0016] (1) Insert data: A new record is added to the running table. At this time, the primary key of the new record does not exist in the backup table;

[0017] (2) Modify data: The value of one or more fields (not the primary key) of a record in the running table has changed. At this time, the record in the running table and the backup table has the same primary key value;

[0018] (3) Delete data: Delete one or more records from the running table. At this time, the backup table has the primary key of the deleted record but the running table does not.

[0019] Difference data locating involves comparing data in the running table with the backup table to precisely locate changed records. Since data columns are fixed and finite, once the data row is located, field value comparison is easy. The primary key uniquely identifies a row, so difference data comparison requires the primary key value and the type of change in the difference data. Depending on the type of change, the difference results are described as follows:

[0020] (1) Insert data: changeType = insert, newId = the primary key value of the new record;

[0021] (2) Modify data: changeType = update, id = the primary key value of the modified record;

[0022] (3) Delete data: changeType = delete, oldId = the primary key value of the deleted record.

[0023] To locate discrepant data, it is necessary to compare the running table with the backup table. This is particularly difficult when the data volume is large (more than one million), as it is difficult to find even small data changes. Current data comparison methods mainly include:

[0024] (1) Full table comparison: Each row of data in the backup table is retrieved sequentially and compared with the data in the running table. This process can locate modified and deleted data, but cannot find newly added data (because this data does not exist in the backup table). Therefore, all uncompared data in the running table must be recorded, which is very time-consuming. For large data volumes, full table comparison cannot meet the time and efficiency requirements.

[0025] (2) Log analysis: Most databases provide log records of data operations. In theory, all data change information can be found in the log, and the location of data differences can be further located. However, in practice, log analysis technology is very difficult. First, log information is "unstructured" and is just a text description. It is very difficult to extract and parse the details of data changes from it. Second, the amount of log information itself is also very large. It is necessary to analyze all data operations in sequence starting from the backup time, which takes a long time. Third, different databases provide different log information formats and data description methods. It may not be possible to accurately locate the location of the primary key value that has changed. Therefore, in the case of large data volumes, log analysis methods cannot meet the requirements of accurately locating differential data.

[0026] (3) Writing trigger programs: Most databases support writing trigger programs. When certain operations (such as adding, modifying, and deleting) occur on the running table, a specific task is performed (such as recording the change information in a specified data table or file), so that data changes can be obtained in real time. Although trigger programs are powerful, their disadvantages are also very obvious: first, trigger programs are executed in real time along with data change operations, so each data operation will execute the trigger program additionally, which greatly increases the burden on the database server and will greatly reduce the performance of the database. This is intolerable for large data volume environments; second, trigger programs are not universal standards, and the trigger program writing formats of different databases are different; third, trigger programs are difficult to write and are prone to bugs, which can affect normal data storage operations and cause system instability or even crashes. Therefore, trigger programs are only suitable for scenarios with small data volumes and are not universal.

[0027] In summary, currently used technical means and methods cannot effectively support the rapid location of differential data between running tables and backup tables, especially in large data environments. Therefore, the methods need to be redesigned. Summary of the Invention

[0028] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a multi-scale progressive difference data location method for the education industry that can quickly and efficiently locate data record items with differences in the running table and the backup table, output the results of the data comparison, and has less performance pressure on the database.

[0029] The object of the present invention is achieved through the following technical solution: a multi-scale progressive difference data positioning method for the education industry, comprising the following steps:

[0030] S1. Data sorting: Sort the running table TR and the backup table TB by the size of the primary key value. The primary key of TR is recorded as KR, and the primary key of TB is recorded as KB.

[0031] S2. Calculate the starting primary key value KStart and the ending primary key value KEnd;

[0032] S3. Divide the primary key value data range [KStart, KEnd] into scales; the scale size is determined according to the following formula:

[0033]

[0034] The symbols Indicates rounding up; denominator n∈[2,p], if p<2, then take n=2;

[0035] Use the selected scale to divide [KStart, KEnd], the i-th group G i The data value range is calculated as follows:

[0036] G i =[KStart i , KEnd i ]=[KStart+(i-1)×D,min(KStart+i×D-1,KEnd)]

[0037] i≥1;

[0038] S4, grouping G according to scale i Making a judgment includes the following sub-steps:

[0039] S4.1. If KStart i <KEnd i , calculate the data records whose primary key values ​​in TR and TB are within the grouping range respectively;

[0040] S4.2. If KStart i =KEnd i , indicating that there is only one primary key value data, let: K = KStart i =KEnd i , directly take out the data records corresponding to K for comparison;

[0041] S5. Execute the operation of step S4 for all groups of [KStart, KEnd] to obtain all difference data items in TR and TB.

[0042] The specific implementation method of step S2 is:

[0043] S2.1. Calculate the minimum value of KR, min(KR);

[0044] S2.2. Calculate the minimum value of KB min(KB);

[0045] S2.3. Calculate the maximum value of KR max(KR);

[0046] S2.4. Calculate the maximum value of KB max(KB);

[0047] S2.5. Calculate the starting primary key value KStart = min(min(KR), min(KB));

[0048] S2.6. Calculate the ending primary key value KEnd = max(max(KR), max(KB)).

[0049] The specific implementation method of step S4.1 is:

[0050] S4.1.1 Query data records in TR, the query condition is: KR∈[KStart i , KEnd i ], get the data record set as DataSetR i ; The corresponding SQL statement is: Select * from TR where id betweenKStart i andKEnd i ;

[0051] S4.1.2、Query the primary key value range in TB in [KStart i , KEnd i ] in the data record set DataSetB i The corresponding SQL statement is: Select * from TB where id between KStart i andKEnd i ;

[0052] S4.1.3、DataSetR i and DataSetB i Perform hash operations and compare the results for equality:

[0053] If hash(DataSetR i)=hash(DataSetB i ), indicating that G i DataSetR in the range i and DataSetB i Equal, there is no difference in data, no need to further subdivide the records in the comparison data set, group G i The operation ends directly;

[0054] If hash(DataSetR i )≠hash(DataSetB i ), indicating that the primary key value is in G i There are differences in the data records within the G i The scale is further subdivided and compared gradually; at this time, G i Consider it as the entire data set, continue to group and divide it, and return to step S3.

[0055] The specific implementation method of step S4.2 is:

[0056] S4.2.1. Query data records in TR. The query condition is: KR=K. The corresponding SQL statement is: Select*fromTR where KR=K. The query result is recorded as DataR i If the primary key value K does not exist in TR, set DataR i =null;

[0057] S4.2.2. Query data records in TB. The query condition is: KB = K. The corresponding SQL statement is: Select * from TB where KB = K. The query result is recorded as DataB i If the primary key value K does not exist in TB, set DataB i =null; Since K is the union of the primary keys of TR and TB, DataR is not possible. i and DataB i The case where both are null;

[0058] S4.2.3 Comparison with DataR i and DataB i Value:

[0059] If DataR i ≠nullandDataB i ≠nullandhash(DataR i )≠hash(DataB i), indicating that records with primary key K exist in both TR and TB, but their hash values ​​are different, indicating that data modification has occurred. The data change is recorded as: changeType = update, id = K;

[0060] If DataR i ≠nullandDataB i ≠nullandhash(DataR i )=hash(DataB i ), indicating that the record of primary key K exists in both TR and TB, but the data has not changed. It is not differential data and no record needs to be made at this time;

[0061] If DataR i ≠nullandDataB i =null, indicating that the running table TR has a primary key K, but the backup table TB does not have K, indicating that data insertion has occurred. The data change is recorded as follows: changeType=insert,newId=K;

[0062] If DataR i =nullandDataB i ≠null indicates that the primary key K does not exist in the running table TR, but exists in the backup table TB, indicating that data deletion has occurred. The data change is recorded as: changeType = delete, oldId = K.

[0063] The present invention has the beneficial effect of quickly and efficiently locating discrepant data records in the running table and the backup table and outputting the data comparison results. Furthermore, the method supports any common relational database and places minimal pressure on database performance, thereby overcoming various drawbacks of traditional data comparison methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flow chart of the multi-scale progressive data comparison method of the present invention. DETAILED DESCRIPTION

[0065] The following definitions are given for the terms used in this invention:

[0066] Data partitioning: All data in a data set can be grouped. Let the number of groups be n, and each group is recorded as G1, G2, G3...G n (n≥2). Let the entire data set be U, and each grouping be a subset of U. A grouping is called a partition of U if and only if the following conditions hold:

[0067] (1)G1∪G2∪G3∪...∪G n= U, that is, the union of all groups is the entire set, so any data record must be in a group;

[0068] (2) For any sequence number i≠j, That is, any two groups do not contain the same data records, so any data record must be in a unique group.

[0069] The grouping method that satisfies the above conditions is called a partition of U, such as dividing into odd and even groups according to the primary key value, or dividing into male and female according to gender, etc.

[0070] Data scale: The data scale definition of the present invention is to measure the data group G i In order to improve data processing efficiency, the balance of groups should be ensured, that is, the number of data records in each group should be equal or as close as possible. The number of data in a certain group is called data scale, which is recorded as |U| represents the total number of records in the complete set. Since |U| may not be divisible by n, this data partitioning method should ensure that most groups have the same number of records (equal to the scale). The last group may deviate by |U| mod n data points, but this does not affect the definition of scale. The size of the scale indicates the level of detail in the partitioning of U: a larger scale indicates a coarser data partitioning, while a smaller scale indicates a finer partitioning.

[0071] If a data group G i As a new complete set, further G i To divide, there is obviously By analogy, any group can be further divided until the scale is equal to 1.

[0072] Multi-scale progression means gradually approaching data records with certain characteristics by adjusting the scale of the division, and those large-scale groups in the division that do not meet the conditions can be eliminated as a whole, which greatly improves the efficiency of data matching and comparison.

[0073] Data hashing: A hash function is used to map one or more data records into information summaries of equal length. The requirements for an ideal hashing operation are:

[0074] (1) Regardless of the size of the original data r, the hash operation is fast and the length of the operation result is equal.

[0075] (2) If two data records r1=r2, then hash(r1)=hash(r2).

[0076] (3) If r1≠r2, then hash(r1)≠hash(r2).

[0077] For any hash function, (1) and (2) must be satisfied, but (3) is satisfied only in an ideal state, that is, r1=r2 is a necessary and sufficient condition for hash(r1)=hash(r2). However, in actual situations, due to the existence of a certain probability of hash collision (different original data calculate the same hash value), r1=r2 is a sufficient but not necessary condition for hash(r1)=hash(r2). However, if the probability of hash collision is low enough (for example, less than 2 -10 ) can be considered that the hash operation is reliable.

[0078] The algorithm process of the hash function can be independently designed or directly use a known public algorithm. The present invention recommends using the MD5 digest algorithm, which has the advantages of fast operation speed and low collision probability (less than 2 -64 ) characteristics.

[0079] In some extreme cases, it is necessary to ensure the reliability of hash operations with a very high probability. Double hashing can be used, that is, using two different hash algorithms to compare data. If: hash1(r1)=hash1(r2) and hash2(r1)=hash2(r2), the probability of a conflict is reduced to the conflict probability 1×the conflict probability 2. This can be considered a very reliable hash operation, and it can be completely assumed that r1=r2 holds at this time.

[0080] The reason for using hash operations is that when comparing one or more data records, we can avoid comparing each data item row by row and column by column, and only need to compare their hash values ​​as a whole.

[0081] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0082] like Figure 1 As shown, the present invention is a multi-scale progressive difference data location method for the education industry, which is used to quickly locate those data records with differences in the running table and the backup table, and output the data comparison results. The prerequisites of the present invention include:

[0083] (1) The running table (denoted as TR) and the backup table (denoted as TB) have exactly the same columns (fields), including the number and type, as well as the order of the fields; the data table actually used in the educational information system is called the running table, such as the student information table in the student affairs system, which records all student information in real time.

[0084] (2) TR and TB have unique primary keys and will not be modified;

[0085] (3) The primary key types of TR and TB are integers, so sorting and addition and subtraction operations can be performed.

[0086] The positioning method of the present invention comprises the following steps:

[0087] S1. Data sorting: Sort the running table TR and the backup table TB by the size of the primary key value (ascending or descending. If there is no special requirement, ascending is generally used because this is the default sorting rule of most databases). The primary key of TR is recorded as KR, and the primary key of TB is recorded as KB. The purpose of sorting is to facilitate the subsequent calculation of the primary key value range.

[0088] S2. Calculate the starting primary key value KStart and the ending primary key value KEnd. The specific implementation method is:

[0089] S2.1. Calculate the minimum value of KR, min(KR);

[0090] S2.2. Calculate the minimum value of KB min(KB);

[0091] S2.3. Calculate the maximum value of KR max(KR);

[0092] S2.4. Calculate the maximum value of KB max(KB);

[0093] Since the database uses indexing technology for primary key values ​​by default, the speed of calculating the minimum and maximum values ​​of the primary key is very fast. According to tests, the query results of tens of millions of data can generally be obtained within 0.1 seconds.

[0094] S2.5. Calculate the starting primary key value KStart = min(min(KR), min(KB));

[0095] S2.6. Calculate the ending primary key value KEnd=max(max(KR), max(KB)).

[0096] That is, the starting primary key value KStart is equal to the minimum primary key value of KR and KB, and the ending primary key value KEnd is equal to the maximum primary key value of KR and KB.

[0097] S3. Divide the primary key value data range [KStart, KEnd] into scales. The scale size is related to the value of the data range (i.e., KEnd-KStart), and can be determined according to the following formula:

[0098]

[0099] The symbols The denominator n∈[2, p], if p<2, then n=2; rounding up ensures that D≥1. When D=1, it means there is only one piece of data in the group, and it cannot be further subdivided.

[0100] The value of n can be adjusted appropriately based on actual conditions. An excessively large n results in a smaller scale, which can lead to an excessive number of groups and increased data comparisons, resulting in reduced efficiency. An excessively small n results in a larger scale, which increases the range of each group, increasing the probability of containing discrepant data and triggering a new round of grouping and comparison. Generally speaking, p can be set to the number of cores in the current computer's CPU to support multithreaded concurrent computing and improve computational efficiency.

[0101] Use the selected scale to divide [KStart, KKnd], the i-th group G i The data value range is calculated as follows:

[0102] G i =[KStart i , KEnd i ]=[KStart+(i-1)×D,min(KStart+i×D-1,KEnd)]

[0103] i≥1; this formula complies with the definition of partitioning and ensures that the upper limit value does not exceed KEnd, thereby avoiding invalid data search range.

[0104] S4, grouping G according to scale i Making a judgment includes the following sub-steps:

[0105] S4.1. If KStart i <KEnd i , respectively calculate the data records in TR and TB whose primary key values ​​are within the grouping range; since the primary key value is the unique identifier of the record, this step actually realizes the division of the data records in TR and TB. The specific implementation method is:

[0106] S4.1.1 Query data records in TR, the query condition is: KR∈[KStart i , KEnd i ], get the data record set as DataSetS i ; The corresponding SQL statement is: Select * from TR where id between andKEnd i Because the primary key value is indexed, this step takes very little time, and results can generally be obtained in seconds for tens of millions of data points.

[0107] S4.1.2、Query the primary key value range in TB in [KStart i , KEnd i ] in the data record set DataSetB iThe corresponding SQL statement is: Select * from TB where id between KStart i andKEnd i ;

[0108] S4.1.3、DataSetR i and DataSetB i Perform hash operations and compare the results for equality:

[0109] If hash(DataSetR i )=hash(DataSetB i ), indicating that G i DataSetR in the range i and DataSetB i Equal, there is no difference in data, no need to further subdivide the records in the comparison data set, group G i The operation ends directly;

[0110] If hash(DataSetR i )≠hash(DataSetB i ), indicating that the primary key value is in G i There are differences in the data records within the G i The scale is further subdivided and compared gradually; at this time, G i Consider it as the entire data set, continue to group and divide it, return to step S3, and divide the data according to G i The data cases take different n values.

[0111] S4.2. If KStart i =KEnd i , indicating that there is only one primary key value data, let: K = KStart i =KEnd i , directly take out the data records corresponding to K for comparison; the specific implementation method is:

[0112] S4.2.1. Query data records in TR. The query condition is: KR=K. The corresponding SQL statement is: Select*fromTR where KR=K. The query result is recorded as DataR i If the primary key value K does not exist in TR, set DataR i =null;

[0113] S4.2.2. Query data records in TB. The query condition is: KB = K. The corresponding SQL statement is: Select * from TB where KB = K. The query result is recorded as DataB i If the primary key value K does not exist in TB, set DataB i =null; Since K is the union of the primary keys of TR and TB, DataR is not possible. i and DataB i The case where both are null;

[0114] S4.2.3 Comparison with DataR i and DataB i Value:

[0115] If DataR i ≠nullandDataB i ≠nullandhash(DataR i )≠hash(DataB i ), indicating that records with primary key K exist in both TR and TB, but their hash values ​​are different, indicating that data modification has occurred. The data change is recorded as: changeType = update, id = K;

[0116] If DataR i ≠nullandDataB i ≠nullandhash(DataR i )=hash(DataB i ), indicating that the record of primary key K exists in both TR and TB, but the data has not changed. It is not differential data and no record needs to be made at this time;

[0117] If DataR i ≠nullandDataB i =null, indicating that the running table TR has a primary key K, but the backup table TB does not have K, indicating that data insertion has occurred. The data change is recorded as follows: changeType=insert,newId=K;

[0118] If DataR i =nullandDatab i ≠null indicates that the primary key K does not exist in the running table TR, but exists in the backup table TB, indicating that data deletion has occurred. The data change is recorded as: changeType = delete, oldId = K.

[0119] S5. Perform the judgment operation of step S4 on all groups of [KStart, KEnd] to obtain all difference data items in TR and TB.

[0120] The methods and steps described in this invention are applicable to large data volumes (over 100,000) with a discrepancy of less than 20%. Large amounts of discrepancies may lead to more incremental partitioning and comparison, which can reduce efficiency. However, tests have shown that searches on data volumes exceeding 10 million (50,000 discrepancies) can be completed in under 2 minutes.

[0121] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multi-scale progressive difference data positioning method for the education industry, characterized by: The following steps are involved: S1. Data sorting: Sort the running table TR and the backup table TB by the size of the primary key value. The primary key of TR is recorded as KR, and the primary key of TB is recorded as KB. S2. Calculate the starting primary key value KStart and the ending primary key value KEnd; S3. Divide the primary key value data range [KStart, KEnd] into scales; the scale size is determined according to the following formula: The symbols Indicates rounding up; denominator n∈[2, p], if p<2, then take n=2; Use the selected scale to divide [KStart, KEnd], the i-th group G ∈ The data value range is calculated as follows: G i =[KStart i ,Known i ]=[KStart+(i-1)×D,min(KStart+i×D-1,KEnd)]i≥1; S4, grouping G according to scale i Making a judgment includes the following sub-steps: S4.

1. If KStart i <KEnd i , calculate the data records whose primary key values ​​in TR and TB are within the grouping range respectively; S4.

2. If KStart i =KEnd i , indicating that there is only one primary key value data, let: K = KStart i =KEnd ∈ , directly take out the data records corresponding to K for comparison; S5. Execute the operation of step S4 for all groups of [KStart, KEnd] to obtain all difference data items in TR and TB.

2. The multi-scale progressive difference data positioning method for the education industry according to claim 1, characterized in that: The specific implementation method of step S2 is: S2.

1. Calculate the minimum value of KR, min(KR); S2.

2. Calculate the minimum value of KB min(KB); S2.

3. Calculate the maximum value of KR max(KR); S2.

4. Calculate the maximum value of KB max(KB); S2.

5. Calculate the starting primary key value KStart = min(min(KR), min(KB)); S2.

6. Calculate the ending primary key value KEnd=max(max(KR), max(KB)).

3. The multi-scale progressive difference data positioning method for the education industry according to claim 1, characterized in that: The specific implementation method of step S4.1 is: S4.1.1 Query data records in TR, the query condition is: KR∈[KStart i , KEnd i ], get the data record set as DataSetR ∈ ; The corresponding SQL statement is: Select * from TR where id between KStart i and KEnd i ; S4.1.2、Query the primary key value range in TB in [KStart i , KEnd i ] in the data record set DataSetB i The corresponding SQL statement is: Select * from TB whereid between KStart i and KEnd i ; S4.1.3、DataSetR i and DataSetB i Perform hash operations and compare the results for equality: If hash(DataSetR i )=hash(DataSetB i ), indicating that G i DataSetR in the range i and DataSetB i If they are equal, there is no difference in data, and there is no need to further subdivide and compare the records in the data set. The operation of group Gi ends directly; If hash(DataSetR i )≠hash(DataSetB i ), indicating that the primary key value is in G i There are differences in the data records within the G i The scale is further subdivided and compared gradually; at this time, G i Consider it as the entire data set, continue to group and divide it, and return to step S3.

4. The multi-scale progressive difference data positioning method for the education industry according to claim 1, characterized in that: The specific implementation method of step S4.2 is: S4.2.

1. Query data records in TR. The query condition is: KR=K. The corresponding SQL statement is: Select * from TR where KR=K. The query result is recorded as DataR i If the primary key value K does not exist in TR, set DataR i =null; S4.2.

2. Query data records in TB. The query condition is: KB = K. The corresponding SQL statement is: Select * from TB where KB = K. The query result is recorded as DataB i If the primary key value K does not exist in TB, set DataB i =null; Since K is the union of the primary keys of TR and TB, DataR is not possible. i and DataB i The case where both are null; S4.2.3 Comparison with DataR i and DataB i Value: If DataR i ≠null and DataB i ≠null and hash(DataR i )≠hash(DataB i ), indicating that records with primary key K exist in both TR and TB, but their hash values ​​are different, indicating that data modification has occurred. The data change is recorded as: changeType = update, id = K; If DataR i ≠null and DataB ∈ ≠null and hash(DataR i )=hash(DataB i ), indicating that the record of primary key K exists in both TR and TB, but the data has not changed. It is not differential data and no record needs to be made at this time; If DataR i ≠null and DataB i =null, indicating that the running table TR has a primary key K, but the backup table TB does not have K, indicating that data insertion has occurred. The data change is recorded as follows: changeType=insert, newId=K; If DataR i =null and DataB i ≠null indicates that the primary key K does not exist in the running table TR, but exists in the backup table TB, indicating that data deletion has occurred. The data change is recorded as: changeType = delete, oldId = K.

Images