A data consistency checking method, device and equipment and readable storage medium

Abstract

The application discloses a data consistency verification method and device, equipment and a readable storage medium, applied to the technical field of data management, comprising: determining a synchronization field rule according to synchronization table metadata; reading data of a to-be-verified business table in an edge end in batches, and determining a current primary key interval according to a primary key value of adjacent page tail data; determining to-be-verified data of two ends based on a table name, the current primary key interval and the synchronization field rule; processing the to-be-verified data of two ends according to the same normalization rule and performing consistency verification; and circularly processing all pages to obtain a difference row set. Compared with current full comparison, the application limits a verification field through a synchronization field rule, forms a primary key interval commonly used by two ends through a page tail primary key value, and unifies the representation form of data of two ends based on a normalization rule, so that a non-synchronization field does not enter a comparison range, and data of two ends will not be misjudged due to different representation forms, thereby improving the accuracy and efficiency of data consistency verification.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a data consistency verification method, apparatus, device, and readable storage medium. Background Technology

[0002] In cloud-edge data synchronization frameworks, the same business table often exists as two copies simultaneously, one in the cloud and one at the edge. To ensure the synchronization link is functioning correctly, the system typically needs to periodically verify the consistency between the cloud and edge data. Traditional consistency checks usually involve directly comparing the entire table or entire rows of fields, or exporting the entire table and comparing each row individually to determine differences. These methods have significant shortcomings in cloud-edge scenarios: First, edge nodes have limited network bandwidth and remote call capabilities, and full table verification can easily lead to significant data transmission overhead; second, cloud-edge synchronization frameworks typically do not perform bidirectional synchronization of all fields in the entire table, and directly comparing entire rows of fields can generate a large number of false alarms.

[0003] It is evident that improving the accuracy and efficiency of data consistency verification is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data consistency verification method, apparatus, device and readable storage medium, which solves the technical problem of low accuracy and efficiency of data consistency verification in the prior art.

[0005] To address the aforementioned technical problems, this invention provides a data consistency verification method applied to edge computing, comprising: The synchronization field rules are determined based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; The data of the business table to be verified in the edge terminal is read in pages according to the preset page size, and the current primary key range corresponding to the current page is determined based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page. Based on the table name, the current primary key range, and the synchronization field rules, determine the edge terminal data to be verified corresponding to the edge terminal, and obtain the center terminal data to be verified returned by the center terminal after searching based on the table name, the current primary key range, and the synchronization field rules; Obtain edge-end and center-end standardized data to be verified after processing the edge-end and center-end data to be verified based on normalization rules, and perform consistency verification on the edge-end and center-end standardized data to be verified to obtain the difference rows in the current primary key interval; Continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

[0006] Optionally, the synchronization field rules can be determined based on the synchronization table metadata of the business table to be verified, including: The set of cloud-to-edge synchronization fields is determined according to the cloud-to-edge synchronization field rules. The set of edge-to-cloud synchronization fields is determined according to the edge-to-cloud synchronization field rules. The union of the cloud-to-edge synchronization field set and the edge-to-cloud synchronization field set is determined as the range of fields that need to be verified for consistency, and the synchronization field rules are formed based on the range of fields.

[0007] Optionally, the set of cloud-to-edge synchronization fields is determined according to the cloud-to-edge synchronization field rules, including: When the cloud-to-edge synchronization field rule is a field inclusion set, the fields in the field inclusion set are determined as the cloud-to-edge synchronization field set in the cloud-to-edge synchronization direction; When the cloud-to-edge synchronization field rule is a field exclusion set, the fields in the field exclusion set are excluded from the business fields that need to be synchronized in the business table to be verified, thus obtaining the cloud-to-edge synchronization field set.

[0008] Optionally, the synchronized table metadata also includes logical deletion rules; determining the edge-end data to be verified corresponding to the edge end based on the table name, the current primary key range, and the synchronization field rules, and obtaining the center-end data to be verified returned by the center end after searching based on the table name, the current primary key range, and the synchronization field rules, including: Obtain the data to be verified at the central end, which is determined by the central end based on the table name, the current primary key range, the synchronization field rules, and the logical deletion rules; The process of determining the data to be verified at the central end includes: based on the current primary key range and the synchronization field rules, the initial data to be verified at the central end is determined, the operation time field is deleted from the initial data to be verified at the central end, and data that matches the logical deletion rules is excluded, so as to obtain the data to be verified at the central end.

[0009] Optionally, data from the business table to be verified in the edge terminal is read in pages according to a preset page size, and the current primary key range corresponding to the current page is determined based on the primary key value of the last record of the current page and the primary key value of the last record of the previous page, including: The data of the business table to be verified is read in pages according to the primary key values ​​corresponding to the primary key fields; When the current page is the first page, the preset initial primary key value is determined as the lower bound of the current primary key range; When the current page is not the first page, the primary key value of the last data entry on the previous page is determined as the lower bound of the current primary key range; The primary key value of the last record on the current page is determined as the upper bound of the current primary key range, and the current primary key range is determined as data whose primary key value is greater than the lower bound and less than or equal to the upper bound.

[0010] Optionally, obtaining edge-end verification standardized data and center-end verification standardized data after processing the edge-end verification data and the center-end verification data respectively based on normalization rules includes: Sort the fields in the edge-end and center-end data to be verified according to the field name respectively, and retain the null value fields in the edge-end and center-end data to be verified. The sorted edge-end data to be verified and the center-end data to be verified are converted into specified string formats to obtain the edge-end data to be verified and the center-end data to be verified.

[0011] Optionally, a consistency check is performed on the edge-end specification data and the center-end specification data to be verified to obtain the difference rows within the current primary key interval, including: An edge data mapping is generated with the primary key value as the key and the corresponding edge end verification specification data as the value, and a center end data mapping is generated with the primary key value as the key and the corresponding center end verification specification data as the value. Based on the edge data mapping and the center data mapping, consistency verification is performed, and data that do not have the same primary key value on one end or whose corresponding data to be verified are different are identified as differential rows. Determine the difference fields in the difference rows; Accordingly, the process continues reading the next page and executing the steps of determining the primary key range, determining the data to be verified, and performing consistency verification, until the consistency verification of all pages of the business table to be verified is completed, resulting in the set of differing rows corresponding to the business table to be verified, including: After completing the consistency check of all primary key ranges of the business table to be checked, the table-level consistency check result is output, including the table name, the set of differing rows, and the set of differing fields.

[0012] The present invention also provides a data consistency verification device, applied at the edge, comprising: The synchronization field rule determination module is used to determine the synchronization field rules based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; The primary key range determination module is used to read the data of the business table to be verified in the edge terminal in pages according to the preset page size, and determine the current primary key range corresponding to the current page based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page. The data to be verified module is used to determine the data to be verified at the edge end corresponding to the edge end based on the table name, the current primary key range and the synchronization field rules, and to obtain the data to be verified at the center end returned by the center end after searching based on the table name, the current primary key range and the synchronization field rules. The difference determination module is used to obtain edge-end verification standard data and center-end verification standard data after processing the edge-end verification data and the center-end verification data respectively based on normalization rules, and to perform consistency verification on the edge-end verification standard data and the center-end verification standard data to obtain the difference rows in the current primary key interval; The loop traversal module is used to continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification, until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

[0013] The present invention also provides a data consistency verification device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data consistency verification method described above.

[0014] The present invention also provides a readable storage medium (i.e., a computer-readable storage medium) on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-described data consistency verification method.

[0015] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method.

[0016] As can be seen, this invention determines synchronization field rules based on the synchronization table metadata of the business table to be verified. The synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules. The synchronization field rules determine the fields that need to be verified for consistency. The primary key values ​​corresponding to the primary key fields have an ordered relationship. Data from the business table to be verified in the edge terminal is read in pages according to a preset page size, and the current primary key range corresponding to the current page is determined based on the primary key value of the last record in the current page and the primary key value of the last record in the previous page. Based on the table name, the current primary key range, and the synchronization field rules, the edge terminal data to be verified is determined. The process involves: obtaining the central data to be verified returned by the central end after searching based on the table name, current primary key range, and synchronization field rules; obtaining the edge-end and central-end standardized data to be verified after processing the edge-end and central-end data to be verified based on normalization rules, and performing consistency verification on the edge-end and central-end standardized data to be verified to obtain the difference rows within the current primary key range; continuing to read the next page and executing the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, resulting in the set of difference rows corresponding to the business table to be verified.

[0017] The beneficial effects of this invention are as follows: Compared with the current full-scale comparison, this invention determines the synchronization field rules based on the synchronized table metadata, forms primary key ranges according to the primary key, and extracts the data to be verified from the records at both ends based on the primary key range and the synchronization field rules, converts it into standardized data to be verified, and then performs the comparison. Because the synchronization field rules can accurately determine the data content that needs consistency verification, and the primary key range limits the range retrieved each time, the same normalization process ensures that the data at both ends forms a unified comparison form. Therefore, there is no comparison of asynchronous fields, and false alarms caused by asynchronous fields are reduced, thereby improving the accuracy and efficiency of data consistency comparison.

[0018] In addition, the present invention also provides a data consistency verification device, equipment and readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A flowchart of a data consistency verification method provided in an embodiment of the present invention; Figure 2 This is a structural framework diagram of a data consistency verification method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a data consistency verification device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a data consistency verification device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please refer to Figure 1 , Figure 1 A flowchart illustrating a data consistency verification method provided in an embodiment of the present invention. The method may include: S101, determine the synchronization field rules based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are the rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key fields have an order relationship.

[0023] The execution entity in this embodiment is the edge terminal. The business table to be verified in this embodiment is a business table that has corresponding data copies on both the central and edge terminals, and whose cloud-edge synchronization results need to be compared for consistency. Verification can be performed on one business table at a time; when multiple business tables exist, the edge terminal can execute the method provided in this embodiment on each business table separately. The synchronization table metadata in this embodiment describes the synchronization attributes of the business table to be verified. The table name is used to ensure that the edge and central terminals locate the same business table; the primary key field is used to uniquely identify each piece of data in the business table; the cloud-to-edge synchronization field rules describe the fields synchronized from the central terminal to the edge terminal; and the edge-to-cloud synchronization field rules describe the fields synchronized from the edge terminal to the central terminal. The primary key value corresponding to the primary key field is a numerical value with an ordered relationship; subsequent data is read according to the order of the primary key values ​​to form the primary key range. This embodiment does not limit the specific method of obtaining the synchronization table metadata; it can be obtained by parsing the synchronization table annotation corresponding to the business table to be verified; or it can be obtained from the synchronization configuration maintained by the cloud-edge synchronization framework. The synchronization table annotation or synchronization configuration can record the synchronization direction and the synchronization fields corresponding to each synchronization direction. By reusing the synchronization configuration used in the cloud-edge synchronization process to determine the consistency verification fields, the verification object can be kept consistent with the actual synchronization object.

[0024] It should be further explained that, based on any of the above embodiments, the determination of synchronization field rules based on the synchronization table metadata of the business table to be verified may include: determining a cloud-to-edge synchronization field set based on cloud-to-edge synchronization field rules; determining an edge-to-cloud synchronization field set based on edge-to-cloud synchronization field rules; determining the range of fields requiring consistency verification by taking the union of the cloud-to-edge synchronization field set and the edge-to-cloud synchronization field set, and forming synchronization field rules based on the field range. In this embodiment, since the business table to be verified can be synchronized bidirectionally, using only the synchronization field set of one direction may miss the fields actually synchronized in the other direction. This embodiment takes the union of the synchronization field sets of the two synchronization directions, ensuring that all fields that need to be consistent in either synchronization direction are included in subsequent verification, thereby improving the completeness of bidirectional synchronization data verification.

[0025] It should be further explained that, based on any of the above embodiments, determining the cloud-to-edge synchronization field set according to the cloud-to-edge synchronization field rules can include: when the cloud-to-edge synchronization field rule is a field inclusion set, the fields in the field inclusion set are determined as the cloud-to-edge synchronization field set; when the cloud-to-edge synchronization field rule is a field exclusion set, the fields in the field exclusion set are excluded from the business fields that need to be synchronized in the business table to be verified, thus obtaining the cloud-to-edge synchronization field set. It should be noted that the edge-to-cloud synchronization field set can also be determined in the same way. For example, when the field inclusion set is includeFields={"status", "useTime"}, the status field and the useTime field can be determined as the synchronization fields for that synchronization direction. When the field exclusion set is excludeFields={"remark"}, the remark field can be deleted from the business fields that need to be synchronized, and the remaining fields are determined as the synchronization fields for the corresponding synchronization direction. The field inclusion set is suitable for business tables with fewer synchronized fields, while the field exclusion set is suitable for business tables with only a few fields that do not participate in synchronization. It should be noted that if the business table to be verified does not have specific synchronization field rules configured, it means that all business data in the business table that belongs to the synchronization object needs to be synchronized. The edge can use business fields belonging to the synchronization object as fields that need to be verified for consistency, and exclude control fields that exist only on one end and do not belong to the synchronized business data.

[0026] S102, read the data of the business table to be verified in the edge terminal in pages according to the preset page size, and determine the current primary key range corresponding to the current page based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page.

[0027] This embodiment reads edge data sequentially according to the primary key values ​​corresponding to the primary key fields. A preset page size is used to limit the number of data records read by the edge at each time; for example, it can be one hundred, five hundred, or other numbers preset based on the available resources at the edge. This embodiment does not limit the method of determining the preset page size; it can be a fixed value shared by all business tables, or it can be a value configured separately for different business tables. In this embodiment, paginated reading is used to obtain the boundaries of the primary key interval from the edge data. Each time the edge reads a page of data, the primary key value of the last record on the current page is determined as the upper bound of the current primary key interval, and the primary key value of the last record on the previous page is determined as the lower bound of the current primary key interval. Therefore, the current primary key interval can be represented as: lowLimitId < primaryKey ≤ upLimitId; where lowLimitId represents the lower bound of the current primary key interval, upLimitId represents the upper bound of the current primary key interval, and primaryKey represents the primary key value of any data in the business table to be verified. This primary key interval includes data whose primary key value is greater than the lower bound and less than or equal to the upper bound. Because the lower bound uses an open interval and the upper bound uses a closed interval, the primary key intervals corresponding to adjacent pages can be connected continuously without repeatedly containing boundary data.

[0028] It should be further explained that, based on any of the above embodiments, the process of reading data from the business table to be verified in the edge terminal in pages according to a preset page size, and determining the current primary key range corresponding to the current page based on the primary key value of the last record of the current page and the primary key value of the last record of the previous page, may include: S1021: Read the data of the business table to be verified in pages according to the primary key value order corresponding to the primary key field; S1022, When the current page is the first page, the preset initial primary key value is determined as the lower bound of the current primary key range; S1023, when the current page is not the first page, the primary key value of the last data in the previous page is determined as the lower bound of the current primary key range; S1024, determine the primary key value of the last data in the current page as the upper bound of the current primary key range, and determine the current primary key range as data whose primary key value is greater than the lower bound and less than or equal to the upper bound.

[0029] In this embodiment, for the first page, a preset initial primary key value can be determined as the lower bound of the current primary key interval. The preset initial primary key value can be less than the minimum valid primary key value in the business table to be verified; or the primary key interval corresponding to the first page can be set to an interval without an actual lower bound. For pages other than the first page, the primary key value of the last record on the previous page is both the upper bound of the previous primary key interval and the lower bound of the current primary key interval. For example, the data read by the edge end according to the primary key value order is 10, 20, 50, 80, 90, and 120, and the preset page size is three records. The data on the first page is 10, 20, and 50, the upper bound of the first primary key interval is 50, and the lower bound can be set to 10; the data on the second page is 80, 90, and 120, the lower bound of the second primary key interval is 50, and the upper bound is 120. In this embodiment, the central end queries data according to the primary key interval determined by the edge end, thereby avoiding pagination misalignment caused by different numbers of data records at both ends. After all pages at the edge have been read, a lower bounded tail primary key range can be set to detect data in the central end whose primary key value is greater than the maximum primary key value at the edge. This tail primary key range can identify tail data that exists in the central end but not at the edge, preventing the omission of extra data in the central end when only verifying based on the existing page boundaries at the edge.

[0030] S103, determine the edge terminal data to be verified corresponding to the edge terminal based on the table name, the current primary key range and the synchronization field rules, and obtain the central terminal data to be verified returned by the central terminal after searching based on the table name, the current primary key range and the synchronization field rules.

[0031] In this embodiment, the edge endpoint, based on synchronization field rules, retains the fields requiring consistency verification from the current page data to obtain the edge endpoint's verification data. Since the current page data is used to form the current primary key range, the edge endpoint's verification data corresponds to the current primary key range. The edge endpoint also sends the table name, the current primary key range, and the synchronization field rules to the central endpoint, enabling the central endpoint to find data within the same business table and the same primary key range, and form the central endpoint's verification data according to the same synchronization field rules.

[0032] It should be further explained that, based on any of the above embodiments, the aforementioned synchronized table metadata may further include logical deletion rules; determining the edge-end data to be verified corresponding to the edge end based on the table name, the current primary key range, and the synchronization field rules, and obtaining the central-end data to be verified returned by the central end after searching based on the table name, the current primary key range, and the synchronization field rules, may include: obtaining the central-end data to be verified determined by the central end based on the table name, the current primary key range, the synchronization field rules, and the logical deletion rules; wherein, the process of the central end determining the central-end data to be verified includes: deleting the operation time field from the initial central-end data to be verified based on the initial central-end data to be verified determined by the current primary key range and the synchronization field rules, and excluding data that hit the logical deletion rules, to obtain the central-end data to be verified.

[0033] The logical deletion rules in this embodiment may include a logical deletion field and a logical deletion value indicating that the data has been logically deleted. The central end can retain the deleted data using logical deletion, while the edge end can remove the corresponding data using physical deletion. If the central end directly returns logically deleted data to the edge end, this data may be mistakenly considered as data existing independently at the central end. Therefore, the central end excludes data that matches the logical deletion rules when determining the data to be verified at the central end. For example, the logical deletion field is deleteFlag, and the logical deletion value is 1. When the deleteFlag field value of the central end data is 1, the central end will not include this data in the central end's data to be verified. This embodiment does not limit the data type of the logical deletion value; it can be a numeric status value or a string value representing the deletion status, as long as the validity of the data can be determined according to the logical deletion rules. The central end data may also include an operation time field formed when the data is written to the central end database. The operation time field is used to record the write time of the data on the central end database side, but this field is not included in the edge end as a synchronization business field. The central end deletes the operation time field from the initial central end data to be verified, so that the central end data to be verified and the edge end data to be verified have the same field range. By deleting this field before forming the data to be verified at the central end, the data to be verified at the central end and the edge end have the same field range, thereby preventing false alarms caused by data write times that only exist at the central end. A process for the central end to determine the data to be verified at the central end can be represented as follows: Query by primaryKey range; for each record: checkData = filterCheckData(record, syncFieldRule, logicDeleteRule); if checkData != null: returnData.put(primaryKey, checkData); return returnData; Among them, Query by primaryKey range means to search for data in the central end based on the current primary key range; filterCheckData means to retain the field according to the synchronization field rules, delete the operation time field and perform logical deletion filtering; when the processing result is not empty, the corresponding data will be returned as the data to be verified in the central end.

[0034] It should be noted that if the data retrieval at the central end corresponding to a certain primary key range fails, the edge end can terminate the consistency check of the business table to be checked in this round and output a failure message to avoid using incomplete segment results as the full table check result. The consistency check of the business table to be checked can then be re-initiated. This embodiment does not limit the re-checking method after failure; a full table consistency check of the business table to be checked can be re-initiated; or, while retaining the completed primary key range processing results, data retrieval and consistency check can be re-executed from the current primary key range where the retrieval failed. The failure message is used to indicate that the check result of this round is incomplete, thereby preventing the output of table-level consistency check results that lose their accuracy.

[0035] S104: Obtain the edge-end and center-end standardized data to be verified after processing the edge-end and center-end data to be verified based on the normalization rules, and perform consistency verification on the edge-end and center-end standardized data to be verified to obtain the difference rows in the current primary key range.

[0036] The normalization rules in this embodiment are used to ensure that edge-end and center-end data to be verified with the same field content form the same data representation. This embodiment does not limit the specific normalization rules; for example, this embodiment may fix the field order and retain null values ​​during normalization; or this embodiment may determine the normalized expression and order of the fields.

[0037] It should be further noted that, based on any of the above embodiments, obtaining the edge-end verification standard data and the center-end verification standard data obtained after processing the edge-end verification data and the center-end verification data according to the normalization rules respectively can include: S1041 Sort the fields in the data to be verified at the edge end and the data to be verified at the center end according to the field name, and retain the null value fields in the data to be verified at the edge end and the data to be verified at the center end. S1042, convert the sorted edge-end data to be verified and center-end data to be verified into the specified string format respectively to obtain edge-end data to be verified and center-end data to be verified.

[0038] In this embodiment, the edge end sorts the fields in the data to be verified at both ends according to the field names, retaining null fields. The sorted data is then converted into a specified string format to obtain the edge-end and center-end data to be verified. This embodiment does not limit the specific type of the specified string format; it can be a JSON (Lightweight Data Interchange) string or a key-value string formed by concatenating field names in order. In one implementation, using a JSON string facilitates the center end's return of data via an interface and allows the edge end to retain differing data. It is understood that if the field order is not standardized, the same set of fields and values ​​may result in different strings depending on the field output order; similarly, omitting null fields on one end and retaining them on the other may also result in different strings. By sorting by field names and uniformly retaining null values, the same fields and values ​​form the same data representation, thereby reducing misjudgments caused by different data representation formats.

[0039] It should be further explained that, based on any of the above embodiments, performing consistency verification on the edge-end and center-end data to be verified to obtain the difference rows within the current primary key range may include: generating edge-end data mappings with primary key values ​​as keys and corresponding edge-end data to be verified as values, and center-end data mappings with primary key values ​​as keys and corresponding center-end data to be verified as values; performing consistency verification based on the edge-end and center-end data mappings, identifying data where no identical primary key values ​​exist at one end or where the data to be verified corresponding to the same primary key value are different as difference rows; determining the difference fields in the difference rows; correspondingly, continuing to read the next page and executing the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, obtaining the set of difference rows corresponding to the business table to be verified, may include: after completing the consistency verification of all primary key ranges of the business table to be verified, outputting the table-level consistency verification result including the table name, the set of difference rows, and the set of difference fields.

[0040] In this embodiment, the primary key value is used to locate the same business data at both ends, and the data to be verified is used to determine whether the content of the fields requiring consistency verification in the corresponding business data is the same. In this embodiment, the edge end can first traverse the central end data mapping and search for the corresponding data in the edge end data mapping based on the primary key value in the central end data mapping. If there is no identical primary key value in the edge end data mapping, the central end data is identified as a missing difference row in the edge end; if there is an identical primary key value but the corresponding data to be verified is different, the corresponding data is identified as a difference row with different content. After completing the traversal of the central end data mapping, the edge end then traverses the unmatched data in the edge end data mapping, and identifies the data in the central end data mapping that does not have the same primary key value as a missing difference row in the central end. This embodiment does not limit the traversal order of the two data mappings; it can traverse the central end data mapping first and then traverse the unmatched data in the edge end data mapping; or it can traverse the edge end data mapping first and then traverse the unmatched data in the central end data mapping. As long as it can identify cases where the data exists only in the central end, exists only in the edge end, and the data to be verified corresponding to the same primary key value is different. For each discrepancy row, the edge endpoint retains the corresponding edge endpoint and central endpoint data to be verified, resulting in discrepancy data for subsequent field-level analysis and manual verification. When discrepancies exist, the edge endpoint can extract the set of field names from the first discrepancy data row to generate a discrepancy field set. Since the discrepancy data originates from data to be verified after being filtered according to synchronization field rules, the discrepancy field set is limited to the fields requiring consistency verification and will not include asynchronous fields. One edge endpoint current page processing procedure can be represented as: while hasNextPage: pageData = selectLocalPage(); upLimitId = lastPrimaryKey(pageData); centerData = pullCheckData(tableName, lowLimitId, upLimitId,syncFieldRule); edgeMap = buildDataMap(pageData); centerMap = buildDataMap(centerData); compare(centerMap, edgeMap); lowLimitId = upLimitId; In this implementation, `selectLocalPage` reads the current page data according to a preset page size; `lastPrimaryKey` retrieves the primary key value of the last record on the current page; `pullCheckData` retrieves the data to be verified within the corresponding primary key range at the central endpoint; `buildDataMap` generates a data mapping with the primary key value as the key and the data to be verified as the value; and `compare` performs a consistency check on the two data mappings. This embodiment improves the efficiency and accuracy of comparison through data mapping.

[0041] S105, continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification, until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

[0042] In this embodiment, after the edge terminal completes the consistency check of the current primary key range, it uses the primary key value of the last record on the current page as the lower bound of the next primary key range and advances the pagination cursor to read the data on the next page. The edge terminal then redefines the primary key range, the data to be checked at the edge terminal, and the data to be checked at the center terminal for the next page, and performs normalization processing and consistency checks. After the consistency checks of the primary key ranges corresponding to all pages are completed, the edge terminal summarizes the differing rows within each primary key range to obtain a set of differing rows, and can also determine a set of differing fields, outputting a table-level consistency check result including the table name, the set of differing rows, and the set of differing fields.

[0043] This invention provides a data consistency verification method, which may include: S101, determining synchronization field rules based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; S102, reading the data of the business table to be verified in the edge terminal in pages according to a preset page size, and determining the current primary key range corresponding to the current page based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page; S103, determining the edge terminal based on the table name, the current primary key range, and the synchronization field rules. The process involves: S104, obtaining the edge-end data to be verified and the central-end data to be verified returned after searching based on the table name, current primary key range, and synchronization field rules; S105, obtaining the edge-end standardized data to be verified and the central-end standardized data to be verified after processing the edge-end data to be verified and the central-end data to be verified based on normalization rules, and performing consistency verification on the edge-end standardized data to be verified and the central-end standardized data to be verified to obtain the difference rows within the current primary key range; S106, continuing to read the next page and executing the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, and obtaining the set of difference rows corresponding to the business table to be verified. Compared with the current full comparison, this invention determines the synchronization field rules based on the synchronized table metadata, forms a primary key range according to the primary key, extracts the data to be verified from the records at both ends based on the primary key range and synchronization field rules, converts it into standardized data to be verified, and then compares them. Since the synchronization field rules can accurately determine the data content that needs to be verified for consistency, the primary key range limits the range of each retrieval, and the same normalization process makes the data at both ends form a unified comparison form, there is no comparison of asynchronous fields, and false alarms caused by asynchronous fields are reduced, thereby improving the accuracy and efficiency of data consistency comparison.

[0044] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A structural framework diagram of a data consistency verification method provided in an embodiment of the present invention may specifically include: (1) Construct synchronization table metadata. Establish synchronization table metadata for business tables participating in cloud-edge synchronization. The metadata includes at least the table name, primary key field, cloud-to-edge synchronization field rules, edge-to-cloud synchronization field rules, and logical deletion rules. Synchronization field rules can be composed of a set of included fields or a set of excluded fields, used to describe which fields in the table belong to the actual synchronization fields.

[0045] (2) Generate data to be verified. At the start of verification, the system does not directly compare the original business objects, but first converts the business objects into comparable data to be verified based on the synchronization field rules determined by the synchronization table metadata. The data to be verified must have at least the deletion operation time field; if the current record hits the logical deletion field and logical deletion, the record will be directly blocked and will not be included in subsequent verification. If synchronization field rules are configured in both the cloud-to-edge and edge-to-cloud directions, the synchronization field sets in both directions will be calculated separately, and the union of the sets will be used as the synchronization field mirror range of this table; if no synchronization field rules are configured, the business fields after removing the interference of operation time and logical deletion will be used as the data to be verified by default.

[0046] (3) Segmentation based on primary key range. When the edge queries the data of the table to be verified locally, it retrieves records in pages according to the preset page size, and uses the primary key value of the last record of each page as the upper limit of the current segment. The system maintains the lower and upper limits of the current segment to form a primary key range where "the primary key is greater than the lower limit and less than or equal to the upper limit". The last segment can retain only the lower limit without setting the upper limit, which is used to cover the remaining records.

[0047] (4) Retrieve the central data mapping with the same primary key range from the remote end. The edge end sends the table name, lower bound of the primary key range, and upper bound of the primary key range to the central end. The central end queries the records of the central end based on the same table metadata, the same primary key range, and the same synchronization field mirroring rules, and generates a data mapping with the primary key as the key and the data to be verified and normalized in the central end as the value, and returns it to the edge end. When normalizing the data to be verified in the central end, the field names are sorted and null values ​​are retained to ensure that the two ends can be compared in a unified format.

[0048] (5) Generate edge data mappings for the same primary key range at the edge. The edge applies the same synchronization field rules as the central end to the local records in the current page, generating a data mapping with the primary key as the key and the edge segment's data to be verified as the value. Since both the cloud and edge use the same synchronization field rules, local extended fields, asynchronous fields, and logically deleted records will not be included in the final comparison set.

[0049] (6) Perform segmented comparison and row-by-row positioning. The edge end first traverses the data mapping returned by the center end, checking item by item whether there is the same primary key and the same data to be verified in the edge end. If the edge end does not have the primary key, or the data to be verified is different, it is recorded as a difference row. Then, the remaining unmatched data mappings in the edge end are traversed, and the primary keys that do not exist in the center end but exist in the edge end are recorded as difference rows. For each difference row, the mirrored data in the center end and the mirrored data in the edge end are retained for subsequent field-level analysis and manual confirmation.

[0050] (7) Extract the set of difference fields. If a difference row is found, the system extracts the set of field names from the first difference mirror data to generate the set of difference fields for the current verification result. This set of fields is naturally limited to the scope of the synchronous field mirror, so asynchronous fields will not be output as difference fields.

[0051] (8) Complete full table consistency check. The edge segment continues to advance the lower bound and pagination cursor according to the primary key segment order until the last segment comparison is completed. After all primary key ranges in the entire table have been processed, the table-level consistency check result is output, including the set of differing rows, the set of differing fields, and the check table name. If fetching remote data for a certain segment fails, the current round of check is terminated and a failure message is returned for subsequent retries.

[0052] For ease of understanding, in Example 1, the edge terminal initiates a table-level consistency check. Upon receiving a consistency check request for the business table to be checked, the edge terminal first reads the synchronized table metadata to obtain the cloud-to-edge field rules, edge-to-cloud field rules, primary key fields, and logical deletion rules. Subsequently, the edge terminal reads local records page by page and generates the upper bound of the current primary key range based on the primary key value of the last record on each page. For each primary key range, the edge terminal calls the central terminal's verification interface, requesting the central terminal to return the data to be checked within the same primary key range.

[0053] In Example 2, the central endpoint generates a synchronized field mirror. Upon receiving a primary key segmentation request, the central endpoint queries the business table records based on the same primary key range. For each record, the central endpoint first deletes the operation time field, then generates a synchronized field mirror according to the synchronized field rules. If a record matches the logical deletion rule, it is skipped; otherwise, the primary key and mirror string are written into the returned mapping. Because the field order is fixed and null values ​​are retained during normalization, the central endpoint can provide stable, unverified data to the edge endpoints.

[0054] Example 3: Bidirectional difference comparison is performed at the edge. The edge applies the same synchronization field rules as the central end to the local paginated records to obtain edge-mapped data. Subsequently, the edge first searches for records with the same primary key and the same verification specification data based on the central end mapping, and then processes the remaining unmatched records locally. If a primary key exists at one end but not at the other, or if the verification specification data of records with the same primary key is different, it is determined to be a difference row, and the verification specification data at both the cloud and edge ends is retained.

[0055] Example 4: Output of the Difference Field Set. If a business table contains differing rows, the system extracts the field set from the first difference mirror and uses it as the difference field set output for this verification. Since the difference field set comes from the synchronous field mirror, the output result is naturally limited to the scope of synchronous semantics. Users can directly use this to determine whether the actual synchronous fields in the cloud and edge are consistent, without having to manually exclude local extended fields and asynchronous fields.

[0056] Through the above implementation methods, the present invention can transform the comparison of the original records of the entire table into a synchronous field mirror comparison in cloud-edge consistency verification, and reduce the overhead of remote verification by combining primary key segmentation control, thereby improving the efficiency and accuracy of consistency verification of large-scale cloud-edge tables.

[0057] The following describes a data consistency verification device provided by an embodiment of the present invention. The data consistency verification device described below can be referred to in correspondence with the data consistency verification method described above.

[0058] Please refer to the details. Figure 3 , Figure 3 A schematic diagram of a data consistency verification device provided in an embodiment of the present invention may include: The synchronization field rule determination module 100 is used to determine the synchronization field rules based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; The primary key range determination module 200 is used to read the data of the business table to be verified in the edge terminal in pages according to the preset page size, and determine the current primary key range corresponding to the current page based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page. The data to be verified module 300 is used to determine the data to be verified at the edge end corresponding to the edge end based on the table name, the current primary key range and the synchronization field rules, and to obtain the data to be verified at the center end returned by the center end after searching based on the table name, the current primary key range and the synchronization field rules; The difference determination module 400 is used to obtain edge-end verification standard data and center-end verification standard data after processing the edge-end verification data and the center-end verification data respectively based on normalization rules, and to perform consistency verification on the edge-end verification standard data and the center-end verification standard data to obtain the difference rows in the current primary key interval; The loop traversal module 500 is used to continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

[0059] Furthermore, based on any of the above embodiments, the synchronization field rule determination module 100 may include: The cloud-to-edge synchronization field set determination unit is used to determine the cloud-to-edge synchronization field set according to the cloud-to-edge synchronization field rules. The edge-to-cloud synchronization field set determination unit is used to determine the edge-to-cloud synchronization field set according to the edge-to-cloud synchronization field rules. The synchronization field rule determination unit is used to determine the range of fields that need to be verified for consistency by the union of the cloud-to-edge synchronization field set and the edge-to-cloud synchronization field set, and to form the synchronization field rule according to the range of fields.

[0060] Furthermore, based on any of the above embodiments, the cloud-to-edge synchronization field set determination unit may include: The first cloud-to-edge synchronization field set determination subunit is used to determine the fields in the field-containing set as the cloud-to-edge synchronization field set in the cloud-to-edge synchronization direction when the cloud-to-edge synchronization field rule is a field-containing set. The second cloud-to-edge synchronization field set determination subunit is used to exclude fields in the field exclusion set from the business fields that need to be synchronized in the business table to be verified when the cloud-to-edge synchronization field rule is a field exclusion set, so as to obtain the cloud-to-edge synchronization field set.

[0061] Furthermore, based on any of the above embodiments, the synchronized table metadata further includes logical deletion rules; the aforementioned data to be verified determination module 300 may include: A central-end data to be verified determination unit is used to obtain the central-end data to be verified determined by the central end based on the table name, the current primary key range, the synchronization field rule, and the logical deletion rule; wherein, the process of the central end determining the central-end data to be verified includes: based on the initial central-end data to be verified determined by the current primary key range and the synchronization field rule, deleting the operation time field from the initial central-end data to be verified, and excluding data that matches the logical deletion rule to obtain the central-end data to be verified.

[0062] Furthermore, based on any of the above embodiments, the primary key interval determination module 200 may include: The pagination reading unit is used to read the data of the business table to be verified in pages according to the primary key value corresponding to the primary key field. The first lower bound determination unit is used to determine the preset initial primary key value as the lower bound of the current primary key range when the current page is the first page; The second lower bound determination unit is used to determine the primary key value of the last data in the previous page as the lower bound of the current primary key interval when the current page is not the first page. The primary key interval determination unit is used to determine the primary key value of the last data record on the current page as the upper bound of the current primary key interval, and to determine the current primary key interval as data whose primary key value is greater than the lower bound and less than or equal to the upper bound.

[0063] Furthermore, based on any of the above embodiments, the difference determination module 400 may include: The normalization processing unit is used to sort the fields in the edge-end data to be verified and the center-end data to be verified according to the field name, and retain the null value fields in the edge-end data to be verified and the center-end data to be verified. The conversion unit is used to convert the sorted edge-end data to be verified and the center-end data to be verified into a set string format, respectively, to obtain the edge-end data to be verified and the center-end data to be verified.

[0064] Furthermore, based on any of the above embodiments, the difference determination module 400 may include: The data mapping determination unit is used to generate edge data mappings with primary key values ​​as keys and corresponding edge data to be verified as values, and center data mappings with primary key values ​​as keys and corresponding center data to be verified as values. The difference row determination unit is used to perform consistency verification based on the edge end data mapping and the center end data mapping, and to determine the data that does not have the same primary key value on one end or the data to be verified that are different according to the same primary key value as the difference row. A difference field determination unit is used to determine the difference fields in the difference row; Correspondingly, the loop traversal module 500 may include: The table-level consistency verification result determination unit is used to output the table-level consistency verification result, including the table name, the set of differing rows, and the set of differing fields, after completing the consistency verification of all primary key ranges of the business table to be verified.

[0065] It should be noted that the order of the modules and units in the above-mentioned data consistency verification device can be changed without affecting the logic.

[0066] This invention provides a data consistency verification device, which may include: a synchronization field rule determination module 100, used to determine synchronization field rules based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; The primary key range determination module 200 is used to read data from the business table to be verified in the edge terminal in pages according to a preset page size, and determine the current primary key range corresponding to the current page based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page; the data to be verified determination module 300 is used to determine the edge terminal data to be verified corresponding to the edge terminal based on the table name, the current primary key range, and the synchronization field rules, and to obtain the central terminal data to be verified returned by the central terminal after searching based on the table name, the current primary key range, and the synchronization field rules; the difference determination module 400 is used to obtain the base The edge-end and center-end data to be verified are processed according to normalization rules to obtain edge-end and center-end normalized data to be verified, respectively. Consistency verification is then performed on these two normalized data to obtain the difference rows within the current primary key range. A loop traversal module 500 continues to read the next page and executes the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, resulting in a set of difference rows corresponding to the business table to be verified. Compared with the current full-scale comparison, this invention determines the synchronization field rules based on the synchronized table metadata, forms primary key ranges according to the primary key, and extracts the data to be verified from the records at both ends based on the primary key range and the synchronization field rules. This data is then converted into normalized data to be verified and compared. Because the synchronization field rules can accurately determine the data content requiring consistency verification, and the primary key range limits the range obtained each time, the same normalization processing makes the data at both ends form a unified comparison form, thus eliminating the comparison of asynchronous fields and reducing false alarms caused by asynchronous fields, thereby improving the accuracy and efficiency of data consistency comparison.

[0067] The following describes a data consistency verification device provided by an embodiment of the present invention. The data consistency verification device described below and the data consistency verification method described above can be referred to in correspondence.

[0068] Please refer to Figure 4 , Figure 4 A schematic diagram of a data consistency verification device provided in an embodiment of the present invention may include: Memory 10 is used to store computer programs; Processor 20 is used to execute computer programs to implement the data consistency verification method described above.

[0069] Communication interface 30 is used to communicate with the central terminal or other devices.

[0070] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0071] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions: The synchronization field rules are determined based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are the rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key fields have an order relationship; The data of the business table to be verified in the edge terminal is read in pages according to the preset page size, and the current primary key range corresponding to the current page is determined based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page. The edge terminal data to be verified is determined based on the table name, the current primary key range, and the synchronization field rules. The center terminal data to be verified is also obtained after searching based on the table name, the current primary key range, and the synchronization field rules. Obtain the edge-end and center-end standardized data to be verified after processing the edge-end and center-end data to be verified based on normalization rules, and perform consistency verification on the edge-end and center-end standardized data to be verified to obtain the difference rows in the current primary key range; Continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

[0072] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0073] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0074] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0075] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0076] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the data consistency verification device in the embodiments of the present invention. In practical applications, the data consistency verification device may include more than Figure 4 Show more or fewer parts, or combine certain parts.

[0077] The readable storage medium provided in the embodiments of the present invention is described below. The readable storage medium described below and the data consistency verification method described above can be referred to each other.

[0078] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data consistency verification method.

[0079] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0081] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0082] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0083] The above provides a detailed description of the data consistency verification method, apparatus, device, and readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data consistency verification method, characterized in that, Applied to the edge, including: The synchronization field rules are determined based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; The data of the business table to be verified in the edge terminal is read in pages according to the preset page size, and the current primary key range corresponding to the current page is determined based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page. Based on the table name, the current primary key range, and the synchronization field rules, determine the edge terminal data to be verified corresponding to the edge terminal, and obtain the center terminal data to be verified returned by the center terminal after searching based on the table name, the current primary key range, and the synchronization field rules; Obtain edge-end and center-end standardized data to be verified after processing the edge-end and center-end data to be verified based on normalization rules, and perform consistency verification on the edge-end and center-end standardized data to be verified to obtain the difference rows in the current primary key interval; Continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

2. The data consistency verification method according to claim 1, characterized in that, The synchronization field rules are determined based on the synchronization table metadata of the business table to be verified, including: The set of cloud-to-edge synchronization fields is determined according to the cloud-to-edge synchronization field rules. The set of edge-to-cloud synchronization fields is determined according to the edge-to-cloud synchronization field rules. The union of the cloud-to-edge synchronization field set and the edge-to-cloud synchronization field set is determined as the range of fields that need to be verified for consistency, and the synchronization field rules are formed based on the range of fields.

3. The data consistency verification method according to claim 2, characterized in that, The set of cloud-to-edge synchronization fields is determined based on the aforementioned cloud-to-edge synchronization field rules, including: When the cloud-to-edge synchronization field rule is a field inclusion set, the fields in the field inclusion set are determined as the cloud-to-edge synchronization field set in the cloud-to-edge synchronization direction; When the cloud-to-edge synchronization field rule is a field exclusion set, the fields in the field exclusion set are excluded from the business fields that need to be synchronized in the business table to be verified, thus obtaining the cloud-to-edge synchronization field set.

4. The data consistency verification method according to any one of claims 1 to 3, characterized in that, The synchronized table metadata also includes logical deletion rules; determining the edge-end data to be verified corresponding to the edge end based on the table name, the current primary key range, and the synchronization field rules; and obtaining the center-end data to be verified returned by the center end after searching based on the table name, the current primary key range, and the synchronization field rules, including: Obtain the data to be verified at the central end, which is determined by the central end based on the table name, the current primary key range, the synchronization field rules, and the logical deletion rules; The process of determining the data to be verified at the central end includes: based on the current primary key range and the synchronization field rules, the initial data to be verified at the central end is determined, the operation time field is deleted from the initial data to be verified at the central end, and data that matches the logical deletion rules is excluded, so as to obtain the data to be verified at the central end.

5. The data consistency verification method according to claim 1, characterized in that, The data from the business table to be verified in the edge terminal is read in pages according to the preset page size. Based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page, the current primary key range corresponding to the current page is determined, including: The data of the business table to be verified is read in pages according to the primary key values ​​corresponding to the primary key fields; When the current page is the first page, the preset initial primary key value is determined as the lower bound of the current primary key range; When the current page is not the first page, the primary key value of the last data entry on the previous page is determined as the lower bound of the current primary key range; The primary key value of the last record on the current page is determined as the upper bound of the current primary key range, and the current primary key range is determined as data whose primary key value is greater than the lower bound and less than or equal to the upper bound.

6. The data consistency verification method according to claim 1, characterized in that, Obtaining edge-end and center-end standardized data to be verified, respectively, after processing the edge-end and center-end data to be verified based on normalization rules, includes: Sort the fields in the edge-end and center-end data to be verified according to the field name respectively, and retain the null value fields in the edge-end and center-end data to be verified. The sorted edge-end data to be verified and the center-end data to be verified are converted into specified string formats to obtain the edge-end data to be verified and the center-end data to be verified.

7. The data consistency verification method according to claim 1, characterized in that, Perform consistency checks on the edge-end and center-end data to be verified to obtain the difference rows within the current primary key range, including: An edge data mapping is generated with the primary key value as the key and the corresponding edge end verification specification data as the value, and a center end data mapping is generated with the primary key value as the key and the corresponding center end verification specification data as the value. Based on the edge data mapping and the center data mapping, consistency verification is performed, and data that do not have the same primary key value on one end or whose corresponding data to be verified are different are identified as differential rows. Determine the difference fields in the difference rows; Accordingly, the process continues reading the next page and executing the steps of determining the primary key range, determining the data to be verified, and performing consistency verification, until the consistency verification of all pages of the business table to be verified is completed, resulting in the set of differing rows corresponding to the business table to be verified, including: After completing the consistency check of all primary key ranges of the business table to be checked, the table-level consistency check result is output, including the table name, the set of differing rows, and the set of differing fields.

8. A data consistency verification device, characterized in that, Applied to the edge, including: The synchronization field rule determination module is used to determine the synchronization field rules based on the synchronization table metadata of the business table to be verified; wherein, the synchronization table metadata includes the table name, primary key field, cloud-to-edge synchronization field rules, and edge-to-cloud synchronization field rules; the synchronization field rules are rules for determining the fields that need to be verified for consistency; the primary key values ​​corresponding to the primary key field have an order relationship; The primary key range determination module is used to read the data of the business table to be verified in the edge terminal in pages according to the preset page size, and determine the current primary key range corresponding to the current page based on the primary key value of the last data in the current page and the primary key value of the last data in the previous page. The data to be verified module is used to determine the data to be verified at the edge end corresponding to the edge end based on the table name, the current primary key range and the synchronization field rules, and to obtain the data to be verified at the center end returned by the center end after searching based on the table name, the current primary key range and the synchronization field rules. The difference determination module is used to obtain edge-end verification standard data and center-end verification standard data after processing the edge-end verification data and the center-end verification data respectively based on normalization rules, and to perform consistency verification on the edge-end verification standard data and the center-end verification standard data to obtain the difference rows in the current primary key interval; The loop traversal module is used to continue reading the next page and execute the steps of determining the primary key range, determining the data to be verified, and performing consistency verification, until the consistency verification of all pages of the business table to be verified is completed, and the set of difference rows corresponding to the business table to be verified is obtained.

9. A data consistency verification device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data consistency verification method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data consistency verification method as described in any one of claims 1 to 7.

Images