Data consistency detection method and device, computer medium and product

Abstract

The invention discloses a data consistency detection method and device, a computer medium and a product, which are applied to the field of data processing, and the method comprises the following steps: marking each row of data of two data tables, and determining inconsistent data rows of a first target data table and a second target data table according to label values of the two data tables; and according to each field in the inconsistent data row, determining an inconsistent field as a data consistency detection result. Based on an optimistic data consistency thought, the data in the first data table and the second data table are regarded as most data as consistency data. According to the data consistency detection method, the two data tables are distinguished through the labels, consistency line data with large data size is eliminated, inconsistent data lines are reserved, the data range needing to be detected is greatly narrowed, and then the data consistency detection efficiency is improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data consistency detection method, apparatus, computer medium and product. Background Technology

[0002] As business demands increase, the amount of data required also gradually grows. To ensure business execution efficiency, the same batch processing task can be executed in parallel on the same source data in two independent business processing systems, and the processing results can be loaded into their respective lakehouse systems to achieve a dual-load batch processing mode.

[0003] When one business processing system encounters a problem and a switch to another is required, inconsistent results from the two batch processing methods under dual-load mode can severely impact the execution of subsequent business processes. Therefore, data consistency checks are necessary.

[0004] Existing data consistency checks require manual configuration of comparison rules, which are then loaded into the database, resulting in a large workload for users and low efficiency. Summary of the Invention

[0005] This application provides a data consistency detection method, apparatus, device, and computer storage medium, which can improve the efficiency of data consistency detection.

[0006] In a first aspect, embodiments of this application provide a data consistency detection method, the method comprising: Retrieve the first data table and the second data table; Each row of data in the first data table and the second data table is labeled to obtain a first target data table including a first label value and a second target data table including a second label value. Determine the hash value of each row of data in the first target data table and the second target data table; Based on the first tag value of the first target data table, the second tag value of the second target data table, and each hash value, determine the inconsistent data rows between the first target data table and the second target data table; Based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, the inconsistent fields are determined as the data consistency detection results.

[0007] In some possible implementations, based on the first tag value of the first target data table, the second tag value of the second target data table, and each of the hash values, the specific implementation includes: Based on each of the hash values, a consistent data row pair is determined, wherein the consistent data row pair includes row data from the first target data table and row data from the second target data table; Based on the first tag value of the first target data table and the second tag value of the second target data table, mirror and eliminate each pair of consistent data rows; The remaining rows of data in the first target data table and the second target data table are designated as inconsistent data rows.

[0008] In some possible implementations, mirroring and eliminating each pair of consistent data rows based on the first tag value of the first target data table and the second tag value of the second target data table specifically includes: For each consistent data row pair, a preset value is obtained by summing the first tag value and the second tag value of the two rows in the consistent data row pair, so as to mirror and eliminate the two rows in the consistent data row pair.

[0009] In some possible implementations, before tagging each row of data in the first data table and the second data table to obtain a first target data table including a first tag value and a second target data table including a second tag value, the method further includes: Copy the first data table and the second data table to the database of the massively parallel processing architecture; Each row of data in the first data table and the second data table is labeled to obtain a first target data table including a first label value and a second target data table including a second label value, specifically including: In the database, each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

[0010] In some possible implementations, before determining the inconsistent data rows between the first target data table and the second target data table based on the first label value of the first target data table and the second label value of the second target data table, the method further includes: When you have permission to inspect the first data table and the second data table, perform table structure inspection on the first data table and the second data table, and obtain the table structure inspection results. If the table structure detection result indicates that the table structures of the first data table and the second data table are inconsistent, a prompt message is generated to indicate that the table structures of the first data table and the second data table are inconsistent, and is then displayed. Each row of data in the first data table and the second data table is labeled to obtain a first target data table including a first label value and a second target data table including a second label value, specifically including: If the table structure detection result shows that the table structures of the first data table and the second data table are consistent, then each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

[0011] In some possible implementations, table structure detection is performed on the first data table and the second data table to obtain table structure detection results, specifically including: Determine the structural parameters of each field in the first data table and the second data table respectively. The structural parameters include at least one of field name, data type and field length. By comparing the structural parameters in the first data table and the second data table, the table structure detection results are obtained.

[0012] In some possible implementations, the database of the massively parallel processing architecture includes several storage nodes; Before determining the inconsistent fields based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, the method further includes: Obtain the first distribution key of the first target data table and the second distribution key of the second target data table; Based on the first distribution key and the second distribution key, determine the first storage node of the first target data table in the database of the massively parallel processing architecture and the second storage node of the second target data table in the database of the massively parallel processing architecture. When the first storage node is different from the second storage node, the first target data table is migrated to the second storage node or the second target data table is migrated to the first storage node.

[0013] Secondly, embodiments of this application provide a data consistency detection device, the device comprising: The data table acquisition module is used to acquire the first data table and the second data table; The tagging module is used to tag each row of data in the first data table and the second data table respectively, to obtain a first target data table including a first tag value and a second target data table including a second tag value; The hash value determination module is used to determine the hash value of each row of data in the first target data table and the second target data table; The inconsistent data row determination module is used to determine the inconsistent data rows between the first target data table and the second target data table based on the first tag value of the first target data table, the second tag value of the second target data table, and each of the hash values. The data consistency detection result determination module is used to determine the inconsistent fields based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, and use these fields as the data consistency detection result.

[0014] Thirdly, embodiments of this application provide a data consistency detection device, the device comprising: A processor and a memory storing computer program instructions; a data consistency detection method that implements any one of the above when the processor executes the computer program instructions.

[0015] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data consistency detection method described above is implemented.

[0016] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by the processor of an electronic device, enable the electronic device to perform any of the above-mentioned data consistency detection methods.

[0017] The data consistency detection method, apparatus, device, computer storage medium, and computer program product of this application embodiment obtain a first data table and a second data table; label each row of data in the first data table and the second data table respectively to obtain a first target data table including a first label value and a second target data table including a second label value; determine the inconsistent data rows between the first target data table and the second target data table based on the first label value of the first target data table and the second label value of the second target data table; and determine the inconsistent fields based on each field in the inconsistent data rows in the first target data table and each field in the inconsistent data rows in the second target data table, as the data consistency detection result.

[0018] In this embodiment, based on the optimistic data consistency concept, the data in the first and second data tables is considered to be mostly consistent data. The two data tables are distinguished by labels, and consistent rows with large amounts of data are eliminated, while inconsistent rows are retained. This significantly narrows the range of data that needs to be detected, thereby improving the efficiency of data consistency detection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a data consistency detection method provided in one embodiment of this application; Figure 2 A flowchart illustrating another data consistency detection method provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a data consistency detection device provided in another embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data consistency detection device provided in another embodiment of this application. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0023] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0024] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0025] Currently, in dual-load batch processing mode, ensuring data consistency between the two business processing systems is crucial. When a business processing system is modified or needs to be switched, data inconsistency between the two systems can affect subsequent business execution and the user experience. In particular, inconsistent data such as opening batch data, product service data, and online access data will impact services like business opening and customer transactions, leading to a poor user experience. Opening batch data refers to the data generated by a batch of critical batch processing tasks that must be completed before the start of daily business operations. In the financial sector, product service data refers to the basic information and service rules of various financial products provided; in the education sector, it refers to course information, teaching materials, etc., for training courses. It's understandable that product service data can refer to information about the corresponding products in different sectors. Online access data refers to the data that users need to access and update in real time when processing transactions through software applications and other channels.

[0026] Existing data consistency checks require manual completion of comparison rule forms, importing the form data into the database, and finally, a background data comparison program to read the form data, perform the comparison, generate the comparison results, and store them in the database. This manual completion of comparison rule forms is wasteful of human resources and involves a large workload. Furthermore, given the massive scale of lake warehouse data (typically reaching petabyte (PB) levels), the workload for configuring the rule forms is even greater. Here, "lake warehouse" refers to an architecture that integrates a data lake and a data warehouse.

[0027] To address the problems of existing technologies, embodiments of this application provide a data consistency detection method, apparatus, device, computer storage medium, and computer program product. The method involves acquiring a first data table and a second data table; tagging each row of data in both tables to obtain a first target data table including a first tag value and a second target data table including a second tag value; determining inconsistent data rows between the two tables based on the first tag value of the first target data table and the second tag value of the second target data table; and identifying inconsistent fields based on each field in the inconsistent data rows of the first and second target data tables, using these fields as the data consistency detection result.

[0028] In this embodiment, based on the optimistic data consistency concept, the data in the first and second data tables is considered to be mostly consistent data. The two data tables are distinguished by labels, and consistent rows with large amounts of data are eliminated, while inconsistent rows are retained. This significantly narrows the range of data that needs to be detected, thereby improving the efficiency of data consistency detection.

[0029] The execution subject of this application embodiment can be any electronic device that can execute a data consistency detection method, such as a server, server cluster, personal computer, tablet, etc. This application embodiment does not limit this; for ease of subsequent explanation, an electronic device will be used as the execution subject for description.

[0030] The data consistency detection method provided in the embodiments of this application will be introduced first below.

[0031] Figure 1 This diagram illustrates a flow chart of a data consistency detection method provided in one embodiment of this application. Figure 1 As shown, the method may include the following steps: S101: Obtain the first data table and the second data table.

[0032] In this embodiment, the first data table and the second data table are data tables from different business processing systems, both capable of processing the same business. Generally, each field in the first data table and the second data table should be identical, meaning they should satisfy data consistency. However, due to reasons specific to the business processing system or other factors, the first data table and the second data table may not meet the data consistency requirement. To ensure that the business processing system can use the data correctly during subsequent system switching, a data consistency check can be performed on the first data table and the second data table.

[0033] S102: Tag each row of data in the first data table and the second data table respectively to obtain a first target data table including the first tag value and a second target data table including the second tag value.

[0034] The first and second label values ​​can be mirror images of each other. The specific label values ​​can be set as needed, and this embodiment does not impose any restrictions. For example, the first label value can be 1 and the second label value can be -1, or the first label value can be 2 and the second label value can be -2. Mirror images mean that the absolute values ​​of the label values ​​are the same, but their mathematical signs are opposite.

[0035] S103: Determine the hash value of each row of data in the first target data table and the second target data table by using a preset database engine.

[0036] The default database engine can be Greenplum or other database engines.

[0037] By using a pre-defined database engine, the hash value of each row of data in the first and second target data tables is determined. Then, based on the hash value, grouping and aggregation operations are performed on each row of data in the first and second target data tables to obtain consistent data row pairs with the same hash value.

[0038] S104: Based on the first label value of the first target data table and the second label value and hash values ​​of the second target data table, determine the inconsistent data rows between the first target data table and the second target data table.

[0039] In this embodiment of the application, consistent data row pairs are determined based on each hash value. The consistent data row pairs include row data from the first target data table and row data from the second target data table. Based on the first tag value of the first target data table and the second tag value of the second target data table, each consistent data row pair is mirrored and eliminated. The remaining row data in the first target data table and the second target data table are taken as inconsistent data rows.

[0040] Hash values ​​are determined based on the fields in a row of data; therefore, identical hash values ​​indicate that the row data is consistent.

[0041] In other words, rows with the same hash value are grouped together, and two rows in a group are considered consistent rows. Generally, a group contains at least one row and at most two rows. For rows with inconsistent hash values, which typically exist in a single group, a group containing only one row is considered an inconsistent row.

[0042] In this embodiment, because the preset database engine itself has strong performance, the hash value determined by using the preset database engine can avoid hash collisions, ensuring the uniqueness of the hash values ​​of the same row data, and thus ensuring the accuracy of consistent data row pairs determined based on the hash values. Mirroring eliminates the amount of data required for subsequent field consistency checks.

[0043] In some embodiments, when mirroring and eliminating each consistent data row pair based on the first tag value of the first target data table and the second tag value of the second target data table, for each consistent data row pair, a preset value is obtained by summing the first tag value and the second tag value of the two rows in the consistent data row pair, so as to mirror and eliminate the two rows in the consistent data row pair.

[0044] It's important to note that mirroring elimination doesn't involve deleting consistent data rows in the physical data tables. Instead, it logically prevents the database engine from performing subsequent field consistency checks on these consistent data rows. This is because physical deletion requires significant computational resources due to the large number of consistent data pairs. Furthermore, physical deletion necessitates traversing the physical data tables to find consistent data rows before deletion can occur. The physical data tables refer to the first and second target data tables stored in the storage nodes.

[0045] In this embodiment, since both the tag value and hash value participate in the grouping aggregation operation, during mirroring elimination, the first tag value and the second tag value of the consistent data row pair can be directly summed, allowing the preset database engine to logically determine the row data that does not require subsequent field consistency checks. This eliminates the need to traverse and search for consistent data row pairs in the actual physical data table, reducing computational resource consumption.

[0046] S105: Based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, determine the inconsistent fields as the data consistency detection results.

[0047] Specifically, the target data table for the inconsistent row data can be determined by the label value of the inconsistent row data. Then, each field in the inconsistent data row in the first target data table is compared with each field in the inconsistent data row in the second target data table to obtain the inconsistent fields, which serve as the data consistency detection result. Each field in the inconsistent data row can also be understood as a business primary key.

[0048] The data consistency detection method of this application embodiment involves obtaining a first data table and a second data table; tagging each row of data in the first data table and the second data table to obtain a first target data table including a first tag value and a second target data table including a second tag value; determining the inconsistent data rows between the first target data table and the second target data table based on the first tag value of the first target data table and the second tag value of the second target data table; and determining the inconsistent fields based on each field in the inconsistent data rows in the first target data table and each field in the inconsistent data rows in the second target data table, which are used as the data consistency detection result.

[0049] In this embodiment, based on the optimistic data consistency concept, the data in the first and second data tables is considered to be mostly consistent data. The two data tables are distinguished by labels, and consistent rows with large amounts of data are eliminated, while inconsistent rows are retained. This significantly narrows the range of data that needs to be detected, thereby improving the efficiency of data consistency detection.

[0050] Figure 2 A flowchart illustrating another data consistency detection method provided in this application embodiment is shown below. Figure 2 As shown.

[0051] In some embodiments, before executing S102, the electronic device may also copy the first data table and the second data table to a database of a massively parallel processing architecture. The massively parallel processing architecture can be an MPP (Massively Parallel Processing) architecture, or other architectures capable of processing large-scale data in parallel; this embodiment does not limit this. Then, when executing S102, it can be performed in the database of this massively parallel processing architecture. That is, in the database, each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

[0052] In this embodiment, by copying the first and second data tables to a database with a massively parallel processing architecture, data consistency checks can be performed on the first and second data tables using this architecture. Since the database has parallel processing capabilities and can handle large amounts of data, the efficiency of data consistency checks is further improved. In other words, having the first and second data tables in the same database allows the database to leverage distributed computing capabilities, providing highly efficient data consistency checks for massive data comparisons.

[0053] In some embodiments, during the copying process, due to device malfunction or other reasons, the first and second data tables may fail to be successfully copied to the database. Therefore, before executing S102, it can be determined whether the first and second data tables exist in the database based on unique identifiers such as the table names. If they exist, it can be further determined whether permissions or other operations exist. If they do not exist, the data table copying operation is performed again, or a prompt message is generated to inform the user that the first and second data tables do not exist in the current database, and then displayed. Here, the user is a technical personnel responsible for data consistency testing, not a user who uses the data in the data tables to conduct business.

[0054] In some embodiments, since the data table may contain encrypted data that requires certain permissions to access for detection, the system can also determine whether the user has the necessary permissions based on a pre-set list of detection permissions. If so, the table structure can be checked. If not, a message indicating that the user does not currently have detection permissions is generated and displayed.

[0055] In some embodiments, when the user has permission to inspect the first data table and the second data table, the user performs table structure inspection on the first data table and the second data table to obtain the table structure inspection result. If the table structure inspection result indicates that the table structures of the first data table and the second data table are inconsistent, a prompt message is generated to indicate that the table structures of the first data table and the second data table are inconsistent, and the message is displayed. If the table structure detection result shows that the table structure of the first data table and the second data table are consistent, then the first data table and the second data table can be tagged, and then the data consistency detection of the first target data table and the second target data table can be performed.

[0056] In other words, if the table structure detection result shows that the table structure of the first data table and the second data table are consistent, then each row of data in the first data table and the second data table is tagged respectively to obtain a first target data table including the first tag value and a second target data table including the second tag value.

[0057] The reason for performing table structure checks on the first and second data tables is that, for different business processing systems, since they are processing the same business, the field names, data types, and other structural parameters in the generated data tables should be consistent. If they are inconsistent, it indicates that the data in the tables generated by the two business processing systems will also be inconsistent, and further data consistency checks are unnecessary.

[0058] In this embodiment, data consistency detection involves a large amount of computation and requires significant computing resources. Therefore, the table structures of the first and second data tables can be checked first for preliminary exclusion. If the table structures are inconsistent, there is no need to tag the data tables or perform data consistency detection, thereby reducing the waste of computing resources.

[0059] In some embodiments, when detecting the table structure, the structural parameters of each field in the first data table and the second data table can be determined respectively. The structural parameters include at least one of field name, data type and field length. The table structure detection result is obtained by comparing the structural parameters in the first data table and the second data table.

[0060] For example, if the field length of the first column in the first data table is set to 2, and the field length of the first column in the second data table is set to 3, this indicates that the table structures of the first and second data tables are inconsistent. Inconsistent table structures may also result in inconsistent data in the first column. Therefore, this can be used to display a message indicating that the table structures of the first and second data tables are inconsistent.

[0061] In this embodiment of the application, due to the inconsistency of the table structure, the subsequent second round of field consistency detection may not be possible. Therefore, the table structure is checked first, and the data consistency detection is performed on the premise that the table structure is consistent, so as to ensure that the subsequent field consistency detection can be performed normally.

[0062] In some embodiments, before executing S102, the electronic device may further perform deduplication processing on the first data table and the second data table to ensure the uniqueness of each row of data in a data table, and avoid the situation where a group contains more than two rows of data when grouping based on hash values ​​in the subsequent process.

[0063] In some embodiments, since the database of a massively parallel processing architecture includes several storage nodes, and in order to ensure the balance of the amount of stored data on each storage node, each data table can be stored in different storage nodes. Therefore, the first target data table and the second target data table may not be in the same storage node. Subsequently, when performing field consistency comparison, that is, when executing S105, the first distribution key of the first target data table and the second distribution key of the second target data table can be obtained; based on the first distribution key and the second distribution key, the first storage node of the first target data table in the database of the massively parallel processing architecture and the second storage node of the second target data table in the database of the massively parallel processing architecture are determined; when the first storage node and the second storage node are different, the first target data table is migrated to the second storage node or the second target data table is migrated to the first storage node.

[0064] The distribution key can be one or more columns of the data table, and can be set as needed.

[0065] It should be noted that since consistent data has been eliminated during the execution of S104, the actual data migrated during the target data table migration are the inconsistent data rows in the first and second target data tables.

[0066] When migrating data tables, the target data table with a smaller remaining data volume can be determined based on the amount of remaining inconsistent data rows in the first target data table and the amount of remaining inconsistent data rows in the second target data table. The inconsistent data rows of the target data table with a smaller remaining data volume can then be migrated to the storage node of the other target data table.

[0067] In this embodiment of the application, by migrating the inconsistencies between the first target data table and the second target data table to the same storage node, data does not need to be transmitted across nodes during field consistency detection, which is highly efficient and consumes less computing resources.

[0068] Of course, the aforementioned data table migration operation can also be performed during S104. However, since the large amount of consistent data has already been mirrored and eliminated during S104, leaving only a small portion of inconsistent data, the computational resources consumed in data table migration are relatively small between S104 and S105 due to the smaller amount of data to be migrated.

[0069] In this embodiment, since the possibility of data inconsistency is low, whether in technical transformation verification or daily high-availability dual-run, assuming no serious defects in the program and process, the proportion of data with inconsistent fields is low. Based on this optimistic data consistency concept, the comparison of massive amounts of data can be transformed into the comparison of small amounts of data, thereby quickly performing data consistency detection and improving comparison performance and timeliness.

[0070] Based on the data consistency detection method provided in the above embodiments, this application also provides specific implementation methods of the data consistency detection device. Please refer to the following embodiments.

[0071] First see Figure 3 The data consistency detection device 300 provided in this application embodiment includes: The data table acquisition module 301 is used to acquire the first data table and the second data table; The tagging module 302 is used to tag each row of data in the first data table and the second data table respectively, to obtain a first target data table including a first tag value and a second target data table including a second tag value; The hash value determination module 303 is used to determine the hash value of each row of data in the first target data table and the second target data table; The inconsistent data row determination module 304 is used to determine the inconsistent data rows between the first target data table and the second target data table based on the first tag value of the first target data table, the second tag value of the second target data table, and each hash value. The data consistency detection result determination module 305 is used to determine the inconsistent fields based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, and use these fields as the data consistency detection result.

[0072] In some possible implementations, the inconsistent data row determination module 304 can also be used for: Based on each of the hash values, a consistent data row pair is determined, wherein the consistent data row pair includes row data from the first target data table and row data from the second target data table; Based on the first tag value of the first target data table and the second tag value of the second target data table, mirror and eliminate each pair of consistent data rows; The remaining rows of data in the first target data table and the second target data table are designated as inconsistent data rows.

[0073] In some possible implementations, the inconsistent data row determination module 304 can also be used for: For each consistent data row pair, a preset value is obtained by summing the first tag value and the second tag value of the two rows in the consistent data row pair, so as to mirror and eliminate the two rows in the consistent data row pair.

[0074] In some possible implementations, the device further includes: The data table copying module is used to copy the first data table and the second data table to a database with a massively parallel processing architecture before tagging each row of data in the first data table and the second data table to obtain a first target data table including a first tag value and a second target data table including a second tag value. Based on this, the marking module 302 can also be used for: In the database, each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

[0075] In some possible implementations, the device further includes: The table structure detection module is used to perform table structure detection on the first data table and the second data table before determining the inconsistent data rows between the first target data table and the second target data table based on the first tag value of the first target data table and the second tag value of the second target data table, when it has the permission to detect the first data table and the second data table, and obtain the table structure detection result. If the table structure detection result indicates that the table structures of the first data table and the second data table are inconsistent, a prompt message is generated to indicate that the table structures of the first data table and the second data table are inconsistent, and is then displayed. Based on this, the marking module 302 can also be used for: If the table structure detection result shows that the table structures of the first data table and the second data table are consistent, then each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

[0076] In some possible implementations, the table structure detection module can also be used for: Determine the structural parameters of each field in the first data table and the second data table respectively. The structural parameters include at least one of field name, data type and field length. By comparing the structural parameters in the first data table and the second data table, the table structure detection results are obtained.

[0077] In some possible implementations, the database of the massively parallel processing architecture includes several storage nodes; The device further includes: The data migration module is used to obtain the first distribution key of the first target data table and the second distribution key of the second target data table before determining the inconsistent fields based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table. Based on the first distribution key and the second distribution key, determine the first storage node of the first target data table in the database of the massively parallel processing architecture and the second storage node of the second target data table in the database of the massively parallel processing architecture. When the first storage node is different from the second storage node, the first target data table is migrated to the second storage node or the second target data table is migrated to the first storage node.

[0078] The various modules of the data consistency detection device provided in this application embodiment can achieve... Figure 1 It provides the functionality for each step of the data consistency detection method and achieves the corresponding technical effects. For the sake of brevity, it will not be elaborated here.

[0079] Based on the data consistency detection method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Figure 4 A schematic diagram of an electronic device 400 provided in an embodiment of this application is shown.

[0080] Electronic device 400 may include processor 401 and memory 402 storing computer program instructions.

[0081] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0082] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to electronic device 400. In a particular embodiment, memory 402 is a non-volatile solid-state memory.

[0083] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this application.

[0084] The processor 401 implements any of the data consistency detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 402.

[0085] In one example, the electronic device 400 may also include a communication interface 403 and a bus 404. For example, Figure 4 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 404 and complete communication with each other.

[0086] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0087] Bus 404 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 404 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0088] For example, the electronic device 400 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0089] The electronic device can execute the data retrieval method in the embodiments of this application, thereby achieving the combination Figures 1 to 3 The data consistency detection method and apparatus described herein, and the beneficial effects of the corresponding method embodiments, will not be elaborated further here.

[0090] The data consistency detection method described in the above embodiments can be implemented using a computer storage medium in this application. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data consistency detection methods described in the above embodiments.

[0091] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data consistency detection methods described in the above embodiments.

[0092] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0093] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0094] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0095] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0096] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data consistency detection method, characterized in that, include: Retrieve the first data table and the second data table; Each row of data in the first data table and the second data table is labeled to obtain a first target data table including a first label value and a second target data table including a second label value. Determine the hash value of each row of data in the first target data table and the second target data table; Based on the first tag value of the first target data table, the second tag value of the second target data table, and each hash value, determine the inconsistent data rows between the first target data table and the second target data table; Based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, the inconsistent fields are determined as the data consistency detection results.

2. The method according to claim 1, characterized in that, Based on the first tag value of the first target data table, the second tag value of the second target data table, and each hash value, specifically including: Based on each of the hash values, a consistent data row pair is determined, wherein the consistent data row pair includes row data from the first target data table and row data from the second target data table; Based on the first tag value of the first target data table and the second tag value of the second target data table, mirror and eliminate each pair of consistent data rows; The remaining rows of data in the first target data table and the second target data table are designated as inconsistent data rows.

3. The method according to claim 2, characterized in that, Based on the first tag value of the first target data table and the second tag value of the second target data table, mirroring and eliminating each pair of consistent data rows specifically includes: For each consistent data row pair, a preset value is obtained by summing the first tag value and the second tag value of the two rows in the consistent data row pair, so as to mirror and eliminate the two rows in the consistent data row pair.

4. The method according to claim 1, characterized in that, Before tagging each row of data in the first data table and the second data table to obtain a first target data table including a first tag value and a second target data table including a second tag value, the method further includes: Copy the first data table and the second data table to the database of the massively parallel processing architecture; Each row of data in the first data table and the second data table is labeled to obtain a first target data table including a first label value and a second target data table including a second label value, specifically including: In the database, each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

5. The method according to claim 1, characterized in that, Before determining the inconsistent data rows between the first target data table and the second target data table based on the first label value of the first target data table and the second label value of the second target data table, the method further includes: When you have permission to inspect the first data table and the second data table, perform table structure inspection on the first data table and the second data table, and obtain the table structure inspection results. If the table structure detection result indicates that the table structures of the first data table and the second data table are inconsistent, a prompt message is generated to indicate that the table structures of the first data table and the second data table are inconsistent, and is then displayed. Each row of data in the first data table and the second data table is labeled to obtain a first target data table including a first label value and a second target data table including a second label value, specifically including: If the table structure detection result shows that the table structures of the first data table and the second data table are consistent, then each row of data in the first data table and the second data table is tagged to obtain a first target data table including a first tag value and a second target data table including a second tag value.

6. The method according to claim 5, characterized in that, The table structure of the first data table and the second data table is checked to obtain the table structure check results, which specifically include: Determine the structural parameters of each field in the first data table and the second data table respectively. The structural parameters include at least one of field name, data type and field length. By comparing the structural parameters in the first data table and the second data table, the table structure detection results are obtained.

7. The method according to claim 4, characterized in that, The database of the massively parallel processing architecture includes several storage nodes; Before determining the inconsistent fields based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, the method further includes: Obtain the first distribution key of the first target data table and the second distribution key of the second target data table; Based on the first distribution key and the second distribution key, determine the first storage node of the first target data table in the database of the massively parallel processing architecture and the second storage node of the second target data table in the database of the massively parallel processing architecture. When the first storage node is different from the second storage node, the first target data table is migrated to the second storage node or the second target data table is migrated to the first storage node.

8. A data consistency detection device, characterized in that, The device includes: The data table acquisition module is used to acquire the first data table and the second data table; The tagging module is used to tag each row of data in the first data table and the second data table respectively, to obtain a first target data table including a first tag value and a second target data table including a second tag value; The hash value determination module is used to determine the hash value of each row of data in the first target data table and the second target data table; The inconsistent data row determination module is used to determine the inconsistent data rows between the first target data table and the second target data table based on the first tag value of the first target data table, the second tag value of the second target data table, and each of the hash values. The data consistency detection result determination module is used to determine the inconsistent fields based on each field in the inconsistent data row of the first target data table and each field in the inconsistent data row of the second target data table, and use these fields as the data consistency detection result.

9. A data consistency detection device, characterized in that, The device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the data consistency detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data consistency detection method as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the data consistency detection method as described in any one of claims 1-7.

Images