A data consistency verification method and related device
Through the data consistency verification method between Hadoop and MPP databases, the field mapping and type conversion relationship are used to realize automated data consistency verification, solving the problem of time-consuming, labor-intensive and accurate verification in the existing technology. , improves efficiency and accuracy.
Patent Information
- Application Number
- CN202110850826.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the data interaction and conversion between Hadoop and MPP databases, it is difficult for the prior art to effectively verify data consistency, which makes the verification process time-consuming and labor-intensive, and the results are not accurate, making it difficult to accurately locate problem fields.
Provide a data consistency verification method. By determining the mapping relationship between the target hive table and the GBase database table, automatically perform field-level data consistency verification based on the preset meaning type or data type conversion relationship of the field, generate verification results and include Difference field information.
It realizes automated data consistency verification, reduces labor costs, improves verification efficiency and accuracy, can accurately locate problem data, and reduces dependence on professional knowledge.
Smart Images

Figure CN113486022B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data verification, and in particular to a data consistency verification method and related devices. Background Art
[0002] In big data applications in various industries, data tables can be formed based on actual needs. Hadoop is a distributed system infrastructure. The Hadoop ecosystem is better at batch processing, multi-table or full-table scanning, real-time stream processing, storage and conversion of unstructured data, etc. Massively Parallel Processing (MPP) The MPP architecture distributes tasks in parallel to multiple servers and nodes. After the calculation is completed on each node, the results of each part are aggregated to obtain the final result. A database using the MPP architecture is called an MPP database. MPP is better at reporting, ad hoc queries, structured data calculation and analysis, etc. Based on the background of complex data, more and more companies choose to use a hybrid architecture of Hadoop and MPP to meet all data business needs, so as to achieve complementary advantages of the two in data processing. For example, Hadoop is generally used to implement a big data platform, and MPP is used to implement a data warehouse.
[0003] Hive is a data warehouse tool based on the Hadoop ecosystem. It is not a database in essence, but a tool. Hive is used to extract, transform, and load data. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files to a database table and provide SQL query functions. It can convert SQL statements into MapReduce tasks to run, and analyze the required content through its own SQL queries. This set of SQL is referred to as HQL (Hive SQL), so that users who are not familiar with MapReduce can easily use SQL language to query, summarize, and analyze data. Since it can map structured data files in Hadoop to database tables and provide HQL (full name: Hive SQL) queries similar to SQL queries, the hive tool can be understood as a database. In the following text, the data tables extracted by the hive tool are referred to as hive tables.
[0004] GBase 8a MPP database (hereinafter referred to as GBase database) is a representative of MPP column storage databases. Column databases are databases that use column-related storage architectures to store data, and are mainly suitable for batch data processing and instant queries. In contrast, row-based databases use row-related storage architectures to allocate space for data, and are mainly suitable for small batch data processing, and are often used for online transactional data processing.
[0005] Hive and GBase have different positioning and uses in enterprise projects, which leads to the need to process the data types in them. In the interaction and conversion of Hadoop and MPP data, that is, the interaction and conversion of hive tables and GBase database tables, it is necessary to verify the data consistency between the two. Due to the differences between hive and GBase databases in storage model, calculation mode, data distribution method, and compatibility with SQL, the query efficiency will be very slow in complex demand testing involving thousands of tables and huge data volume. During verification, HQL and SQL statements need to be written in hive tables and GBase respectively for query, and the query results need to be manually compared to obtain the verification results. The verification process will be very time-consuming and labor-intensive. In addition, this comparison method requires the verification personnel to have a certain understanding of SQL and HQL, and the comparison results cannot guarantee the accuracy of data in all columns. When there are many tables involved, the manpower and time costs required will increase exponentially. And once there is a problem with the comparison result, it will be very difficult to locate the specific field. Summary of the invention
[0006] Based on the above problems, the present application provides a data consistency verification method and related devices to automatically realize accurate verification of data consistency between hive tables and GBase database tables.
[0007] The embodiments of the present application disclose the following technical solutions:
[0008] In a first aspect, the present application provides a data consistency verification method, comprising:
[0009] Determine a target hive table and a target GBase database table, wherein the target hive table and the target GBase database table have a mapping relationship;
[0010] According to the first target field in the target hive table and the second target field in the target GBase database table, determining a verification pass condition for consistency verification of data in the first target field and data in the second target field, wherein the first target field and the second target field have a mapping relationship;
[0011] According to the verification pass condition, the data of the first target field and the data of the second target field, a consistency verification result of the data of the first target field and the data of the second target field is obtained.
[0012] Optionally, the determining, according to the first target field in the target hive table and the second target field in the target GBase database table, a verification pass condition for consistency verification of data in the first target field and data in the second target field includes:
[0013] When the first target field and the second target field are fields of preset meaning types, taking the meaning of the data in the first target field and the data in the second target field to be the same as the verification pass condition;
[0014] When the first target field and the second target field are fields of non-preset meaning types, the verification pass condition is that the data of the first target field and the data of the second target field satisfy a preset data type conversion relationship.
[0015] Optionally, before determining the target hive table and the target GBase database table, the method further includes:
[0016] The mapping relationship between the hive table to be verified and the GBase database table to be verified is configured according to business requirements, and the mapping relationship between the fields to be verified in the hive table to be verified and the fields to be verified in the GBase database table is configured according to the business requirements.
[0017] Optionally, obtaining a consistency verification result of the data in the first target field and the data in the second target field according to the verification pass condition, the data in the first target field, and the data in the second target field includes:
[0018] When the data of the first target field and the data of the second target field meet the verification pass condition, generating a consistency verification result, the consistency verification result indicating that the data of the first target field and the data of the second target field pass the consistency verification;
[0019] When the data of the first target field and the data of the second target field do not meet the verification pass condition, a consistency verification result is generated, and the consistency verification result indicates that the data of the first target field and the data of the second target field have not passed the consistency verification; the consistency verification result includes difference field information, and the difference field information includes the first target field and the second target field.
[0020] Optionally, the above method further includes:
[0021] A first file is generated according to the target hive table, and a second file is generated according to the target GBase database table. The first file and the second file have the same file type.
[0022] Optionally, the above method further includes:
[0023] The file type is determined according to the target verification efficiency and the data volume of the target hive table and the data volume of the target GBase database table.
[0024] Optionally, the preset data type conversion relationship includes:
[0025] The data type of the first target field is int, integer, tinyint or smallint, and the data type of the second target field is int; or,
[0026] The data type of the first target field is text, clob or blob, and the data type of the second target field is varchar; or,
[0027] The data type of the first target field is string, and the data type of the second target field is varchar.
[0028] In a second aspect, the present application provides a data consistency verification device, comprising:
[0029] A data table determination module is used to determine a target hive table and a target GBase database table, wherein the target hive table and the target GBase database table have a mapping relationship;
[0030] A condition determination module, used to determine a verification pass condition for consistency verification of data in the first target field and data in the second target field according to a first target field in the target hive table and a second target field in the target GBase database table, wherein the first target field and the second target field have a mapping relationship;
[0031] The verification module is used to obtain a consistency verification result of the data in the first target field and the data in the second target field according to the verification pass condition, the data in the first target field and the data in the second target field.
[0032] Optionally, the condition determination module includes: a first condition determination unit and a second condition determination unit;
[0033] The first condition determination unit is configured to, when the first target field and the second target field are fields of a preset meaning type, take the meaning of the data in the first target field and the data in the second target field to be the same as the verification pass condition;
[0034] The second condition determination unit is used to, when the first target field and the second target field are fields of non-preset meaning types, take the data of the first target field and the data of the second target field satisfying a preset data type conversion relationship as the verification pass condition.
[0035] Optionally, the verification module includes: a first verification unit and a second verification unit;
[0036] The first verification unit is used to generate a consistency verification result when the data of the first target field and the data of the second target field meet the verification pass condition, and the consistency verification result indicates that the data of the first target field and the data of the second target field pass the consistency verification;
[0037] The second verification unit is used to generate a consistency verification result when the data of the first target field and the data of the second target field do not meet the verification pass condition, and the consistency verification result indicates that the data of the first target field and the data of the second target field have not passed the consistency verification; the consistency verification result includes difference field information, and the difference field information includes the first target field and the second target field.
[0038] Compared with the prior art, this application has the following beneficial effects:
[0039] In a data consistency verification method provided by the present application, the target hive table and the target GBase database table are first determined, and the target hive table and the target GBase database table have a mapping relationship, thereby determining the two data tables that need to be verified for data consistency. Thereafter, according to the first target field in the target hive table and the second target field in the target GBase database table, the verification pass condition for consistency verification of the data of the first target field and the data of the second target field is determined, and the first target field and the second target field have a mapping relationship. According to the verification pass condition determined by the first target field and the second target field, the data consistency verification of the first target field and the second target field can be assisted in a purposeful and well-founded manner. Finally, according to the verification pass condition, the data of the first target field and the data of the second target field, the consistency verification result of the data of the first target field and the data of the second target field is obtained. This method can automatically realize data consistency verification with field-level accuracy, accurately locate the field to which the problem data belongs, reduce dependence on personnel expertise, and save labor costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1A A flowchart of a data consistency verification method provided in an embodiment of the present application;
[0042] Figure 1B A flowchart of another data consistency verification method provided in an embodiment of the present application;
[0043] Figure 2 A schematic diagram of the structure of a data consistency verification device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] At present, more and more enterprises choose to use a hybrid architecture of Hadoop and MPP to meet all data business needs. Hive is an enterprise big data platform that stores data from various source systems of the enterprise. It is necessary to ensure that the data is completely consistent with the data in the source system. The data of each source system is stored in various databases. In other words, the data sources of the source system are diverse, resulting in various data types, such as Oracle, MySQL, and DB2. GBase is generally used to build a data warehouse platform or an analysis system. There is data integration, processing, and modeling. Its data source may be extracted from a big data platform, which involves the conversion from hive to GBase. Whether it is building a data warehouse or a backend database for other analysis systems, it is necessary to ensure that its data is accurate and thus its availability. Data consistency verification between hive tables and GBase database tables is a key operation to verify the accuracy of data in GBase database tables. Only through verification can it be verified whether the data in the GBase database table is distorted. Distorted data may seriously affect data availability.
[0045] Currently, the operation of verifying the data consistency between hive tables and GBase database tables is usually performed manually by personnel with certain verification experience, which has problems such as unstable verification result accuracy, low positioning accuracy of problem data, high labor costs and low efficiency. Based on the above problems, the present application provides a targeted solution. Specifically, the present application provides a data consistency verification method and related devices.
[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] See also Figure 1A , which is a flow chart of a data consistency verification method provided by an embodiment of the present application. Figure 1A The data consistency verification method shown includes:
[0048] Step 101: Determine a target hive table and a target GBase database table, where the target hive table and the target GBase database table have a mapping relationship.
[0049] In an embodiment of the present application, the target hive table and the target GBase database table are respectively the hive table and the GBase database table that need to verify the consistency of data with each other. The target hive table and the target GBase database table can be determined specifically depending on the data usage requirements of the enterprise project. For example, if the data that needs to be used belongs to a certain hive table, the hive table is determined as the target hive table, and the GBase database table that has a mapping relationship with the target hive table is determined as the target GBase database table. Similarly, when it is determined that the data that needs to be used belongs to a certain GBase database table, the GBase database table is determined as the target GBase database table, and the hive table that has a mapping relationship with the target GBase database table is determined as the target hive table. Here, the mapping relationship refers to a relationship between two tables that have corresponding data and can be mutually verified.
[0050] Between the target hive table and the target GBase database table, the corresponding data belongs to the field to be verified with a mapping relationship. The name of the field to be verified with a mapping relationship may be the same or different between the target hive table and the target GBase database table. For ease of distinction, in this application, the field to be verified in the target hive table is defined as the first field, and the field to be verified in the target GBase database table that has a mapping relationship with the first field is defined as the second field.
[0051] In an optional implementation, the mapping relationship between the hive table to be verified and the GBase database table to be verified can be configured in advance according to business requirements, and the mapping relationship between the fields to be verified in the hive table to be verified and the fields to be verified in the GBase database table can be configured in advance according to business requirements. Thereafter, in this step, the target hive table and the target GBase database table are determined based on the pre-configured mapping relationship and actual business requirements.
[0052] Step 102: According to the first target field in the target hive table and the second target field in the target GBase database table, a verification pass condition for consistency verification of data in the first target field and data in the second target field is determined, and the first target field and the second target field have a mapping relationship.
[0053] In practical applications, it is necessary to determine the verification pass conditions of data consistency verification in combination with specific fields, and then perform field consistency verification operations between the target hive table and the target GBase database table in combination with the matching verification pass conditions.
[0054] The following introduces several implementation methods for determining the verification pass conditions of data consistency verification based on specific fields.
[0055] In a possible implementation, when the first target field and the second target field are fields of preset meaning types, the verification condition is that the meaning of the data in the first target field and the data in the second target field are the same. As examples, the preset meaning types include: amount type or date type.
[0056] As an example, in the reports of various source systems, there will be fields such as certain amounts. Different source systems may enter hive, and the data types defined there may be various, such as number, char, decimal, etc. Then, in GBase, in order to make the subsequent data uniformly available, the data types of all amount type fields need to be converted to decimal. In this way, the amount fields of these tables can be correctly calculated when the system performs analysis and calculation. The result of this conversion may be that a value changes from 0.1 to 0.10000000. Although the data looks different in form, it is actually correct because the two data 0.1 (from the hive table) and 0.10000000 (from the GBase database table) with the meaning of amount are consistent.
[0057] As another example, the date field in the source system may have multiple designs such as 2021 / 7 / 15, 20210715, 15 / 07 / 21, etc. There are multiple data types to store them, including varchar, date, string, and even non-standard systems will store it as NUMBER. In GBase, you may need to uniformly convert date fields into dates. Otherwise, if you want to summarize the data of a certain day in the future, different types will not be counted. Then, for this conversion comparison, you need to identify that 20210715 and 2021 / 7 / 15 are consistent data.
[0058] In combination with the above example, in a possible implementation, when the first target field and the second target field are fields of preset meaning types, the same meaning of the data in the first target field and the data in the second target field can be used as a verification pass condition.
[0059] In another possible implementation, when the first target field and the second target field are fields of non-preset meaning types, it is used as a verification pass condition that the data of the first target field and the data of the second target field satisfy a preset data type conversion relationship.
[0060] For example, in Gbase, in order to more conveniently and accurately integrate, process, and model data, some data types may be uniformly converted. For example, all int, integer, tinyint, and smallint types in the hive table are converted to int type; all text, clob, and blob types in the hive table are converted to varchar type. This type conversion is based on the compliant preset data type conversion relationship defined by a certain project design. In this case, verifying data consistency does not require considering the specific meaning of the field, but only checking whether the data types of the first target field and the second target field conform to the preset data type conversion relationship. In addition, there is a special case where the data types supported by the GBase database table and the hive table themselves are different. For example, a string type field in the hive table needs to be converted to a varchar type field in the GBase database table. This is because the GBase database table does not support the string type.
[0061] In combination with the above example, in another possible implementation, when the first target field and the second target field are fields of non-preset meaning types, the data of the first target field and the data of the second target field satisfying the preset data type conversion relationship is used as a verification pass condition. Specifically, the preset data type conversion relationship includes:
[0062] The data type of the first target field is int, integer, tinyint, or smallint, and the data type of the corresponding second target field is int; or,
[0063] The data type of the first target field is text, clob, or blob, and the data type of the corresponding second target field is varchar; or,
[0064] The data type of the first target field is string, and the corresponding data type of the second target field is varchar.
[0065] Step 103: Obtain a consistency verification result of the data in the first target field and the data in the second target field according to the verification pass condition, the data in the first target field, and the data in the second target field.
[0066] It is understandable that when the data of the first target field and the data of the second target field are verified according to the verification pass condition, the obtained consistency verification results may be divided into two situations, namely, passing the verification and failing the verification.
[0067] When the data of the first target field and the data of the second target field meet the verification pass condition, a consistency verification result is generated, and the consistency verification result indicates that the data of the first target field and the data of the second target field pass the consistency verification.
[0068] For example, the consistency verification result includes a verification result presentation area corresponding to each pair of the first target field and the second target field to be verified, and a blank or a check mark or the word "pass" is displayed in the area.
[0069] When the data of the first target field and the data of the second target field do not satisfy the verification pass condition, a consistency verification result is generated, and the consistency verification result indicates that the data of the first target field and the data of the second target field have not passed the consistency verification.
[0070] For example, the consistency verification result includes a verification result presentation area corresponding to each pair of the first target field and the second target field to be verified. The deletion difference field information is displayed in this area, and the difference field information may include: the field names of the first target field and the second target field, and may also be specific to the inconsistent data in the first target field and the second target field or the number of rows of the inconsistent data in the table.
[0071] The above is a data consistency verification method provided by an embodiment of the present application. In this method, the target hive table and the target GBase database table are first determined, and the target hive table and the target GBase database table have a mapping relationship, thereby determining the two data tables that need to be verified for data consistency. Thereafter, according to the first target field in the target hive table and the second target field in the target GBase database table, the verification pass condition for consistency verification of the data of the first target field and the data of the second target field is determined, and the first target field and the second target field have a mapping relationship. According to the verification pass condition determined by the first target field and the second target field, the data consistency verification of the first target field and the second target field can be assisted in a purposeful and well-founded manner. Finally, according to the verification pass condition, the data of the first target field and the data of the second target field, the consistency verification result of the data of the first target field and the data of the second target field is obtained. This method can automatically realize data consistency verification with field-level accuracy, accurately locate the field to which the problem data belongs, reduce dependence on personnel professionalism, save manpower costs and save time.
[0072] In actual applications, before verification, and after determining the target hive table and the target GBase database table based on the mapping relationship, read the data in the target hive table to generate a first file, put the first file into a specified directory, and read the data in the target GBase database table through the GBaseConnector structure to generate a second file, and put the second file into the aforementioned specified directory. In order to facilitate verification of consistency, optionally, the generated first file and the second file have the same file type and consistent format, so that it is convenient to compare and verify data consistency. As an example, the file type can be a text file with a custom delimiter.
[0073] It is worth mentioning that the process of reading the target hive table and the target GBase database table and generating files respectively needs to fully consider its performance, evaluate the size of the verified data and its impact on efficiency, and then decide on the technology selection. The file types of the first file and the second file generated in this application can be determined according to actual conditions (such as data size and verification efficiency). The file type is determined based on actual parameters or actual needs such as data volume and verification efficiency, and the first file and the second file are generated based on this file type, which can effectively match the actual needs and ensure verification efficiency.
[0074] Figure 1B A flowchart of another data consistency verification method provided in an embodiment of the present application. Figure 1B As shown, first configure the mapping relationship between tables and between fields; then read the data of the first target field and the second target field in the target hive table and the target GBase library table with the mapping relationship respectively, and generate the first file and the second file; then compare and verify the consistency of the data of the first target field and the second target field based on the verification pass condition to obtain the consistency verification result; finally output the consistency verification result. The present application provides a solution for realizing accurate verification of data consistency at the field level of hive tables and GBase database tables (batch verification can be realized). Before comparison, you only need to configure the table information of the required comparison range to generate and output the consistency verification result. At present, this solution has been implemented and applied in projects. Practice has shown that this solution can save a lot of manpower and time.
[0075] As a verification example, the data of the first target field in the target hive table includes: 00386, 01022, 01095, 01187, 01206, 01309, 01311, etc., and the data of the second target field in the target GBase database table includes: 386, 1022, 1095, 1187, 1206, 1309, 1311, etc. It is obvious that the data type conversion between the first target field and the second target field is incorrect. The first target field may be defined as a character type in the hive table, but it may be defined as a numeric type in the GBase database table, resulting in the loss of the leading 0. This field may be an institution number or a number type value. In subsequent calculations or modeling, this field of this table may need to be associated with other tables, so the distorted data cannot be associated. If the meaning of the field is the amount, if the defined data type is incorrect, it may cause data errors in the subsequent calculation of indicators and reports using this field. If data consistency verification is not performed and data is used directly, other failures may occur, such as the inability to ensure data accuracy when analyzing data. In addition, if the field type length is incorrectly defined after conversion and the field is too long, data loading failure and system batch failure may occur. By verifying the consistency of the data of the two data tables in the embodiment of the present application, abnormal fields can be located in a timely, effective, efficient and high-precision manner, thereby ensuring the validity of the data.
[0076] Based on the method provided in the above embodiment, the present application also provides a data consistency verification device. Figure 2 , which is a schematic diagram of the structure of a data consistency verification device provided by an embodiment of the present application. Figure 2 As shown, the data consistency verification device 200 includes:
[0077] The data table determination module 201 is used to determine a target hive table and a target GBase database table, wherein the target hive table and the target GBase database table have a mapping relationship;
[0078] A condition determination module 202 is used to determine a verification pass condition for performing consistency verification on data in the first target field and data in the second target field according to the first target field in the target hive table and the second target field in the target GBase database table, wherein the first target field and the second target field have a mapping relationship;
[0079] The verification module 203 is used to obtain a consistency verification result of the data in the first target field and the data in the second target field according to the verification pass condition, the data in the first target field and the data in the second target field.
[0080] Optionally, the condition determination module 202 includes: a first condition determination unit and a second condition determination unit;
[0081] The first condition determination unit is configured to, when the first target field and the second target field are fields of a preset meaning type, take the meaning of the data in the first target field and the data in the second target field to be the same as the verification pass condition;
[0082] The second condition determination unit is used to, when the first target field and the second target field are fields of non-preset meaning types, take the data of the first target field and the data of the second target field satisfying a preset data type conversion relationship as the verification pass condition.
[0083] Optionally, the data consistency verification device 200 further includes:
[0084] The mapping relationship configuration module is used to configure the mapping relationship between the hive table to be verified and the GBase database table to be verified according to business requirements, and configure the mapping relationship between the fields to be verified in the hive table to be verified and the fields to be verified in the GBase database table according to the business requirements.
[0085] Optionally, the verification module 203 includes: a first verification unit and a second verification unit;
[0086] The first verification unit is used to generate a consistency verification result when the data of the first target field and the data of the second target field meet the verification pass condition, and the consistency verification result indicates that the data of the first target field and the data of the second target field pass the consistency verification;
[0087] The second verification unit is used to generate a consistency verification result when the data of the first target field and the data of the second target field do not meet the verification pass condition, and the consistency verification result indicates that the data of the first target field and the data of the second target field have not passed the consistency verification; the consistency verification result includes difference field information, and the difference field information includes the first target field and the second target field.
[0088] Optionally, the data consistency verification device 200 further includes:
[0089] The file generation module is used to generate a first file according to the target hive table and to generate a second file according to the target GBase database table. The first file and the second file have the same file type.
[0090] Optionally, the data consistency verification device 200 further includes:
[0091] The file type determination module is used to determine the file type according to the target verification efficiency and the data volume of the target hive table and the data volume of the target GBase database table.
[0092] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0093] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data consistency verification method, characterized in that: include: Determine a target hive table and a target GBase database table, wherein the target hive table and the target GBase database table have a mapping relationship; According to the first target field in the target hive table and the second target field in the target GBase database table, determining a verification pass condition for consistency verification of data in the first target field and data in the second target field, wherein the first target field and the second target field have a mapping relationship; Generate a first file according to the target hive table, and generate a second file according to the target GBase database table, wherein the first file and the second file have the same file type; wherein the file type is determined based on the target verification efficiency, the data volume of the target hive table, and the data volume of the target GBase database table; According to the verification pass condition, the data of the first target field and the data of the second target field, a consistency verification result of the data of the first target field and the data of the second target field is obtained.
2. The method according to claim 1, characterized in that: The determining, according to the first target field in the target hive table and the second target field in the target GBase database table, a verification pass condition for consistency verification of data in the first target field and data in the second target field comprises: When the first target field and the second target field are fields of preset meaning types, taking the meaning of the data in the first target field and the data in the second target field to be the same as the verification pass condition; When the first target field and the second target field are fields of non-preset meaning types, the verification pass condition is that the data of the first target field and the data of the second target field satisfy a preset data type conversion relationship.
3. The method according to claim 1, characterized in that: Before determining the target hive table and the target GBase database table, the method further includes: The mapping relationship between the hive table to be verified and the GBase database table to be verified is configured according to business requirements, and the mapping relationship between the fields to be verified in the hive table to be verified and the fields to be verified in the GBase database table is configured according to the business requirements.
4. The method according to claim 1, characterized in that: The obtaining, according to the verification pass condition, the data of the first target field, and the data of the second target field, a consistency verification result of the data of the first target field and the data of the second target field includes: When the data of the first target field and the data of the second target field meet the verification pass condition, generating a consistency verification result, the consistency verification result indicating that the data of the first target field and the data of the second target field pass the consistency verification; When the data of the first target field and the data of the second target field do not meet the verification pass condition, a consistency verification result is generated, and the consistency verification result indicates that the data of the first target field and the data of the second target field have not passed the consistency verification; the consistency verification result includes difference field information, and the difference field information includes the first target field and the second target field.
5. The method according to claim 2, characterized in that: The preset data type conversion relationship includes: The data type of the first target field is int, integer, tinyint or smallint, and the data type of the second target field is int; or, The data type of the first target field is text, clob or blob, and the data type of the second target field is varchar; or, The data type of the first target field is string, and the data type of the second target field is varchar.
6. A data consistency verification device, characterized in that: include: A data table determination module is used to determine a target hive table and a target GBase database table, wherein the target hive table and the target GBase database table have a mapping relationship; A condition determination module, used to determine a verification pass condition for consistency verification of data in the first target field and data in the second target field according to a first target field in the target hive table and a second target field in the target GBase database table, wherein the first target field and the second target field have a mapping relationship; A file generation module, used to generate a first file according to the target hive table, and to generate a second file according to the target GBase database table, wherein the first file and the second file have the same file type; wherein the file type is determined by a file type determination module according to target verification efficiency, the data volume of the target hive table, and the data volume of the target GBase database table; The verification module is used to obtain a consistency verification result of the data in the first target field and the data in the second target field according to the verification pass condition, the data in the first target field and the data in the second target field.
7. The device according to claim 6, characterized in that The condition determination module includes: a first condition determination unit and a second condition determination unit; The first condition determination unit is configured to, when the first target field and the second target field are fields of a preset meaning type, take the same meaning of the data in the first target field and the data in the second target field as the verification pass condition; The second condition determination unit is used to, when the first target field and the second target field are fields of non-preset meaning types, take the data of the first target field and the data of the second target field satisfying a preset data type conversion relationship as the verification pass condition.
8. The device according to claim 6, characterized in that The verification module includes: a first verification unit and a second verification unit; The first verification unit is used to generate a consistency verification result when the data of the first target field and the data of the second target field meet the verification pass condition, and the consistency verification result indicates that the data of the first target field and the data of the second target field pass the consistency verification; The second verification unit is used to generate a consistency verification result when the data of the first target field and the data of the second target field do not meet the verification pass condition, and the consistency verification result indicates that the data of the first target field and the data of the second target field have not passed the consistency verification; the consistency verification result includes difference field information, and the difference field information includes the first target field and the second target field.
