Big data offline data quality inspection method and device
Automatic verification of Hive and Hbase data through Python solves the inefficiency and complexity of offline data quality inspection in big data environments, improves the accuracy and maintainability of verification, and adapts to the needs of complex business scenarios.
Patent Information
- Application Number
- CN202410953460.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-15
AI Technical Summary
The existing technology has inefficient efficiency, complex verification logic development, difficult processing of complex business scenarios and high verification error rate during offline data quality inspection in big data environments, and has failed to effectively solve these problems.
Read the user ID and update time in the Hive table through Python, obtain the data to be checked in the Hbase table, perform data verification and write inconsistent data to the CSV file, simplify the verification logic with Python's automation and mapping relationships, and improve the verification logic Automation level and accuracy of verification.
It improves the automation level, accuracy and maintainability of data verification, significantly reduces development and maintenance costs, reduces human errors, and adapts to data processing in complex business scenarios.
Smart Images

Figure CN118964338A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology applications, and in particular to a big data offline data quality inspection method and device. Background Art
[0002] In the related art, the offline data quality inspection process is as follows:
[0003] 1). Write the processed data in the Hive database into the Hbase database;
[0004] 2). Use the PySpark big data processing framework to read data from the latest partition of the Hive table into the memory of the big data cluster;
[0005] 3). Get the user user_no and update time last_modified in the second step data to read the corresponding data in the Hbase library into the big data cluster memory;
[0006] 4). Create temporary tables Hive_table and Hbase_table for Hive data and Hbase data to facilitate subsequent SQL processing;
[0007] 5) Perform full outer join on Hive_table and Hbase_table to compare whether there are more or less data, and fields and values corresponding to different data values.
[0008] However, when the relevant technology is executed, there are relatively few data verification tables at the beginning, only two or three Hive tables. However, with the increase of business scenarios and tables, more and more tables need to be verified. If there are more than 500 Hive tables that need to be verified and compared with the data in the Hbase table, it is very time-consuming and inefficient to develop verification according to this method, and it is very easy to make mistakes. For a single Hive table with 400 to 500 fields, it is necessary to write 400 to 500 lines of verification logic SQL. This amount of work is very huge, and if the correct SQL is not written, there may be problems such as the fields in the library not corresponding to the actual fields. Later, colleagues in the analysis use the wrong data to draw wrong conclusions, and then make wrong decisions, causing economic losses to the company. If there is a complex business scenario that requires data to be processed before writing to the Hbase library, this method is very difficult to implement, and the amount of SQL is very huge, and it is very difficult to maintain later. For example, the field with a denominator of zero in the calculation is stored as null in the Hive table, and it needs to be converted to -999 when written to the corresponding Hbase table. This part needs to be mapped when comparing the corresponding fields. This method greatly increases the complexity of SQL.
[0009] At present, no effective solution has been proposed for the problems of low efficiency, complex verification logic development and maintenance, difficulty in handling complex business scenarios, and high verification error rate in the offline data quality verification process in the big data environment in related technologies. Summary of the invention
[0010] The purpose of the present invention is to provide a big data offline data quality verification method and device to address the deficiencies in the prior art, so as to solve the problems of low efficiency, complex verification logic development and maintenance, difficulty in handling complex business scenarios and high verification error rate in the offline data quality verification process in a big data environment in the related technology.
[0011] To achieve the above object, the technical solution adopted by the present invention is:
[0012] In a first aspect, the present invention provides a big data offline data quality verification method, comprising: reading the user identifier user_no and the update time last_modified in the Hive table through Python; obtaining the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; verifying the data in the Hive table and the data to be verified through Python; if the data in the Hive table and the data to be verified are inconsistent in the verification result, writing the inconsistent data into a CSV format file named after the Hive table name to facilitate problem troubleshooting.
[0013] In some of the embodiments, before reading the user identifier user_no and the update time last_modified in the Hive table through Python, the method further includes: writing the data of the Hive table into the Hbase table through the ImportTsv method.
[0014] In some of the embodiments, reading the user identifier user_no and the update time last_modified in the Hive table through Python includes: reading the user identifier user_no and the update time last_modified in the Hive table through Python; writing the user identifier user_no and the update time last_modified into a CSV format file, and storing the column names of the data corresponding to the user identifier user_no and the update time last_modified into the field list columns_list.
[0015] In some embodiments, obtaining the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified includes: obtaining the name of the Hbase table corresponding to the Hive table from the dictionary of the mapping relationship according to the mapping relationship between the Hive library and the Hbase library; and obtaining the corresponding data from the Hbase table according to the name of the Hbase table, the user identifier user_no and the update time last_modified through Python.
[0016] In some embodiments, verifying the data in the Hive table and the data to be verified through Python includes: associating the data corresponding to the user identifier user_no and the update time last_modified with the corresponding data obtained from the Hbase table, and placing the data with the same user identifier in the same row according to the user identifier for verification, wherein the verification includes: determining whether the data corresponding to the user identifier user_no and the update time last_modified are consistent with the corresponding data in the Hbase table.
[0017] In some of the embodiments, writing inconsistent data into a file in CSV format named after the Hive table name includes: writing data that fails the verification in the verification results into a file in CSV format named after the Hive table name, and generating an alarm message, wherein the alarm message is used to indicate the data in the Hbase table that fails the verification.
[0018] In some of the embodiments, the problem troubleshooting includes: troubleshooting files in CSV format and solving problems in the files in CSV format; and verifying the Hbase table after solving the problems until the verification passes.
[0019] In a second aspect, the present invention provides a big data offline data quality inspection device, comprising: a reading module, used to read the user identifier user_no and the update time last_modified in the Hive table through Python; an acquisition module, used to obtain the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; a verification module, used to verify the data in the Hive table and the data to be verified through Python; a troubleshooting module, used to write the inconsistent data into a CSV format file named after the Hive table name if the data in the Hive table and the data to be verified are inconsistent in the verification result, so as to facilitate problem troubleshooting.
[0020] In some of the embodiments, the device further includes: a writing module for writing the data of the Hive table into the Hbase table through the ImportTsv method before reading the user identifier user_no and the update time last_modified in the Hive table through Python.
[0021] In some embodiments, the reading module includes: a reading unit, used to read the user identifier user_no and the update time last_modified in the Hive table through Python; a writing unit, used to write the user identifier user_no and the update time last_modified into a CSV format file, and store the column names of the data corresponding to the user identifier user_no and the update time last_modified into the field list columns_list.
[0022] The present invention adopts the above technical scheme, reads the user identifier user_no and the update time last_modified in the Hive table through Python; obtains the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; verifies the data in the Hive table and the data to be verified through Python; if the data in the Hive table and the data to be verified are inconsistent in the verification result, the inconsistent data is written into a CSV format file named after the Hive table name for troubleshooting. Compared with the prior art, the present invention has the following technical effects: improving the automation level, accuracy and maintainability of data verification. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a flowchart of a method for offline data quality inspection of big data according to an embodiment of the present invention;
[0024] Figure 2 is a flow chart of another method for offline data quality inspection of big data according to an embodiment of the present invention;
[0025] Figure 3 It is a schematic diagram of a verification process of whether data field values are consistent in another big data offline data quality verification method according to an embodiment of the present invention;
[0026] Figure 4 is a schematic diagram of a big data offline data quality inspection device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0028] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0029] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0030] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or units (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0031] Example 1
[0032] An exemplary embodiment of the present invention is as follows Figure 1 As shown, Figure 1 1 is a flow chart of a method for offline data quality inspection of big data according to an embodiment of the present invention. The method for offline data quality inspection of big data provided by the embodiment of the present application includes:
[0033] Step S102: Read the user ID user_no and update time last_modified in the Hive table through Python;
[0034] In some of the embodiments, before reading the user identifier user_no and the update time last_modified in the Hive table through Python, a big data offline data quality inspection method provided by an embodiment of the present application also includes: writing the data of the Hive table into the Hbase table through the ImportTsv method.
[0035] The data written from Hive tables to Hbase are all important feature data, and the user requires that the data used be non-repetitive, non-missing, and error-free. Hive table data is imported through the ImportTsv method provided by the Hbase framework, where ImportTsv is a command in the HBase Shell that is used to import TSV (Tab Separated Value) files into HBase tables. ImportTsv can be used to quickly load data from external files into HBase tables, facilitating data migration and batch data import.
[0036] In some of the embodiments, reading the user identifier user_no and the update time last_modified in the Hive table through Python includes: reading the user identifier user_no and the update time last_modified in the Hive table through Python; writing the user identifier user_no and the update time last_modified into a CSV format file, and storing the column names of the data corresponding to the user identifier user_no and the update time last_modified into the field list columns_list.
[0037] Among them, the user_no (i.e., the user ID in the embodiment of the present application) and the last_modified field (i.e., the update time in the embodiment of the present application) in the hive_df data are obtained, and then written into the hive_user.csv file (i.e., the CSV format file in the embodiment of the present application) for the next step of obtaining the corresponding data in the hbase_table table; at the same time, the column names of the data corresponding to the user ID user_no and the update time last_modified in hive_df are obtained and saved in a list named columns_list;
[0038] Specifically, the data to be verified in the Hive table can be read as follows:
[0039] hive_df=pd.read_sql(get_hive_data_sql(database,h_table,dt,flag),hive_conn)
[0040] def get_hive_data_sql(database,hive_table,dt,flag):
[0041] update_time = 'update_time'
[0042] if flag == 1:
[0043] update_time = 'last_modified'
[0044] sql=f"'
[0045] select*
[0046] from{database}.{hive_table}
[0047] where dt={dt}and UdfFromUnixTime({update_time},"yyyyMMdd")={dt}
[0048] ”'
[0049] return sql
[0050] Among them, pd.read_sql is a method for reading data, which is provided by Python; the get_hive_data_sql method is the SQL for obtaining hive_table data, database is the Hive database, h_table is the Hive table, that is, hive_table here, dt is the partition time, and flag is the flag.
[0051] In the embodiment of the present application, user_no is the user's unique number, similar to the role of an ID card, one number for each user.
[0052] Step S104, obtaining the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified;
[0053] In some embodiments, obtaining the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified includes: obtaining the name of the Hbase table corresponding to the Hive table from the dictionary of the mapping relationship according to the mapping relationship between the Hive library and the Hbase library; and obtaining the corresponding data from the Hbase table according to the name of the Hbase table, the user identifier user_no and the update time last_modified through Python.
[0054] Among them, the data in the corresponding Hbase table is obtained through user_no (that is, the user ID in the embodiment of the present application) and last_modified (that is, the update time in the embodiment of the present application).
[0055] Specifically, the corresponding data is obtained from the Hbase table as follows:
[0056] need_columns={hive_hbase.get(hive_table):columns_list}
[0057] tab=NewBigTableClass(infile,flag,'third_data',tables=tables,need_columns=need_columns)
[0058] f = tab.get_vars()[0]
[0059] Among them, hive_hbase.get(hive_table) is to obtain the name of hbase_table (that is, the Hbase table in the embodiment of the present application) corresponding to the table hive_table (that is, the Hive table in the embodiment of the present application) from the dictionary hive_hbase, which is the mapping relationship between Hive and Hbase. Columns_list is the field that needs to be verified. NewBigTableClass is a self-developed Python framework for reading Hbase data. Tab.get_vars() is used to execute data acquisition, and f is the file path where the data is written after acquisition.
[0060] In the above code in the embodiment of the present application, a dictionary need_columns is first created, which contains the columns that need to be retrieved from the Hbase table. Then, the NewBigTableClass class (for interacting with Hbase) is instantiated and the get_vars method is called to obtain data.
[0061] Step S106: Verify the data in the Hive table and the data to be verified through Python;
[0062] In some embodiments, verifying the data in the Hive table and the data to be verified through Python includes: associating the data corresponding to the user identifier user_no and the update time last_modified with the corresponding data obtained from the Hbase table, and placing the data with the same user identifier in the same row according to the user identifier for verification, wherein the verification includes: determining whether the data corresponding to the user identifier user_no and the update time last_modified are consistent with the corresponding data in the Hbase table.
[0063] Specifically, the retrieved hive_table data is associated with the hbase_table data (i.e., the data corresponding to the user identifier user_no and the update time last_modified in the embodiment of the present application is associated with the corresponding data obtained from the Hbase table), so that the same user is in one row of data for subsequent verification (i.e., the data with the same user identifier is in the same row according to the user identifier in the embodiment of the present application for verification). The logic is as follows:
[0064] merged_df=pd.merge(data_df,hbase_df,on=['user_id','app_time'],how='outer',indicator=True)
[0065] Among them, pd.merge is provided by Python, data_df and hbase_df are hive_table data and hbase_table data respectively;
[0066] In the embodiment of the present application, the merge method is used to merge the Hive table data (data_df) and the Hbase table data (hbase_df) according to the user_id and app_time fields. how = 'outer' indicates an outer join, and indicator = True will add a column named _merge to the result DataFrame to indicate the source of each row of data.
[0067] After the data is prepared, the next step is to verify the data. The core logic is as follows:
[0068] def customcompy(merged_df,columns_list,flag):
[0069] temp_df=merged_df[merged_df["_merge"]=="both"]
[0070] differences = []
[0071] for index,row in temp_df.iterrows():
[0072] column=custom_compare(row,columns_list)
[0073] if column:
[0074] value1=temp_df.loc[index,column+'_x']
[0075] value2=temp_df.loc[index,column+'_y']
[0076] if flag==0:
[0077] user_id=temp_df.loc[index,'user_id']
[0078] difference={'user_id':user_id,'filed':column,'hive_value':value1,'hbase_value':value2}
[0079] differences.append(difference)
[0080] elif flag==1:
[0081] user_no=temp_df.loc[index,'user_no']
[0082] channel_id=temp_df.loc[index,'channel_id']
[0083] device_no=temp_df.loc[index,'device_no']
[0084] difference={'user_no':user_no,'channel_id':channel_id,'device_no':device_no,'filed':column,'hive_value':value1,
[0085] 'hbase_value':value2}
[0086] differences.append(difference)
[0087] else:
[0088] user_no=temp_df.loc[index,'user_no']
[0089] difference={'user_no':user_no,'filed':column,'hive_value':value1,'hbase_value':value2}
[0090]
[0091] pass
[0092] if pd.isna(data_row[cln_x])and pd.isna(data_row[cln_y]):
[0093] continue
[0094] return cln
[0095] Among them, customcompy traverses all the data in columns_list and records the fields with problems, merged_df is the merged data, and flag is the flag. The core verification logic is custom_compare, which verifies whether each field is consistent and includes part of the rule conversion verification logic.
[0096] In the embodiment of the present application, the function def customcompy() traverses the rows in merged_df and uses the custom_compare function to compare the values of the corresponding fields in the Hive and Hbase tables. If inconsistencies are found, it will record these differences.
[0097] In the embodiment of the present application, the function def custom_compare() compares each field in data_row to check whether their values in Hive and Hbase tables are consistent. It includes some specific rules, such as handling null values and numerical comparisons in specific formats.
[0098] In the customcompy function, if the field comparison finds inconsistencies, it creates a dictionary differences to record these differences and adds them to the list differences. Finally, these differences are converted into a new DataFrame differences_df.
[0099] The above code in the big data offline data quality inspection method in the embodiment of this application shows how to use Python to automate the data verification process in a big data environment, improve efficiency and reduce human errors. The specific implementation of the code may need to be adjusted according to the actual business logic and data structure.
[0100] Step S108: If the data in the Hive table in the verification result is inconsistent with the data to be verified, the inconsistent data is written into a CSV format file named after the Hive table name to facilitate problem troubleshooting.
[0101] In some of the embodiments, writing inconsistent data into a file in CSV format named after the Hive table name includes: writing data that fails the verification in the verification results into a file in CSV format named after the Hive table name, and generating an alarm message, wherein the alarm message is used to indicate the data in the Hbase table that fails the verification.
[0102] In some of the embodiments, the problem troubleshooting includes: troubleshooting files in CSV format and solving problems in the files in CSV format; and verifying the Hbase table after solving the problems until the verification passes.
[0103] Specifically, if the check fails, the incorrect data will be written into a CSV format file named after the Hive table name, which is convenient for the subsequent problem troubleshooting. After the problem is troubleshooted, steps S102 to S106 need to be performed again. Among them, if the data that fails the field check is written into the file hive_table.csv (i.e., the file in CSV format named after the Hive table name in the embodiment of the present application), and an alarm will be issued to determine the Hbase table that has not been checked; if an alarm occurs, then check the file hive_table.csv, solve the corresponding problem, and then repeat steps S102 to S106.
[0104] In summary, in combination with step S102 to step S108, Figure 2 FIG. 1 is a flow chart of another method for offline data quality inspection of big data according to an embodiment of the present invention. Figure 2 As shown, business feature data is written from the Hive database to the Hbase database; Hive database data is read through Python; Hbase database data is read through Python; determine whether the data field values are consistent, if so, complete, if not, re-check; Among them, Figure 3 FIG. 1 is a schematic diagram of a verification process for determining whether data field values are consistent in another method for offline data quality verification of big data according to an embodiment of the present invention. Figure 3 As shown, Hive data and Hbase data are passed to the Python development method to filter out fields with more or less rows and inconsistent corresponding values of the same fields according to fixed logic.
[0105] In a big data offline data quality verification method provided in an embodiment of the present application, the Hive table hive_table that needs to be verified is first obtained, and the metadata corresponding to the hive_table in the Hive library is read through the package PyHive provided by Python. It only needs to specify the Hive table name. The metadata of the obtained hive_table is stored in a list list named table_filed_list, and the elements in table_filed_list are the field names that need to be verified. Traverse all elements of table_filed_list and perform field verification. In this way, no matter what table it is, no matter how many fields the table corresponds to, the field verification can be automatically completed. Compared with the original need to write as many SQL logics as there are fields, the efficiency and accuracy are greatly improved.
[0106] With regard to the problem of the increasing number of Hive tables that need to be verified in the related art, the offline data quality verification method for big data provided in the embodiment of the present application solves the problem of many verification fields by obtaining the fields by specifying the Hive table name, and only needs to provide the Hive table name to solve the problem of the increasing number of Hive tables that need to be verified. Therefore, for the increasing number of Hive tables, only the Hive table name needs to be provided, and no other operations need to be performed. Compared with the original need to develop an SQL verification logic for adding a new table, the efficiency is greatly improved, and the subsequent maintenance cost is also very low.
[0107] For the problems of complex business scenarios existing in the related technologies, in the big data offline data quality inspection method provided by the embodiment of the present application, only the corresponding conversion logic needs to be developed for the data conversion in each type of business scenario, because the conversion logic rules are unified, and each rule does not affect each other. Even if there is no such data or rule conversion, the rule will not be hit, resulting in problems with the verification logic. Compared with the previous SQL verification, which customized the development of SQL logic for each field, this solution does not focus on specific fields, but only on rule conversion logic, and the accuracy and efficiency are maximized.
[0108] The development efficiency of the big data offline data quality inspection method provided in the embodiment of the present application is maximized. Although SQL is a basic skill that must be possessed in the field of big data, it can meet the needs of this business scenario, but its execution efficiency is very low. Every time a new table is added, SQL needs to be developed once. In addition, if there are many table fields to be verified and many data conversion logic rules, it may take half a day or even a day for one table.
[0109] The later maintenance cost of the big data offline data quality inspection method provided by the embodiment of the present application becomes very low, because the verification logic is universal, which makes the functional hierarchy of the developed Python code clear and clear, and the function of each piece of code is very clear. If there is a problem, the problem can be located very quickly. I don't want to need to check the logic of hundreds of monitoring tables, each table may have dozens or even hundreds or thousands of lines of SQL statements, if there is a problem, it is very difficult to troubleshoot, or it is also extremely difficult to modify the logic.
[0110] The error rate of the development of the big data offline data quality inspection method provided by the embodiment of the present application is close to zero. The solutions in the related art are prone to incorrect judgments that should be correct, and failure to verify errors that should be wrong. The code in the big data offline data quality inspection method provided by the embodiment of the present application is clearly layered and has a clear structure. If the same logical judgment occurs, no addition or modification is required.
[0111] In addition, the big data offline data quality inspection method of the embodiment of the present application can be implemented by a computer device. The components of the computer device may include but are not limited to a processor and a memory storing computer program instructions.
[0112] In some embodiments, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0113] In some embodiments, the memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a specific embodiment, the memory is a non-volatile memory. In a specific embodiment, the memory includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.
[0114] The memory may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor.
[0115] The processor implements any one of the big data offline data quality inspection methods in the above embodiments by reading and executing computer program instructions stored in the memory.
[0116] In some embodiments, the computer device may further include a communication interface and a bus, wherein the processor, the memory, and the communication interface are connected via the bus and communicate with each other.
[0117] The communication interface is used to realize the communication between each unit, device, unit and / or device in the embodiment of the present application. The communication interface can also realize data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.
[0118] The bus includes hardware, software or both, and couples the components of the computer device to each other. The bus includes but is not limited to at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
[0119] The computer device can execute the big data offline data quality inspection method in the embodiment of the present application.
[0120] In addition, in combination with the big data offline data quality inspection method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the big data offline data quality inspection methods in the above embodiment is implemented.
[0121] The present invention adopts the above technical scheme, reads the user identifier user_no and the update time last_modified in the Hive table through Python; obtains the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; verifies the data in the Hive table and the data to be verified through Python; if the data in the Hive table and the data to be verified are inconsistent in the verification result, the inconsistent data is written into a CSV format file named after the Hive table name for troubleshooting. Compared with the prior art, the present invention has the following technical effects: improving the automation level, accuracy and maintainability of data verification.
[0122] Example 2
[0123] An exemplary embodiment of the present invention is as follows Figure 4 As shown, Figure 4 Schematic diagram of a big data offline data quality inspection device according to an embodiment of the present invention. The big data offline data quality inspection device provided by the embodiment of the present application includes:
[0124] A reading module 42 is used to read the user identifier user_no and update time last_modified in the Hive table through Python; an acquisition module 44 is used to obtain the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; a verification module 46 is used to verify the data in the Hive table and the data to be verified through Python; a troubleshooting module 48 is used to write the inconsistent data into a CSV format file named after the Hive table name if the data in the Hive table and the data to be verified are inconsistent in the verification result, so as to facilitate problem troubleshooting.
[0125] In some of the embodiments, a big data offline data quality inspection device provided by the embodiments of the present application also includes: a writing module, which is used to write the data of the Hive table into the Hbase table through the ImportTsv method before reading the user identifier user_no and update time last_modified in the Hive table through Python.
[0126] In some embodiments, the reading module includes: a reading unit, used to read the user identifier user_no and the update time last_modified in the Hive table through Python; a writing unit, used to write the user identifier user_no and the update time last_modified into a CSV format file, and store the column names of the data corresponding to the user identifier user_no and the update time last_modified into the field list columns_list.
[0127] The present invention adopts the above technical scheme, reads the user identifier user_no and the update time last_modified in the Hive table through Python; obtains the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; verifies the data in the Hive table and the data to be verified through Python; if the data in the Hive table and the data to be verified are inconsistent in the verification result, the inconsistent data is written into a CSV format file named after the Hive table name for troubleshooting. Compared with the prior art, the present invention has the following technical effects: improving the automation level, accuracy and maintainability of data verification.
[0128] The present invention adopts the above technical solution, and has the following technical effects compared with the prior art:
[0129] 1) Greatly improved development efficiency: The versatility and automation of Python code reduces the need for SQL development and significantly improves development efficiency.
[0130] 2) Reduced maintenance costs: Due to the universality of the verification logic and the clear code structure, the subsequent maintenance costs are greatly reduced, and problem location and logic modification become easier.
[0131] 3) Reduced error rate: The new solution has a clear structure, which reduces the possibility of misjudgment and improves the accuracy of verification.
[0132] The above description is only a preferred embodiment of the present invention, and does not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.
Claims
1. A big data offline data quality inspection method, characterized in that: include: Read the user ID user_no and update time last_modified in the Hive table through Python; Obtain the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; Verify the data in the Hive table and the data to be verified through Python; If the data in the Hive table in the verification result is inconsistent with the data to be verified, the inconsistent data is written to a CSV format file named after the Hive table name to facilitate troubleshooting.
2. The big data offline data quality inspection method according to claim 1 is characterized in that: Before reading the user ID user_no and the update time last_modified in the Hive table through Python, the method further includes: The data of the Hive table is written into the Hbase table through the ImportTsv method.
3. The big data offline data quality inspection method according to claim 1 or 2, characterized in that: The method of reading the user ID user_no and update time last_modified in the Hive table through Python includes: Read the user ID user_no and the update time last_modified in the Hive table through Python; The user identifier user_no and the update time last_modified are written into a CSV format file, and the column names of the data corresponding to the user identifier user_no and the update time last_modified are stored in the field list columns_list.
4. The method for offline data quality inspection of big data according to claim 3 is characterized in that: The obtaining of the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified includes: According to the mapping relationship between the Hive library and the Hbase library, obtain the name of the Hbase table corresponding to the Hive table from the dictionary of the mapping relationship; Obtain corresponding data from the Hbase table through Python according to the name of the Hbase table, the user identifier user_no and the update time last_modified.
5. The method for offline data quality inspection of big data according to claim 4 is characterized in that: Verifying the data in the Hive table and the data to be verified by Python includes: The data corresponding to the user identifier user_no and the update time last_modified are associated with the corresponding data obtained from the Hbase table, and the data with the same user identifier are placed in the same row according to the user identifier for verification, wherein the verification includes: determining whether the data corresponding to the user identifier user_no and the update time last_modified are consistent with the corresponding data in the Hbase table.
6. The big data offline data quality inspection method according to claim 5 is characterized in that: Writing the inconsistent data into a CSV file named after the Hive table name includes: The data that failed the verification in the verification result is written into a file in CSV format named after the Hive table name, and an alarm message is generated, wherein the alarm message is used to indicate the data that failed the verification in the Hbase table.
7. The big data offline data quality inspection method according to claim 6 is characterized in that: The troubleshooting includes: Check the file in the CSV format and solve the problems in the file in the CSV format; Verify the Hbase table after solving the problem until it passes the verification.
8. A big data offline data quality inspection device, characterized in that: include: The read module is used to read the user ID user_no and update time last_modified in the Hive table through Python; An acquisition module, used to acquire the corresponding data to be verified in the Hbase table according to the user identifier user_no and the update time last_modified; A verification module, used to verify the data in the Hive table and the data to be verified through Python; The troubleshooting module is used to write the inconsistent data into a CSV file named after the Hive table name if the data in the Hive table in the verification result is inconsistent with the data to be verified, so as to troubleshoot the problem.
9. The big data offline data quality inspection device according to claim 8, characterized in that: The device also includes: The writing module is used to write the data of the Hive table into the Hbase table through the ImportTsv method before reading the user identifier user_no and the update time last_modified in the Hive table through Python.
10. The big data offline data quality inspection device according to claim 8 or 9, characterized in that: The reading module comprises: A reading unit, used for reading the user identifier user_no and the update time last_modified in the Hive table through Python; The writing unit is used to write the user identifier user_no and the update time last_modified into a CSV format file, and store the data corresponding to the user identifier user_no and the update time last_modified into a to-be-checked list columns_list.
