A method and device for processing multi-source data

Abstract

The present invention discloses a multi-source data processing method and device. A specific implementation of the method includes: according to a data loading configuration file, data from multiple data sources are uniformly connected to a first database to be stored in the form of a data table; according to a data review configuration file, the data table of the data connected from each of the multiple data sources is reviewed to generate a data review result; according to a data comparison configuration file, the reviewed data is compared to generate a data comparison result; and according to a data output configuration file, the data review result or the data comparison result is output. This implementation method realizes rapid analysis of data from a single data source by reviewing the data according to the data review configuration file; and by comparing the data from multiple data sources according to the data comparison configuration file, the difference analysis of the data from multiple data sources is realized, so that users can quickly identify data differences, thereby better meeting business needs.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-source data processing method and device. Background Art

[0002] As new technologies such as big data and artificial intelligence rapidly enhance their strategic position in the country's Internet construction, more and more applications and scenarios are built on big data platforms, and there are many designs and solutions for data access, processing, conversion, cleaning and application. Faced with a variety of complex data, as time goes by, the amount of data is getting larger and larger. How to quickly form an understanding of various data has become a pain point in data analysis. In particular, the data sources are complex and the structure is quite different, making it difficult to compare and analyze various data.

[0003] There are some patents related to multi-source data processing and big data comparison in the market, but they are either for the specific use of a certain tool, such as "A big data comparison method and device based on kettle", which emphasizes the use of kettle for data comparison, regardless of the data being compared and the technical background; or the processing means and solutions are different, "A preprocessing method for multi-source heterogeneous big data" proposes to connect multi-source heterogeneous data to HDFS and then perform some entity recognition and de-redundancy.

[0004] In the current construction of the trading subsystem of CCB's large asset management system, there is a large demand for product quotation information, basic information, valuation information and other data every day. The current market situation is that there are multiple data suppliers providing relevant data, such as Wande, Caihui, Juyuan, etc. The data of each company is similar, but the classification, naming and quality are different, and the storage databases are also different. This is very unfriendly to data users, and it is often impossible to determine the accuracy of the data. For trading systems that have high data quality requirements, it causes many inconveniences. Summary of the invention

[0005] In view of this, the embodiment of the present invention adopts big data technology to uniformly access multi-source heterogeneous inputs, then performs data quality analysis on the accessed data and compares and analyzes the differences of multi-source data based on configuration files, supports users to customize their data tables and fields of interest according to their needs, and provides interface access so that users can quickly identify data differences, analyze data accuracy, and support correction of erroneous data, thereby better meeting business needs.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a multi-source data processing method is provided.

[0007] A multi-source data processing method according to an embodiment of the present invention includes:

[0008] According to the data loading configuration file, the data from the multiple data sources are uniformly connected to the first database to be stored in the form of a data table;

[0009] Inspecting the data table of the data accessed from each of the plurality of data sources according to the data inspection configuration file to generate a data inspection result;

[0010] Comparing the inspected data according to the data comparison configuration file to generate a data comparison result; and

[0011] The data inspection result or the data comparison result is output according to the data output configuration file.

[0012] Optionally, the data source is selected from one or more of the following: Hive, Hbase, HDFS, MySQL, Oralce and Kafka.

[0013] Optionally, each of the multiple data sources loads data into the first database by calling a corresponding loader based on a corresponding data loading configuration file.

[0014] Optionally, the data table includes one or more of the following: the data source of the data, the data type of the data source, the ID of the data table and its name, storage location, data supply frequency, data capacity, field name and its ID, field type and field length.

[0015] Optionally, inspecting a data table of data accessed from each of the plurality of data sources according to the data inspection configuration file to generate a data inspection result further includes:

[0016] Loading a data table of data accessed from each of the plurality of data sources from the first database to register as a temporary table;

[0017] Load the data inspection configuration file to read the fields to be inspected and the indicators of interest;

[0018] Analyzing the temporary table based on the field to be inspected and the index of interest to generate the data inspection result; and

[0019] The data inspection result is output as a data quality report of the data source.

[0020] Optionally, after viewing the data table of the data accessed from each of the plurality of data sources according to the data viewing configuration file to generate the data viewing result, the method further includes:

[0021] De-duplication and cleaning of the data inspection results; and

[0022] The deduplicated and cleaned data inspection results are stored in the first database.

[0023] Optionally, the data viewing configuration file includes SQL configuration, indicator configuration, UDF configuration and field configuration.

[0024] Optionally, the review further includes:

[0025] At the table level of the data table, it supports viewing the library name, table name, statistical frequency, and increase / full amount.

[0026] Optionally, the review further includes:

[0027] At the row level of the data table, it supports viewing the number of rows, separator exceptions, blank rows, header missing, and sensitive row information.

[0028] Optionally, the review further includes:

[0029] At the field level of the data table, it supports checking the field name, field type error, field non-empty, field code value error, field length error, and field value range error.

[0030] Optionally, the data inspection configuration file also includes a data inspection result output configuration, wherein the data inspection result includes a data table size, a number of rows, a number of error fields, a field missing rate, and whether to include details based on the data inspection result output configuration.

[0031] Optionally, comparing the inspected data according to the data comparison configuration file to generate a data comparison result further includes:

[0032] Based on the data comparison configuration file, a first data table of a first data source and a second data table of a second data source among the multiple data sources are loaded from the first database to construct a first partition and bucket table and a second partition and bucket table respectively; and

[0033] Associating the first partition and bucket table with the second partition and bucket table to construct a difference processing chain according to the data comparison configuration file;

[0034] Passing the associated first partition and bucket table and second partition and bucket table through the difference processing chain to generate the data comparison result; and

[0035] The data comparison result is stored in a second database based on the data comparison configuration file.

[0036] Optionally, the data comparison configuration file includes comparison source table configuration, comparison field configuration, partition and bucket configuration, comparison content configuration, comparison storage configuration, comparison difference configuration, alarm template configuration and output template configuration.

[0037] Optionally, the difference processing chain includes null value difference, data precision difference, date type difference and noun difference.

[0038] Optionally, the noun differences are identified using a financial graph.

[0039] Optionally, using the financial graph to identify the noun difference includes:

[0040] Establish nodes and corresponding edge relationships based on multiple nouns with the same meaning to form a financial graph;

[0041] Compare the nodes formed by each pair of nouns with the nodes of the financial graph;

[0042] If there is an identical node or an edge relationship exists between the two nodes, then the compared nouns are determined to be identical fields; and

[0043] If there is no identical node and there is no edge relationship between the two nodes, it is determined that the compared nouns are not identical fields.

[0044] Optionally, the method further comprises:

[0045] A data comparison report containing warning information is output based on the alarm template configuration and the output template configuration.

[0046] Optionally, associating the first partition and bucket table with the second partition and bucket table further includes:

[0047] Designing a first data table of a first data source and a second data table of a second data source among the multiple data sources into a first partition table and a second partition table according to time settings for separate storage;

[0048] Splitting the fields in the first partition table and the second partition table into bucket keys to form a first bucket temporary table and a second bucket temporary table; and

[0049] The bucket key in the first bucket temporary table is directly associated with the same bucket key in the second bucket temporary table for data comparison.

[0050] Optionally, if there is an error in the data comparison result, the data is corrected to eliminate the error.

[0051] Optionally, each comparison result in the data comparison result is stored in the second database in a bit-by-bit order according to a configuration comparison order.

[0052] Optionally, the method further comprises:

[0053] The data comparison results are checked to determine whether there are errors in the data comparison results.

[0054] Optionally, if there is an error in the data comparison result, the data is corrected to eliminate the error.

[0055] To achieve the above objective, according to another aspect of an embodiment of the present invention, a multi-source data processing device is provided.

[0056] The multi-source data processing device of the embodiment of the present invention includes:

[0057] A data unified access module, used to uniformly access data from multiple data sources into the first database according to the data loading configuration file so as to be stored in the form of a data table;

[0058] A data inspection module, configured to inspect a data table of data accessed from each of the plurality of data sources according to a data inspection configuration file to generate a data inspection result;

[0059] A data comparison module, for comparing the inspected data according to a data comparison configuration file to generate a data comparison result; and

[0060] The data output module is used to output the data inspection result or the data comparison result according to the data output configuration file.

[0061] Optionally, the data inspection module is further used to:

[0062] Loading a data table of data accessed from each of the plurality of data sources from the first database to register as a temporary table;

[0063] Load the data inspection configuration file to read the fields to be inspected and the indicators of interest;

[0064] Analyzing the temporary table based on the field to be inspected and the index of interest to generate the data inspection result; and

[0065] The data inspection result is output as a data quality report of the data source.

[0066] Optionally, the data comparison module is further used for:

[0067] Based on the data comparison configuration file, a first data table of a first data source and a second data table of a second data source among the multiple data sources are loaded from the first database to construct a first partition and bucket table and a second partition and bucket table respectively; and

[0068] Associating the first partition and bucket table with the second partition and bucket table to construct a difference processing chain according to the data comparison configuration file;

[0069] Passing the associated first partition and bucket table and second partition and bucket table through the difference processing chain to generate the data comparison result; and

[0070] The data comparison result is stored in a second database based on the data comparison configuration file.

[0071] To achieve the above objective, according to another aspect of an embodiment of the present invention, a multi-source data processing electronic device is provided.

[0072] A multi-source data processing electronic device according to an embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a multi-source data processing method according to an embodiment of the present invention.

[0073] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0074] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a multi-source data processing method according to an embodiment of the present invention is implemented.

[0075] An embodiment of the above invention has the following advantages or beneficial effects: by viewing the data according to the data viewing configuration, the data of a single data source can be quickly analyzed; and by comparing the data of multiple data sources according to the data comparison configuration, the difference analysis of the data of multiple data sources can be realized, thereby supporting users to customize the data tables and fields of interest of users according to their needs, and providing interface access so that users can quickly identify data differences, analyze data accuracy, and support correction of erroneous data, thereby better meeting business needs.

[0076] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0078] Figure 1 is a schematic diagram of main steps of a multi-source data processing method according to an embodiment of the present invention;

[0079] Figure 2 is a schematic diagram of main steps of another multi-source data processing method according to an embodiment of the present invention;

[0080] Figure 3is a schematic diagram of main modules of a multi-source data processing device according to an embodiment of the present invention;

[0081] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0082] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0084] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0085] Figure 1 It is a schematic diagram of main steps of a multi-source data processing method according to an embodiment of the present invention.

[0086] like Figure 1 As shown, a multi-source data processing method according to an embodiment of the present invention mainly includes the following steps:

[0087] Step S101: data from multiple data sources are uniformly connected to a first database according to a data loading configuration file to be stored in the form of a data table.

[0088] In one example, the multiple data sources may be data suppliers that can provide various data. The multiple data sources may be selected from one or more of Hive, Hbase, HDFS, MySQL, Oracle, and Kafka, but the present invention is not limited thereto. Each of the multiple data sources loads data to access the first database by calling a corresponding loader based on a corresponding data loading configuration file.

[0089] In one example, the data loading configuration file can be customized by the user according to the needs. The data for unified access can be stored as a data table in the first database according to the data loading configuration file. The data table can be a two-dimensional table or other forms, etc. For example, if the data for unified access comes from a relational database and is therefore in the form of a two-dimensional table, the data can be stored directly in the first database. For example, if the data for unified access is in a file format, the access data can be encapsulated through a data loading configuration file and then stored in the first database in the form of a data table. The data table includes one or more of the following: the data source of the data, the data type of the data source, the ID of the data table and its name, storage location, data supply frequency, data capacity, field name and its ID, field type and field length.

[0090] In one example, the first database may be a Hive database known to those skilled in the art. Hive is a data warehouse tool based on Hadoop that is used to extract, transform, and load data. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution.

[0091] Step S102: Viewing a data table of data accessed from each of the plurality of data sources according to a data viewing configuration file to generate a data viewing result.

[0092] Data inspection is to monitor the quality of data from a single data source, analyze the number of fields in the data table, the overall missing rate, the missing rate of each field, outliers, etc., to help users quickly understand the overall picture of the accessed data, so as to control the data sample situation of the data table as a whole.

[0093] In one example, the data viewing configuration file includes SQL configuration, index configuration, UDF configuration and field configuration. Table 1 shows details of an exemplary data viewing configuration file for viewing a data table of data in each of a plurality of data sources.

[0094]

[0095] Table 1

[0096] In one example, the data inspection configuration may support inspection of library name, table name, statistical frequency, and increment / full quantity at the table level of the data table. This is for illustrative purposes only and the present invention is not limited thereto. For example, the data inspection configuration may only support data row quantity statistics and field missing row statistics at the row level of the data table, but not support separator abnormal row statistics and sensitive information row statistics. For example, the data inspection configuration may not only support data row quantity statistics, field missing row statistics, separator abnormal row statistics, and sensitive information row statistics at the row level of the data table, but may also support other types of data row statistics.

[0097] In one example, the data inspection configuration can also support inspection of the number of rows, separator anomalies, blank rows, header missing, and sensitive row information at the row level of the data table. This is for illustrative purposes only, and the present invention is not limited thereto.

[0098] In one example, the data inspection configuration can also support inspection of field name, field type error, field non-empty, field code value error, field length error, and field value range error at the field level of the data table. This is for illustrative purposes only, and the present invention is not limited thereto.

[0099] In one example, the data inspection configuration file further includes a data inspection result output configuration, wherein the data inspection result includes a data table size, a number of rows, a number of error fields, a field missing rate, and whether to include details based on the data inspection result output configuration.

[0100] In one example, after viewing the data table of the data accessed from each of the multiple data sources according to the data viewing configuration file to generate the data viewing results, it can also include: deduplicating and cleaning the data viewing results; and storing the deduplicated and cleaned data viewing results in the first database.

[0101] Step S103: Compare the inspected data according to the data comparison configuration file to generate a data comparison result.

[0102] Among them, the data comparison configuration file may include comparison source table configuration, comparison field configuration, partition and bucket configuration, comparison content configuration, comparison storage configuration, comparison difference configuration, alarm template configuration and output template configuration.

[0103] Each comparison result in the data comparison result is stored in the second database in a bit-by-bit order according to the configuration comparison order.

[0104] The user can use SparkSQL to compare the fields of interest in the selected data table to be compared based on the data comparison configuration file to generate data comparison results. Spark is a fast and general computing engine designed for large-scale data processing, and SparkSQL is a Spark component used to process structured data.

[0105] The data comparison result is stored in the second database according to the comparison storage configuration. The second database may be an Hbase database, which is a distributed column-oriented open source database.

[0106] Each comparison result in the data comparison result is stored in the second database in a bit position according to the order of configuration comparison. In one example, each comparison result is represented by 1 or 0, and the comparison result is encoded and stored in the second database according to the bit position, which can save a lot of storage space. For example, the number of comparison fields is first counted, and then the number of bits is initialized according to the number of fields, and the comparison results are stored in the bit position in order, which actually greatly reduces the storage space of the comparison results.

[0107] Among them, users can view the comparison results and their details through the data comparison interface.

[0108] The user can determine the accuracy of the data based on the data comparison result. If there is an error in the data, the error in the data is corrected to store the modified data in the first database.

[0109] Step S104: outputting the data inspection result or the data comparison result according to the data output configuration file.

[0110] Furthermore, according to an embodiment of the present invention, by viewing the data according to the data viewing configuration, rapid analysis of data from a single data source is achieved; and by comparing the data from multiple data sources according to the data comparison configuration, difference analysis of data from multiple data sources is achieved, thereby supporting users to customize their data tables and fields of interest according to their needs, and providing interface access so that users can quickly identify data differences, analyze data accuracy, and support correction of erroneous data, thereby better meeting business needs.

[0111] Figure 2 FIG. 1 is a schematic diagram of the main steps of another multi-source data processing method according to an embodiment of the present invention. Figure 2 As shown, the main steps of another multi-source data processing method provided by an embodiment of the present invention may include the following steps:

[0112] Step S201: according to the data loading configuration file, data from multiple data sources are uniformly connected to the first database to be stored in the form of a data table.

[0113] Step S202: Loading a data table of data accessed from each of the plurality of data sources from the first database to register it as a temporary table.

[0114] Step S203: Load the data viewing configuration file to read the fields to be viewed and the indicators of interest.

[0115] Step S204: Analyze the temporary table based on the field to be inspected and the indicator of interest to generate the data inspection result.

[0116] Step S205: Output the data inspection result as a data quality report of the data source.

[0117] Step S206: Based on the data comparison configuration file, a first data table of a first data source and a second data table of a second data source among the multiple data sources are loaded from the first database to construct a first partition and bucket table and a second partition and bucket table respectively.

[0118] Step S207: Associating the first partition and bucket table with the second partition and bucket table to construct a difference processing chain according to the data comparison configuration file.

[0119] In one example, associating the first partition and bucket tables with the second partition and bucket tables may also include: designing the first data table of the first data source and the second data table of the second data source in the multiple data sources into the first partition table and the second partition table according to time settings for separate storage; splitting the fields in the first partition table and the second partition table into bucket keys to form a first bucket temporary table and a second bucket temporary table; and directly associating the bucket keys in the first bucket temporary table with the same bucket keys in the second bucket temporary table for data comparison.

[0120] In one example, the difference processing chain may include null value difference, data precision difference, date type difference, and noun difference.

[0121] In one example, the noun difference can be identified using a financial graph. Specifically, identifying the noun difference using a financial graph may include: establishing nodes and corresponding edge relationships based on multiple nouns of the same meaning to form a financial graph; comparing the nodes formed by each pair of nouns with the nodes of the financial graph; if there are identical nodes or an edge relationship between the two nodes, determining that the paired nouns are identical fields; and if there are no identical nodes and no edge relationship between the two nodes, determining that the paired nouns are different fields.

[0122] In one example, the method further includes: outputting a data comparison report containing warning information based on the alarm template configuration and the output template configuration.

[0123] Step S208: The associated first partition and bucket table and second partition and bucket table are passed through the difference processing chain to generate the data comparison result.

[0124] Step S209: storing the data comparison result in a second database based on the data comparison configuration file.

[0125] Step S210: Outputting the data inspection result or the data comparison result according to the data output configuration.

[0126] It can be seen from a multi-source data processing method according to an embodiment of the present invention that by viewing the data according to the data viewing configuration, a rapid analysis of the data of a single data source is achieved; and by comparing the data of multiple data sources according to the data comparison configuration, a difference analysis of the data of multiple data sources is achieved, thereby supporting users to customize their data tables and fields of interest according to their needs, and providing interface access so that users can quickly identify data differences, analyze data accuracy, and support correction of erroneous data, thereby better meeting business needs.

[0127] Figure 3 is a schematic diagram of a multi-source data processing device according to an embodiment of the present invention.

[0128] like Figure 3 As shown, a multi-source data processing device 300 according to an embodiment of the present invention includes: a data receiving module 301, a data inspection module 302, a data comparison module 303 and a data output module 304; wherein,

[0129] The data unified access module 301 is used to uniformly access data from multiple data sources into the first database according to the data loading configuration file so as to be stored in the form of a data table;

[0130] A data inspection module 302, configured to inspect a data table of data accessed from each of the plurality of data sources according to a data inspection configuration file to generate a data inspection result;

[0131] A data comparison module 303, for comparing the inspected data according to the data comparison configuration file to generate a data comparison result; and

[0132] The data output module 304 is used to output the data inspection result or the data comparison result according to the data output configuration file.

[0133] In one embodiment of the present invention, the data viewing module 302 is further used to:

[0134] Loading a data table of data accessed from each of the plurality of data sources from the first database to register as a temporary table;

[0135] Load the data inspection configuration file to read the fields to be inspected and the indicators of interest;

[0136] Analyzing the temporary table based on the field to be inspected and the index of interest to generate the data inspection result; and

[0137] The data inspection result is output as a data quality report of the data source.

[0138] In one embodiment of the present invention, the data comparison module 303 is further used for:

[0139] Based on the data comparison configuration file, a first data table of a first data source and a second data table of a second data source among the multiple data sources are loaded from the first database to construct a first partition and bucket table and a second partition and bucket table respectively; and

[0140] Associating the first partition and bucket table with the second partition and bucket table to construct a difference processing chain according to the data comparison configuration file;

[0141] Passing the associated first partition and bucket table and second partition and bucket table through the difference processing chain to generate the data comparison result; and

[0142] The data comparison result is stored in a second database based on the data comparison configuration file.

[0143] It can be seen from a multi-source data processing device according to an embodiment of the present invention that by viewing the data according to the data viewing configuration, a rapid analysis of the data of a single data source is achieved; and by comparing the data of multiple data sources according to the data comparison configuration, a difference analysis of the data of multiple data sources is achieved, thereby supporting users to customize the data tables and fields of interest of users according to their needs, and providing interface access, so that users can quickly identify data differences, analyze data accuracy, and support correction of erroneous data, thereby better meeting business needs.

[0144] Figure 4 An exemplary system architecture 400 is shown to which a multi-source data processing method or a multi-source data processing device according to an embodiment of the present invention can be applied.

[0145] like Figure 4As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0146] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0147] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0148] Server 405 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 401, 402, and 403. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (such as target push information and product information) to the terminal device.

[0149] It should be noted that the multi-source data processing method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the multi-source data processing device is generally disposed in the server 405 .

[0150] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0151] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0152] like Figure 5As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0153] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0154] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0155] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, they may be described as follows: a processor includes a data receiving module, a data inspection module, a data comparison module, and a data output module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, a data comparison module may also be described as a "module for comparing data."

[0158] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: according to the data loading configuration file, the data from multiple data sources are uniformly connected to the first database to be stored in the form of a data table; according to the data inspection configuration file, the data table of the data connected from each of the multiple data sources is inspected to generate a data inspection result; according to the data comparison configuration file, the inspected data is compared to generate a data comparison result; and according to the data output configuration file, the data inspection result or the data comparison result is output.

[0159] According to the technical solution of the embodiment of the present invention, by reviewing the data according to the data review configuration, the data of a single data source can be quickly analyzed; and by comparing the data of multiple data sources according to the data comparison configuration, the difference analysis of the data of multiple data sources can be realized, thereby supporting users to customize the data tables and fields of interest of users according to their needs, and providing interface access, so that users can quickly identify data differences, analyze data accuracy, and support correction of erroneous data, thereby better meeting business needs.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-source data processing method, characterized in that: include: According to the data loading configuration file, the data from the multiple data sources are uniformly connected to the first database to be stored in the form of a data table; Inspecting the data table of the data accessed from each of the plurality of data sources according to the data inspection configuration file to generate a data inspection result; Invoke a big data distributed computing engine to compare the inspected data according to a data comparison configuration file to generate a data comparison result; wherein the fields of interest in the inspected data are compared according to the data comparison configuration file; and Output the data inspection result or the data comparison result according to the data output configuration file; Comparing the inspected data according to the data comparison configuration file to generate a data comparison result also includes: loading the first data table of the first data source and the second data table of the second data source among the multiple data sources from the first database based on the data comparison configuration file to construct a first partition and bucket table and a second partition and bucket table respectively; and associating the first partition and bucket table with the second partition and bucket table to construct a difference processing chain according to the data comparison configuration file; passing the associated first partition and bucket table and the second partition and bucket table through the difference processing chain to generate the data comparison result; and storing the data comparison result in the second database based on the data comparison configuration file.

2. The method according to claim 1, characterized in that The data source is selected from one or more of the following: Hive, Hbase, HDFS, MySQL, Oracle, and Kafka.

3. The method according to claim 1, characterized in that Each of the plurality of data sources loads data into the first database by calling a corresponding loader based on a corresponding data loading configuration file.

4. The method according to claim 3, characterized in that The data table includes one or more of the following: the data source of the data, the data type of the data source, the ID and name of the data table, storage location, data supply frequency, data capacity, field name and ID, field type and field length.

5. The method according to claim 1, characterized in that Inspecting the data table of the data accessed from each of the plurality of data sources according to the data inspection configuration file to generate a data inspection result also includes: Loading a data table of data accessed from each of the plurality of data sources from the first database to register as a temporary table; Load the data inspection configuration file to read the fields to be inspected and the indicators of interest; Analyzing the temporary table based on the field to be inspected and the index of interest to generate the data inspection result; and The data inspection result is output as a data quality report of the data source.

6. The method according to claim 1, characterized in that After viewing the data table of the data accessed from each of the plurality of data sources according to the data viewing configuration file to generate the data viewing result, the method further includes: De-duplication and cleaning of the data inspection results; and The deduplicated and cleaned data inspection results are stored in the first database.

7. The method according to claim 1, characterized in that The data viewing configuration file includes SQL configuration, indicator configuration, UDF configuration and field configuration.

8. The method according to claim 5, characterized in that The review also includes: At the table level of the data table, it supports viewing the library name, table name, statistical frequency, and increase / full amount.

9. The method according to claim 5, characterized in that The review also includes: At the row level of the data table, it supports viewing the number of rows, separator exceptions, blank rows, header missing, and sensitive row information.

10. The method according to claim 5, characterized in that The review also includes: At the field level of the data table, it supports checking the field name, field type error, field non-empty, field code value error, field length error, and field value range error.

11. The method according to claim 5, characterized in that The data inspection configuration file also includes a data inspection result output configuration, wherein the data inspection result includes a data table size, a number of rows, a number of error fields, a field missing rate, and whether to include details based on the data inspection result output configuration.

12. The method according to claim 1, characterized in that The data comparison configuration file includes comparison source table configuration, comparison field configuration, partition and bucket configuration, comparison content configuration, comparison storage configuration, comparison difference configuration, alarm template configuration and output template configuration.

13. The method according to claim 1, characterized in that The difference processing chain includes null value difference, data precision difference, date type difference and noun difference.

14. The method according to claim 13, characterized in that The noun differences are identified using financial graphs.

15. The method according to claim 14, characterized in that Using financial graphs to identify these noun differences includes: Establish nodes and corresponding edge relationships based on multiple nouns with the same meaning to form a financial graph; Compare the nodes formed by each pair of nouns with the nodes of the financial graph; If there is an identical node or an edge relationship exists between the two nodes, then the compared nouns are determined to be identical fields; and If there is no identical node and there is no edge relationship between the two nodes, it is determined that the compared nouns are not identical fields.

16. The method according to claim 12, characterized in that Also includes: A data comparison report containing warning information is output based on the alarm template configuration and the output template configuration.

17. The method according to claim 1, characterized in that Associating the first partition and bucket table with the second partition and bucket table further includes: Designing a first data table of a first data source and a second data table of a second data source among the multiple data sources into a first partition table and a second partition table according to time settings for separate storage; Splitting the fields in the first partition table and the second partition table into bucket keys to form a first bucket temporary table and a second bucket temporary table; and The bucket key in the first bucket temporary table is directly associated with the same bucket key in the second bucket temporary table for data comparison.

18. The method according to claim 1, characterized in that Each comparison result in the data comparison result is stored in the second database in a bit-by-bit order according to the configuration comparison order.

19. The method according to claim 1, characterized in that Also includes: The data comparison results are checked to determine whether there are errors in the data comparison results.

20. The method according to claim 19, characterized in that If there is an error in the data comparison result, the data is corrected to eliminate the error.

21. A multi-source data processing device, characterized in that: include: A data unified access module, used to uniformly access data from multiple data sources into the first database according to the data loading configuration file so as to be stored in the form of a data table; A data inspection module, configured to inspect a data table of data accessed from each of the plurality of data sources according to a data inspection configuration file to generate a data inspection result; wherein the fields of interest in the inspected data are compared according to a data comparison configuration file; A data comparison module, used to call the big data distributed computing engine to compare the inspected data according to the data comparison configuration file to generate a data comparison result; and A data output module, used to output the data inspection result or the data comparison result according to a data output configuration file; The data comparison module is further used to: load the first data table of the first data source and the second data table of the second data source among the multiple data sources from the first database based on the data comparison configuration file to construct a first partition and bucket table and a second partition and bucket table respectively; and associate the first partition and bucket table with the second partition and bucket table to construct a difference processing chain according to the data comparison configuration file; pass the associated first partition and bucket table and the second partition and bucket table through the difference processing chain to generate the data comparison result; and store the data comparison result in the second database based on the data comparison configuration file.

22. The multi-source data processing device according to claim 21, characterized in that: The data inspection module is also used for: Loading a data table of data accessed from each of the plurality of data sources from the first database to register as a temporary table; Load the data inspection configuration file to read the fields to be inspected and the indicators of interest; Analyzing the temporary table based on the field to be inspected and the index of interest to generate the data inspection result; and The data inspection result is output as a data quality report of the data source.

23. A multi-source data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 20.

24. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 20 is implemented.

Images