A method to change Hive data based on index

Abstract

The present invention particularly relates to a method for changing Hive data based on an index. The method for changing Hive data based on an index obtains the primary key Keys of the stock data HData and the primary key Keys of the incremental data SData through Spark; Spark obtains the file path of the corresponding historical stock data HData from the HBase database index table according to the primary key Keys of the incremental data SData; then, Spark merges the incremental data SData with the corresponding historical stock data HData according to the primary key Keys and the flag bit to form a new file; finally, Spark marks whether the file is valid according to the primary key Keys and the file path, and marks it invalid if it is invalid. The method for changing Hive data based on an index uses the characteristics of the HBase index table recording the file location and Spark's fast file merging to quickly read and locate the file involved in the modification, realizes the merging of the newly added data and the historical stock data, and thus realizes the incremental update of the Hive data.

Description

Technical Field

[0001] The present invention relates to the technical field of big data computing applications, and in particular to a method for modifying Hive data based on an index. Background Art

[0002] The construction of a new generation of information technology infrastructure, the development of the digital economy, and the industrial Internet are the concepts that are currently receiving the most attention. These concepts involve all aspects that cannot be separated from the support of big data. For example, 5G networks, data centers, artificial intelligence, etc. are all driven by technological innovation and based on information networks to provide support for social digital transformation, intelligent upgrades, and basic services. These technologies are based on data fusion and big data integration, and based on artificial intelligence and machine learning, they provide data foundation and computing power support for enterprises, governments, education, and other industries. How to collect and store data to ensure that subsequent data fusion and calculations provide complete data are important technical means to achieve the above technologies.

[0003] Hadoop is the most mainstream ecological framework for big data. Based on Hadoop, Hive, HBase, Spark and other excellent frameworks for big data storage and computing have been derived. Building a data lake based on the Hadoop ecosystem is the current mainstream big data technology trend. Its advantages of being distributed, fault-tolerant and easy to expand make it stand out in the big data field.

[0004] However, the current mainstream data storage and computing of Hadoop are based on the premise that data remains unchanged, and provide data processing services that take a long time. For example, data storage based on Hive cannot be changed, or changed by partition, which limits its scope of use. For example, in the construction of big data integration, all business libraries must be collected in the big data center to open up the connectivity between data and make a copy of the data available across the entire network. Break through industry barriers, regional barriers, and even department barriers. However, there is an important problem in this scenario: system construction processing can collect historical data at one time, but how to collect the incremental data of subsequent business departments into the lake, so as to achieve accurate data entry into the lake and ensure data consistency. Because the main problem of using HDFS and Hive as the storage engine of the data lake is that the data modified by the business department cannot be accurately updated one by one when synchronized to Hive.

[0005] In order to solve the dilemma that Hive cannot support data changes when users synchronize data from traditional databases to data lakes, resulting in the inability to synchronize data changes to the database, the present invention proposes a method for changing Hive data based on indexes. Summary of the invention

[0006] In order to overcome the defects of the prior art, the present invention provides a simple and efficient method for changing Hive data based on indexes.

[0007] The present invention is achieved through the following technical solutions:

[0008] A method for changing Hive data based on an index, characterized by comprising the following steps:

[0009] The first step is to obtain the primary keys of the existing data HData and the primary keys of the incremental data SData through Spark (computing engine);

[0010] Step 2: Spark obtains the file path of the corresponding historical stock data HData from the HBase database index table according to the primary key Keys of the incremental data SData; then, Spark merges the incremental data SData with the corresponding historical stock data HData according to the primary key Keys and the flag bit to form a new file;

[0011] In the fourth step, Spark marks whether the file is valid based on the primary key and file path. If invalid, it is marked as invalid.

[0012] In the first step, the unique primary key or joint primary key of the existing data HData table is specified. Spark obtains the value of the primary key or joint primary key column when writing data, writes the value of the joint primary key column as the primary key Keys into the HBbase database, and saves the real path of the file where the row of data is located in the HDFS (Hadoop Distributed File System) database.

[0013] In the first step, the historical stock data HData in the data source is obtained through Spark, and the primary key data set primary key Keys is obtained according to the primary key or joint primary key information; while writing the historical stock data Hdata into Hive, the file path Locals where each row of data exists is recorded, and the primary key data set primary key Keys and file path Locals are written into the HBase database; and all valid files are recorded in the Log file under a separate Logs folder.

[0014] In the second step, when the new incremental data SData needs to be updated through Spark, the HBase database is first queried for the file path of the corresponding historical stock data HData in the HDFS database based on the primary key Keys, and then the data in the file is merged with the incremental data SData to form a new file and save it in the HDFS database.

[0015] In the second step, after Spark obtains the incremental data SData, it first extracts the primary key Keys, and then searches the HBase database index table for the file Files that has saved the current row based on the primary key value, and merges the incremental data SData into the file Files; Spark reads all the data MData in the file Files, and merges all the data Mdata in the file Files with the incremental data Sdata, and the incremental data Sdata includes deleted data, modified data, and newly inserted data.

[0016] In the third step, the obsolete files are marked as obsolete in the Log file in the Log directory, and the absolute path of the new file is saved and updated in the corresponding HBase database, thereby completing the incremental update of the Hive data.

[0017] In the second step, the merged data is written into the file NewFiles, and the file name and absolute path of the corresponding primary key Keys in the HBase database are updated; the file Files is marked as discarded in the Log file, and the file NewFiles is marked as a new file.

[0018] The specific implementation process of this method of changing Hive data based on index is as follows:

[0019] (1) Spark obtains historical stock data HData from the database and obtains the value of the primary key based on the primary key information;

[0020] (2) Spark writes the historical stock data HData into Hive and records which file each row of data is written to, forming a key-value pair (key, filelocals); and records the file path filelocals in the Log directory, and sets the file path filelocals as a valid file in the log file;

[0021] (3) Spark saves the (key, fileslocal) key-value pair as a data index into the HBase database table;

[0022] (4) Spark Streaming (batch streaming (real-time) computing framework) obtains the incremental data StreamRDD (queue stream) from the message framework and obtains the primary key Keys set;

[0023] (5) Spark Streaming retrieves the file path filelocals collection that already exists before the data to be modified from the HBase database based on the primary key Keys collection;

[0024] (6) The HBase database returns the file path filelocals collection;

[0025] (7) Spark Streaming retrieves the file to be modified from Hive based on the file path filelocals collection;

[0026] (8) Return the data file set RDD that needs to be modified. Spark Streaming merges the incremental data StreamRDD and the data file set RDD, deletes data, updates data, and / or inserts data, and finally merges them into a new data file set New RDD;

[0027] (9) Spark Streaming writes the data file set NewRDD into the Hive table, updates the new file set written by the data file set NewRDD to the Log table, marks it as valid, and marks the file path filelocals set in step (7) as invalid;

[0028] (10) Update the primary key Keys and file path filelocals of the updated data to a new (keys, filelocals) key-value pair and save it to the HBase database index table.

[0029] The beneficial effect of the present invention is that the method for changing Hive data based on indexes utilizes the characteristics of HBase index table recording file locations and Spark's fast file merging, quickly locates the files involved in the modification, realizes the merging of new data and historical stock data, and thus realizes the incremental update of Hive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Attached Figure 1 The figure is a schematic diagram of a method for modifying Hive data based on an index according to the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] The method for changing Hive data based on an index includes the following steps:

[0034] The first step is to obtain the primary keys of the existing data HData and the primary keys of the incremental data SData through Spark (computing engine);

[0035] Step 2: Spark obtains the file path of the corresponding historical stock data HData from the HBase database index table according to the primary key Keys of the incremental data SData; then, Spark merges the incremental data SData with the corresponding historical stock data HData according to the primary key Keys and the flag bit to form a new file;

[0036] In the fourth step, Spark marks whether the file is valid based on the primary key and file path. If invalid, it is marked as invalid.

[0037] In the first step, the unique primary key or joint primary key of the existing data HData table is specified. Spark obtains the value of the primary key or joint primary key column when writing data, writes the value of the joint primary key column as the primary key Keys into the HBbase database, and saves the real path of the file where the row of data is located in the HDFS (Hadoop Distributed File System) database.

[0038] In the first step, the historical stock data HData in the data source is obtained through Spark, and the primary key data set primary key Keys is obtained according to the primary key or joint primary key information; while writing the historical stock data Hdata into Hive, the file path Locals where each row of data exists is recorded, and the primary key data set primary key Keys and file path Locals are written into the HBase database; and all valid files are recorded in the Log file under a separate Logs folder.

[0039] In the second step, when the new incremental data SData needs to be updated through Spark, the HBase database is first queried for the file path of the corresponding historical stock data HData in the HDFS database based on the primary key Keys, and then the data in the file is merged with the incremental data SData to form a new file and save it in the HDFS database.

[0040] In the second step, after Spark obtains the incremental data SData, it first extracts the primary key Keys, and then searches the HBase database index table for the file Files that has saved the current row based on the primary key value, and merges the incremental data SData into the file Files; Spark reads all the data MData in the file Files, and merges all the data Mdata in the file Files with the incremental data Sdata, and the incremental data Sdata includes deleted data, modified data, and newly inserted data.

[0041] In the third step, the obsolete files are marked as obsolete in the Log file in the Log directory, and the absolute path of the new file is saved and updated in the corresponding HBase database, thereby completing the incremental update of the Hive data.

[0042] In the second step, the merged data is written into the file NewFiles, and the file name and absolute path of the corresponding primary key Keys in the HBase database are updated; the file Files is marked as discarded in the Log file, and the file NewFiles is marked as a new file.

[0043] The specific implementation process of this method of changing Hive data based on index is as follows:

[0044] (1) Spark obtains historical stock data HData from the database and obtains the value of the primary key based on the primary key information;

[0045] (2) Spark writes the historical stock data HData into Hive and records which file each row of data is written to, forming a key-value pair (key, filelocals); and records the file path filelocals in the Log directory, and sets the file path filelocals as a valid file in the log file;

[0046] (3) Spark saves the (key, fileslocal) key-value pair as a data index into the HBase database table;

[0047] (4) Spark Streaming (batch streaming (real-time) computing framework) obtains the incremental data StreamRDD (queue stream) from the message framework and obtains the primary key Keys set;

[0048] (5) Spark Streaming retrieves the file path filelocals collection that already exists before the data to be modified from the HBase database based on the primary key Keys collection;

[0049] (6) The HBase database returns the file path filelocals collection;

[0050] (7) Spark Streaming retrieves the file to be modified from Hive based on the file path filelocals collection;

[0051] (8) Return the data file set RDD that needs to be modified. Spark Streaming merges the incremental data StreamRDD and the data file set RDD, deletes data, updates data, and / or inserts data, and finally merges them into a new data file set New RDD;

[0052] (9) Spark Streaming writes the data file set NewRDD into the Hive table, updates the new file set written by the data file set NewRDD to the Log table, marks it as valid, and marks the file path filelocals set in step (7) as invalid;

[0053] (10) Update the primary key Keys and file path filelocals of the updated data to a new (keys, filelocals) key-value pair and save it to the HBase database index table.

[0054] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for changing Hive data based on an index, characterized in that: The following steps are involved: The first step is to obtain the primary key set through Spark, which includes the primary key of the existing data and the primary key of the incremental data; Specify the unique primary key or joint primary key of the existing data table. When writing data, Spark obtains the value of the specified primary key or joint primary key column, writes the obtained primary key or joint primary key column value as the primary key to the HBbase database, and saves the real path of the file where the corresponding row data is located in the HDFS database. Use Spark to obtain historical stock data from the data source, and obtain the primary key set based on the primary key or joint primary key information; while writing the historical stock data into Hive, record the file path Locals where each row of data exists, and write the primary key set and file path Locals into the HBase database; And all files marked as valid are recorded in the Log file in a separate Logs folder; Step 2: Spark obtains the file path of the corresponding historical stock data from the HBase database index table according to the primary key of the incremental data; then, Spark merges the incremental data with the corresponding historical stock data according to the primary key and flag to form a new file; When Spark obtains incremental data and needs to update the incremental data to Hive, it first searches the HBase database for the file path of the corresponding historical stock data in the HDFS database based on the primary key, then merges the data in the file with the incremental data to form a new file and saves it in the HDFS database. After Spark obtains the incremental data, it first extracts the primary key of the incremental data, then searches the HBase database index table for the file Files that has saved the current row based on the primary key value, and merges the incremental data into the file Files; Spark reads all the data MData in the file Files, and merges all the data Mdata in the file Files with the incremental data, which includes deleted data, modified data, and newly inserted data; Write the merged data to the file NewFiles, and update the file name and absolute path of the corresponding primary key in the HBase database; Mark the file Files as discarded in the Log file, and mark the file NewFiles as a new file; In the third step, Spark marks whether the file is valid based on the incremental data primary key and file path. If invalid, it is marked as invalid. Mark the obsolete files as obsolete in the Log file in the Log directory, save and update the absolute path of the new files to the corresponding HBase database, and then complete the incremental update of Hive data; The specific implementation process is as follows: (1) Spark obtains historical stock data from the database and obtains the value of the primary key based on the primary key information; (2) Spark writes historical stock data into Hive and records which file each row of data is written to. It combines the primary key of each row of historical stock data to form a key-value pair of (key, filelocals); And record the file path filelocals in the Log directory, and set the file path filelocals as a valid file in the log file; (3) Spark saves the (key, filelocals) key-value pair as a data index to the HBase database table; (4) Spark Streaming obtains the incremental data stream StreamRDD from the message framework and obtains the primary key set of historical stock data; (5) Spark Streaming retrieves the file path filelocals set of the corresponding historical stock data from the HBase database based on the primary key set; (6) The HBase database returns the file path filelocals collection; (7) Spark Streaming retrieves the files that need to be modified from Hive based on the file path filelocals collection; (8) Return the data file collection RDD that needs to be modified. Spark Streaming merges the acquired incremental data stream StreamRDD as incremental data with the data file collection RDD. The incremental data includes deleted data, updated data, and / or inserted data. Finally, the data is merged into a new data file collection New RDD. (9) Spark Streaming writes the data file set New RDD into the Hive table, updates the new file set in the data file set NewRDD to the Log table, marks it as valid, and marks the file path filelocals set of the files that need to be modified in step (7) as invalid; (10) Update the primary key KEY and file path filelocals of the updated data to a new key-value pair (KEY, filelocals) and save it in the HBase database index table.

Images