A method and device for dynamically reading hive table data in a stream processing mode
By setting partitionMap and partitionValueList in the stream processing engine, we judge and dynamically read the old partition new files in the hive table, the problem that the stream processing engine cannot read the old partition new files, and real-time and complete reading of the hive table data is achieved.
Patent Information
- Application Number
- CN202111194393.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-13
- Publication Date
- 2025-02-28
- Estimated Expiration
- 2041-10-13
AI Technical Summary
The stream processing engine cannot dynamically read the old partition new files in the hive table, resulting in partition mismatch and some new partitions misread.
By setting partitionMap and partitionValueList, determine whether the file to be read is a new file in the old partition, and dynamically read it according to the file path. At the same time, use a lackPartition to handle missing partition names to ensure that all partition files are read.
The stream processing engine dynamic reading of new files in the old partition of hive table is realized, avoiding partition mismatch and misreading problems, and ensuring real-time and completeness of data.
Smart Images
Figure CN113886330B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method and device for supporting dynamic reading of hive table data in a stream processing manner. Background Art
[0002] With the development of science and technology, the data of many application scenarios is unlimited. Batch processing can no longer meet the needs of these scenarios. As market demand increases, stream processing engines are becoming increasingly popular. Stream processing engines are used for real-time computing. They process data as it arrives, instead of waiting for a batch of data to arrive before processing, as in batch processing. The advantage of real-time computing by stream processing engines makes them widely used in network monitoring and e-commerce.
[0003] Hive is a data warehouse analysis and processing tool based on Hadoop, and can be queried with SQL statements. Because the bottom layer of Hive is encapsulated stream Hadoop, Hive data is stored in Hadoop files, so it has good batch processing capabilities, but Hive does not support stream processing. For data in different partitions, each partition is a folder, and the files in the folder are the stored data. In Hive, for new data added to old partitions, new files will be automatically created in HDFS, and data cannot be added to existing files. Therefore, the stream processing engine cannot read newly added files in old partitions of the Hive table, that is, it does not support dynamic reading of new files in old partitions in Hive.
[0004] When a large number of new partition files are added at one time, the stream processing engine cannot read all the new partition files at once. Due to the time difference, the partitions in the stream processing engine and the partitions in Hive will not match, resulting in some new partitions being missed.
[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention
[0006] The technical problems to be solved by the present invention are:
[0007] In Hive, for newly added data in old partitions, new files will be automatically created in HDFS. Data cannot be added under existing files. Therefore, the stream processing engine cannot read newly added files in old partitions of the Hive table. In other words, it does not support dynamic reading of new files in old partitions in Hive.
[0008] To achieve the above object, the present invention adopts the following technical solution:
[0009] In a first aspect, the present invention provides a method for supporting dynamic reading of hive table data in a stream processing manner, comprising:
[0010] Before the stream processing engine starts reading the hive table, set the partitionMap to load the partition name that has been read in the entire reading process and the maximum file modification time that has been read in the corresponding partition; set the partitionValueList to load the partition name to be read in the current round, the file modification time of the file to be read in the corresponding partition, and the path of the corresponding file to be read; set the timestamp in the first round of reading to the preset initial value;
[0011] When the stream processing engine reads the hive table, if the partition name to be read in partitionValueList can be found in partitionMap, the corresponding partition to be read is determined to be an old partition; if the file modification time of the file to be read in the old partition is greater than the maximum file modification time that has been read in the corresponding partition in partitionMap, and the file modification time of the file to be read in the old partition is greater than the current round timestamp, the file to be read in the old partition is determined to be a new file in the old partition, and the new file in the old partition is read according to the path of the corresponding file to be read in partitionValueList;
[0012] After the files to be read in the current round are read, clear the partitionValueList, update the maximum file modification time that has been read in the corresponding partition in the partitionMap, and update the timestamp to the file modification time corresponding to the file with the largest file modification time among all the files that have been read in the current round.
[0013] Preferably, in the process of the stream processing engine reading the hive table, if the partition name to be read in partitionValueList cannot be found in patitionMap, the corresponding partition to be read is determined to be a new partition, the file to be read in the corresponding partition is a new partition file, the path of the corresponding file to be read is set to an empty string, and the new partition file is read in Hadoop according to the new partition name.
[0014] Preferably, when the stream processing engine reads the hive table, hadoop takes all the partition names from the hive table, compares them with the partition names in the partitionMap, finds the missing partition names, and writes the missing partition names and the maximum file modification time in the corresponding partitions into lackPartition;
[0015] If lackPartition is not empty, the stream processing engine will first read the files of the missing partitions according to lackPartition. After completing the reading of the files of the corresponding partitions, it will delete the corresponding partition name from lackPartition until lackPartition is empty.
[0016] Preferably, the stream processing engine reads the files of the missing partition according to the missing partition name in lackPartition.
[0017] Preferably, when lackPartition is empty, the stream processing engine directly reads the file to be read according to the path of the corresponding file to be read in partitionValueList.
[0018] Preferably, before the stream processing engine reads the file to be read according to the path of the corresponding file to be read in partitionValueList, it first sorts the file to be read in the corresponding partition in partitionValueList in ascending order according to the file modification time, and reads the file to be read according to the sorted order using the path of the corresponding file to be read in partitionValueList.
[0019] Preferably, the updated partitionMap is used to determine whether the partition to be read in the next round of reading is a new partition or an old partition.
[0020] Preferably, the updated timestamp is used to determine whether the file to be read in the next round of reading is a new file in the old partition.
[0021] Preferably, if the file modification time of the file to be read in the old partition is less than or equal to the maximum file modification time that has been read in the corresponding partition in the partitionMap, or the file modification time of the file to be read in the old partition is less than or equal to the current round timestamp, then the file to be read in the old partition is determined to be an old file of the old partition, and the reading of the old file of the old partition is skipped.
[0022] In a second aspect, the present invention provides a device for dynamically reading hive table data in a stream processing manner, which is used to implement the method for dynamically reading hive table data in a stream processing manner described in the first aspect. The device for dynamically reading hive table data in a stream processing manner includes:
[0023] At least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for supporting dynamic reading of hive table data in a stream processing manner as described in the first aspect.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] The present invention compares the file modification time of the file to be read with the maximum file modification time that has been read in the corresponding partition and the current round timestamp, finds the old partition new file in the file to be read, and reads the old partition new file according to the path of the corresponding file to be read in partitionValueList; the hive data source can be executed on the stream mechanism, so that the stream processing engine can dynamically read the old partition new file in hive. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0027] Figure 1 It is a flow chart of a method for dynamically reading hive table data in a stream processing manner provided by an embodiment of the present invention;
[0028] Figure 2 It is a flow chart of another method for supporting dynamic reading of hive table data in a stream processing manner provided by an embodiment of the present invention;
[0029] Figure 3 It is a structural diagram of a device that supports dynamic reading of hive table data in a stream processing manner provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0031] In the description of the present invention, the terms "inside", "outside", "longitudinal", "lateral", "upper", "lower", "top", "bottom" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on the present invention.
[0032] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0033] Embodiment 1:
[0034] The embodiment of the present invention provides a method for dynamically reading hive table data in a stream processing manner, such as Figure 1 As shown, including:
[0035] Step 201, before the stream processing engine starts to read the hive table, set the partitionMap to load the partition name that has been read in the entire reading process and the maximum file modification time that has been read in the corresponding partition; each partition in the partitionMap corresponds to the maximum file modification time that has been read in the corresponding partition; set the partitionValueList to load the partition name to be read in the current round, the file modification time of the file to be read in the corresponding partition and the path of the corresponding file to be read; the information in the partitionValueList in each round of reading is different, and each partition in the partitionValueList corresponds to one or at least two files to be read in the corresponding partition, which mainly depends on the number of files in the corresponding partition; set the timestamp in the first round of reading to the preset initial value; wherein the preset initial value can be selected according to actual needs, generally set to a negative number, for example, can be set to -1.
[0036] The maximum file modification time that has been read in the corresponding partition is specifically: the file modification time corresponding to the file with the largest file modification time among the files that have been read in the corresponding partition.
[0037] Step 202, during the process of the stream processing engine reading the hive table, if the partition name to be read in partitionValueList can be found in patitionMap, the corresponding partition to be read is determined to be an old partition; if the partition name to be read in partitionValueList cannot be found in patitionMap, the corresponding partition to be read is determined to be a new partition.
[0038] Step 203, if the file modification time of the file to be read in the old partition is greater than the maximum file modification time that has been read in the corresponding partition in the partitionMap, and the file modification time of the file to be read in the old partition is greater than the current round timestamp, then it is determined that the file to be read in the old partition is a new file in the old partition.
[0039] Step 204, read the new file in the old partition according to the path of the corresponding file to be read in partitionValueList.
[0040] Step 205, after the files to be read in the current round are read, clear partitionValueList, update the maximum file modification time that has been read in the corresponding partition in partitionMap, and update the timestamp to the file modification time corresponding to the file with the largest file modification time among all the files that have been read in the current round.
[0041] The above descriptions of the number of rounds are for the purpose of describing the present invention clearly rather than for limiting the present invention. In fact, before the stream processing engine starts the hive reading service, the hive table is empty; after the stream processing engine starts the hive reading service, an infinite loop is adopted to continuously scan the hdfs path corresponding to the table specified by hive to read the data in the hive table; the reading of data is divided into two parts, the first part is that Hadoop obtains the partition name from the hive table, and the second part is that the stream processing engine obtains the data from the hive table.
[0042] In an embodiment of the present invention, during the process of the stream processing engine reading the hive table, if the partition name to be read in the partitionValueList cannot be found in the patitionMap, the corresponding partition to be read is determined to be a new partition, the file to be read in the corresponding partition is a new partition file, the path of the corresponding file to be read is set to an empty string, and the new partition file is read in Hadoop according to the new partition name.
[0043] Among them, in the first round of reading, patitionMap is empty, and all partitions to be read in partitionValueList cannot be found in this round of patitionMap, that is, in the first round of reading, all partitions to be read are new partitions, and the paths of the corresponding files to be read in partitionValueList are set to empty strings. The stream processing engine reads all files in the corresponding partition in Hadoop according to the partition name to be read in partitionValueList.
[0044] When a large number of new partition files are added at one time, since the stream processing engine cannot read all the new partition files at once, the partitions in the stream processing engine and the partitions in Hive will not match due to time difference, resulting in some new partitions being missed.
[0045] In the embodiment of the present invention, in order to solve the problem that the partitions in the stream processing engine and the partitions in hive do not match due to the time difference, as shown in Figure 2 As shown, the stream processing method supports the following methods for dynamically reading hive table data:
[0046] Step 301, when the stream processing engine reads the hive table, Hadoop takes all the partition names from the hive table, compares them with the partition names in the partitionMap, and finds the missing partition names.
[0047] Step 302: Write the missing partition name and the maximum file modification time in the corresponding partition into lackPartition.
[0048] Step 303: If lackPartition is not empty, the stream processing engine will first read the files of the missing partition according to lackPartition.
[0049] Step 304: After completing reading of the file in the corresponding partition, delete the corresponding partition name from lackPartition until lackPartition is empty.
[0050] The maximum file modification time in the corresponding partition is specifically the file modification time corresponding to the file with the largest file modification time among the files in the missing partition.
[0051] For example, in the third round of reading, Hadoop obtains all the partition names part1, part2, part3, and part4 from the hive table, and compares them with [(part2,2020-10-2 12:43:37,””),(part1,2020-10-2 12:33:37,””)] stored in the third round of partitionMap to find the missing partition names part3 and part4, and write (part3,2020-10-2 13:33:37) and (part4,2020-10-2 13:34:37) to lackPartition; the stream processing engine prioritizes the [(part3,2020-10-2 13:33:37),(part4,2020-10-213:34:37)] to read the files of the missing partitions. When the stream processing engine can read (part3,2020-10-2 13:33:37) and (part4,2020-10-2 13:34:37), (part3,2020-10-2 13:33:37) and (part4,2020-10-2 13:34:37) are deleted from the lackPartition of the third round. The lackPartition is empty, and the stream processing engine starts to read the files to be read according to [(part2,2020-10-212:43:37,””),(part1,2020-10-2 12:33:37,””)] stored in the patitionMap of the third round.
[0052] In the embodiment of the present invention, the stream processing engine reads the file of the missing partition according to the missing partition name in lackPartition.
[0053] Specifically, the stream processing engine reads the files of the missing partitions from the next round of stream processing engines based on the missing partition names in lackPartition; for example, in the second round of reading, hadoop takes all the partition names from the hive table, compares them with the partition names in the second round of partitionMap, finds the missing partition names, and writes the missing partition names and the maximum file modification time in the corresponding partitions into lackPartition.
[0054] In the third round of reading, the stream processing engine first reads the files of the missing partitions from the third round of stream processing engines based on the missing partition names in lackPartition. After completing the reading of the files of the corresponding partitions, the corresponding partition names are deleted from lackPartition until lackPartition is empty, and then reading is started according to the partitionValueList of the third round.
[0055] In the embodiment of the present invention, when lackPartition is empty, the stream processing engine directly reads the file to be read according to the path of the corresponding file to be read in partitionValueList.
[0056] Specifically, the stream processing engine directly reads the file to be read from the stream processing engine according to the path of the corresponding file to be read in the partitionValueList; the stream processing engine stores not only the partitionMap and the partitionValueList, but also the corresponding file to be read.
[0057] In an embodiment of the present invention, before the stream processing engine reads the file to be read according to the path of the corresponding file to be read in partitionValueList, it first sorts the file to be read in the corresponding partition in partitionValueList in ascending order according to the file modification time, and reads the file to be read according to the sorted order using the path of the corresponding file to be read in partitionValueList.
[0058] In the embodiment of the present invention, the updated partitionMap is used to determine whether the partition to be read in the next round of reading is a new partition or an old partition.
[0059] For example, in the third round of reading, after the stream processing engine completes reading the file to be read according to the path of the corresponding file to be read in the partitionValueList of the third round, it clears the partitionValueList of the third round, updates the patitionMap of the third round and the timestamp of the third round; wherein the updated patitionMap is used as the patitionMap of the fourth round, and the updated timestamp is used as the timestamp of the fourth round; in the fourth round of reading, if the partition name to be read in the partitionValueList of the fourth round cannot be found in the patitionMap of the fourth round, the corresponding partition to be read is determined to be a new partition; if the partition name to be read in the partitionValueList of the fourth round can be found in the patitionMap of the fourth round, the corresponding partition to be read is determined to be an old partition.
[0060] The following is a specific example. There are 4 files under part1, namely file1, flie2, flie3, and file4, and their modification times are 2020-10-2, 2020-10-3, 2020-10-4, and 2020-10-5, respectively. Among them, file1 and file2 are old files in the old partition, and file3 and file4 are new files in the old partition. The content stored in partitionValueList at this moment is [(part1,2020-10-2,"part1 / file1"), (part1,2020-10-3,"part1 / file2"), (part1,2020-10-4,"part1 / file3"), (part1,2020-10-5,"part1 / file4")], and partiti The content stored in onMap is (part1, 2020-10-3), and the timestamp is 2020-10-3; because part1 can be found in partitionMap, it means that part1 is the old partition; because 2020-10-4 and 2020-10-5 are greater than 2020-10-3, file3 and file4 are determined to be new files in the old partition; the stream processing engine reads file3 and file4 according to "part1 / file3" and "part1 / file4" in partitionValueList; after completing the reading of file3 and file4, clear partitionValueList, update the content stored in partitionMap to (part1, 2020-10-5), and the update timestamp is 2020-10-5.
[0061] In the embodiment of the present invention, the updated timestamp is used to determine whether the file to be read in the next round of reading is a new file in the old partition.
[0062] For example, in the third round of reading, after the stream processing engine completes reading of the file to be read according to the path of the corresponding file to be read in the partitionValueList of the third round, it clears the partitionValueList of the third round, updates the patitionMap of the third round and the timestamp of the third round; wherein, the updated patitionMap is used as the patitionMap of the fourth round, and the updated timestamp is used as the timestamp of the fourth round; in the fourth round of reading, if the partition name to be read in the partitionValueList of the fourth round can be found in the patitionMap of the fourth round, then the corresponding partition to be read is determined to be an old partition; if the file modification time of the file to be read in the old partition is greater than the maximum file modification time that has been read in the corresponding partition in the patitionMap of the fourth round, and the file modification time of the file to be read in the old partition is greater than the timestamp of the fourth round, then the file to be read in the old partition is determined to be a new file in the old partition.
[0063] In an embodiment of the present invention, if the file modification time of the file to be read in the old partition is less than or equal to the maximum file modification time that has been read in the corresponding partition in the partitionMap, or the file modification time of the file to be read in the old partition is less than or equal to the current round timestamp, then the file to be read in the old partition is determined to be an old file of the old partition, and the reading of the old file of the old partition is skipped.
[0064] Embodiment 2:
[0065] The embodiment of the present invention provides a device that supports dynamic reading of hive table data in a stream processing manner, such as Figure 3 As shown, it includes one or more processors 21 and a memory 22. Among them, Figure 3 A processor 21 is taken as an example.
[0066] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.
[0067] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for dynamically reading hive table data in a stream processing manner in Example 1. The processor 21 executes the method for dynamically reading hive table data in a stream processing manner by running the non-volatile software programs and instructions stored in the memory 22.
[0068] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. The memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0069] The program instructions / modules are stored in the memory 22. When executed by the one or more processors 21, the method for dynamically reading hive table data in the stream processing mode in the above embodiment 1 is executed. For example, the method described above is executed. Figure 1 and Figure 2 The steps shown.
[0070] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.
[0071] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for dynamically reading hive table data in a stream processing manner, characterized in that: include: Before the stream processing engine starts reading the hive table, set the partitionMap to load the partition name that has been read in the entire reading process and the maximum file modification time that has been read in the corresponding partition; Set partitionValueList to load the partition name to be read in the current round, the file modification time of the file to be read in the corresponding partition, and the path of the corresponding file to be read; Set the timestamp in the first round of reading to the preset initial value; In the process of the stream processing engine reading the hive table, if the partition name to be read in the partitionValueList can be found in the partitionMap, the corresponding partition to be read is determined to be an old partition; if the file modification time of the file to be read in the old partition is greater than the maximum file modification time that has been read in the corresponding partition in the partitionMap, and the file modification time of the file to be read in the old partition is greater than the current round timestamp, then the file to be read in the old partition is determined to be a new file in the old partition, and the new file in the old partition is read according to the path of the corresponding file to be read in the partitionValueList; in the process of the stream processing engine reading the hive table, hadoop takes all the partition names from the hive table, compares them with the partition names in the partitionMap, finds the missing partition names, and writes the missing partition names and the maximum file modification time in the corresponding partition into the lackPartition; If lackPartition is not empty, the stream processing engine will first read the files of the missing partitions according to lackPartition. After completing the reading of the files of the corresponding partitions, it will delete the corresponding partition name from lackPartition until lackPartition is empty. After the files to be read in the current round are read, clear the partitionValueList, update the maximum file modification time that has been read in the corresponding partition in the partitionMap, and update the timestamp to the file modification time corresponding to the file with the largest file modification time among all the files that have been read in the current round.
2. According to claim 1, the method for dynamically reading hive table data in a stream processing mode is characterized in that: When the stream processing engine reads the hive table, if the partition name to be read in partitionValueList is not found in patitionMap, the corresponding partition to be read is determined to be a new partition, the file to be read in the corresponding partition is a new partition file, the path of the corresponding file to be read is set to an empty string, and the new partition file is read in Hadoop according to the new partition name.
3. The method according to claim 1, wherein the stream processing method supports dynamic reading of hive table data, The stream processing engine reads the files of the missing partitions according to the missing partition names in lackPartition.
4. The method according to claim 1, wherein the stream processing method supports dynamic reading of hive table data, When lackPartition is empty, the stream processing engine directly reads the file to be read according to the path of the corresponding file to be read in partitionValueList.
5. The method according to claim 1, wherein the stream processing method supports dynamic reading of hive table data, Before the stream processing engine reads the file to be read according to the path of the corresponding file to be read in partitionValueList, it first sorts the file to be read in the corresponding partition in partitionValueList in ascending order according to the file modification time, and reads the file to be read according to the sorted order using the path of the corresponding file to be read in partitionValueList.
6. The method according to claim 1, wherein the stream processing method supports dynamic reading of hive table data, The updated partitionMap is used to determine whether the partition to be read in the next round of reading is a new partition or an old partition.
7. The method for dynamically reading hive table data in a stream processing mode according to claim 1 is characterized in that: The updated timestamp is used to determine whether the file to be read in the next round of reading is a new file in the old partition.
8. The method for dynamically reading hive table data in a stream processing mode according to claim 1, characterized in that: If the file modification time of the file to be read in the old partition is less than or equal to the maximum file modification time that has been read in the corresponding partition in the partitionMap, or the file modification time of the file to be read in the old partition is less than or equal to the current round timestamp, then the file to be read in the old partition is determined to be an old file of the old partition, and the reading of the old file of the old partition is skipped.
9. A device that supports dynamic reading of hive table data in a stream processing manner, characterized in that: The device comprises: At least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for supporting dynamic reading of hive table data in a stream processing manner as described in any one of claims 1 to 8.