Method and system for quickly writing Hive partition table

Abstract

The invention discloses a method and system for quickly writing in a Hive partition table, and relates to the technical field of big data processing, and the method comprises the following steps: obtaining to-be-written data, and carrying out data processing and automatic merging on the obtained data according to a preset rule; the method comprises the following steps: acquiring and analyzing schema information of a target table, extracting and caching a partition field list, constructing a partition creation statement, and pre-judging and creating a partition before data writing so as to realize automatic management of a Hive partition table; before the data is written into the target table, mapping fields of the to-be-written data with fields of the target table through an intelligent data mapping technology; according to the method, data to be written are written into a target table, in the process, a multi-thread parallel writing strategy is adopted, and each thread is responsible for a set number of data writing tasks. According to the method, on the premise of ensuring data consistency and integrity, the data storage speed can be remarkably increased, high-speed execution of data synchronization is realized, and the problem of low data writing efficiency of the Hive partition table in a big data scene is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method and system for quickly writing into a Hive partition table. Background Art

[0002] With the rapid development of big data technology, big data applications have penetrated into the core business of various industries and become an important force in promoting digital transformation. In this context, data synchronization technology has become a bridge connecting data sources and data warehouses, playing a vital role in ensuring the real-time, completeness and accuracy of data. Especially in enterprise-level applications, data warehouses are not only the center of data storage, but also the key to data analysis and decision support. Therefore, the performance of data warehouses is directly related to the decision-making efficiency of enterprises and the smoothness of business operations.

[0003] Hive, as a distributed data warehouse system based on Hadoop, has won the favor of many enterprises with its powerful data processing capabilities, flexible scalability and friendly SQL interface. Hive is particularly suitable for processing massive data and can support a variety of complex data analysis tasks. Among them, Hive's partition table feature is crucial to optimizing query performance. It allows users to partition data according to specified fields, thereby speeding up data retrieval and improving system responsiveness.

[0004] However, despite the many advantages of Hive, the traditional way of writing data to Hive partitioned tables faces many challenges in practical applications. First, the data processing speed is often unsatisfactory, especially when dealing with large-scale data sets, the write performance will be seriously affected. Second, the traditional method usually requires a lot of manual intervention, such as manually creating partitions and managing data files, which not only increases the workload but also easily introduces human errors. In addition, insufficient resource utilization is also a problem that needs to be solved urgently. During peak hours, the system may not be able to fully utilize its performance due to resource bottlenecks; and during low hours, resources may be idle, resulting in waste.

[0005] These problems are particularly prominent when dealing with high-concurrency, large-volume writes. Due to the large amount of data and frequent write requests, traditional methods are often unable to respond to all requests in a timely manner, resulting in serious impact on the timeliness of data processing. At the same time, the system may experience performance bottlenecks under high load, resulting in insufficient resource utilization, further reducing system efficiency.

[0006] Therefore, in order to solve these problems, a method and system for quickly writing Hive partition tables is explored to improve the efficiency and quality of data processing and ensure the stable operation and efficient use of the data warehouse. Summary of the invention

[0007] Aiming at the problem of low efficiency of writing data into Hive partition table in big data scenario, the present invention provides a method and system for quickly writing data into Hive partition table, which accelerates data storage speed and realizes high-speed execution of data synchronization under the premise of ensuring data consistency and integrity.

[0008] In the first aspect, the present invention provides a method for quickly writing into a Hive partition table. The technical solution adopted to solve the above technical problem is as follows:

[0009] A method for quickly writing into a Hive partition table includes the following steps:

[0010] S1. Obtain the data to be written, process the acquired data according to preset demand rules, and automatically merge the processed data according to preset merging conditions;

[0011] S2. Obtain and parse the schema information of the target table, extract and cache the partition field list, build partition creation statements, and pre-judge and create partitions before writing data to achieve automatic management of Hive partition tables.

[0012] S3. Before writing data into the target table, the fields of the data to be written are mapped with the fields of the target table through intelligent data mapping technology;

[0013] S4. Write the data to be written into the target table. In this process, a multi-threaded parallel writing strategy is adopted, and each thread is responsible for a set number of data writing tasks.

[0014] Optionally, step S1 is performed, and the preset merging conditions are as follows:

[0015] (a) Number of data items: Set a data item threshold based on business requirements and system processing capabilities. When the accumulated number of data items reaches the data item threshold, a data merge operation is triggered to merge the current accumulated number of data items into one file.

[0016] (b) Overall data size: A threshold for the overall data size is set based on the utilization efficiency of the storage space and the performance of the file system. When the accumulated data size reaches the threshold for the overall data size, a data merge operation is triggered to merge the data of the current accumulated size into one file.

[0017] (c) Waiting time: according to the timeliness of the data, a maximum waiting time is set; when the accumulated data time reaches the maximum waiting time, the data merging operation is triggered to merge the data of the current accumulated time into one file;

[0018] After the acquired data is processed according to the preset demand rules, the data merging operation is triggered when the acquired data meets any merging condition among (a), (b), and (c);

[0019] When multiple data merging operations are triggered at the same time, multiple data merging tasks are processed simultaneously using multi-threading or distributed concurrent processing, and multiple merged files in the specified format are obtained and stored in the specified location.

[0020] Optionally, step S2 specifically includes the following operations:

[0021] S2.1. Obtain the schema information of the target table from Hive, and parse and extract the obtained schema information;

[0022] S2.2. Extract the partition field list from the schema information of the target table and cache it. For multiple partition fields, extract the partition field list and record the original order of the partition fields.

[0023] S2.3. Construct a partition creation statement based on the extracted partition field list and Hive partition rules;

[0024] S2.4. Before writing data into the target table, pre-judge and create the target table data partition according to the constructed partition creation statement.

[0025] Further optionally, step S3 specifically includes the following operations:

[0026] S3.1. Parse the metadata information of the data to be written, build a data schema for writing to the target table, and cache the parsed schema information;

[0027] S3.2. Before writing data into the target table, detect the field type of the data to be written, compare it with the field type of the target table, and automatically convert the mismatched field types to convert the shard data of the data to be written into a data structure that matches the target table;

[0028] S3.3. After completing the data conversion, the converted data is constructed into an HDFS data file according to the storage format of the target table.

[0029] Optionally, execute step S4, and in the process of writing the data to be written into the target table, adopt data verification and error retry mechanism to ensure the accuracy and reliability of the data; adopt transaction management and logging mechanism to ensure the consistency and recoverability of the data during the writing process.

[0030] In a second aspect, the present invention provides a system for quickly writing into a Hive partition table. The technical solution adopted to solve the above technical problem is as follows:

[0031] A system for quickly writing into a Hive partition table, comprising:

[0032] The acquisition preprocessing module is used to acquire the data to be written, process the acquired data according to the preset demand rules, and automatically merge the processed data according to the preset merging conditions;

[0033] Automatic partition management module, which is used to obtain and parse the schema information of the target table, extract and cache the partition field list, build partition creation statements, and pre-judge and create partitions before writing data to achieve automatic management of Hive partition tables;

[0034] A mapping processing module is used to map the fields of the data to be written with the fields of the target table by using intelligent data mapping technology before the data is written into the target table;

[0035] The data writing module is used to adopt a multi-threaded parallel writing strategy, set a specified number of data writing tasks for each thread, and write the data to be written into the target table.

[0036] Optionally, the system involved also includes:

[0037] The demand setting module is used to pre-set corresponding demand rules and merging conditions according to the demand;

[0038] The preset merge conditions are as follows:

[0039] (a) Number of data items: Set a data item threshold based on business requirements and system processing capabilities. When the accumulated number of data items reaches the data item threshold, a data merge operation is triggered to merge the current accumulated number of data items into one file.

[0040] (b) Overall data size: A threshold for the overall data size is set based on the utilization efficiency of the storage space and the performance of the file system. When the accumulated data size reaches the threshold for the overall data size, a data merge operation is triggered to merge the data of the current accumulated size into one file.

[0041] (c) Waiting time: according to the timeliness of the data, a maximum waiting time is set; when the accumulated data time reaches the maximum waiting time, the data merging operation is triggered to merge the data of the current accumulated time into one file;

[0042] After the acquisition preprocessing module processes the acquired data according to the preset demand rules, the data merging operation is triggered when the acquired data meets any one of the merging conditions (a), (b), and (c); when multiple data merging operations are triggered at the same time, the acquisition preprocessing module uses multi-threading or distributed concurrent processing to simultaneously process multiple data merging tasks, and obtains multiple merged files in the specified format, and stores them in the specified location.

[0043] Optionally, the automatic partition management module involved specifically includes:

[0044] The acquisition parsing unit is used to obtain the schema information of the target table from Hive and parse and extract the obtained schema information;

[0045] Extract record unit, used to extract the partition field list from the schema information of the target table and cache it. For multiple partition fields, the partition field list is extracted and the original order of the partition fields is recorded.

[0046] The statement building unit is used to build a partition creation statement based on the extracted partition field list and Hive partition rules;

[0047] The judgment creation unit is used to pre-judge and create the target table data partition according to the constructed partition creation statement before writing the data into the target table.

[0048] Further optionally, the mapping processing module involved specifically includes:

[0049] The parsing and building unit is used to parse the metadata information of the data to be written, build the data schema for writing into the target table, and cache the parsed schema information;

[0050] A detection and conversion unit, before writing the data into the target table, is used to detect the field type of the data to be written, compare it with the field type of the target table, and automatically convert the mismatched field types to convert the shard data of the data to be written into a data structure that matches the target table;

[0051] The file construction unit is used to construct the converted data into an HDFS data file according to the storage format of the target table.

[0052] Optionally, in the process of the data writing module writing the data to be written into the target table, a data verification and error retry mechanism is adopted to ensure the accuracy and reliability of the data, and a transaction management and logging mechanism is adopted to ensure the consistency and recoverability of the data during the writing process.

[0053] Compared with the prior art, the method and system for quickly writing into a Hive partition table of the present invention have the following beneficial effects:

[0054] 1. The present invention can significantly speed up the data storage speed and realize high-speed execution of data synchronization under the premise of ensuring data consistency and integrity, and solve the problem of low efficiency of data writing in Hive partition table in big data scenarios;

[0055] 2. The present invention reduces unnecessary data conversion and I / O operations, optimizes resource usage, and reduces resource consumption through intelligent matching and concurrent processing; automated field matching and partition identification improve system adaptability, reduce maintenance costs, and enhance data processing flexibility; built-in error handling and automatic retry mechanisms ensure data consistency and integrity, and enhance data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0057] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0059] Embodiment 1:

[0060] Combined with Figure 1 This embodiment proposes a method for quickly writing into a Hive partition table, which includes the following steps:

[0061] S1. Obtain the data to be written through streaming real-time monitoring and collection or timed batch collection;

[0062] Perform data processing operations such as data cleaning and data conversion on the acquired data according to preset demand rules;

[0063] The processed data is automatically merged according to the preset merging conditions. The preset merging conditions are as follows:

[0064] (a) Number of data items: Set a data item threshold based on business requirements and system processing capabilities. When the accumulated number of data items reaches the data item threshold, a data merge operation is triggered to merge the current accumulated number of data items into one file.

[0065] (b) Overall data size: A threshold for the overall data size is set based on the utilization efficiency of the storage space and the performance of the file system. When the accumulated data size reaches the threshold for the overall data size, a data merge operation is triggered to merge the data of the current accumulated size into one file.

[0066] (c) Waiting time: A maximum waiting time is set based on the timeliness of the data. When the accumulated data time reaches the maximum waiting time, the data merge operation is triggered to merge the data of the current accumulated time into one file.

[0067] After the acquired data is processed according to the preset demand rules, the data merging operation is triggered when the acquired data meets any merging condition among (a), (b), and (c);

[0068] When multiple data merging operations are triggered at the same time, multiple data merging tasks are processed simultaneously using multi-threading or distributed concurrent processing, and multiple merged files in the specified format are obtained and stored in the specified location.

[0069] It should be added that the real-time system load refers to the current system resource usage, such as CPU utilization, memory usage, disk I / O, etc. When the system load is high, it means that resource competition is intensifying and data processing efficiency may decrease. The data aggregation rate refers to the rate or speed at which data flows into the system. If the data aggregation rate is fast, it means that more data needs to be processed and stored per unit time.

[0070] According to the real-time load of the system and the rate of data aggregation, the thresholds of the number of data items, the overall data size and the waiting time are dynamically adjusted.

[0071] Under high load or fast data aggregation rate: appropriately increase the threshold of the number of data items and the overall data size. The purpose of this is to reduce the number of merges and improve the merge efficiency. At the same time, it increases the size of the merged single file and reduces the generation of small files, which is beneficial to improve the utilization efficiency of the big data system storage space and the subsequent data processing performance.

[0072] When the data aggregation rate is slow: appropriately lower the waiting time threshold to improve the timeliness of data processing. When data aggregation is slow, it may cause the number of data items and the threshold of the total data size to fail to reach for a long time, thus affecting the timeliness of data processing. By lowering the waiting time threshold, the existing data can be merged in time when the source data does not meet the threshold of the number of data items and the total data size for a long time.

[0073] In simple terms, the core purpose of this dynamic adjustment strategy (dynamically adjusting the number of data items, the overall data size, and the waiting time threshold) is to optimize the use of storage space and ensure the timeliness of data while ensuring data processing efficiency. By reasonably setting the number of data items, the overall data size, and the waiting time threshold, a balance point can be found under different loads and data aggregation rates to achieve efficient data processing and storage management.

[0074] S2. Obtain and parse the schema information of the target table, extract and cache the partition field list, construct partition creation statements, and pre-judge and create partitions before writing data to achieve automatic management of Hive partition tables. The specific operations include the following:

[0075] S2.1. Obtain the schema information of the target table from Hive, and parse and extract the obtained schema information. It should be added that: Schema is the structural definition of the table in the database, including table name, column name, data type, constraint and other information. In Hive, each table has a corresponding schema, which is used to describe the metadata information such as the structure and data type of the table. The content of metadata parsing includes but is not limited to: table name, column name, data type, comment information, etc. This information can help understand the key information such as the structure and data type of the table, and provide guidance for subsequent data operations and processing.

[0076] S2.2. Extract the partition field list from the schema information of the target table and cache it. For multiple partition fields, extract the partition field list and record the original order of the partition fields.

[0077] S2.3. Construct a partition creation statement based on the extracted partition field list and Hive partition rules;

[0078] S2.4. Before writing data into the target table, pre-judge and create the target table data partition according to the constructed partition creation statement.

[0079] S3. Before writing data into the target table, the fields of the data to be written are mapped with the fields of the target table through intelligent data mapping technology, which specifically includes the following operations:

[0080] S3.1. Parse the metadata information of the data to be written, build a data schema for writing to the target table, and cache the parsed schema information;

[0081] S3.2. Before writing data into the target table, detect the field type of the data to be written, compare it with the field type of the target table, and automatically convert the mismatched field types to convert the shard data of the data to be written into a data structure that matches the target table;

[0082] S3.3. After completing the data conversion, the converted data is constructed into an HDFS data file according to the storage format of the target table.

[0083] S4. Write the data to be written into the target table. In this process, a multi-threaded parallel writing strategy is adopted, and each thread is responsible for a set number of data writing tasks.

[0084] In the process of writing the data to be written into the target table, data verification and error retry mechanisms are used to ensure the accuracy and reliability of the data; transaction management and logging mechanisms are used to ensure the consistency and recoverability of the data during the writing process.

[0085] It should be added that the above target table refers to the Hive partition table, and the data to be written after the merger in step S1 is applied in subsequent steps.

[0086] Embodiment 2:

[0087] Combined with Figure 2 This embodiment provides a system for quickly writing into a Hive partition table, which includes:

[0088] The demand setting module is used to pre-set corresponding demand rules and merging conditions according to the demand;

[0089] The acquisition preprocessing module is used to acquire the data to be written, process the acquired data according to the preset demand rules, and automatically merge the processed data according to the preset merging conditions;

[0090] Automatic partition management module, which is used to obtain and parse the schema information of the target table, extract and cache the partition field list, build partition creation statements, and pre-judge and create partitions before writing data to achieve automatic management of Hive partition tables;

[0091] A mapping processing module is used to map the fields of the data to be written with the fields of the target table by using intelligent data mapping technology before the data is written into the target table;

[0092] The data writing module is used to adopt a multi-threaded parallel writing strategy, set a specified number of data writing tasks for each thread, and write the data to be written into the target table.

[0093] In this embodiment, the merging conditions pre-set by the demand setting module are as follows:

[0094] (a) Number of data items: Set a data item threshold based on business requirements and system processing capabilities. When the accumulated number of data items reaches the data item threshold, a data merge operation is triggered to merge the current accumulated number of data items into one file.

[0095] (b) Overall data size: A threshold for the overall data size is set based on the utilization efficiency of the storage space and the performance of the file system. When the accumulated data size reaches the threshold for the overall data size, a data merge operation is triggered to merge the data of the current accumulated size into one file.

[0096] (c) Waiting time: A maximum waiting time is set based on the timeliness of the data. When the accumulated data time reaches the maximum waiting time, the data merge operation is triggered to merge the data of the current accumulated time into one file.

[0097] After the acquisition preprocessing module processes the acquired data according to the preset demand rules, the data merging operation is triggered when the acquired data meets any one of the merging conditions (a), (b), and (c); when multiple data merging operations are triggered at the same time, the acquisition preprocessing module uses multi-threading or distributed concurrent processing to simultaneously process multiple data merging tasks, and obtains multiple merged files in the specified format, and stores them in the specified location.

[0098] It should be added that the real-time system load refers to the current system resource usage, such as CPU utilization, memory usage, disk I / O, etc. When the system load is high, it means that resource competition is intensifying and data processing efficiency may decrease. The data aggregation rate refers to the rate or speed at which data flows into the system. If the data aggregation rate is fast, it means that more data needs to be processed and stored per unit time.

[0099] In the process of pre-setting the merging conditions through the demand setting module, the real-time load of the system and the data aggregation rate are taken into consideration to dynamically adjust the thresholds of the number of data items, the overall data size and the waiting time.

[0100] Under high load or fast data aggregation rate: appropriately increase the threshold of the number of data items and the overall data size. The purpose of this is to reduce the number of merges and improve the merge efficiency. At the same time, it increases the size of the merged single file and reduces the generation of small files, which is beneficial to improve the utilization efficiency of the big data system storage space and the subsequent data processing performance.

[0101] When the data aggregation rate is slow: appropriately lower the waiting time threshold to improve the timeliness of data processing. When data aggregation is slow, it may cause the number of data items and the threshold of the total data size to fail to reach for a long time, thus affecting the timeliness of data processing. By lowering the waiting time threshold, the existing data can be merged in time when the source data does not meet the threshold of the number of data items and the total data size for a long time.

[0102] In simple terms, the core purpose of this dynamic adjustment strategy (dynamically adjusting the number of data items, the overall data size, and the waiting time threshold) is to optimize the use of storage space and ensure the timeliness of data while ensuring data processing efficiency. By reasonably setting the number of data items, the overall data size, and the waiting time threshold through the demand setting module, you can find a balance point under different loads and data aggregation rates to achieve efficient data processing and storage management.

[0103] In this embodiment, the automatic partition management module involved specifically includes:

[0104] The acquisition parsing unit is used to obtain the schema information of the target table from Hive and parse and extract the obtained schema information;

[0105] Extract record unit, used to extract the partition field list from the schema information of the target table and cache it. For multiple partition fields, the partition field list is extracted and the original order of the partition fields is recorded.

[0106] The statement building unit is used to build a partition creation statement based on the extracted partition field list and Hive partition rules;

[0107] The judgment creation unit is used to pre-judge and create the target table data partition according to the constructed partition creation statement before writing the data into the target table.

[0108] In this embodiment, the mapping processing module involved specifically includes:

[0109] The parsing and building unit is used to parse the metadata information of the data to be written, build the data schema for writing into the target table, and cache the parsed schema information;

[0110] A detection and conversion unit, before writing the data into the target table, is used to detect the field type of the data to be written, compare it with the field type of the target table, and automatically convert the mismatched field types to convert the shard data of the data to be written into a data structure that matches the target table;

[0111] The file construction unit is used to construct the converted data into an HDFS data file according to the storage format of the target table.

[0112] In this embodiment, the data writing module uses data verification and error retry mechanisms to ensure the accuracy and reliability of the data during the process of writing the data to be written into the target table, and uses transaction management and logging mechanisms to ensure the consistency and recoverability of the data during the writing process.

[0113] It should be added that the above target table refers to the Hive partition table, and the data to be written after being merged by the preprocessing module is obtained and applied in the subsequent modules.

[0114] In summary, the method and system for quickly writing Hive partition tables of the present invention can significantly speed up the data storage speed and achieve high-speed execution of data synchronization under the premise of ensuring data consistency and integrity, thus solving the problem of low efficiency of writing data to Hive partition tables in big data scenarios.

[0115] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for quickly writing into a Hive partition table, characterized in that: The steps include: S1. Obtain the data to be written, process the acquired data according to preset demand rules, and automatically merge the processed data according to preset merging conditions; S2. Obtain and parse the schema information of the target table, extract and cache the partition field list, build partition creation statements, and pre-judge and create partitions before writing data to achieve automatic management of Hive partition tables. S3. Before writing data into the target table, the fields of the data to be written are mapped with the fields of the target table through intelligent data mapping technology; S4. Write the data to be written into the target table. In this process, a multi-threaded parallel writing strategy is adopted, and each thread is responsible for a set number of data writing tasks.

2. A method for quickly writing into a Hive partition table according to claim 1, characterized in that: Execute step S1, the preset merging conditions are as follows: (a) Number of data items: Set a data item threshold based on business requirements and system processing capabilities. When the accumulated number of data items reaches the data item threshold, a data merge operation is triggered to merge the current accumulated number of data items into one file. (b) Overall data size: A threshold for the overall data size is set based on the utilization efficiency of the storage space and the performance of the file system. When the accumulated data size reaches the threshold for the overall data size, a data merge operation is triggered to merge the data of the current accumulated size into one file. (c) Waiting time: according to the timeliness of the data, a maximum waiting time is set; when the accumulated data time reaches the maximum waiting time, the data merging operation is triggered to merge the data of the current accumulated time into one file; After the acquired data is processed according to the preset demand rules, the data merging operation is triggered when the acquired data meets any merging condition among (a), (b), and (c); When multiple data merging operations are triggered at the same time, multiple data merging tasks are processed simultaneously using multi-threading or distributed concurrent processing, and multiple merged files in the specified format are obtained and stored in the specified location.

3. A method for quickly writing into a Hive partition table according to claim 1, characterized in that: The step S2 specifically includes the following operations: S2.

1. Obtain the schema information of the target table from Hive, and parse and extract the obtained schema information; S2.

2. Extract the partition field list from the schema information of the target table and cache it. For multiple partition fields, extract the partition field list and record the original order of the partition fields. S2.

3. Construct a partition creation statement based on the extracted partition field list and Hive partition rules; S2.

4. Before writing data into the target table, pre-judge and create the target table data partition according to the constructed partition creation statement.

4. A method for quickly writing into a Hive partition table according to claim 3, characterized in that: Step S3 specifically includes the following operations: S3.

1. Parse the metadata information of the data to be written, build a data schema for writing to the target table, and cache the parsed schema information; S3.

2. Before writing data into the target table, detect the field type of the data to be written, compare it with the field type of the target table, and automatically convert the mismatched field types to convert the shard data of the data to be written into a data structure that matches the target table; S3.

3. After completing the data conversion, the converted data is constructed into an HDFS data file according to the storage format of the target table.

5. The method for quickly writing into a Hive partition table according to claim 1, characterized in that: Execute step S4. In the process of writing the data to be written into the target table, a data verification and error retry mechanism is used to ensure the accuracy and reliability of the data; a transaction management and log recording mechanism is used to ensure the consistency and recoverability of the data during the writing process.

6. A system for quickly writing into a Hive partition table, characterized in that: It includes: The acquisition preprocessing module is used to acquire the data to be written, process the acquired data according to the preset demand rules, and automatically merge the processed data according to the preset merging conditions; Automatic partition management module, which is used to obtain and parse the schema information of the target table, extract and cache the partition field list, build partition creation statements, and pre-judge and create partitions before writing data to achieve automatic management of Hive partition tables; A mapping processing module is used to map the fields of the data to be written with the fields of the target table by using intelligent data mapping technology before the data is written into the target table; The data writing module is used to adopt a multi-threaded parallel writing strategy, set a specified number of data writing tasks for each thread, and write the data to be written into the target table.

7. A system for quickly writing into a Hive partition table according to claim 6, characterized in that: It also includes: The demand setting module is used to pre-set corresponding demand rules and merging conditions according to the demand; The preset merge conditions are as follows: (a) Number of data items: Set a data item threshold based on business requirements and system processing capabilities. When the accumulated number of data items reaches the data item threshold, a data merge operation is triggered to merge the current accumulated number of data items into one file. (b) Overall data size: A threshold for the overall data size is set based on the utilization efficiency of the storage space and the performance of the file system. When the accumulated data size reaches the threshold for the overall data size, a data merge operation is triggered to merge the data of the current accumulated size into one file. (c) Waiting time: according to the timeliness of the data, a maximum waiting time is set; when the accumulated data time reaches the maximum waiting time, the data merging operation is triggered to merge the data of the current accumulated time into one file; After the acquisition preprocessing module processes the acquired data according to the preset demand rules, the data merging operation is triggered when the acquired data meets any one of the merging conditions (a), (b), and (c); when multiple data merging operations are triggered at the same time, the acquisition preprocessing module uses multi-threading or distributed concurrent processing to simultaneously process multiple data merging tasks, and obtains multiple merged files in the specified format, and stores them in the specified location.

8. The system for quickly writing into a Hive partition table according to claim 6, characterized in that: The automatic partition management module specifically includes: The acquisition parsing unit is used to obtain the schema information of the target table from Hive and parse and extract the obtained schema information; Extract record unit, used to extract the partition field list from the schema information of the target table and cache it. For multiple partition fields, the partition field list is extracted and the original order of the partition fields is recorded. The statement building unit is used to build a partition creation statement based on the extracted partition field list and Hive partition rules; The judgment creation unit is used to pre-judge and create the target table data partition according to the constructed partition creation statement before writing the data into the target table.

9. A system for quickly writing into a Hive partition table according to claim 8, characterized in that: The mapping processing module specifically includes: The parsing and building unit is used to parse the metadata information of the data to be written, build the data schema for writing into the target table, and cache the parsed schema information; A detection and conversion unit, before writing the data into the target table, is used to detect the field type of the data to be written, compare it with the field type of the target table, and automatically convert the mismatched field types to convert the shard data of the data to be written into a data structure that matches the target table; The file construction unit is used to construct the converted data into an HDFS data file according to the storage format of the target table.

10. The system for quickly writing into a Hive partition table according to claim 6, characterized in that: When the data writing module writes the data to be written into the target table, it adopts data verification and error retry mechanism to ensure the accuracy and reliability of the data, and adopts transaction management and log recording mechanism to ensure the consistency and recoverability of the data during the writing process.

Images