Small file merging method and device based on Spark, computer equipment and storage medium
By repartitioning the data table to be written in Spark, the performance and stability problems caused by too many small files in Spark tasks are solved, and more efficient data processing and storage are achieved.
Patent Information
- Application Number
- CN202411132826.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-27
AI Technical Summary
When using Spark for financial data ETL processing, too many small files are generated, which affects the performance and stability of the distributed cluster, resulting in unstable Spark tasks.
By repartitioning the data table to be written before data is written, the write logical plan is determined based on the data query instructions and the data table to be written, the repartition operator is used to reduce the generation of small files, and the merge of small files is realized.
It effectively reduces the generation of small files, improves the stability and performance of Spark tasks, and optimizes the storage and query performance of data.
Smart Images
Figure CN119202043A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of data processing and financial technology, and in particular to a method, device, computer equipment and storage medium for merging small files based on Spark. Background Art
[0002] With the development of computer technology, big data technology is increasingly used in the financial field, which can help financial institutions analyze and process data to better identify, evaluate and manage risks, including credit risk, market risk, operational risk, etc. As a fast and general large-scale data processing engine, Apache Spark plays an increasingly important role in data analysis and system construction in the financial field. It can be applied to big data application scenarios in the financial field such as offline batch computing, financial data ETL (Extract, Transform, Load) processing, data mining, machine learning, etc. Using Spark's data processing capabilities, financial data from different data sources can be integrated and cleaned to obtain a standardized data set. However, when using Spark sql to process data for ETL and write it to distributed storage, a large number of small files will be generated. When the Spark task reaches a certain scale, the number of small files generated is too large, which will seriously affect the performance and stability of the entire distributed cluster and increase the instability factor for the Spark task. Therefore, how to reduce the generation of small files and improve the stability of Spark tasks has become an urgent problem to be solved. Summary of the invention
[0003] The present application provides a Spark-based small file merging method, device, computer equipment and storage medium to reduce the generation of small files and thereby improve the stability of Spark tasks.
[0004] In a first aspect, the present application provides a small file merging method based on Spark, the method comprising:
[0005] When receiving the data to be written, obtaining a data query instruction corresponding to the data to be written and a data table to be written;
[0006] Determine a write logic plan based on the data query instruction and the data table to be written;
[0007] Based on the write logic plan, the data table to be written is repartitioned to obtain a repartition result, and the data to be written is written into the target data table to merge small files.
[0008] In a second aspect, the present application further provides a small file merging device based on Spark, the device comprising:
[0009] A module for acquiring a table to be written, used for acquiring a data query instruction corresponding to the data to be written and a table of data to be written when receiving the data to be written;
[0010] A logic plan determination module, used to determine a write logic plan based on the data query instruction and the data table to be written;
[0011] The data writing module is used to repartition the data table to be written based on the writing logic plan, obtain a repartition result, and write the data to be written into the target data table to merge small files.
[0012] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the above-mentioned Spark-based small file merging method when executing the computer program.
[0013] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the Spark-based small file merging method as described above.
[0014] The present application discloses a method, device, computer equipment and storage medium for merging small files based on Spark. When receiving data to be written, the data query instruction corresponding to the data to be written and the data table to be written are obtained; based on the data query instruction and the data table to be written, a write logic plan is determined; based on the write logic plan, the data table to be written is repartitioned to obtain the repartition result, and the data to be written is written into the target data table to merge small files. The present application can repartition the data table to be written after obtaining the data to be written and before the data is written, and write the data to be written into the repartitioned data table to be written, so that small files are automatically merged, the generation of small files is reduced, and the stability of Spark tasks is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0016] Figure 1 This is a schematic flow chart of a first embodiment of a method for merging small files based on Spark provided in an embodiment of the present application;
[0017] Figure 2 is a schematic flow chart of a second embodiment of a method for merging small files based on Spark provided in an embodiment of the present application;
[0018] Figure 3 A schematic block diagram of a small file merging device based on Spark provided in an embodiment of the present application;
[0019] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0021] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0022] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0023] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0024] The embodiments of the present application provide a method, device, computer equipment and storage medium for merging small files based on Spark. The method for merging small files based on Spark can be applied to a server, by repartitioning the data table to be written after obtaining the data to be written and before the data is written, and writing the data to be written into the repartitioned data table to be written, so that small files are automatically merged, the generation of small files is reduced, and the stability of Spark tasks is improved. The server can be an independent server or a server cluster.
[0025] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0026] See also Figure 1 , Figure 1 This is a schematic flow chart of a small file merging method based on Spark provided by an embodiment of the present application. The small file merging method based on Spark can be applied to a server, and is used to repartition a data table to be written after obtaining the data to be written and before the data is written, and write the data to be written into the repartitioned data table to be written, so that small files are automatically merged, the generation of small files is reduced, and the stability of Spark tasks is improved.
[0027] like Figure 1 As shown, the Spark-based small file merging method specifically includes steps S101 to S104.
[0028] S101, when receiving data to be written, obtaining a data query instruction corresponding to the data to be written and a data table to be written;
[0029] In one embodiment, the data to be written is the data obtained after the Spark sql data query instruction performs ETL (Extract, Transform, Load) processing on the data, and needs to be written into the distributed storage. Spark sql is a computer language for accessing and processing databases for data operations in a large data distributed cluster, which is designed in Spark and basically follows the structured query language standard. It is the most widely used module in Spark, and users can use simple and easy-to-operate sql statements to process large-scale data.
[0030] In one embodiment, when receiving data to be written, a data query instruction is obtained (such as an SQL query or a DataFrame operation. DataFrame is a core data structure in Apache Spark that provides a rich API (application programming interface) for structured data processing. DataFrame can be viewed as a tabular data structure with named columns that can be of different data types.), the data query instruction is analyzed, and the data table to be written is determined.
[0031] In one embodiment, the data table to be written can be a table specified by the user in the data query instruction, or a data table obtained randomly or by a preset mapping relationship. Among them, the data table to be written can be an RDD (ResilientDistributed Dataset) or a specified Hive table. The Hive table stores structured data, which is similar to a table in a relational database and consists of rows and columns, each column having a specific data type.
[0032] S102, determining a write logic plan based on the data query instruction and the data table to be written;
[0033] In one embodiment, the logical plan is an intermediate representation of the query that the Spark sql execution engine understands and optimizes.
[0034] In one embodiment, a write logic plan for data writing is determined according to the data query instruction, specifically, including the steps of parsing, verifying and optimizing the data query instruction. The logic plan will define how to repartition the data table to be written and write the data to be written into the repartitioned data table.
[0035] In one embodiment, the data query instruction includes relevant information of the write logic plan, and the write logic plan corresponding to the data query instruction can be determined based on the relevant information. The write logic plan can be one of InsertIntoHiveTable, createHiveTableAsSelectCommand, or OptimizedCreateHiveTableAsSelectCommand, or other customized write logic plans.
[0036] S103. Based on the write logic plan, repartition the data table to be written to obtain a repartition result, and write the data to be written into the target data table to merge small files.
[0037] In one embodiment, the write logic plan is checked to determine whether a repartition operation is required. For example, when data writing involves a large number of small files or data distribution is uneven, a repartition operation is performed.
[0038] In one embodiment, a repartition operation is inserted into the write logic plan, for example, the repartition operator is used to redistribute data. Specifically, a Spark repartition operator is introduced before the result stage (usually the last stage in a Spark task, outputting the final result data), which adds a layer of shuffle, so that the number of final output files is reasonably changed. The repartition operator can repartition the data table to be written to modify the number of partitions, thereby changing the distribution of data in the cluster. After repartition, the data is written into the data table according to the rules after repartitioning, reducing the generation of small files.
[0039] In one embodiment, the optimized logical plan is converted into a physical plan, which describes in detail the specific execution steps of the data writing operation. According to the physical plan, the data writing operation is performed to write the repartitioned data into the corresponding partition of the data table. After the data writing is completed, the metadata of Hive is updated to ensure that the newly written data can be correctly identified and managed.
[0040] The above embodiment provides a method, device, computer equipment and storage medium for merging small files based on Spark. When receiving the data to be written, the data query instruction corresponding to the data to be written and the data table to be written are obtained; based on the data query instruction and the data table to be written, a write logic plan is determined; based on the write logic plan, the data table to be written is repartitioned to obtain the repartition result, and the data to be written is written into the target data table to merge small files. The present application can repartition the data table to be written after obtaining the data to be written and before the data is written, and write the data to be written into the repartitioned data table to be written, so that small files are automatically merged, the generation of small files is reduced, and the stability of Spark tasks is improved.
[0041] See also Figure 2 , Figure 2 This is a schematic flow chart of a small file merging method based on Spark provided by an embodiment of the present application. The small file merging method based on Spark can be applied to a server, and is used to obtain the dynamic partition column of the data table to be written when the data query instruction and the data table to be written meet the preset conditions, and repartition the data table to be written according to the dynamic partition column, obtain the repartition result, and write the data to be written into the repartitioned data table to be written, so that small files are automatically merged, the generation of small files is reduced, and the stability of Spark tasks is improved.
[0042] like Figure 2As shown, the Spark-based small file merging method specifically includes steps S201 to S204.
[0043] S201, based on the write logic plan, determining whether the data query instruction and the to-be-written data table meet preset conditions;
[0044] Furthermore, the preset conditions include that the data query instruction has been parsed, the bucket specification of the data table to be written is empty, and an expression for repartitioning exists in the data query instruction.
[0045] In one embodiment, it is checked whether the query instruction has been resolved. Specifically, it can be implemented by checking the state of the logical plan or using a specific function (such as query.resolved).
[0046] In one embodiment, metadata of the target data table is obtained and checked to see if it contains bucket specifications. This typically involves checking table properties or using a specific function (e.g., table.bucketSpec.isEmpty).
[0047] In one embodiment, bucket specifications refer to data that has been pre-partitioned according to certain keys when it is written. If the table has bucket specifications, data will follow these preset partitioning rules when it is written, and no additional repartitioning operations are required. If the table does not have bucket specifications, repartitioning operations can be performed when the data is used.
[0048] In one embodiment, the expression in the query instruction is analyzed to determine whether there is an expression that can be used for repartitioning. Specifically, the relationship between the query output field and the table partition field is checked, or a specific function (such as canInsertRepartitionByExpression(query)) is used.
[0049] In one embodiment, repartition expressions are the basis for performing repartition operations, and they determine how data is redistributed to different partitions or nodes.
[0050] S202: when the data query instruction and the data table to be written meet preset conditions, repartition the data table to be written to obtain the repartition result;
[0051] In one embodiment, if the data query instruction and the data table to be written meet preset conditions, a repartition operation is performed on the data table to be written to obtain a repartition result, so as to optimize the physical storage layout of the data according to the repartition result.
[0052] Furthermore, the repartitioning the data table to be written to obtain the repartitioning result includes: obtaining the dynamic partitioning column of the data table to be written based on the write logical plan; constructing a repartitioning logical plan based on the data to be written and the dynamic partitioning column; and repartitioning the data table to be written based on the repartitioning logical plan to obtain the repartitioning result.
[0053] In one embodiment, the acquisition process of the dynamic partition column is determined according to the type of the write logic plan. Specifically, the write logic plan can be one of InsertIntoHiveTable, createHiveTableAsSelectCommand, or OptimizedCreateHiveTableAsSelectCommand, or other customized write logic plans. Each write logic plan corresponds to an acquisition process of a dynamic partition column.
[0054] In one embodiment, based on the acquired dynamic partition columns and the write logical plan, a repartition logical plan is constructed, which includes a repartition operation. Specifically, the Spark repartition operator or similar operation is used to repartition the data table to be written according to the dynamic partition columns to optimize data distribution.
[0055] In one embodiment, after executing the repartitioning logical plan, repartitioning results are obtained, which are used for subsequent data writing operations.
[0056] Furthermore, the obtaining of the dynamic partition column of the data table to be written based on the write logical plan includes: when the write logical plan is a first logical plan, obtaining partition information of the data table to be written, wherein the partition information includes a partition value; based on the partition information, determining a partition column whose partition value is empty; searching for a field corresponding to the partition column whose partition value is empty in a field list of the data to be written, and converting the field into a sequence to obtain the dynamic partition column.
[0057] In one embodiment, when the write logical plan is the first logical plan (for example, InsertIntoHiveTable), the partition information of the data table to be written is obtained. The partition information is in the form of a key-value pair. The partition key usually includes an identifier or name of the partition, and the partition value is usually a specific attribute or range of the data contained in the partition.
[0058] In one embodiment, a column with an empty partition value is found from the partition information, a field corresponding to the empty partition value column is found in a field list of data to be written, and these fields are converted into sequences to obtain a dynamic partition column.
[0059] Furthermore, obtaining the dynamic partition column of the data table to be written based on the write logical plan includes: when the write logical plan is the second logical plan or the third logical plan, obtaining a field list of the data to be written; comparing the field name with the partition column name in the data table to be written, and obtaining at least one field whose field name is the same as the partition column name as the dynamic partition column.
[0060] In one embodiment, when the write logical plan is the second logical plan or the third logical plan (for example, createHiveTableAsSelectCommand or OptimizedCreateHiveTableAsSelectCommand), a field list of data to be written is obtained.
[0061] In one embodiment, the field names are compared with the partition column names in the data table to be written, and fields with the same field names as the partition column names are found, and these fields are used as dynamic partition columns.
[0062] Furthermore, constructing a repartitioning logical plan based on the data to be written and the dynamic partition columns includes: obtaining an original partitioning state and an original data distribution rule of the data to be written; performing a comprehensive analysis on the original partitioning state, the original data distribution rule and the dynamic partition columns to obtain a target number of partitions and a target data distribution rule of the data table to be written; and constructing the repartitioning logical plan based on the target number of partitions and the data distribution rule.
[0063] In one embodiment, the raw partition state is used to determine how the data to be written is currently partitioned, including how the data is divided into different partitions and the amount of data contained in each partition.
[0064] In one embodiment, the new number of partitions (target number of partitions) is determined by analyzing the original partition state and data distribution rules in combination with the characteristics of the dynamic partition column.
[0065] In a specific embodiment, based on the data volume and cluster resources, the ideal number of partitions is calculated to optimize the processing speed and resource utilization. According to the value of the dynamic partition column, the target data distribution rule is designed to ensure that the data can be evenly distributed to the new partitions. According to the target number of partitions and the data distribution rule, a repartitioning logic plan is generated to perform the actual repartitioning operation under the guidance of the repartitioning logic plan.
[0066] In the above embodiments, the data table to be written can be effectively repartitioned to optimize data storage and query performance, especially when processing large-scale data sets, to reduce the generation of small files and improve the stability of Spark tasks.
[0067] S203, generating a data write node based on the write logic plan and the repartitioning result;
[0068] In one embodiment, the data writing node is a specific type of node in the logical plan, which is responsible for performing the actual data writing operation and is used to write data to a specified storage location according to the instructions of the logical plan.
[0069] In one embodiment, a data write node is generated according to the structure of the repartitioned data table and the target data table. When generating a data write node, relevant attributes such as file format, compression options, storage path, schema (structure) of the data table, etc. need to be configured.
[0070] S204. Based on the data writing node, write the data to be written into the repartitioned data table to be written.
[0071] In one embodiment, the execution engine of Spark is used to perform the actual data writing operation according to the generated data writing node, and the data to be written is written into the repartitioned data table according to the target data distribution rule. Specifically, Spark writes the data into the repartitioned data table according to the target data distribution rule in the data writing node. After the data is written, the metadata of the data table is updated to ensure that the newly written data can be correctly recognized and managed by Hive.
[0072] Exemplarily, the key codes for implementing the above embodiment are as follows:
[0073]
[0074]
[0075] In one embodiment, in Spark SQL, the Union operation is used to merge multiple query result sets. case u@Union(children) is a Scala pattern matching statement used to identify the Union node in the logical plan. u is the instance of the currently matched Union node, and children is a list of all subqueries (or child nodes) contained in the Union node. u.copy(children=children.map(addRepartition)) uses the copy method to create a new instance of the Union node. children.map(addRepartition) recursively applies the addRepartition function to each child node in the Union node. The map function applies the addRepartition function to each element in the children list.
[0076] In the above embodiment, when the data query instruction and the data table to be written meet the preset conditions, the data table to be written is repartitioned to obtain the repartitioning result, and the data to be written is written into the repartitioned data table to be written, so that small files are automatically merged, the generation of small files is reduced, and the stability of the Spark task is improved.
[0077] See also Figure 3 , Figure 3 The embodiment of the present application provides a schematic block diagram of a small file merging device based on Spark, which is used to execute the aforementioned small file merging method based on Spark. The small file merging device based on Spark can be configured on a server.
[0078] like Figure 3 As shown, the Spark-based small file merging device 300 includes:
[0079] The to-be-written table acquisition module 301 is used to acquire the data query instruction and the to-be-written data table corresponding to the to-be-written data when receiving the to-be-written data;
[0080] A logic plan determination module 302, configured to determine a write logic plan based on the data query instruction and the data table to be written;
[0081] The data writing module 303 is used to repartition the data table to be written based on the writing logic plan, obtain a repartition result, and write the data to be written into the target data table to merge small files.
[0082] Furthermore, the data writing module 303 includes:
[0083] A condition judgment unit, used to judge whether the data query instruction and the to-be-written data table meet preset conditions based on the write logic plan;
[0084] A repartitioning result obtaining unit, configured to repartition the data table to be written to obtain the repartitioning result when the data query instruction and the data table to be written meet preset conditions;
[0085] A write node generation unit, used to generate a data write node based on the write logic plan and the repartitioning result;
[0086] A data writing unit is used to write the to-be-written data into the repartitioned to-be-written data table based on the data writing node.
[0087] Furthermore, the preset conditions include that the data query instruction has been parsed, the bucket specification of the data table to be written is empty, and an expression for repartitioning exists in the data query instruction.
[0088] Furthermore, the repartitioning result obtaining unit includes:
[0089] A dynamic partition column acquisition unit, used for acquiring the dynamic partition column of the data table to be written based on the write logic plan;
[0090] A logical plan building unit, used to build a repartitioning logical plan based on the data to be written and the dynamic partition column;
[0091] A repartitioning result obtaining unit is used to repartition the to-be-written data table based on the repartitioning logic plan to obtain the repartitioning result.
[0092] Furthermore, the dynamic partition column acquisition unit includes:
[0093] A partition information acquisition subunit, used for acquiring the partition information of the data table to be written when the write logical plan is the first logical plan, wherein the partition information includes a partition value;
[0094] A partition column determination subunit, configured to determine a partition column whose partition value is empty based on the partition information;
[0095] The dynamic partition column obtaining subunit is used to search the field corresponding to the partition column whose partition value is empty in the field list of the data to be written, and convert the field into a sequence to obtain the dynamic partition column.
[0096] Furthermore, the dynamic partition column acquisition unit includes:
[0097] A field list acquisition subunit, used for acquiring the field list of the data to be written when the write logical plan is the second logical plan or the third logical plan;
[0098] The field obtaining subunit is used to compare the field name with the partition column name in the data table to be written, and obtain at least one field whose field name is the same as the partition column name as the dynamic partition column.
[0099] Furthermore, the logic plan construction unit includes:
[0100] An original rule acquisition subunit, used to acquire the original partition state and original data distribution rule of the data to be written;
[0101] A target rule obtaining subunit is used to comprehensively analyze the original partition state, the original data distribution rule and the dynamic partition column to obtain the target partition number and target data distribution rule of the data table to be written;
[0102] The logical plan building subunit is used to build the repartitioning logical plan based on the target number of partitions and the data distribution rule.
[0103] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0104] The above-mentioned device can be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.
[0105] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0106] See also Figure 4 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0107] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any Spark-based small file merging method.
[0108] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0109] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any Spark-based small file merging method.
[0110] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0111] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0112] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0113] When receiving the data to be written, obtaining a data query instruction corresponding to the data to be written and a data table to be written;
[0114] Determine a write logic plan based on the data query instruction and the data table to be written;
[0115] Based on the write logic plan, the data table to be written is repartitioned to obtain a repartition result, and the data to be written is written into the target data table to merge small files.
[0116] In one embodiment, when the processor repartitions the data table to be written based on the write logic plan, obtains a repartition result, and writes the data to be written into the target data table, it is configured to implement:
[0117] Based on the write logic plan, determining whether the data query instruction and the to-be-written data table meet preset conditions;
[0118] When the data query instruction and the data table to be written meet preset conditions, repartitioning the data table to be written to obtain the repartitioning result;
[0119] Generate a data write node based on the write logic plan and the repartition result;
[0120] Based on the data writing node, the to-be-written data is written into the repartitioned to-be-written data table.
[0121] In one embodiment, the preset conditions include that the data query instruction has been parsed, the bucket specification of the data table to be written is empty, and an expression for repartitioning exists in the data query instruction.
[0122] In one embodiment, when the processor repartitions the to-be-written data table and obtains the repartition result, it is configured to implement:
[0123] Based on the write logic plan, obtain the dynamic partition column of the data table to be written;
[0124] Constructing a repartitioning logic plan based on the data to be written and the dynamic partition column;
[0125] The to-be-written data table is repartitioned based on the repartitioning logical plan to obtain the repartitioning result.
[0126] In one embodiment, when the processor obtains the dynamic partition column of the data table to be written based on the write logic plan, it is used to implement:
[0127] When the write logical plan is the first logical plan, obtaining partition information of the data table to be written, wherein the partition information includes a partition value;
[0128] Based on the partition information, determine a partition column whose partition value is empty;
[0129] The field corresponding to the partition column whose partition value is empty is searched in the field list of the data to be written, and the field is converted into a sequence to obtain the dynamic partition column.
[0130] In one embodiment, when the processor obtains the dynamic partition column of the data table to be written based on the write logic plan, it is used to implement:
[0131] When the write logical plan is the second logical plan or the third logical plan, obtaining a field list of the data to be written;
[0132] The field name is compared with the partition column name in the data table to be written, and at least one field whose field name is the same as the partition column name is obtained as the dynamic partition column.
[0133] In one embodiment, when constructing a repartitioning logic plan based on the data to be written and the dynamic partition column, the processor is used to implement:
[0134] Obtaining the original partition state and original data distribution rule of the data to be written;
[0135] Comprehensively analyzing the original partition state, the original data distribution rule, and the dynamic partition column to obtain the target partition number and target data distribution rule of the data table to be written;
[0136] The repartitioning logical plan is constructed based on the target number of partitions and the data distribution rule.
[0137] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any Spark-based small file merging method provided in the embodiment of the present application.
[0138] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A small file merging method based on Spark, characterized in that: include: When receiving the data to be written, obtaining a data query instruction corresponding to the data to be written and a data table to be written; Determine a write logic plan based on the data query instruction and the data table to be written; Based on the write logic plan, the data table to be written is repartitioned to obtain a repartition result, and the data to be written is written into the target data table to merge small files.
2. The small file merging method based on Spark according to claim 1 is characterized in that: The step of repartitioning the data table to be written based on the write logic plan, obtaining a repartition result, and writing the data to be written into the target data table includes: Based on the write logic plan, determining whether the data query instruction and the to-be-written data table meet preset conditions; When the data query instruction and the data table to be written meet preset conditions, repartitioning the data table to be written to obtain the repartitioning result; Generate a data write node based on the write logic plan and the repartition result; Based on the data writing node, the to-be-written data is written into the repartitioned to-be-written data table.
3. The small file merging method based on Spark according to claim 2 is characterized in that: The preset conditions include that the data query instruction has been parsed, the bucket specification of the data table to be written is empty, and an expression for repartitioning exists in the data query instruction.
4. The method for merging small files based on Spark according to claim 2, characterized in that: The repartitioning the to-be-written data table to obtain the repartitioning result includes: Based on the write logic plan, obtain the dynamic partition column of the data table to be written; Constructing a repartitioning logic plan based on the data to be written and the dynamic partition column; The to-be-written data table is repartitioned based on the repartitioning logical plan to obtain the repartitioning result.
5. The method for merging small files based on Spark according to claim 4, characterized in that: The acquiring the dynamic partition column of the data table to be written based on the write logic plan includes: When the write logical plan is the first logical plan, obtaining partition information of the data table to be written, wherein the partition information includes a partition value; Based on the partition information, determine a partition column whose partition value is empty; The field corresponding to the partition column whose partition value is empty is searched in the field list of the data to be written, and the field is converted into a sequence to obtain the dynamic partition column.
6. The method for merging small files based on Spark according to claim 4, characterized in that: The acquiring the dynamic partition column of the data table to be written based on the write logic plan includes: When the write logical plan is the second logical plan or the third logical plan, obtaining a field list of the data to be written; The field name is compared with the partition column name in the data table to be written, and at least one field whose field name is the same as the partition column name is obtained as the dynamic partition column.
7. The method for merging small files based on Spark according to any one of claims 4 to 6, characterized in that: The constructing a repartitioning logic plan based on the data to be written and the dynamic partition column includes: Obtaining the original partition state and original data distribution rule of the data to be written; Comprehensively analyzing the original partition state, the original data distribution rule, and the dynamic partition column to obtain the target partition number and target data distribution rule of the data table to be written; The repartitioning logical plan is constructed based on the target number of partitions and the data distribution rule.
8. A small file merging device based on Spark, characterized in that: include: A module for acquiring a table to be written, used for acquiring a data query instruction corresponding to the data to be written and a table of data to be written when receiving the data to be written; A logic plan determination module, used to determine a write logic plan based on the data query instruction and the data table to be written; The data writing module is used to repartition the data table to be written based on the writing logic plan, obtain a repartition result, and write the data to be written into the target data table to merge small files.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the Spark-based small file merging method as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the Spark-based small file merging method according to any one of claims 1 to 7.