Spark-based file size adaptive processing method and system

Abstract

The application provides a Spark-based file size adaptive processing method and system, wherein the method comprises: scanning a list of historical input files and historical output files to obtain sizes of the historical input files and the historical output files; determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files; scanning a list of current input files to obtain sizes of the current input files; determining sizes of current output files based on the average input-output conversion ratio and the sizes of the current input files; determining partition parameters based on the sizes of the current output files; and adaptively controlling the sizes of the files using the partition parameters.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data processing, and more particularly, to a method and system for controlling file size by adjusting partition parameters for a Spark task. BACKGROUND

[0002] With the continuous development of the Internet and 5G technology, the development momentum of big data applications is very rapid, and has a positive and extensive application in telecommunications, e-commerce, media, transportation and other aspects, for example, it can be applied to processing massive data in the field of video networking, such as video monitoring data, etc. Hadoop is a distributed system infrastructure developed by the Apache Foundation. This offline high-throughput and low-response framework can no longer meet some needs, and people's requirements for the efficiency and latency of big data computing have also increased. Spark, a distributed big data computing engine based on memory, has emerged as the times require. As a fast and general-purpose computing engine designed for large-scale data processing, it is more than 100 times faster than Hadoop.

[0003] However, while Spark provides speed and convenience, it can cause a large number of small files and large single files due to dynamic partitioning and unreasonable partitioning quantity. Small files can cause long disk addressing time, and because Spark frequently creates tasks to read data, it can lead to low task efficiency. Large files can cause the time for Spark programs to transfer data from the disk to be significantly greater than the time required to locate the start position of the block, and for large files that cannot be blocked (for example, snappy compressed format files), Spark cannot take advantage of distributed parallel processing, and the program can run out of memory due to the large amount of data, which not only affects efficiency but also causes the program to fail. SUMMARY

[0004] The summary is provided to introduce some concepts in a simplified form that will be further described in the following detailed embodiments. The summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0005] To solve the above problems, the present application provides a file size adaptive processing method and system based on Spark, which pre-generates Spark partition output parameters by analyzing the current input data and combining historical input data and output data, so that the size of the Spark task output file is controlled within a certain range, thereby realizing adaptive output file size and solving the problems of too many small files and too large single files.

[0006] According to one aspect of the present application, there is provided a method for adaptive processing of file size based on Spark, comprising:

[0007] scanning a list of historical input files and historical output files to obtain sizes of the historical input files and the historical output files;

[0008] determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files;

[0009] scanning a list of current input files to obtain sizes of the current input files;

[0010] determining sizes of current output files based on the average input-output conversion ratio and the sizes of the current input files;

[0011] determining partition parameters based on the sizes of the current output files; and

[0012] adaptively controlling the sizes of the files using the partition parameters.

[0013] According to one embodiment of the present application, determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files comprises:

[0014] summing the sizes of the historical input files and the historical output files respectively;

[0015] determining an input-output conversion ratio based on the sum of the sizes of the historical input files and the sum of the sizes of the historical output files; and

[0016] averaging the input-output conversion ratio to determine the average input-output conversion ratio.

[0017] According to another embodiment of the present application, determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files comprises:

[0018] setting a historical threshold number;

[0019] in a case where the number of the historical input files and the historical output files is greater than the historical threshold number, determining an average input-output conversion ratio based on sizes of the historical threshold number of the historical input files and the historical output files most recently; or

[0020] in a case where the number of the historical input files and the historical output files is less than or equal to the historical threshold number, determining an average input-output conversion ratio based on sizes of all the historical input files and the historical output files.

[0021] According to yet another embodiment of the present application, determining the partition parameter based on the size of the current output file comprises calculating the partition parameter according to the following formula:

[0022] The partition parameter = the size of the current output file / the single file size + 1

[0023] wherein the single file size is pre-set, and " / " represents integer division.

[0024] According to yet another embodiment of the present application, wherein adaptively controlling the size of the file using the partition parameter comprises:

[0025] passing the partition parameter into a Spark task, and applying the partition parameter to a repartition operator when the Spark task outputs, thereby controlling the size of the file output by the Spark task.

[0026] According to yet another embodiment of the present application, the method further comprises:

[0027] setting a default partition parameter in the absence of scanning the historical input files and the historical output files.

[0028] According to another aspect of the present application, there is provided a Spark-based file size adaptive processing system, comprising:

[0029] a memory configured to store input files and output files;

[0030] at least one processor coupled to the memory and configured to:

[0031] scan a list of historical input files and historical output files, thereby obtaining sizes of the historical input files and the historical output files;

[0032] determine an average input-output conversion ratio based on the sizes of the historical input files and the historical output files;

[0033] scan a list of current input files, thereby obtaining sizes of the current input files;

[0034] determine a size of a current output file based on the average input-output conversion ratio and the sizes of the current input files;

[0035] determine a partition parameter based on the size of the current output file; and

[0036] adaptively control the size of the file using the partition parameter.

[0037] According to one embodiment of the present application, determining an average input-output conversion ratio based on sizes of the historical input files and the historical output files comprises:

[0038] setting a historical threshold number;

[0039] in a case that the number of the historical input files and the historical output files is greater than the historical threshold number, determining an average input-output conversion ratio based on sizes of the historical threshold number of the latest historical input files and the historical output files; or

[0040] in a case that the number of the historical input files and the historical output files is less than or equal to the historical threshold number, determining an average input-output conversion ratio based on sizes of all the historical input files and the historical output files.

[0041] According to another embodiment of the present application, determining a partition parameter based on a size of the current output file comprises calculating the partition parameter according to the following formula:

[0042] the partition parameter = the size of the current output file / single file size + 1

[0043] wherein the single file size is preset, and " / " represents integer division.

[0044] According to still another embodiment of the present application, the at least one processor is further configured to:

[0045] in a case that the historical input files and the historical output files are not scanned, setting a default partition parameter.

[0046] These and other features and advantages will be apparent from a reading of the following detailed description and a review of the associated drawings. It is to be understood that both the foregoing general description and the following detailed description are meant only to illustrate and not to limit the various aspects claimed. BRIEF DESCRIPTION OF DRAWINGS

[0047] So that the above-recited features and advantages of the present application can be understood in detail, a more particular description will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings represent only typical aspects of this application and are therefore not to be considered limiting of its scope, for the description can admit to other equally effective aspects.

[0048] Figure 1 is a flow chart of a Spark-based file size adaptive processing method according to one embodiment of the present application.

[0049] Figure 2is an example flowchart of a Spark-based HDFS file size adaptive method according to an embodiment of the present application.

[0050] Figure 3 is a partition parameter generation diagram according to an embodiment of the present application.

[0051] Figure 4 is a partition effect diagram according to an embodiment of the present application.

[0052] Figure 5 is an example architecture diagram of a Spark-based file size adaptive processing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be described in detail below with reference to the drawings. The features of the present application will be further apparent from the following detailed description of the embodiments and the accompanying drawings. The detailed description and the accompanying drawings of the embodiments are intended to illustrate the principles of the present application by way of example only, and are not intended to limit the scope of the present application, i.e., the present application is not limited to the described embodiments.

[0054] In the description of the present application, it should be noted that, unless otherwise specified, the meaning of "a plurality of" is more than two; the orientation or position relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer" and the like only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance. "Vertical" is not strictly vertical, but within the allowable range of error. "Parallel" is not strictly parallel, but within the allowable range of error.

[0055] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0056] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0057] For the problem of too many small files or too large single file of Spark task, the existing solution is often to start a new task for secondary merging / splitting processing, but in the case of too large input data source, it is easy to bring additional overhead. The present application provides a file size adaptive method based on Spark, which reasonably controls the number and size of Spark output files, thereby reducing the overhead of Spark task, reasonably utilizing the distributed and multi-threaded properties of Spark, and thereby improving the efficiency of Spark task.

[0058] Figure 1 is a flow chart of a file size adaptive processing method 100 based on Spark according to an embodiment of the present application. As shown in Figure 1 Method 100 starts at step 102, which can scan the list of historical input files and historical output files to obtain the size information of the historical input files and the historical output files.

[0059] Then, the method proceeds to step 104, which can determine the average input-output conversion ratio based on the size of the historical input files and the historical output files.

[0060] As an example, the size of the historical input files and the historical output files can be summed up respectively, and the input-output conversion ratio can be determined based on the sum of the size of the historical input files and the sum of the size of the historical output files. By this method, the size information of multiple historical input and output files of the Spark task is calculated respectively, and multiple corresponding input-output conversion ratios are obtained, and then the input-output conversion ratios are averaged to determine the average input-output conversion ratio. For example, specifically, the size information of the historical input and output files can be aggregated, and the input-output conversion ratio of the Spark task can be calculated by the following formula:

[0061] Input-output conversion ratio = SUM(historical input file size) / SUM(historical output file size) * 100%

[0062] Where SUM() represents the summation of each term in the parentheses. Then the input-output conversion ratios are averaged to obtain the average input-output conversion ratio of the Spark task.

[0063] In further non-limiting embodiments, a history threshold number can be set. In a case where the number of history input files and history output files is greater than the history threshold number, the average input-output conversion ratio is determined based on the sizes of the most recent history threshold number of history input files and history output files. Specifically, the size information of the most recent history threshold number of history input, output files is aggregated to obtain a history threshold number of corresponding input-output conversion ratios, and the input-output conversion ratios are averaged to determine the average input-output conversion ratio. Alternatively, in a case where the number of history input files and history output files is less than or equal to the history threshold number, the average input-output conversion ratio can be determined based on the sizes of all history input files and history output files. Specifically, the size information of all history input, output files is aggregated to obtain a plurality of corresponding input-output conversion ratios, and the input-output conversion ratios are averaged to determine the average input-output conversion ratio.

[0064] Next, at step 106, the list of current input files can be scanned to obtain the size of the current input files.

[0065] Subsequently, at step 108, the size of the current output file can be determined based on the average input-output conversion ratio determined at step 104 and the size of the current input file obtained at step 106. As an example, the size of the current output file can be predicted by:

[0066] Current output file size = Current input file size / Average input-output conversion ratio

[0067] At step 110, the partition parameter can be determined based on the size of the current output file obtained at step 108. As an example, the partition parameter for the Spark task can be generated according to the following formula:

[0068] Partition parameter = Current output file size / Single file size + 1

[0069] where the single file size can be pre-set, for example, the single file size can be set as 128 MB by default, but it can be appreciated that the single file size can also be adjusted to other values; in addition, " / " represents integer division.

[0070] Finally, in step 112, the partition parameter can be used to adaptively control the size of the file. As an example, when running a Spark task, the partition parameter obtained in step 110 can be passed into the Spark task, and the partition parameter can be applied to the repartition operator of the Spark task output (the repartition operator is an operator in Spark for resetting the number of partitions), thereby controlling the size and number of files of the Spark task output, achieving the effect of adaptation.

[0071] It can be appreciated that, as a non-limiting example, when no historical input file and historical output file is scanned in step 102, a default partition parameter can be set. For example, if no scanning is successful in step 102, i.e., the task is run for the first time, there is no historical input and output file, steps 104-110 can be skipped, and the default partition parameter can be generated directly.

[0072] The following will be described in conjunction with Figure 2 A specific embodiment of the file size adaptive processing method of the present application will be further described below with an example of a Spark-based Hadoop Distributed File System (HDFS). However, it can be appreciated that, although Figure 2 Although a distributed file size adaptive processing method is described, the method of the present application is not limited to processing distributed files, and can be applied to processing other file systems according to actual needs. Figure 2 is an example flowchart of a Spark-based HDFS file size adaptive method 200 according to an embodiment of the present application. Hadoop Distributed File System (HDFS) refers to a distributed file system (Distributed File System) designed to run on commodity hardware. HDFS has the characteristics of high fault tolerance, and it can provide high throughput data access, which is very suitable for large-scale data sets. The present application pre-generates a partition parameter by combining the historical input and output size of a Spark task with the current input of the task, thereby achieving the adaptation of the HDFS file size of the Spark task output, effectively improving the running efficiency and stability of the subsequent Spark task.

[0073] Method 200 starts in step 202, and the historical input and output file list information configured by a Spark task can be obtained, wherein the historical input and output file list of the Spark task can be pre-configured.

[0074] In step 204, it is determined whether the historical input and output file list exists. If not, it means that the task is run for the first time, and the process jumps to step 206; if yes, the process jumps to step 204.

[0075] At step 206, the partition parameter of the Spark task is generated by default, and the process jumps to step 218. As a non-limiting example, the partition parameter of the Spark task can be set to 200. It can be appreciated that in actual use, any default partition parameter can also be set arbitrarily according to the data output. For example, when the data volume is small at the first time, a smaller default partition parameter can be manually set. After the Spark task is executed using the default partition parameter, the link is closed due to the generation of the input and output file information, and the normal step process (i.e., steps 208-218) is executed.

[0076] At step 208, the size information of the historical input and output file list can be obtained through the HDFS command or API, and the unit of the file size can be bytes (Byte).

[0077] At step 210, for a single history, the historical input and output file sizes can be summed up respectively, and the input-output conversion ratio can be calculated by SUM (input size) / SUM (output size) * 100%. As a non-limiting example, for multiple histories, the input-output conversion ratio of the selected number of histories is averaged to obtain the average input-output conversion ratio. It can be appreciated that other number of historical input and output file sizes can also be selected according to actual needs.

[0078] At step 212, the size information of the current task input file list can be obtained through the HDFS command or API, and the current input file size can be summed up.

[0079] At step 214, based on the average input-output conversion ratio obtained at step 210 and the current input file size obtained at step 212, the current output file size can be budgeted according to the following formula:

[0080] Predicted current output file size = current input file size / average input-output conversion ratio

[0081] At step 216, the number of partitions can be calculated by the following formula:

[0082] Number of partitions = predicted current output file size / (1024*1024*128)+1

[0083] Where 1024*1024*128 refers to the conversion of the file size unit from bytes to gigabytes (GB), as described in the reference Figure 1The default single file size 128MB in the table is converted to Byte; " / " represents division and rounding down, and adding 1 is to prevent the partition parameter from being 0, and the final number of partitions is the partition parameter of the Spark task.

[0084] In step 218, the partition parameter can be passed into the Spark task, and the partition parameter is applied to the repartition operator of the Spark task output, so that the HDFS file size of the Spark task output is finally adaptive.

[0085] Figure 3 is a partition parameter generation diagram according to an embodiment of the present application. As can be seen, the present application can obtain an average input-output conversion ratio of 478.566% based on the input and output file list of the same Spark task history. In combination with the preferred average input-output conversion ratio and the input file list of the current Spark task (the input file size is 105.43GB), the current output file size (22.0303GB) can be predicted. In order to ensure that the output single file is close to the preset file output size (the default is 128MB), a preferred partition parameter 177 is generated.

[0086] Figure 4 is a partition effect diagram according to an embodiment of the present application. As shown in Figure 4 , in the case that the partition parameter 117 obtained in Figure 3 is applied to the repartition operator of the Spark task output, the actual file size of the output is close to 128MB. As can be seen, the partition parameter obtained by the Spark-based HDFS file size adaptive method of the present application can control the size of the Spark task output file within a certain range, so that the output file size is adaptive, and the problems of too large small files and too large single files are solved.

[0087] Figure 5 is an example architecture diagram of a Spark-based file size adaptive processing system 500 according to an embodiment of the present application. As shown in Figure 5 , the Spark-based file size adaptive processing system 500 can include a memory 502 and one or more processors 504.

[0088] The memory 502 can be configured to store input files and output files, including historical input files and historical output files, and current input files and output files.

[0089] The processor 504 can be coupled to the memory 502 and configured to perform as Figure 1The Spark-based file size adaptive processing method 100 is implemented as follows: scanning a list of historical input files and historical output files to obtain sizes of the historical input files and the historical output files; determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files; scanning a list of current input files to obtain sizes of the current input files; determining sizes of current output files based on the average input-output conversion ratio and the sizes of the current input files; determining partition parameters based on the sizes of the current output files; and adaptively controlling the file sizes using the partition parameters.

[0090] As an example, determining the average input-output conversion ratio based on the sizes of the historical input files and the historical output files can include: setting a historical threshold number; in a case where a number of the historical input files and the historical output files is greater than the historical threshold number, determining the average input-output conversion ratio based on sizes of the most recent historical threshold number of historical input files and historical output files; or in a case where the number of the historical input files and the historical output files is less than or equal to the historical threshold number, determining the average input-output conversion ratio based on sizes of all the historical input files and the historical output files.

[0091] As another example, determining the partition parameters based on the sizes of the current output files can include calculating the partition parameters according to the following formula:

[0092] Partition parameters = sizes of the current output files / single file size + 1

[0093] wherein the single file size can be pre-set, and “ / ” represents integer division.

[0094] As a further non-limiting embodiment, the at least one processor 504 can be further configured to directly set a default partition parameter without scanning the historical input files and the historical output files.

[0095] The above describes the Spark-based file size adaptive processing method and system of the present application, which has at least the following advantages compared with the prior art:

[0096] For the problem of too many small files or too large single file of Spark task, the prior art often uses relatively static setting or starts a new processing task. However, the static setting is not easy for long-term development and needs to be updated regularly. In addition, starting a new task also needs to consume a part of extra computing resources and management resources, and does not take the means to radically cure from the source. The present application pre-generates the partition parameters of Spark based on the analysis of the current Spark task input data and the historical input data and output data, so that the partition parameters can dynamically change within a certain range, adapt to the current scene, and make the output file size adaptive without affecting the logic of the current Spark task, so as to solve the problems of too many small files and too large single file within a certain range.

[0097] The present application controls the size and number of files reasonably by self-adapting the output file size of Spark task, thereby avoiding too many small files or too large single file, solving the problems of too large computing resource consumption, too large cluster Namenode pressure and the like caused by too many small files, and solving the problems of low utilization efficiency of Spark distributed and multi-thread advantages, memory overflow and the like caused by too large single file, effectively improving the running efficiency and program stability of Spark task, reducing the load of cluster Namenode, conforming to the current development direction of saving resources and high efficiency, and having strong practicality and application range in the field of big data processing such as video monitoring.

[0098] The above-described content includes examples of aspects of the claimed subject matter. Of course, it is impossible to describe every conceivable combination of components or methods for purposes of describing the claimed subject matter, but one of ordinary skill in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A Spark-based adaptive file size processing method, comprising: scanning a list of historical input files and historical output files to obtain sizes of the historical input files and the historical output files; determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files; scanning a list of current input files to obtain sizes of the current input files; determining sizes of current output files based on the average input-output conversion ratio and the sizes of the current input files; determining a partition parameter based on the sizes of the current output files; and controlling sizes of the files adaptively using the partition parameter, wherein controlling sizes of the files adaptively using the partition parameter comprises: passing the partition parameter into a Spark task and applying the partition parameter to a repartition operator when the Spark task outputs, to control sizes of the files output by the Spark task. determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files comprises:

2. The method of claim 1, wherein, summing the sizes of the historical input files and the historical output files respectively; determining input-output conversion ratios based on the sum of the sizes of the historical input files and the sum of the sizes of the historical output files; and averaging the input-output conversion ratios to determine the average input-output conversion ratio. determining an average input-output conversion ratio based on the sizes of the historical input files and the historical output files comprises:

3. The method of claim 1, wherein, setting a historical threshold number; in a case where a number of the historical input files and the historical output files is greater than the historical threshold number, determining an average input-output conversion ratio based on sizes of the historical threshold number of the historical input files and the historical output files most recently; or in a case where the number of the historical input files and the historical output files is less than or equal to the historical threshold number, determining an average input-output conversion ratio based on sizes of all the historical input files and the historical output files. determining a partition parameter based on the sizes of the current output files comprises calculating the partition parameter according to a formula:

4. The method of claim 1, wherein, the partition parameter = the size of the current output file / single file size + 1 wherein the single file size is preset, and " / " represents integer division. The method further comprises:

5. The method of claim 1, wherein, in a case where no historical input files and historical output files are scanned, setting a default partition parameter. 6.A Spark-based adaptive file size processing system, comprising: a memory configured to store input files and output files; at least one processor coupled to the memory and configured to: scan a list of historical input files and historical output files to obtain sizes of the historical input files and the historical output files; determine an average input-output conversion ratio based on the sizes of the historical input files and the historical output files; ​ scanning a list of current input files to obtain sizes of the current input files; determining a size of a current output file based on the average input-output conversion ratio and the sizes of the current input files; determining a partition parameter based on the size of the current output file; and controlling the size of the file adaptively using the partition parameter, wherein controlling the size of the file adaptively using the partition parameter comprises: passing the partition parameter into a Spark task and applying the partition parameter to a repartition operator when the Spark task outputs, thereby controlling the size of the file output by the Spark task.

7. The system of claim 6, wherein, determining the average input-output conversion ratio based on the sizes of the historical input files and the historical output files comprises: setting a historical threshold number; in a case where the number of the historical input files and the historical output files is greater than the historical threshold number, determining the average input-output conversion ratio based on the sizes of the historical threshold number of the most recent historical input files and the historical output files; or in a case where the number of the historical input files and the historical output files is less than or equal to the historical threshold number, determining the average input-output conversion ratio based on the sizes of all the historical input files and the historical output files.

8. The system of claim 6, wherein, determining the partition parameter based on the size of the current output file comprises calculating the partition parameter according to the following formula: the partition parameter = the size of the current output file / single file size + 1 wherein the single file size is preset, and " / " represents integer division.

9. The system of claim 6, wherein, the at least one processor is further configured to: in a case where the historical input files and the historical output files are not scanned, setting a default partition parameter.

Images