A Spark memory replacement method based on dynamic capacity

Abstract

The Spark memory replacement method based on dynamic capacity disclosed in the present application includes: configuring a first replacement algorithm and a second replacement algorithm; when a new RDD needs to be stored, selecting a replacement algorithm according to the degree of memory tension; when the second replacement algorithm is triggered, dividing the existing RDD into two tables with dependency count and without dependency count for maintenance; then obtaining the weight of the existing RDD; then confirming whether the RDD table without dependency count is empty: if it is not empty, then evict the existing RDDs from the memory one by one in the RDD table without dependency count according to the weight from small to large until the new RDD is cached enough and then stop; if it is empty, then traverse the RDD table with dependency count and evict the RDD in the same way until the space is enough to accommodate the new RDD. This solution can reduce the read and write overhead of Spark during operation and the impact on performance to a greater extent under the conditions of different memory environments, improve the cache hit rate, and improve the operating efficiency of Spark.

Description

Technical Field

[0001] The present application relates to the technical field of big data Spark computing engines, and more specifically, to a Spark memory replacement method based on dynamic capacity. Background Art

[0002] Apache Sprak is a fast and general computing engine designed for large-scale data processing. It has the advantages of Hadoop MapReduce, but different from MapReduce, the intermediate output of the job can be stored in memory, so there is no need to read and write HDFS. Therefore, Spark is widely used in MapReduce algorithms that require iteration, such as data mining and machine learning. In the original Spark, the computing engine stores data files in memory to reduce the read and write overhead of the data when it is used next time. When the memory space is insufficient to store a new RDD (resilient distributed dataset) data file, the Spark engine will call a replacement algorithm to eliminate the old RDD and read in the new RDD. The replacement algorithm commonly used by the current Spark computing engine is LRU (least recently used). This algorithm records the time t that each RDD has experienced since it was last used. When an RDD needs to be eliminated, the algorithm will select the RDD with the largest t value among the existing RDDs, that is, the RDD that has been used least recently to eliminate it. However, this method has the problem of low Spark cache hit rate and high read and write overhead.

[0003] To address the issue of Spark cache algorithm replacement, researchers have provided solutions such as the Least Recently / Frequently Used (LRFU) algorithm, RDD partition-based weighted cache replacement algorithms such as the WSCRP weighted algorithm, and the LRC (Least Reference Count) algorithm. The above solutions provide new eviction priority calculation methods in their own ways, and have made improvements to Spark's LRU-based cache replacement solutions in certain scenarios. Each has its own unique advantages, and both reduce the read and write burden of Spark's workloads. However, the above algorithms still have their own shortcomings, resulting in unsatisfactory results in certain specific scenarios. For example, LRFU is an improved algorithm based on the LRU algorithm. In essence, it adds the information of the data blocks that have been used on the basis of the LRU algorithm, and gives a greater weight to the data blocks that have been used more recently, so as to judge the possibility of the RDD being used in the future. Therefore, in order for the algorithm to be able to relatively accurately judge the possibility of the data block being used, a certain number of iterations is required. When the number of iterations is small or most of the work is linear, the algorithm is easy to evict the data blocks that need to be reused from the memory, resulting in a decrease in the cache hit rate, affecting the performance of Spark work, and the other algorithms also have the above-mentioned effects. It can be seen that the cache replacement scheme in the prior art still lacks comprehensive consideration of different situations in design, as well as the optimization capability corresponding to different usage scenarios, so there is still room for optimization of Spark's read and write overhead performance at work.

[0004] Therefore, how to provide a Spark memory replacement method based on dynamic capacity, which can minimize the read and write overhead and performance impact of Spark when performing operations under different memory environments, improve cache hit rate, and enhance the operating efficiency of Spark, has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] To solve the above technical problems, the present application provides a Spark memory replacement method based on dynamic capacity, which can greatly reduce the read and write overhead and performance impact of Spark when performing operations under different memory environments.

[0006] The technical solutions provided by this application are as follows:

[0007] The present application provides a Spark memory replacement method based on dynamic capacity, comprising the following steps: S1, configuration algorithm: configuring a first replacement algorithm: LRU algorithm; and a second replacement algorithm: based on weight and dependency counting algorithm; S2, cache calculation: when a new RDD data file needs to be stored, traverse the maintained RDD table, and calculate the RDD average value therein, and select the replacement algorithm according to the ratio of the RDD average value to the available memory space of StorageManager in Spark; S3, RDD division: when the second replacement algorithm is triggered, maintain an RDD dependency count table when analyzing the Stage using DAGScheduler in Spark, and divide the existing RDD in the Stage into two tables with dependency counts and without dependency counts for maintenance based on this table; whenever S When a Task in park is calculated, the dependency count of the used RDD is reduced by one. If the dependency count of the RDD is reduced to 0, the RDD is transferred from the dependency count queue to the non-dependency count queue; S4, weight calculation: then obtain the parameters - RDD data size, RDD dependency and RDD usage times; calculate the weight of the existing RDD by weighting the obtained parameter values; S5, data cache: confirm whether the RDD table without dependency count is empty: if it is not empty, then in the RDD table without dependency count, the existing RDDs are evicted from the memory one by one in the order of weight from small to large, and the remaining space is checked after each eviction. If it is empty, traverse the RDD table with dependency count and evict the existing RDDs one by one in the same way until there is enough space to accommodate the new RDD.

[0008] Further, in a preferred embodiment of the present invention, in step S2, the specific rule for selecting the replacement algorithm is: if the RDD average value is less than 20% of the available memory space of the StorageManager, the first replacement algorithm is selected for cache replacement;

[0009] If the RDD average value is greater than or equal to 20% of the available memory space of the StorageManager, the second replacement algorithm is selected for cache replacement.

[0010] Further, in a preferred embodiment of the present invention, in step S4, the specific steps of obtaining the RDD data size are as follows:

[0011] S1. Partition the RDD calculation and distribute the tasks to multiple Executors of Spark for execution.

[0012] S2. During the task execution, RDD needs to be stored. Spark's BlockManager is used to request data cache, and then the acquireMemory method is used to test whether there is enough cache space in the memory.

[0013] S3. Then, the RDD data size is calculated based on the cache space obtained by the test.

[0014] Furthermore, in a preferred embodiment of the present invention, the RDD data size calculation expression is specifically:

[0015] Among them, S acquireMemory The cache space size obtained by testing with the acquireMemory method.

[0016] Furthermore, in a preferred embodiment of the present invention, in step S4, the specific steps of obtaining the RDD dependency relationship include:

[0017] S1. Determine the wide and narrow dependency relationship between the current RDD and its parent RDD. Based on the communication rules of the Executors in the cluster, convert the RDD dependency into the number of communications between Executors for calculation.

[0018] S2. When the dependency is a wide dependency, the number of communications is the product of the number of current RDD partitions and the number of its parent RDD partitions, that is, the RDD dependency is equal to the product;

[0019] S3. When the dependency is a narrow dependency, the number of communications is equal to the number of partitions of the parent RDD corresponding to the current RDD, that is, the RDD dependency is equal to the number of partitions of its parent RDD.

[0020] Furthermore, in a preferred embodiment of the present invention, the communication rule of the Executor is specifically as follows: if there is a narrow dependency relationship between the current RDD and its parent RDD, each Executor in the cluster only needs to communicate with one corresponding Executor to complete the calculation;

[0021] If the current RDD has a wide dependency relationship with its parent RDD, each Executor in the cluster must communicate with all other Executors.

[0022] Furthermore, in a preferred embodiment of the present invention, the calculation formula of the RDD dependency is:

[0023]

[0024] in, The number of communications when it is wide dependency;

[0025] The number of communications for narrow dependencies.

[0026] Furthermore, in a preferred embodiment of the present invention, the specific step of obtaining the RDD usage count includes: S1, when Spark executes a task, before the Spark action operation triggers the runJob method to be called by SparkContext, the application program is used to suspend the specific processing of the data set;

[0027] S2. Then, according to the analysis results of the code logic, determine the RDD dependency generated by the transformation operation of the data set, and build it into a DAG graph;

[0028] S3. According to the connection relationship of the RDD action operations in the DAG graph, count the out-degree of the RDD in the graph; and determine the number of times the RDD is used based on the out-degree of the RDD.

[0029] Furthermore, in a preferred embodiment of the present invention, the specific rule for determining the number of times the RDD is used by the RDD out-degree is:

[0030] In the DAG graph, RDD data corresponds to a point, and an RDD action operation corresponds to an edge;

[0031] Each variable corresponds to an RDD, and the functional relationship between different variables can be regarded as a transformation or action operation. Based on this, the RDD out-degree, that is, the number of outgoing edges of the RDD in the DAG graph, is equal to the number of times the RDD is used.

[0032] Furthermore, in a preferred embodiment of the present invention, the calculation expression of the RDD usage times is specifically:

[0033] in, The out-degree of the current RDD.

[0034] Compared with the prior art, the present invention provides a Spark memory replacement method based on dynamic capacity, which includes the following steps: configuring a first replacement algorithm: an LRU algorithm; and a second replacement algorithm: an algorithm based on weight and dependency counting; S2, cache calculation: when a new RDD data file needs to be stored, traverse the maintained RDD table, and calculate the RDD average value therein, and select the replacement algorithm according to the ratio of the RDD average value to the available memory space of StorageManager in Spark; S3, RDD division: when the second replacement algorithm is triggered, when using DAGScheduler in Spark to analyze the Stage, maintain an RDD dependency counting table, and based on this table, divide the existing RDD in the Stage into two tables with dependency counting and without dependency counting for maintenance; whenever S When a Task in park is calculated, the dependency count of the used RDD is reduced by one. If the dependency count of the RDD is reduced to 0, the RDD is transferred from the dependency count queue to the non-dependency count queue; S4, weight calculation: then obtain the parameters - RDD data size, RDD dependency and RDD usage times; calculate the weight of the existing RDD by weighting the obtained parameter values; S5, data cache: confirm whether the RDD table without dependency count is empty: if it is not empty, then in the RDD table without dependency count, the existing RDDs are evicted from the memory one by one in the order of weight from small to large, and the remaining space is checked after each eviction. If it is empty, traverse the RDD table with dependency count and evict the existing RDDs one by one in the same way until there is enough space to accommodate the new RDD. In the Spark memory replacement method based on dynamic capacity provided in the present application, the first replacement algorithm and the second replacement algorithm are configured. According to the ratio of the average value of RDD to the available memory space of StorageManager, a ratio of 20% is used as a threshold, and a more appropriate cache replacement method can be selected according to the intensity of memory usage in Spark; secondly, when the second replacement algorithm is triggered, the RDD is divided into two groups for maintenance, one group has a dependency count of 0, which will not be used in this Stage, and the other group has a dependency count of not 0, which will be used in this Stage; weight calculation and sorting are performed separately in each group of RDDs, and the weight of the RDD is weighted and calculated in three directions: RDD size, RDD dependency, and RDD usage times. When the RDD cache replacement mechanism of Spark is triggered, it is first checked whether the queue with a dependency count of 0 is empty. If it is not empty, the RDD is first expelled from the group with a dependency count of 0 according to the weight from small to large. If it is empty, the RDD is expelled in the group with a dependency count of not 0 in the same way. This operation can more accurately eliminate RDDs with lower usage possibilities and read-write overheads to release enough memory to store new RDDs.Compared with the prior art, the technical solution involved in the present invention can, under different memory environments, reduce the read and write overhead of Spark when performing operations and the impact on performance to a greater extent, improve the cache hit rate, and enhance the operating efficiency of Spark.

[0035] Beneficial effects:

[0036] 1. The Spark memory replacement method based on dynamic capacity provided by this application can select a more appropriate cache replacement method according to the intensity of memory usage in Spark;

[0037] 2. The dynamic capacity-based Spark memory replacement method provided in this application can more accurately eliminate RDDs with lower usage possibilities and read-write overhead in the iterative algorithm, so that in complex jobs, it can effectively reduce Spark's read-write overhead, improve cache hit rate, and improve Spark's operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of the steps of the Spark memory replacement method based on dynamic capacity provided in an embodiment of the present invention;

[0040] Figure 2 A framework flow chart of the Spark memory replacement method based on dynamic capacity provided in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the connection structure of the RDD wide and narrow dependency relationship provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the RDD conversion relationship in the iterative algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of this application.

[0044] It should be noted that when an element is referred to as being "fixed on" or "set on" another element, it can be directly on the other element or indirectly set on the other element; when an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0045] It should be understood that the terms "length", "width", "up", "down", "front", "back", "first", "second", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0046] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "multiple" and "several" mean two or more, unless otherwise clearly and specifically defined.

[0047] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which this application can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in this application without affecting the effects and purposes that can be achieved by this application.

[0048] Please Figures 1 to 4As shown, the Spark memory replacement method based on dynamic capacity provided by the embodiment of the present application includes the following steps: configuring a first replacement algorithm: an LRU algorithm; and a second replacement algorithm: an algorithm based on weight and dependency counting; S2, cache calculation: when a new RDD data file needs to be stored, traverse the maintained RDD table, and calculate the RDD average value therein, and select the replacement algorithm according to the ratio of the RDD average value to the available memory space of StorageManager in Spark; S3, RDD division: when the second replacement algorithm is triggered, when using DAGScheduler in Spark to analyze the Stage, maintain an RDD dependency count table, and based on this table, divide the existing RDD in the Stage into two tables with dependency counts and without dependency counts for maintenance; whenever Spa When the calculation of a Task in rk is completed, the dependency count of the used RDD is reduced by one. If the dependency count of the RDD is reduced to 0, the RDD is transferred from the dependency count queue to the non-dependency count queue; S4, weight calculation: then obtain the parameters - RDD data size, RDD dependency and RDD usage times; calculate the weight of the existing RDD by weighting the obtained parameter values; S5, data cache: confirm whether the RDD table without dependency count is empty: if it is not empty, then in the RDD table without dependency count, the existing RDDs are evicted from the memory one by one in the order of weight from small to large, and the remaining space is checked after each eviction. If it is empty, traverse the RDD table with dependency count and evict the existing RDDs one by one in the same way until there is enough space to accommodate the new RDD. In the Spark memory replacement method based on dynamic capacity provided in the present application, the first replacement algorithm and the second replacement algorithm are configured. According to the ratio of the average value of RDD to the available memory space of StorageManager, a ratio of 20% is used as a threshold, and a more appropriate cache replacement method can be selected according to the intensity of memory usage in Spark; secondly, when the second replacement algorithm is triggered, the RDD is divided into two groups for maintenance, one group has a dependency count of 0, which will not be used in this Stage, and the other group has a dependency count of not 0, which will be used in this Stage; weight calculation and sorting are performed separately in each group of RDDs, and the weight of the RDD is weighted and calculated in three directions: RDD size, RDD dependency, and RDD usage times. When the RDD cache replacement mechanism of Spark is triggered, it is first checked whether the queue with a dependency count of 0 is empty. If it is not empty, the RDD is first expelled from the group with a dependency count of 0 according to the weight from small to large. If it is empty, the RDD is expelled in the group with a dependency count of not 0 in the same way. This operation can more accurately eliminate RDDs with lower usage possibilities and read-write overheads to release enough memory to store new RDDs.Compared with the prior art, the technical solution involved in the present invention can reduce the reading and writing overhead of Spark when performing operations and the impact on performance to a greater extent under different memory environments.

[0049] Preferred embodiment details:

[0050] The present invention provides a Spark memory replacement method based on dynamic capacity, which specifically includes the following steps: S1, configuring a first replacement algorithm: LRU algorithm;

[0051] And the second replacement algorithm: based on weight and dependency counting algorithm.

[0052] Among them, the LRU (least recently used) algorithm is a commonly used simple cache replacement algorithm, which evicts the least recently used RDD from the memory each time a new RDD is read. In the iterative algorithms commonly used in big data such as PageRank, K-means, neural networks, etc., there are some RDDs that need to be used repeatedly; its own time complexity is low, and there is sufficient memory in Spark jobs, which can give full play to its advantage of short computing time.

[0053] S2, cache calculation: when a new RDD data file needs to be stored, traverse the maintained RDD table and calculate the RDD average value therein. According to the ratio of the RDD average value to the available memory space of StorageManager in Spark, select the replacement algorithm.

[0054] Specifically, in an embodiment of the present invention, the specific rule for selecting the replacement algorithm is: if the RDD average value is less than 20% of the available memory space of the StorageManager, the first replacement algorithm is selected for cache replacement; if the RDD average value is greater than or equal to 20% of the available memory space of the StorageManager, the second replacement algorithm is selected for cache replacement.

[0055] Among them, in an embodiment of the present invention, when the memory of a Spark job is tight, the cache replacement method will be called more frequently, and the requirement for the cache hit rate will be higher than when the memory is sufficient. At this time, if an algorithm such as LRU that predicts the possibility of future reuse based on historical conditions is used for replacement, it is easy to cause data blocks that need to be repeatedly used to be evicted due to short-term non-use, resulting in a decrease in cache hit rate and efficiency. Therefore, the present invention replaces the cache scheme according to the degree of memory tightness: when the memory of a Spark job is sufficient, the memory can accommodate one or more RDDs cached in iterations at the same time. At this time, the accuracy of the LRU algorithm based on historical conditions will be greatly improved. At the same time, the LRU itself has a low time complexity, and the advantage of a short algorithm time can also be reflected;

[0056] The ratio of the RDD average value to the StorageManager's available memory space is used as the judgment criterion; when the RDD average value is less than 20% of the StorageManager's available memory space, it means that the cache of the current Spark job is relatively sufficient, and the same memory pool contains one or more iteration cycles. LRU has a better performance, so subsequent cache replacements are performed using LRU; when the RDD average size is greater than 20% of the StorageManager's available memory, it means that the cache hit rate of the LRU algorithm in this case may decrease rapidly as the RDD increases, and a cache replacement solution based on weight and dependency count is subsequently adopted.

[0057] S3, RDD division: When the second replacement algorithm is triggered, an RDD dependency count table is maintained when using DAGScheduler in Spark to analyze the Stage. Based on this table, the existing RDDs in the Stage are divided into two tables with dependency counts and without dependency counts for maintenance;

[0058] Whenever a Task in Spark is completed, the dependency count of the used RDD is reduced by one. If the dependency count of the RDD is reduced to 0, the RDD is transferred from the dependency count queue to the non-dependency count queue.

[0059] Among them, in an embodiment of the present invention, a cache replacement scheme based on weights and cache counts will first divide RDDs into RDDs with dependency counts (Reference Count, RC) and RDDs without dependency counts. For RDDs with Reference Counts, no matter how long it will be needed in the future, expelling them from the cache will definitely result in retransmission or calculation consumption in the future; for RDDs without Reference Counts, this part of the RDD will no longer be used by the current Stage in the future, so it can be expelled from the cache in a short period of time without affecting performance. Therefore, in this scheme, an improved weight replacement method is preferentially used for RDDs with RC of 0, and a small number of parameters that have a greater impact on the re-reading and writing of RDDs are used for weight calculation. The existing RDDs are divided into two tables with dependency counts and without dependency counts for maintenance. Whenever a new RDD needs to be cached and the space in the memory is insufficient to store the new RDD, the second cache replacement algorithm will be triggered.

[0060] S4, weight calculation: Then obtain the parameters - RDD data size, RDD dependency and RDD usage times; calculate the weight of the existing RDD by weighting the obtained parameter values.

[0061] In an embodiment of the present invention, weight calculation and sorting are performed in each group of RDDs respectively: the weight of the RDD is calculated by weighting in three directions: RDD size, RDD dependency, and RDD usage times.

[0062] Specifically, in an embodiment of the present invention, the specific steps of obtaining the RDD data size are as follows:

[0063] S1. Partition the RDD calculation and distribute the tasks to multiple Executors of Spark for execution.

[0064] S2. During the task execution, RDD needs to be stored. Spark's BlockManager is used to request data cache, and then the acquireMemory method is used to test whether there is enough cache space in the memory.

[0065] S3. Then, the RDD data size is calculated based on the cache space obtained by the test.

[0066] Specifically, in an embodiment of the present invention, the RDD data size calculation expression is specifically:

[0067]

[0068] Among them, S acquireMemory The cache space size obtained by testing with the acquireMemory method.

[0069] Among them, the impact of the size of RDD on the read-write overhead is undoubted. The larger the RDD is, the greater the impact on Spark's operating efficiency and read-write overhead when writing to memory. When arbitrarily evicting RDD, it is necessary to consider the read-write overhead consumed by its re-writing. In Spark, RDD calculations are performed on a partitioned basis, and tasks are divided into multiple Executors for execution. If RDD needs to be stored during the running of the task, the data cache will be requested through BlockManager. During this process, the acquireMemory method will be used to test whether there is enough cache space in the memory, so the size of the RDD can be obtained through this method.

[0070] Specifically, in an embodiment of the present invention, the specific steps of obtaining the RDD dependency relationship include:

[0071] S1. Determine the wide and narrow dependency relationship between the current RDD and its parent RDD. Based on the communication rules of the Executors in the cluster, convert the RDD dependency into the number of communications between Executors for calculation.

[0072] S2. When the dependency is a wide dependency, the number of communications is the product of the number of current RDD partitions and the number of its parent RDD partitions, that is, the RDD dependency is equal to the product;

[0073] S3. When the dependency is a narrow dependency, the number of communications is equal to the number of partitions of the parent RDD corresponding to the current RDD, that is, the RDD dependency is equal to the number of partitions of its parent RDD.

[0074] Among them, the cost of recalculating RDDs with different dependencies after they are lost is different. When recalculating RDDs, the communication between clusters will generate different amounts of overhead depending on the different dependencies. Figure 3 As shown, it can be seen that the communication rules of the Executor are specifically as follows: if the current RDD has a narrow dependency relationship with its parent RDD, each Executor in the cluster only needs to communicate with one corresponding Executor to complete the calculation; if the current RDD has a wide dependency relationship with its parent RDD, each Executor in the cluster must communicate with all other Executors. Therefore, the dependency of RDD can be defined as the number of communications between Executors. Under wide dependency, this number can be obtained by multiplying the number of current RDD partitions by the number of its parent RDD partitions; when the dependency is narrow, the dependency is equal to the number of parent RDD partitions.

[0075] Specifically, in an embodiment of the present invention, the calculation formula of the RDD dependency is:

[0076]

[0077] in, The number of communications when it is wide dependency;

[0078] The number of communications for narrow dependencies.

[0079] Specifically, in an embodiment of the present invention, the specific steps of obtaining the RDD usage count include:

[0080] S1. When Spark executes a task, before the Spark action triggers the runJob method to be called by SparkContext, the application program puts aside the specific processing of the data set;

[0081] S2. Then, according to the analysis results of the code logic, determine the RDD dependency generated by the transformation operation of the data set, and build it into a DAG graph;

[0082] S3. According to the connection relationship of the RDD action operations in the DAG graph, count the out-degree of the RDD in the graph; and determine the number of times the RDD is used based on the out-degree of the RDD.

[0083] Among them, for the number of times RDD is used, as shown in the attached Figure 4 The RDD transformation relationship in the typical iterative algorithm shown in the figure: Spark uses a delayed processing mechanism when executing tasks. Before Spark's action operation triggers the runJob method to be called by SparkContext, the application will put aside the specific processing of the data set, and first determine the RDD dependency generated by the Transformation operation based on the analysis results of the code logic, and build it into a DAG graph. In the DAG graph, RDD data corresponds to a point, and an operation corresponds to an edge. In Spark, each variable corresponds to an RDD, and the functional relationship between different variables can be regarded as a transformation or action operation. Based on this, the RDD out-degree, that is, the number of outgoing edges of the RDD in the DAG graph is equal to the number of times the RDD is used.

[0084] Specifically, in an embodiment of the present invention, the calculation expression of the RDD usage times is specifically: in, The out-degree of the current RDD.

[0085] S5. Data cache: confirm whether the RDD table without dependency count is empty: if it is not empty, then evict the existing RDDs from the memory one by one in the RDD table without dependency count in ascending order of weight. After each eviction, check the remaining space until there is enough space to cache the new RDD, then stop eviction. If it is empty, traverse the RDD table with dependency count and evict the existing RDDs one by one in the same way until there is enough space to accommodate the new RDD.

[0086] Among them, when the method is executed, it first confirms whether the RDD table without dependency count is empty. If it is not empty, the RDD is evicted from the memory in ascending order of weight until there is enough space to cache the new RDD; if it is empty, the RDD queue whose RC is not empty is traversed, and the memory is evicted in ascending order of weight until there is enough space to cache the new RDD.

[0087] Specifically, in an embodiment of the present invention, the Spark cache replacement method is as follows: when DAGScheduler parses Task, the algorithm reads the RDD in the Stage to convert the DAG graph, and stores the RDD sequence number and the number of dependencies in the table RCs in the form of key-value pairs. During the task process, each time a new RDD is stored, the algorithm calls acquireMemory, RDD.partitions().size() and other methods to obtain the size of the RDD, the number of partitions and other information, and after calculating the RDD weight, stores it in the weight table w in the form of key-value pairs. After each Task is completed, the dependency count of the used RDD is reduced by one. An RDD with a dependency count of 0 means that it will not be used again in the Stage. Its RDD sequence number and usage count key-value pair are transferred to the RCs_0 table for maintenance. When BlockManager caches a new RDD, if there is not enough space to cache the new RDD, first check whether the RCs_0 table is empty. If it is not empty, evict the RDDs one by one in the RCs_0 table in order from low to high weight. After each eviction, check the remaining space until there is enough space to cache the new RDD, then stop eviction. If the RCs_0 table is empty, evict the RDDs one by one in the RCs_1 table in the same way until there is enough space to accommodate the new RDD.

[0088] As described above, in the Spark memory replacement method based on dynamic capacity provided in the present application, the first replacement algorithm and the second replacement algorithm are configured. According to the ratio of the RDD average value to the available memory space of the StorageManager, a ratio of 20% is used as a threshold, and a more appropriate cache replacement method can be selected according to the intensity of memory usage in Spark; secondly, when the second replacement algorithm is triggered, the RDD is divided into two groups for maintenance, one group has a dependency count of 0, which will not be used in this Stage, and the other group has a dependency count of not 0, which will continue to be used in this Stage; weight calculation and sorting are performed separately in each group of RDDs, and the weight of the RDD is weightedly calculated in three directions: RDD size, RDD dependency, and RDD usage times. When the RDD cache replacement mechanism of Spark is triggered, it is first checked whether the queue with a dependency count of 0 is empty. If it is not empty, the RDD is first expelled from the group with a dependency count of 0 according to the weight from small to large. If it is empty, the RDD is expelled in the group with a dependency count of not 0 in the same way. This operation can more accurately eliminate RDDs with lower usage possibilities and read-write overheads to release enough memory to store new RDDs. Compared with the prior art, the technical solution involved in the present invention can, under different memory environments, reduce the read and write overhead of Spark when performing operations and the impact on performance to a greater extent, improve the cache hit rate, and enhance the operating efficiency of Spark.

[0089] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A Spark memory replacement method based on dynamic capacity, characterized in that: The following steps are involved: S1. Configuration algorithm: configure the first replacement algorithm: LRU algorithm; and the second replacement algorithm: based on weight and dependency counting algorithm; S2, cache calculation: when a new RDD data file needs to be stored, traverse the maintained RDD table and calculate the RDD average value therein, and select the replacement algorithm based on the ratio of the RDD average value to the available memory space of StorageManager in Spark; S3, RDD division: When the second replacement algorithm is triggered, an RDD dependency count table is maintained when using DAGScheduler in Spark to analyze the Stage. Based on this table, the existing RDDs in the Stage are divided into two tables with dependency counts and without dependency counts for maintenance; whenever a Task in Spark is completed, the dependency count of the used RDD is reduced by one. If the dependency count of the RDD is reduced to 0, the RDD is transferred from the dependency count queue to the non-dependency count queue; S4, weight calculation: then obtain the parameters - RDD data size, RDD dependency and RDD usage times; calculate the weight of the existing RDD by weighting the obtained parameter values; S5, data cache: confirm whether the RDD table without dependency count is empty: if it is not empty, then evict the existing RDDs from the memory one by one in the RDD table without dependency count in ascending order of weight. After each eviction, check the remaining space until there is enough space to cache new RDDs, then stop eviction; If it is empty, traverse the RDD table with dependent counts and evict existing RDDs one by one in the same way until there is enough space to accommodate the new RDD.

2. The Spark memory replacement method based on dynamic capacity according to claim 1, characterized in that: In step S2, the specific rule selected by the replacement algorithm is: If the RDD average value is less than 20% of the available memory space of the StorageManager, the first replacement algorithm is used for cache replacement; If the RDD average value is greater than or equal to 20% of the available memory space of the StorageManager, the second replacement algorithm is selected for cache replacement.

3. The Spark memory replacement method based on dynamic capacity according to claim 1, characterized in that: In step S4, the specific steps of obtaining the RDD data size are as follows: S1. Partition the RDD calculation and distribute the tasks to multiple Executors of Spark for execution. S2. During the task execution, RDD needs to be stored. Spark's BlockManager is used to request data cache, and then the acquireMemory method is used to test whether there is enough cache space in the memory. S3. Then, the RDD data size is calculated based on the cache space obtained by the test.

4. The Spark memory replacement method based on dynamic capacity according to claim 3, characterized in that: The RDD data size calculation expression is specifically: Among them, S acquireMemory The cache space size obtained by testing with the acquireMemory method.

5. The Spark memory replacement method based on dynamic capacity according to claim 3, characterized in that: In step S4, the specific steps of obtaining the RDD dependency relationship include: S1. Determine the wide and narrow dependency relationship between the current RDD and its parent RDD. Based on the communication rules of the Executors in the cluster, convert the RDD dependency into the number of communications between Executors for calculation. S2. When the dependency is a wide dependency, the number of communications is the product of the number of current RDD partitions and the number of its parent RDD partitions, that is, the RDD dependency is equal to the product; S3. When the dependency is a narrow dependency, the number of communications is equal to the number of partitions of the parent RDD corresponding to the current RDD, that is, the RDD dependency is equal to the number of partitions of its parent RDD.

6. The Spark memory replacement method based on dynamic capacity according to claim 5, characterized in that: The communication rules of the Executor are as follows: If the current RDD has a narrow dependency relationship with its parent RDD, each Executor in the cluster only needs to communicate with one corresponding Executor to complete the calculation; If the current RDD has a wide dependency relationship with its parent RDD, each Executor in the cluster must communicate with all other Executors.

7. The Spark memory replacement method based on dynamic capacity according to claim 5, characterized in that: The calculation expression of the RDD dependency is specifically: in, The number of communications when it is wide dependency; The number of communications for narrow dependencies.

8. The Spark memory replacement method based on dynamic capacity according to claim 1, characterized in that: The specific steps of obtaining the RDD usage count include: S1. When Spark executes a task, before the Spark action triggers the runJob method to be called by SparkContext, the application program puts aside the specific processing of the data set; S2. Then, according to the analysis results of the code logic, determine the RDD dependency generated by the transformation operation of the data set, and build it into a DAG graph; S3. According to the connection relationship of the RDD action operations in the DAG graph, count the out-degree of the RDD in the graph; and determine the number of times the RDD is used based on the out-degree of the RDD.

9. The Spark memory replacement method based on dynamic capacity according to claim 8, characterized in that: The specific rule for determining the number of times an RDD is used by the RDD out-degree is: In the DAG graph, RDD data corresponds to a point, and an RDD action operation corresponds to an edge; Each variable corresponds to an RDD, and the functional relationship between different variables can be regarded as a transformation or action operation. Based on this, the RDD out-degree, that is, the number of outgoing edges of the RDD in the DAG graph, is equal to the number of times the RDD is used.

10. The Spark memory replacement method based on dynamic capacity according to claim 9, characterized in that: The calculation expression of the RDD usage times is specifically: F RDDi =OutNum RDDi Among them, OutNum RDDi The out-degree of the current RDD.

Images