A dynamic cache replacement method and device for big data processing

Abstract

The application discloses a dynamic cache replacement method and equipment for big data processing, and the method comprises the steps of: abstracting a big data processing application into a directed acyclic graph G=(V, E), wherein a node set V represents data calculated in the big data processing application, and an edge set E represents a dependency relationship between the data; based on the data contained in the directed acyclic graph G=(V, E), a cache replacement problem mathematical model is established with the aim of minimizing the overall execution time of the big data processing application, and the model decides data to be cached at each time t; the cache replacement problem is simplified based on the characteristics of the big data processing; and the simplified cache replacement problem is solved based on the dynamic programming idea. The application realizes dynamic adaptation of the cache replacement of the data access mode in the data processing process, can improve the memory use efficiency, and greatly reduces the execution time of the big data processing application.

Description

Technical Field

[0001] This invention relates to big data processing systems and memory optimization techniques for big data processing applications, specifically a dynamic cache replacement method and device for big data processing applications. Background Technology

[0002] With the rapid development of the modern information society and the internet, data is experiencing explosive exponential growth. The 2018 IDCS white paper, "Data Age 2025," predicted that the total global data volume would reach 175 ZB by 2025. Thanks to the development of big data processing systems—computing software based on large-scale clusters—a large amount of work in industry and academia is analyzing and processing massive amounts of data, aiming to extract the rich and complex information contained within, thereby promoting modern economic development, social progress, and technological innovation. Unlike the simple parallel operations of traditional data processing systems, modern big data systems offer richer functions, stronger expressive capabilities, and are more user-friendly. Specifically: 1) Big data processing systems provide functions such as storage, computation, analysis, and mining of massive amounts of data; 2) Big data processing systems support user-defined methods, allowing for flexible combined operations such as merging, grouping, slicing, joining, and sorting of data; 3) Big data processing systems provide users with an application layer framework, supporting users to use a simpler and clearer Structured Query Language (SQL) for data processing and analysis.

[0003] Thanks to the aforementioned advantages, big data processing systems (Hadoop, Spark, Flink, etc.) have been widely applied in fields such as graph computing, machine learning, and stream processing. An experimental report from a Microsoft production cluster revealed that up to 60% of jobs in the cluster experienced duplicate data computation, meaning multiple tasks received the same data and executed the same computational logic. To maximize the utilization of this duplicate computation, big data processing systems utilize caching techniques to accelerate the execution of data processing applications, writing intermediate data generated during application execution into memory or disk. Since cached data no longer needs to be repeatedly computed, the completion time of data processing applications is significantly reduced. We observed that the caching process in memory-based big data processing systems has caching requirements under limited memory. Because memory resources are often the bottleneck of big data processing systems, big data processing frameworks cannot cache all intermediate data into memory during application execution; instead, they need to dynamically replace cached data during application runtime. That is, during data processing application runtime, it is necessary to dynamically decide which data to cache and replace cached data based on priority. This problem is known as the cache replacement problem. Since big data processing applications are often accompanied by parallel task execution and parallel data computation, the benefits of data caching are difficult to predict, which in turn affects the decision-making of the cache replacement problem. Therefore, there is an urgent need for a cache replacement strategy that takes into account this parallel phenomenon. Summary of the Invention

[0004] Purpose of the invention: This invention proposes a dynamic cache replacement method and device for big data processing applications, which realizes cache replacement that dynamically adapts to the data access mode in the data processing process, thereby improving memory utilization efficiency and significantly reducing the execution time of big data processing applications.

[0005] Technical solution: To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Firstly, a dynamic cache replacement method for big data processing applications includes the following steps:

[0007] (1) Abstract the big data processing application into a directed acyclic graph G = (V, E), where each node in the set of nodes V represents the data of the big data processing application, and any element v in the set V contains two attributes: the memory space occupied by the data s. v and the time c required to calculate this data v Each edge <u,v> in the edge set E represents that the computation of data v in a big data processing application depends on data u.

[0008] (2) Based on the directed acyclic graph G=(V,E) and the hierarchical relationship between big data processing applications, jobs, stages, and data, obtain the i-th executed job J in the big data processing application. i The execution latency is expressed as f(S) i,j ,CS), indicating that when the cached data set CS is used, job J i Mid-stage S i,j Calculation completion time and task J i The difference between the start times of execution;

[0009] (3) A mathematical model for the cache replacement problem is established with the goal of minimizing the overall execution time of big data processing applications. The cache replacement problem P1 is expressed as deciding on the set of data CS to be stored in the cache space at each time t after data computation is completed. new,t To minimize the application's latency from p-th... t The total completion time for this task and subsequent tasks;

[0010] (4) Solve the cache replacement problem P1 to obtain the dynamic cache replacement strategy.

[0011] Further, in step (2), the hierarchical relationship is as follows: the big data processing application consists of multiple big data processing jobs executed serially, the big data processing job includes multiple big data processing stages executed serially or in parallel, and the big data processing stage includes multiple abstract datasets computed serially or in parallel.

[0012] Furthermore, in step (2), f(S) i,j ,CS) is calculated using the following method:

[0013]

[0014] Among them, S i,j Let x represent the j-th stage of the i-th task in the application. i,j,k Indicates stage S i,j The abstract dataset of the k-th computation, N i,j Indicates stage S i,j The number of abstract datasets in D(S) i,j ) indicates stage S i,j In assignment S i The set of stages that depend on, g(S) i,j (x, CS) represents stage S i,j The execution latency under the cached data set CS is equal to the stage S. i,j Final calculated data The time and stage S of calculation completion i,j The difference at the start time is calculated as follows:

[0015]

[0016] Where c(x,S) i,j ) indicates that in stage S i,j The time required to calculate data x, P(x,S) i,j ) indicates that data x is in stage S i,j A collection of data that it depends on.

[0017] Furthermore, in step (3), the cache replacement problem P1 is expressed as:

[0018]

[0019]

[0020]

[0021]

[0022] Among them, CS old,t This indicates that x has not been stored in the cache space at time t. t The previously cached data set, CS new,t This indicates that the cache space stores x at time t. t The cached data set after that, x t This represents the abstract dataset that has been computed at time t. p represents the set of times when all abstract datasets in the application have been computed. t The index of the job executed at time t is represented, L represents the total memory limit of the cache space, and |J| represents the number of jobs in the application.

[0023] Furthermore, in step (4), solving the cache replacement problem P1 includes: simplifying the cache replacement problem P1 based on the characteristics of big data processing, and solving the simplified cache replacement problem based on the idea of ​​dynamic programming.

[0024] Furthermore, the simplified cache replacement problem P1 based on big data processing characteristics includes:

[0025] Replace all stages of each job in the application with the "job critical path", change the execution mode of the data processing stage from parallel to serial, and simplify problem P1 to problem P2, where the "job critical path" is defined as the computation chain of the stage with the longest execution time in the job.

[0026] The abstract dataset computed in each stage is replaced with "hotspot access data". The data computation mode in the data processing stage changes from parallel to serial. Problem P2 is simplified to problem P3. "Hotspot access data" is defined as the abstract dataset represented by nodes with an out-degree greater than 1 in a directed acyclic graph formed by application abstraction.

[0027] By replacing the "hotspot access data" in each data processing stage with the execution result of "stage representative calculation", problem P3 is simplified to problem P4, which is equivalent to the 0-1 knapsack problem. Here, "stage representative calculation" refers to the data processing operator represented by the "hotspot access data" that is finally calculated in the data processing stage.

[0028] Furthermore, the solution to the simplified cache replacement problem based on dynamic programming includes:

[0029] Preprocessing based on the idea of ​​problem simplification: Receive the set of received parameters J, and the existing data CS in the buffer space at time t. old,t and the data x to be added to the cache space at time t t By calculating the "critical path of the job", analyzing the "hot access data", and statistically analyzing the "representative calculation of the stage" and its caching benefits, it returns the caching benefits RRT of the "representative data set of the stage" x' and the "representative data of the stage".

[0030] Cache replacement based on dynamic programming: Based on the preprocessed "representative data set x'" and the cache benefit RRT of the "representative data of the stage", and combined with the upper limit of cache space L as input, the cache replacement problem is decomposed into multiple subproblems through dynamic programming as each value of the upper limit of cache space L and each element in the "representative data set x'" is traversed, thus obtaining the optimal cache decision CS for problem P4. new,t .

[0031] Furthermore, the upper limit of cache memory L and the memory s occupied by each data x are considered. x When the value is a natural number, the result of this cache replacement algorithm based on the idea of ​​dynamic programming is the optimal cache decision.

[0032] In a second aspect, a computer device includes one or more processors; and a memory for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to perform the steps of the dynamic cache replacement method for big data processing as described in the first aspect of the invention.

[0033] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic cache replacement method for big data processing as described in the first aspect of the invention.

[0034] Beneficial Effects: In big data processing systems, the difficulty of cache replacement lies in the unpredictable benefits of caching due to the frequent parallel execution patterns in data processing applications. To address this challenge, this invention mathematically models the multi-stage parallel execution phenomenon of big data processing jobs and the multi-data parallel computation phenomenon during big data processing stages. Based on this, it defines an operator-level cache replacement problem for multi-stage parallel big data processing applications. The modeled problem is NP-hard. This invention simplifies the problem based on characteristics of "critical path of the job," "hot access data," and "representative computation of each stage." Since the simplified problem has an optimal substructure, this invention proposes a solution algorithm based on dynamic programming to determine the dynamic cache replacement strategy. This invention achieves cache replacement that dynamically adapts to data access patterns during data processing, improving memory utilization efficiency and significantly reducing the execution time of big data processing applications. Attached Figure Description

[0035] Figure 1 This is a flowchart of the dynamic cache replacement method of the present invention;

[0036] Figure 2 It is a directed acyclic graph describing big data processing applications in specific implementations;

[0037] Figure 3 This is a simplified diagram illustrating the cache replacement problem in a specific implementation.

[0038] Figure 4 This is a flowchart of the algorithm based on the concept of dynamic programming in a specific implementation. Detailed Implementation

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0040] This invention addresses the cache replacement problem in big data processing applications. For example... Figure 1 As shown, this invention proposes a dynamic cache replacement method for big data processing applications, comprising the following steps:

[0041] (1) Abstract big data processing applications into directed acyclic graphs.

[0042] This invention abstracts big data processing applications as a directed acyclic graph G = (V, E) to describe the data processing process of big data processing applications. In the directed acyclic graph, each element of the node set V represents the data of the big data processing application—an abstract dataset. In the big data processing system, the abstract dataset is defined as the execution result of data processing operators, consisting of memory data blocks distributed across different nodes in the cluster. For any element v (abstract dataset) in set V, it contains two attributes: the memory space s occupied by the data. v and the time c required to calculate this data v Each edge <u,v> in the edge set E represents a computational dependency between abstract datasets in a big data processing application. For example, edge <u,v> indicates that the computation of data v depends on data u. In fact, the dependencies between data in big data processing applications have practical significance, namely, the execution process of specific operators. However, the edge set E does not record the above information because the attribute c of each node v in the node set V... v —The time required to calculate data v has already taken into account the execution process of the operator.

[0043] (2) Based on the information contained in the directed acyclic graph, analyze the phenomenon of parallel execution of jobs in big data processing applications.

[0044] A directed acyclic graph (DAG) contains hierarchical information consisting of big data processing applications, big data processing jobs, big data processing stages, and abstract datasets. Specifically, a big data processing application comprises multiple serially executed big data processing jobs; a big data processing job contains multiple serially or parallelly executed big data processing stages; and a big data processing stage contains multiple serially or parallelly computed abstract datasets. Based on the node set V, the edge set E, and the aforementioned hierarchical information, the i-th executed job J in the big data processing application can be obtained. iThe execution latency is thus used to provide a mathematical model for optimizing the cache replacement problem. Since data processing jobs consist of stages, and data processing stages consist of abstract datasets (hereinafter collectively referred to as "data"), the following will formally describe the execution latency according to the hierarchy of data, stages, and jobs, based on the concepts in Table 1.

[0045]

[0046]

[0047] Table 1. Symbol table of the abstract directed acyclic graph of big data processing jobs

[0048] In the directed acyclic graph abstracted from big data processing applications, the function c(x,S) is used. i,j ) indicates that in stage S i,j The time required to calculate data x (compared to c mentioned earlier) x (equivalent), then in stage S i,j Cache benefits of data x in stage S i,j The time saved by caching data x can be expressed as r(x,S) i,j Hypothetical function P(x,S) i,j ) is defined as data x in stage S i,j The set of data that the middle depends on. c(x,S) i,j ) and r(x,S i,j The relationship is as follows:

[0049]

[0050] In each stage of a big data processing application, there exists a set of data that is independent of any other data in that stage; this is called the final computational data for that stage. Based on this, the execution latency of a stage can be correlated with the final computational data for that stage. Assume the set of cached data in the cache space is CS, and the variable x... i,j,k . indicates stage S i,j The k-th calculated data, variable N i,j For stage S i,j The number of data included, then stage S i,j The execution latency under the cached data set CS is equal to the stage S. i,j Final calculated data The time and stage S of calculation completion i,j The difference at the start time of execution is formally expressed as: Among them, the recursive function g(S) i,j (x, CS) is used for multi-data parallel computation during the modeling phase, and its formal description is as follows:

[0051]

[0052] In big data processing applications, when the execution of one stage depends on two or more other stages, parallel execution of stages occurs. The job execution latency modeling in this invention primarily focuses on this phenomenon. Considering that each data processing job contains a stage that is not depended upon by any other stage in the job, referred to as the final execution stage, we correlate the job's execution latency with this final execution stage. Assuming the set of cached data in the cache space is CS, and variable J... i Let J represent the i-th job executed in a big data processing application. i The number of stages included is M i function D(S) i,j ) indicates stage S i,j In assignment J i The set consisting of the stages that the work depends on. Then, job J... i The execution latency under the cached dataset CS is equal to that of job J. i Final Execution Phase Execution completion time and task J i The difference at the start time of execution is formally expressed as: Among them, the recursive function f(S) i,j (CS) is used to model the phenomenon of multi-stage parallel execution within a job, and its formal description is as follows:

[0053]

[0054] like Figure 2 As shown, this invention abstracts big data processing applications as directed acyclic graphs (DAGs). A DAG contains hierarchical information consisting of applications, jobs, stages, and data. Furthermore, nodes in the DAG represent data within the big data processing application, and edges represent data dependencies. Based on the DAG, job latency under different caching states can be obtained, which can be used to model the cache replacement problem for big data processing applications. When the application executes, the big data processing framework stores intermediate data generated during runtime into the cache space. The big data processing framework faces a trade-off problem when utilizing limited memory for cache acceleration, namely the cache replacement problem. For example, when... Figure 2 When the data 27 in stage 4 is completed and needs to be cached, if the cache space is about to run out, a decision needs to be made on which data should be stored in the cache space.

[0055] (3) Establish a mathematical model for the cache replacement problem.

[0056] The mathematical model of the cache replacement problem for big data processing frameworks has the following characteristics: For any data x tThe time t after calculation is completed is used with decision variable CS. new,t This represents the data that should be placed in the cache space. Assume CS... old,t This means that without considering data x t Data already placed in the previous cache space, variables Let p be the set of times when all data computations in a data processing application are completed. t The index of the job executed at time t is given, and the variable |J| is the set of jobs for big data processing applications. The function s(x) (and the variable s in step (1)) x Let (equivalent) represent the memory space occupied by data x, and let L represent the upper limit of the cache space. In a limited memory scenario, the operator-level cache replacement problem P1 for big data processing applications can be expressed as deciding the set of data CS to be stored in the cache space at each time t after data computation is completed. new,t To minimize the application's latency from p-th... t The total completion time of each task and subsequent tasks. Based on the concepts in Table 2, problem P1 is formally described as follows:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] Table 2. Symbol table related to the cache replacement problem P1 for big data processing applications

[0063] The constraints have the following characteristics: First, when considering the cache replacement problem for a specific big data processing application, the optimization objective only needs to minimize the p-th iteration of the application. t The overall completion time of the first and subsequent tasks, variable p. t The value range does not exceed the number of data processing jobs in the application. The following constraint describes the set of jobs whose completion time needs to be considered in the optimization objective of the cache replacement problem:

[0064]

[0065] Secondly, the decision variable CS new,t For set CS old,t ∪{x t The subset of}, related to the scope of consideration for decision variables, is described below:

[0066]

[0067] Secondly, it is necessary to ensure that the set of cached data in the caching decision does not exceed the memory capacity of the cache space. The constraints related to the upper limit of memory for cached data are as follows:

[0068]

[0069] (4) Simplify cache replacement problem based on big data processing characteristics.

[0070] Assuming that in a big data processing application, the number of stages in a job and the amount of data in each stage are both 1, this special case of problem P1 is defined as problem P1. * Observation reveals that problem P1 * Equivalent to the 0-1 knapsack problem, the 0-1 knapsack problem can be reduced to problem P1, thus proving that the problem is NP-hard. Therefore, this invention simplifies problem P1 based on the characteristics of "critical path of the task," "hotspot access data," and "stage representativeness calculation" in the data processing process, making it easier to solve. The specific steps are as follows:

[0071] A. In big data processing applications, the "critical path" of a job is defined as the computational chain of stages with the longest execution time within that job. Since the latency of the "critical path" can approximate the overall job latency, and the stages within the "critical path" are executed sequentially, the set of stages in a big data processing job... Replace with "critical path of the task" Then, in problem P1, the recursive function f(S) used to describe the multi-stage parallel phenomenon... i,j (CS) can be removed, and problem P1 is simplified to problem P2, as shown below:

[0072]

[0073]

[0074]

[0075]

[0076] Wherein, variable M′ i ,S′ i,j , Used to represent the concept of "critical path of operation", detailed definition is shown in Table 3, function g(S i,j ,x i,j,k ,CS) has been defined in step (2).

[0077] symbol definition <![CDATA[M′ i ]]> <![CDATA[Job J i The number of stages included in the critical path]]> <![CDATA[N′ i,j ]]> <![CDATA[Stage S′ i,j The number of data that appears]]> <![CDATA[S′ i,j ]]> The j-th stage of the "critical path" of the i-th job in the application <![CDATA[x′ i,j,k ]]> <![CDATA[Phase S′ i,j The k-th calculated data in]]>

[0078] Table 3. Symbols related to question P2

[0079] B. In big data processing applications, "hotspot access data" is defined as data represented by nodes with an out-degree greater than 1 in a directed acyclic graph abstracted by the application. Since the caching benefit of "hotspot access data" can approximately substitute for the caching benefit of all other data, and the computation of "hotspot access data" can be approximated as serial computation, replacing the data computed in each stage with "hotspot access data" changes the data computation mode in the data processing stage from parallel to approximately serial. The recursive function in question P2 used to describe the phenomenon of multi-data parallel computation... This can be removed. To facilitate the formalization of the problem, this invention replaces "minimizing the overall job completion time" in the optimization objective with "maximizing the time saved by cached data." Based on this, problem P2 is simplified to problem P3, as follows:

[0080]

[0081]

[0082]

[0083]

[0084] The set of "hotspot access data" in the application is defined as function μ(i,j,CS) new,t ) and φ(i,j,CS new,t The stage S′ is defined. i,j The set of indices for "hotspot access data" is formally described as follows:

[0085]

[0086]

[0087] C. Due to the "lazy computation" characteristic of big data processing frameworks, the operator represented by the "hot access data" in the final computation of the data processing stage is called "stage representative computation." It can be observed that "stage representative computation" can approximately replace the overall computation of the stage. Therefore, after replacing the "hot access data" in each data processing stage with the execution result of "stage representative computation"—"stage representative data"—problem P3 is simplified to problem P4, as shown below:

[0088]

[0089]

[0090]

[0091]

[0092] k * =max(μ(i,j,CS) old,t ∪{x t})).

[0093] In problem P4, variable k * This represents the representative data for each stage, i.e., "stage representativeness calculation," where k represents the representative data for each specific stage. * To determine the value, the function η(x,CS) indicates whether the data x is one of the elements of the set CS, with 1 for yes and 0 for no.

[0094] D. Based on the characteristics of "critical path of the job", "hotspot access data", and "stage representative calculation", the operator-level cache replacement problem P1 for multi-stage parallel big data processing applications can be simplified to problem P4. Furthermore, assuming that at each time t, the dataset CS... old,t ∪{x t The number of data in} is Data Collection CS old,t ∪{x t The j-th calculated data in} is x′ k Decision variable z k Decide whether to cache data x′ k (1 represents caching, 0 represents no caching), then problem P4 can be equivalently transformed into the following problem:

[0095]

[0096]

[0097] k * =max(μ(i,j,CS) old,t ∪{x t})).

[0098] In the 0-1 knapsack problem, assume the total number of items is... The weight and value of item j are w respectively. j and p j It can be observed that the decision variables in the 0-1 knapsack problem... The weight of the item w j Item value p j The decision variables for question P4 are respectively Data x′ k Memory occupied s(x′) k ), cache each data x′ k The time saved Correspondingly (when data x′) k (When it is not a "stage representative calculation", the time saved by caching this data is 0). Therefore, problem P4 is equivalent to the 0-1 knapsack problem.

[0099] Figure 3 The document showcases the specific content of "critical path of the task," "hotspot access data," and "stage representative calculation." Based on this, it can be observed that the cache replacement problem considering only the data related to "stage representative calculation"—"stage representative data"—can approximately replace the cache replacement problem considering all data, significantly reducing the complexity of the problem.

[0100] (5) Design an optimal algorithm based on the idea of ​​dynamic programming.

[0101] Based on the characteristic that the simplified problem P4 has an optimal substructure, this invention designs a cache replacement algorithm based on the idea of ​​dynamic programming. Figure 4 This paper demonstrates the flow of a cache replacement algorithm based on dynamic programming. After a preprocessing step based on problem simplification, the cache replacement module based on dynamic programming receives the cache benefit RRT of the "stage representative data set" x' and the cache memory limit L as its input. By iterating through each value of the cache memory limit L and each element in the "stage representative data" set x', the module decomposes the cache replacement problem into multiple subproblems using dynamic programming, thus obtaining the optimal cache decision CS for problem P4. new,t Specifically, the algorithm includes the following modules:

[0102] (5.1) Preprocessing module based on problem simplification: This module receives the parameter job set J and the existing data CS in the buffer space at time t. old,t and the data x to be added to the cache space at time t t By calculating the "critical path of the task," analyzing "hotspot access data," and statistically analyzing "stage representative calculations" and their caching benefits, the system returns the caching benefit RRT of the "stage representative data set" x' and the "stage representative data." The returned values ​​are then used as input to a cache replacement module based on dynamic programming. The specific steps are as follows:

[0103] A. Receive inputs J and CS old,t and x t

[0104] B. Initialize the output "Representative Data of the Stage" and its cached benefits: x' and RRT

[0105] C. Calculate the "critical path" (CP) of task J using the longest path algorithm.

[0106] D. Calculate "hotspot access data" based on the directed acyclic graph represented by J.

[0107] E. Statistically determine the unfinished jobs J in the job set J at time t. t

[0108] F. To J t Each job J i Each unfinished stage S' i,j Perform the following operations

[0109] a) Statistical phase S' i,j The topological sequence TP of all data in

[0110] b) Obtain the sequence TP∩(CS) old,t ∪{x t The last element x of})∩HD u

[0111] c) Update x': x'←x'∪{x u}

[0112] d) Update RRT: RRT u ←RRT u +In stage S' i,j Cached data x u Benefits

[0113] G. Output "Representative Data of the Stage" and its caching benefits: x' and RRT

[0114] (5.2) Cache Replacement Module Based on Dynamic Programming: This module receives the output of the preprocessing module based on the problem simplification idea, namely the cache benefit RRT of the "stage representative data set" x' and the "stage representative data", and combines the upper limit L of the cache space as its input. As it iterates through each value of the upper limit L of the cache space and each element in the "stage representative data" set x', this module decomposes the cache replacement problem into multiple subproblems through dynamic programming, thereby obtaining the optimal cache decision CS for problem P4. new,t The specific steps are as follows:

[0115] A. Receive inputs: "Stage Representative Data" x', Cache Revenue RRT, and Cache Space Memory Limit L.

[0116] B. Initialize the dynamic programming array dp and the set of optimal results for subproblems. C.

[0117] C. Execute the following operations iteratively from i=1 to |x'| and j=1 to L.

[0118] a) If Perform the following operations:

[0119] dp i,j C i,j ←dp i-1,j C i-1,j

[0120] b) Otherwise, perform the following operation:

[0121]

[0122] D. Output the optimal caching decision: C N,L

[0123] The computational complexity of this algorithm is jointly determined by the preprocessing module based on problem simplification and the cache replacement module based on dynamic programming. Among these, the steps of calculating the "critical path of the job" and analyzing "hotspot access data" in the preprocessing module based on problem simplification only need to be executed once before the big data processing application begins, thus having a relatively small impact. Therefore, the key step in the preprocessing module based on problem simplification lies in calculating the "stage representativeness" and its caching benefits, with a computational complexity of O(|V|). 2 Furthermore, the computational complexity of the cache replacement module based on dynamic programming is determined by the search space of the dynamic programming algorithm, and is directly related to the size of the "hot access data" and the upper limit of the cache memory. Its computational complexity is O(|V|). 2 Therefore, the overall computational complexity of the dynamic cache replacement algorithm for big data processing applications is O(|V|). 2 ×L).

[0124] Due to limited memory resources, there are trade-offs in using memory resources for caching acceleration in big data processing applications. This invention proposes a dynamic cache replacement method for multi-stage parallel big data processing applications. This method uses an abstract dataset containing multiple data blocks as the cache object, dynamically deciding which data to cache during application execution. To overcome the difficulty of formally describing parallel computing phenomena in data processing applications, this invention establishes a mathematical model for the caching process of multi-stage parallel data processing applications and proves that the operator-level cache replacement problem is NP-hard. Addressing the complexity of the problem, this invention reduces the difficulty based on the characteristics of "critical path of the job," "hot access data," and "representative computation of each stage" by observing the actual caching process in big data processing systems, making the problem easier to solve. Since the simplified problem has an optimal substructure, this invention designs a dynamic cache replacement algorithm based on dynamic programming. This invention fills the gap in cache replacement work for parallel computing within the big data processing framework, achieving dynamic adaptation of cache replacement to data access patterns during data processing, improving memory utilization efficiency, and significantly reducing the execution time of big data processing applications.

[0125] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. In the context of this invention, the computer-readable medium can be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or masked read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital magnetic tape or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs), etc.

[0126] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0128] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A dynamic cache replacement method for big data processing applications, characterized in that, Includes the following steps: (1) Abstract big data processing applications into a directed acyclic graph. , where the set of nodes Each node represents data from a big data processing application, a collection. any element in It includes two attributes: the memory space occupied by the data. and the time required to calculate this data edge set Each edge This refers to data in big data processing applications. The calculation depends on the data ; (2) Based on directed acyclic graph And the hierarchical relationships between big data processing applications, tasks, stages, and data, to obtain the first... One executed job The execution latency is expressed as , representing a cached data set Under these circumstances, the assignment Mid-stage Calculate the completion time and task The difference between the start times of execution; Calculated using the following method: ； in, Indicates the first in the application The first assignment Each stage Representation phase The Middle An abstract dataset of computations, Representation phase The number of abstract datasets in the dataset. Representation phase In the homework The set of stages that depend on it. Representation phase In cached data collection The execution latency under the current stage is equal to the stage Final calculated data The time and stage of calculation completion The difference at the start time is calculated as follows: ； in Indicates the stage Chinese calculation data Time required Representing data In the stage The collection of data that it depends on; (3) A mathematical model for the cache replacement problem is established with the goal of minimizing the overall execution time of big data processing applications. It is expressed as the moment when each data calculation is completed. The data set for storing decisions in the cache space To minimize the application's latency from the first The total completion time for this task and subsequent tasks; (4) Solving the cache replacement problem This allows you to obtain a dynamic cache replacement strategy.

2. The dynamic cache replacement method according to claim 1, characterized in that, In step (2), the hierarchical relationship is as follows: the big data processing application consists of multiple big data processing jobs executed serially, the big data processing jobs include multiple big data processing stages executed serially or in parallel, and the big data processing stages include multiple abstract datasets computed serially or in parallel.

3. The dynamic cache replacement method according to claim 1, characterized in that, In step (3), the cache replacement problem The expression is as follows: ； in, Indicates the cache space at time Not deposited The previously cached data set Indicates the cache space at time deposit The subsequent cached data set Indicates at time The calculated data, This represents the set of times when all data in the application has been calculated. Indicates at time The index of the task being performed. Indicates the total memory limit of the cache space, | | indicates the number of jobs in the application. Representing data The amount of memory space occupied.

4. The dynamic cache replacement method according to claim 1, characterized in that, In step (4), the cache replacement problem is solved. Including: Simplifying cache replacement issues based on big data processing characteristics And based on the idea of ​​dynamic programming, the simplified cache replacement problem is solved.

5. The dynamic cache replacement method according to claim 4, characterized in that, The problem of simplifying cache replacement based on big data processing characteristics include: By replacing all stages of each job in the application with the "job critical path," the execution mode of the data processing stage changes from parallel to serial. (The problem is...) Simplified into a problem The "critical path of the task" is defined as the computational chain of the stage with the longest execution time in the task. The abstract dataset computed in each stage is replaced with "hotspot access data," and the data computation mode in the data processing stage changes from parallel to serial. Simplified into a problem "Hot access data" is defined as an abstract dataset represented by nodes with an out-degree greater than 1 in a directed acyclic graph formed by the application. Replace the "hotspot access data" in each data processing stage with the execution result of "stage representative calculation". (The problem is...) Simplified into a problem ,question Equivalent to the 0-1 knapsack problem, where "stage representativeness calculation" refers to the data processing operator represented by the final calculated data in the data processing stage.

6. The dynamic cache replacement method according to claim 5, characterized in that, The simplified cache replacement problem solved using dynamic programming includes: Preprocessing based on the idea of ​​problem simplification: Receiving parameter job set ,exist Data already existing in the time cache space and in Data waiting to be added to the cache space By calculating the "critical path of the task," analyzing "hotspot access data," and statistically analyzing "representative calculations of the stage" and their caching benefits, a "representative data set of the stage" is returned. Cache benefits of "representative data for a particular stage" ; Cache replacement based on dynamic programming: Based on the "stage-representative data set" returned by preprocessing. Cache benefits of "representative data for a particular stage" At the same time, combined with the memory limit of the cache space As input, with the upper limit of the cache space memory... Each value and the set of "stage representative data" The traversal of each element in the table, using dynamic programming to decompose the cache replacement problem into multiple subproblems, yields the solution. Optimal caching decision .

7. The dynamic cache replacement method according to claim 6, characterized in that, Maximum memory limit in cache space With each data Memory occupied When the value is a natural number, the result of this cache replacement algorithm based on the idea of ​​dynamic programming is the optimal cache decision.

8. A computer device, characterized in that, include: One or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, wherein the programs, when executed by the processor, implement the steps of the method as described in any one of claims 1-7.

9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

Images