A shadow computing method based on big data platform

Abstract

The present invention discloses a shadow computing method based on a big data platform, comprising: receiving a processing task from a client; calculating the ratio between the size of the dataset of the processing task and the size of the executor memory required for the dataset of the processing task; if the ratio is greater than a shadow computing startup threshold, generating multiple shadow tasks and multiple shadow datasets based on the ratio and the processing task; wherein the shadow datasets have the same ratio of executor memory; sending the multiple shadow tasks and multiple shadow datasets to a slave node, so that the slave node applies a cache replacement algorithm to the shadow dataset according to each shadow task; receiving the cache hit rate of each shadow task replied by the slave node, and selecting an optimal cache replacement algorithm for the shadow task as the cache replacement algorithm for the dataset of the processing task based on the cache hit rates of all shadow tasks. The present invention adaptively selects a more appropriate cache replacement algorithm, and has high flexibility and optimization effect.

Description

Technical Field

[0001] The present invention relates to the field of computing engine technology, and in particular to a shadow computing method based on a big data platform. Background Art

[0002] The Spark distributed computing framework employs a least recently used (LRU) cache replacement strategy. The basic idea behind LRU is to predict the likelihood of future use of a data block based on its recent usage. To implement the LRU cache replacement algorithm, Spark's Memory Store class implements a distributed in-memory database based on blocks, managing block data that needs to be written to memory. Block data can be stored in either serialized or deserialized form, but both storage methods must use the Spark Memory Entry interface. Specifically, the Memory Store manages memory objects whose parent interface is the Memory Entry interface. In the current version of Spark, the Memory Store manages memory objects using a doubly linked list, a Linked Hash Map. A Linked Hash Map is an ordered hash map that manages elements in the list in the order they are inserted, forming a doubly linked list. When new data is cached, it is inserted at the head of the Linked Hash Map. When data in the linked list is accessed again, it is moved to the head of the linked list. When new data needs to be cached and Spark storage memory is insufficient, Linked Hash Map discards data from the end of the list forward.

[0003] Therefore, the current cache replacement strategy has the following problems: (1) The LRU (least recently used) algorithm will evict the least recently used RDD from the memory each time a new RDD is read. In the iterative algorithms commonly used in big data, such as PageRank, K-means, and neural networks, there are some cases where some RDDs need to be used repeatedly. The simple judgment mechanism of LRU is also easy to evict such RDDs from the memory, resulting in repeated reading and eviction of the same RDD, resulting in a low cache hit rate. (2) It does not take into account the reuse of RDD data blocks and RDD execution efficiency. The LRU cache replacement method only roughly selects the RDD that has not been used the longest to eliminate. Such an elimination strategy is likely to eliminate RDDs with high data repetition but not recently used from the memory, and may also cause multiple reused data to replace each other, resulting in problems such as a decrease in cache hit rate and unsatisfactory execution efficiency in some multi-iteration tasks. (3) It does not consider the algorithm differences when the cache size is different. When the memory space of the cluster machine is sufficient, the cache replacement frequency is low, and the iterative algorithm can be completed more smoothly without the need for cache replacement operations in one iteration. The LRU algorithm can achieve a better effect of reducing Spark's read and write overhead with a smaller amount of computation. However, when available memory space is tight, the optimization effect of the LRU cache replacement mechanism on Spark's read and write overhead will drop sharply. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a shadow computing method based on a big data platform, which can consider multiple cache strategies and select a cache replacement strategy with high cache hit rate and fast execution efficiency through shadow computing.

[0005] In a first aspect, an embodiment of the present invention provides a shadow computing method based on a big data platform, the shadow computing method based on the big data platform comprising:

[0006] Receive processing tasks from the client;

[0007] Calculating a ratio between a data set size of the processing task and an executor memory size required for the data set of the processing task;

[0008] If the ratio is greater than a shadow computing start threshold, generating a plurality of shadow tasks and a plurality of shadow data sets according to the ratio and the processing task; wherein the shadow data sets have executor memories of the same ratio;

[0009] Sending a plurality of the shadow tasks and a plurality of the shadow data sets to a slave node, so that the slave node applies a cache replacement algorithm to calculate the shadow data sets according to each of the shadow tasks;

[0010] Receive the cache hit rate replied by each shadow task from the slave node, and select an optimal cache replacement algorithm of the shadow task as the cache replacement algorithm of the data set of the processing task according to the cache hit rates of all the shadow tasks.

[0011] The method according to the embodiment of the present invention has at least the following beneficial effects:

[0012] First, by calculating the ratio of the dataset of the processing task and the executor memory corresponding to the dataset, the storage memory occupied by the small dataset of the shadow calculation during the calculation process is ensured to be as close as possible to the calculation of the large dataset of the processing task, and the cache replacement algorithm is simulated more realistically and accurately; then, by calculating the cache hit rate of the shadow calculation under each cache replacement algorithm, the most intuitive performance of each cache replacement algorithm is provided, and multiple cache replacement algorithms are considered for comparison to improve the robustness; finally, the cache replacement algorithm is selected according to the optimal cache hit rate, and the optimal cache replacement algorithm is obtained. The selection of the cache replacement algorithm does not depend on the actual scenario, and the most appropriate cache replacement algorithm can be adaptively selected in different workflows, which is highly flexible.

[0013] According to some embodiments of the present invention, calculating the ratio between the size of the data set of the processing task and the size of the executor memory required for the data set of the processing task includes:

[0014]

[0015] Among them, ratio represents the ratio of the dataset of the processing task to the executor memory required for the dataset of the processing task; datasetSize is the size of the dataset of the processing task; ExecutorMemory represents the memory size of the executor initialization; fra represents the ratio of storage memory to execution memory.

[0016] According to some embodiments of the present invention, generating a plurality of shadow tasks and a plurality of shadow data sets according to the ratio and the processing task includes:

[0017] The ratio and the processing task are scaled by sampling to generate a plurality of shadow tasks and a plurality of shadow data sets.

[0018] According to some embodiments of the present invention, after selecting an optimal cache replacement algorithm for the shadow task as the cache replacement algorithm for the data set of the processing task based on the cache hit rates of all the shadow tasks, the shadow computing method based on the big data platform further includes:

[0019] A history library is established, and the history library is used to store records of the cache replacement algorithm selected by the processing task.

[0020] According to some embodiments of the present invention, the record of the cache replacement algorithm selected by the processing task is stored in the following manner:

[0021] Extracting a DAG graph of the RDD structure tree of the processing task, the size of the data set of the processing task and the ratio of the executor memory size required by the data set of the processing task, and the cache replacement algorithm selected by the processing task;

[0022] The DAG graph is stored in the history library in the form of network data text, and the ratio of the data set size of the processing task to the executor memory size required by the data set of the processing task and the cache replacement algorithm selected by the processing task are stored in the history library through the file address.

[0023] According to some embodiments of the present invention, before generating a plurality of shadow tasks and a plurality of shadow data sets according to the ratio and the processing task, the shadow computing method based on the big data platform further includes:

[0024] It is determined whether the processing task matches the historical tasks in the history library. If the processing task matches the historical tasks in the history library, the cache replacement algorithm of the historical task is selected as the cache replacement algorithm of the processing task.

[0025] According to some embodiments of the present invention, determining whether the processing task matches the historical tasks in the history library includes:

[0026] If the DAG graph of the processing task has multiple repeated stages, the stage is used as the feature DAG; if the DAG graph of the processing task does not have multiple repeated stages, the entire algorithm is used as the feature DAG;

[0027] Extracting the number of nodes and the number of edges from the feature DAG as algorithm features;

[0028] Calculating a historical ratio of a dataset size of the historical task to an executor memory size required by the dataset of the historical task;

[0029] Determine whether the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the historical library. If the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the historical library, and the difference between the ratio and the historical ratio is less than a threshold, then the processing task matches the historical tasks in the historical library.

[0030] According to some embodiments of the present invention, determining whether the algorithmic features of the processing task are consistent with the algorithmic features of historical tasks in the history library includes:

[0031] Calculating the longest critical path vector of the feature DAG of the processing task and the feature DAG of the historical task and calculating the similarity between the longest critical path vectors;

[0032] If the similarity between the longest critical path vectors is less than a similarity threshold, it is determined that the algorithm feature of the processing task is consistent with the algorithm feature of the historical tasks in the history library.

[0033] According to some embodiments of the present invention, after selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task, the method further includes:

[0034] The record of selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task is not stored in the history library.

[0035] According to some embodiments of the present invention, the cache replacement algorithm includes LRU algorithm, LFU algorithm and FIFO algorithm.

[0036] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0038] Figure 1 This is a flowchart of a shadow computing method based on a big data platform provided by one embodiment of the present invention;

[0039] Figure 2 This is a flow chart of establishing a history library provided by one embodiment of the present invention;

[0040] Figure 3 This is a flowchart of how the history library is stored according to an embodiment of the present invention;

[0041] Figure 4 This is a flowchart for determining whether a processing task matches a historical task, provided by an embodiment of the present invention;

[0042] Figure 5 This is a specific flow chart for determining whether a processing task matches a historical task, provided by an embodiment of the present invention;

[0043] Figure 6 This is a flowchart of how to avoid being stored in the history library provided by an embodiment of the present invention;

[0044] Figure 7 This is a flowchart of a shadow computing method based on a big data platform provided by one embodiment of the present invention;

[0045] Figure 8 This is a schematic diagram of a shadow computing method based on a big data platform provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0047] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0048] In the description of the present invention, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0050] Reference Figure 1 In some embodiments of the present invention, a shadow computing method based on a big data platform is provided, comprising:

[0051] Step S100: receiving a processing task from the client.

[0052] Step S200: Calculate the ratio between the size of the data set of the processing task and the size of the executor memory required for the data set of the processing task.

[0053] Step S300: If the ratio is greater than the shadow computing start threshold, multiple shadow tasks and multiple shadow data sets are generated according to the ratio and the processing task; wherein the shadow data sets have the same ratio of executor memory.

[0054] Step S400: Send multiple shadow tasks and multiple shadow data sets to the slave node, so that the slave node applies a cache replacement algorithm according to each shadow task to calculate the shadow data set.

[0055] Step S500: Receive the cache hit rate replied by each shadow task from the node, and select an optimal cache replacement algorithm for the shadow task as the cache replacement algorithm for the data set of the processing task according to the cache hit rates of all shadow tasks.

[0056] In this embodiment, first, step S200 calculates the ratio of the data set of the processing task and the executor memory corresponding to the data set, and ensures that the storage memory occupied by the small data set of the shadow calculation during the calculation process is as close as possible to the calculation of the large data set of the processing task, thereby simulating the cache replacement algorithm more realistically and accurately; then, step S400 calculates the cache hit rate of the shadow calculation under each cache replacement algorithm, provides the most intuitive performance of each cache replacement algorithm, and considers multiple cache replacement algorithms for comparison, thereby improving robustness; finally, step S500 selects the cache replacement algorithm according to the optimal cache hit rate, obtains the optimal cache replacement algorithm, and the selection of the cache replacement algorithm does not depend on the actual scenario, and can adaptively select the most appropriate cache replacement algorithm in different workflows, with high flexibility.

[0057] In some embodiments of the present invention, calculating the ratio between the size of a data set of a processing task and the size of an executor memory required to process the data set of the task includes:

[0058]

[0059] Among them, ratio represents the ratio of the dataset of the processing task to the executor memory required for the dataset of the processing task; datasetSize is the size of the dataset of the processing task; ExecutorMemory represents the memory size of the executor initialization; fra represents the ratio of storage memory to execution memory.

[0060] It should be noted that the formula for generating the corresponding shadow tasks and shadow datasets based on the ratio and processing tasks is as follows:

[0061]

[0062] Here, shadowMemory indicates the Spark executor memory size required for shadow computing; shadowSize indicates the size of the shadow dataset.

[0063] To ensure that the data block access and replacement process in shadow computing more closely resembles actual computation, the storage memory usage of small datasets during computation must be as close as possible to that of large datasets. Storage memory manages storage space in the big data platform Spark. In Spark's cache management, programs require a fixed 300MB of reserved space on the executor to ensure basic program execution. Of the remaining space, 60% is used as storage memory for cached data blocks and as execution memory for intermediate data during Spark's data shuffle operations. These two portions of space can be dynamically allocated according to specific rules. Therefore, the ratio calculated using the above formula optimizes the balance between the shadow dataset and the executor memory required by the shadow dataset, providing excellent hardware conditions for subsequent shadow task computations.

[0064] In some embodiments of the present invention, generating multiple shadow tasks and multiple shadow data sets according to the ratio and processing task includes:

[0065] The ratio and processing tasks are scaled by sampling to generate multiple shadow tasks and multiple shadow data sets.

[0066] It should be noted that in shadow computing, because the calculation does not require the validity of the calculation results, it only needs to simulate the data access rules during the calculation process. Therefore, the scaling sampling of the data set can be lossy scaling. For text-type data sets, shadow data sets can be generated by sampling the source data set. For example, for the classic text word counting algorithm, the shadow data set can be obtained by sampling the text. For image-type data sets, such as image restoration in machine learning, the images can be losslessly scaled and the image set can be sampled and reduced. For data sets with special algorithm requirements, such as requiring a certain correlation between the input data sets, direct sampling processing may result in errors such as dead loops. In this case, the user can actively provide a smaller data set as a shadow data set for shadow computing.

[0067] Sampling scaling can be used to easily and efficiently generate shadow tasks and shadow data sets without taking up too much computing power, saving a lot of computing resources.

[0068] Reference Figure 2 In some embodiments of the present invention, after selecting an optimal cache replacement algorithm for a shadow task as the cache replacement algorithm for the data set of the processing task based on the cache hit rates of all shadow tasks, the shadow computing method based on the big data platform further includes:

[0069] Step S600: Establish a history library, which is used to store records of selecting a cache replacement algorithm for a processing task.

[0070] The establishment of a history library makes it easier for the shadow computing framework to search for similar historical tasks from the history library and select the corresponding cache replacement algorithm when submitting the same or similar processing tasks again, so as to save the time of repeating shadow computing for known tasks.

[0071] Reference Figure 3 In some embodiments of the present invention, the record of the cache replacement algorithm selected by the processing task is stored in the following manner:

[0072] Step S601: extract the DAG graph of the RDD structure tree of the processing task, the ratio of the dataset size of the processing task to the executor memory size required for the dataset of the processing task, and the cache replacement algorithm selected for the processing task.

[0073] Step S602: Store the DAG graph in the history library in the form of network data text, and store the ratio of the data set size of the processing task to the executor memory size required for the data set of the processing task and the cache replacement algorithm selected for the processing task in the history library through the file address.

[0074] Storing the DAG graph in the history library as network data text can save memory resources. Storing it in the history library as a file address can achieve fast call and efficient comparison.

[0075] Reference Figure 4 In some embodiments of the present invention, before generating multiple shadow tasks and multiple shadow data sets according to the ratio and the processing task, the shadow computing method based on the big data platform further includes:

[0076] Step S310: determine whether the processing task matches the historical tasks in the history library. If the processing task matches the historical tasks in the history library, select the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task.

[0077] Searching for similar historical tasks from the history library and selecting the corresponding cache replacement algorithm can save the time of repeated shadow calculations for known tasks.

[0078] Reference Figure 5 In some embodiments of the present invention, determining whether a processing task matches a historical task in a history library includes:

[0079] Step S311: If the DAG graph of the processing task has multiple repeated stages, the stages are used as the feature DAG; if the DAG graph of the processing task does not have multiple repeated stages, the entire algorithm is used as the feature DAG;

[0080] Step S312: extract the number of nodes and edges from the feature DAG as algorithm features;

[0081] Step S313: Calculate the historical ratio of the dataset size of the historical task to the executor memory size required by the dataset of the historical task;

[0082] Step S314: Determine whether the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the history library. If the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the history library, and the difference between the ratio and the historical ratio is less than the threshold, then the processing task matches the historical tasks in the history library.

[0083] By comparing algorithm features and the ratio of processing tasks with the historical ratio of historical tasks, it is judged whether the processing task and the historical task are similar. This can accurately judge whether the processing task and the historical task are similar and has a certain degree of robustness.

[0084] In some embodiments of the present invention, determining whether the algorithmic features of the processing task are consistent with the algorithmic features of historical tasks in the history library includes:

[0085] Calculate the longest critical path vector of the feature DAG of the processing task and the feature DAG of the historical task and calculate the similarity between the longest critical path vectors;

[0086] If the similarity between the longest critical path vectors is less than the similarity threshold, it is determined that the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the history library.

[0087] By judging whether the algorithm features are consistent through the similarity between the longest critical path vectors, simple and high-precision calculations can be achieved, avoiding misjudgment of algorithm features.

[0088] Reference Figure 6 In some embodiments of the present invention, after selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task, the method further includes:

[0089] Step S320: The record of selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task is not stored in the history library.

[0090] Selecting the cache replacement algorithm of historical tasks as the cache replacement algorithm for processing tasks. Processing tasks are not stored in the historical library, which can save memory resources and prevent historical tasks from being too redundant.

[0091] In some embodiments of the present invention, the cache replacement algorithm includes an LRU algorithm, an LFU algorithm, and a FIFO algorithm.

[0092] It should be noted that three cache replacement algorithms are used in this scheme: LRU (least recently used algorithm), LFU (most recently used algorithm) and FIFO (first in first out algorithm), and the time complexity of the three cache replacement algorithms is O(1).

[0093] Using the LRU algorithm, LFU algorithm, and FIFO algorithm can minimize the read and write overhead and performance impact of Spark big data platform operations under different memory environments.

[0094] Reference Figure 7 and Figure 8 To facilitate understanding by those skilled in the art, a specific embodiment of the present invention provides a shadow computing method based on a big data platform, comprising the following steps:

[0095] The first step is to perform executor memory calculation in the shadow computing framework of the big data Spark platform.

[0096] First, let's explain the executor memory calculations within the shadow computing framework of the big data Spark platform. To ensure that the shadow computing data block access and replacement process is more realistically aligned with actual computation, the shadow dataset's storage memory usage during the computation process must be as close as possible to the computation of the task's dataset. Storage space in Spark is primarily managed by Storage Memory. In Spark's cache management, programs require a fixed 300MB of reserved space for executors to ensure basic program execution. Of the remaining space, 60% is used as storage memory for cached data blocks and as execution memory for intermediate data during Spark's data shuffle operations. These two portions of space can be dynamically allocated based on specific rules. The "60%" ratio can be set using the spark.memory.fraction parameter, denoted as fra. Therefore, we can first calculate the ratio of the task's dataset to the configured Spark executor storage memory space:

[0097]

[0098]

[0099] Among them, ratio represents the ratio of the dataset of the processing task to the executor memory required for the dataset of the processing task; datasetSize is the size of the dataset of the processing task; ExecutorMemory represents the memory size of the executor initialization; fra represents the ratio of storage memory to execution memory.

[0100] After calculating the ratio of the large dataset size to the executor storage memory, you can use this ratio to set the Spark executor memory required for shadow dataset calculations. The formula is as follows:

[0101]

[0102] The formula is transformed into:

[0103]

[0104] You can also write directly:

[0105]

[0106] Here, shadowMemory indicates the Spark executor memory size required for shadow computing, in MB; shadowSize indicates the size of the shadow dataset, in MB.

[0107] Step 2: Generate shadow dataset.

[0108] In shadow computing, since the calculation does not require the validity of the calculation results, but only needs to simulate the data access rules during the calculation process, the scaling sampling of the data set of the processing task can be lossy scaling. For data sets of text-type processing tasks, shadow data sets can be generated by sampling the source data set. For example, for the classic text word counting algorithm, the shadow data set can be obtained by sampling the text. For data sets of image-type processing tasks, such as image restoration in machine learning, the images can be losslessly scaled and the image sets can be sampled and reduced. For data sets of processing tasks with special algorithm requirements, such as those that require a certain correlation between the data in the input data sets, direct sampling processing may result in errors such as dead loops. In this case, the user can actively provide a smaller data set as a shadow data set for shadow computing.

[0109] The third step is to perform shadow calculations on the shadow dataset and select a suitable cache replacement algorithm.

[0110] The shadow computing framework of the big data Spark platform starts shadow computing. Shadow computing will submit multiple calculation programs based on low time complexity cache replacement algorithms to replace the original data set with the shadow data set (three cache replacement algorithms are used in this solution: LRU (least recently used), LFU (most recently used) and FIFO (first in first out). The time complexity of the three cache replacement algorithms is O(1)). Based on the cache hit rate of the shadow computing under different cache replacement strategies, the cache replacement algorithm suitable for the current workload is selected for the original calculation program. The basic idea is as follows:

[0111] Step 1: After the client submits a task, the master node traverses the DAG Scheduler (directed acyclic graph scheduler) when initializing the Spark Context (the entry point for the Spark client to connect to the cluster) and generates an RDD structure tree containing the RDD information for the task. When the algorithm detects a data read operation, it reads the program dataset size information based on the input path and marks RDDs with a reference count greater than a threshold N as RDDs to be cached.

[0112] Step 2: The master node calculates the ratio of the job based on the dataset size and the memory size of the executor processing the task. When the ratio is greater than the shadow computing start threshold R, the shadow computing task is started.

[0113] Step 3: The master node saves a copy of the source dataset (the dataset for processing tasks) and samples and scales the dataset to generate a shadow dataset.

[0114] Step 4: The master node calculates the executor memory size required for shadow computing using the ratio value and submits multiple shadow computing jobs using different cache replacement algorithms to the slave nodes for computation. The cache replacement algorithms used in this solution are LRU, LFU, and FIFO.

[0115] Step 5: After the slave node completes the shadow calculation, it returns the cache hit rate of the calculation process to the master node.

[0116] Step 6: The master node selects the appropriate cache replacement algorithm based on the collected cache hit rate information under different cache replacement algorithms and sends it to the slave node for actual calculation. The pseudo code implementation is shown in Table 1:

[0117] Table 1

[0118]

[0119] Step 4: Optimize the processing of repeatedly submitted tasks.

[0120] In actual production, it is often the case that the same workflow performs calculations on different data sets. In order to avoid repeated shadow computing processes, the cache replacement strategy based on shadow computing maintains a historical library locally. The historical library stores the DAG of the extracted RDD structure tree and the corresponding ratio information and the cache replacement algorithm used in the historical library. The DAG information is stored in the form of network data text, and the corresponding ratio and the cache replacement algorithm information used are stored in the library together with the file address. When a new processing task is submitted, after extracting the RDD structure tree, its DAG is compared with the historical job DAG in the local library. When the DAG is consistent and the ratio is similar, the cache replacement strategy based on shadow computing will directly call the cache replacement algorithm selected by the historical task in the historical library to save the time required for shadow computing. The specific processing steps are as follows:

[0121] Step 1: Receive the submitted processing task and extract the DAG graph of the processing task from the DAG Scheduler.

[0122] Step 2: Check the type of the RDD conversion DAG graph in the submitted processing task. If there are multiple repeated stages, it is an iterative algorithm. The repeated stages are taken as the feature DAG. If it is a non-iterative algorithm, the entire algorithm is taken as the feature DAG.

[0123] In Step 3, the number of nodes and edges extracted from the feature DAG in Step 2 is used as the algorithmic features of the algorithm. These are compared with the algorithmic features of historical tasks stored in the history library. If there are historical tasks with consistent features and a ratio difference of less than 10%, the process proceeds to Step 4. If there are no records of the same algorithmic features in the history library, the DAG features and the corresponding ratio are saved in the history library and the process proceeds to Step 5.

[0124] In Step 4, the longest critical path between the submitted job's DAG and the DAG found in the history database is vectorized. A metric is then calculated between the two vectors to determine the similarity between the two DAGs. If the similarity difference is within S, the cache replacement algorithm used for the corresponding DAG in the history database is read for formal calculation, and the process ends.

[0125] Step 5: Start shadow computing. The master node distributes shadow computing tasks using different cache replacement algorithms to the slave nodes, and records the final selected cache replacement algorithm in the entry corresponding to the algorithm in the history library. The process ends.

[0126] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing data (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired data and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any data delivery media.

[0127] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0128] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A shadow computing method based on a big data platform, characterized in that: The shadow computing method based on the big data platform includes: Receive processing tasks from the client; Calculating a ratio between a data set size of the processing task and an executor memory size required for the data set of the processing task; If the ratio is greater than a shadow computing start threshold, generating a plurality of shadow tasks and a plurality of shadow data sets according to the ratio and the processing task; wherein the shadow data sets have executor memories of the same ratio; Sending a plurality of the shadow tasks and a plurality of the shadow data sets to a slave node, so that the slave node applies a cache replacement algorithm to calculate the shadow data sets according to each of the shadow tasks; Receive the cache hit rate replied by each shadow task of the slave node, and select an optimal cache replacement algorithm of the shadow task as the cache replacement algorithm of the data set of the processing task according to the cache hit rates of all the shadow tasks; After selecting an optimal cache replacement algorithm for the shadow task as the cache replacement algorithm for the data set of the processing task according to the cache hit rates of all the shadow tasks, the shadow computing method based on the big data platform further includes: Establishing a history database, the history database is used to store records of the cache replacement algorithm selected by the processing task; The record of selecting the cache replacement algorithm for the processing task is stored in the following manner: Extracting a DAG graph of the RDD structure tree of the processing task, the size of the data set of the processing task and the ratio of the executor memory size required by the data set of the processing task, and the cache replacement algorithm selected by the processing task; The DAG graph is stored in the history library in the form of network data text, and the ratio of the data set size of the processing task to the executor memory size required by the data set of the processing task and the cache replacement algorithm selected by the processing task are stored in the history library via a file address; Before generating a plurality of shadow tasks and a plurality of shadow data sets according to the ratio and the processing task, the shadow computing method based on the big data platform further includes: determining whether the processing task matches the historical tasks in the history library, and if the processing task matches the historical tasks in the history library, selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task; The determining whether the processing task matches the historical tasks in the history library includes: If the DAG graph of the processing task has multiple repeated stages, the stage is used as the feature DAG; if the DAG graph of the processing task does not have multiple repeated stages, the entire algorithm is used as the feature DAG; Extracting the number of nodes and the number of edges from the feature DAG as algorithm features; Calculating a historical ratio of a dataset size of the historical task to an executor memory size required by the dataset of the historical task; Determine whether the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the historical library. If the algorithm characteristics of the processing task are consistent with the algorithm characteristics of the historical tasks in the historical library, and the difference between the ratio and the historical ratio is less than a threshold, then the processing task matches the historical tasks in the historical library.

2. The shadow computing method based on a big data platform according to claim 1, characterized in that: The calculating the ratio between the size of the data set of the processing task and the size of the executor memory required for the data set of the processing task includes: in, Indicates the ratio of the dataset of the task to the executor memory required to process the dataset of the task; The size of the dataset for the processing task; Indicates the memory size of the executor initialization; Indicates the ratio of storage memory to execution memory.

3. The shadow computing method based on a big data platform according to claim 1, characterized in that: Generating a plurality of shadow tasks and a plurality of shadow data sets according to the ratio and the processing task includes: The ratio and the processing task are scaled by sampling to generate a plurality of shadow tasks and a plurality of shadow data sets.

4. The shadow computing method based on a big data platform according to claim 1, characterized in that: The determining whether the algorithm features of the processing task are consistent with the algorithm features of the historical tasks in the history library includes: Calculating the longest critical path vector of the feature DAG of the processing task and the feature DAG of the historical task and calculating the similarity between the longest critical path vectors; If the similarity between the longest critical path vectors is less than a similarity threshold, it is determined that the algorithm feature of the processing task is consistent with the algorithm feature of the historical tasks in the history library.

5. The shadow computing method based on a big data platform according to claim 1, characterized in that: After selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task, the method further includes: The record of selecting the cache replacement algorithm of the historical task as the cache replacement algorithm of the processing task is not stored in the history library.

6. The shadow computing method based on a big data platform according to any one of claims 1 to 5, characterized in that: The cache replacement algorithms include LRU algorithm, LFU algorithm and FIFO algorithm.

Images