Data processing system and data processing method

Abstract

The present disclosure relates to a data processing system and a data processing method, the data processing system comprising: a data computing node configured to: generate an execution plan based on a data processing job submitted by a user; analyzing the execution plan to generate multiple pieces of key information corresponding to the multiple stages, wherein the key information comprises stage identifiers and data type information of the corresponding stages; target cache prompt information is generated based on the target key information corresponding to the target stage, the target cache prompt information is used for indicating at least one data prefetching strategy corresponding to at least one data identifier included in the target stage, and the data prefetching strategy is determined based on the data type information; sending the target cache prompt information to a data storage client; the data storage client is configured to pre-fetch target data indicated by a target data identifier from a distributed storage layer based on a target data pre-fetching strategy corresponding to the target data identifier; and storing the target data in a local cache of the data storage client.

Description

Technical Field

[0001] This disclosure relates to data processing technology. More specifically, it relates to a data processing system and a data processing method. Background Technology

[0002] With the rapid growth in demand for big data analytics, Apache Spark has become a mainstream distributed data processing framework. Spark typically runs on a Yarn cluster environment, using distributed storage systems (such as HDFS and JuiceFS) for data reading and writing. Data access performance has become a key factor affecting the efficiency of Spark tasks. File caching and prefetching techniques can effectively reduce data access latency and improve computational performance, but related technical solutions all prefetch and cache data in Spark memory, relying heavily on memory and being subject to significant resource limitations. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a data processing system and a data processing method.

[0004] In a first aspect, embodiments of this disclosure provide a data processing system, comprising: a data computing node configured to: generate an execution plan based on a data processing job submitted by a user, the execution plan including multiple stages; parse the execution plan to generate multiple key information corresponding to each of the multiple stages, the key information including stage identifiers and data type information of the corresponding stages; generate target cache prompt information based on target key information corresponding to a target stage, the target cache prompt information indicating at least one data prefetching strategy corresponding to at least one data identifier included in the target stage, the data prefetching strategy being determined based on the data type information, the target stage being one of the multiple stages; send the target cache prompt information to a data storage client; the data storage client configured to: prefetch target data indicated by the target data identifier from a distributed storage layer based on the target data prefetching strategy corresponding to the target data identifier; store the target data in a local cache of the data storage client, so that when the data computing node executes a target computing task corresponding to the target data, it reads the target data from the local cache and executes the computing task based on the target data.

[0005] In this embodiment, a data processing system consisting of a "data computing node - data storage client" is constructed. The data computing node parses the execution set plan corresponding to the data processing job to obtain key information for each stage, and generates cache prompts for the target stage based on this key information. The data storage client can then prefetch and cache the target data required for the calculation of the target stage based on these cache prompts. This allows the data computing node to read the target data from its local cache and execute the calculation task based on that target data when executing the target calculation task corresponding to the target data. By caching the prefetched data in the local cache of the data storage client, and considering the larger storage space required by both the data computing node and the data storage client, more data can be prefetched and cached, improving prefetching and caching efficiency.

[0006] In some embodiments of this disclosure, the key information further includes target information for the data at the corresponding stage. The target information includes at least one of the following: locality attribute, expected number of accesses, and processing operator type. The target cache prompt information is also used to indicate at least one priority information corresponding to the at least one piece of data. The priority information is determined based on the target information of the corresponding data and a first mapping relationship. The first mapping relationship is used to indicate the correspondence between the target information and the priority information. The priority information is used to indicate the cache priority of the corresponding data. The data storage client is further configured to delete the target data identifier from the local cache when the cache priority indicated by the target priority information corresponding to the target data identifier is lower than the priority threshold.

[0007] In this embodiment of the disclosure, by adding priority information of the data to be cached to the cache prompt information and marking the data already cached in the local cache with the priority information, low-value data can be deleted periodically in order to improve cache space and cache utilization.

[0008] In some embodiments of this disclosure, the key information also includes the data processing operator type for the corresponding stage, and the data prefetching strategy is determined based on the data type information and the processing operator type.

[0009] In this embodiment of the disclosure, determining the data prefetching strategy based on the data type information and the processing operator type can make the data prefetching strategy more accurate.

[0010] In some embodiments of this disclosure, the target cache hint information is further used to indicate at least one prefetch aggressive coefficient corresponding to the at least one data identifier, the prefetch aggressive coefficient being determined based on the data type information; the data storage client is further configured to: determine the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and a second mapping relationship, the second mapping relationship being used to indicate the correspondence between the target prefetch aggressive coefficient, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and the current prefetch window size; the data storage client is specifically configured to: prefetch the target data corresponding to the target data identifier from the distributed storage layer based on the target data prefetch strategy and the current prefetch window size.

[0011] In this embodiment of the disclosure, by adding a prefetch aggressive coefficient to the cache prompt information, and then dynamically determining the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the basic prefetch window size, the prefetch hit rate of the previous prefetch window, and the second mapping relationship, the resource utilization rate is improved.

[0012] In some embodiments of this disclosure, the data storage client is specifically configured to: determine the current prefetch window size based on the target prefetch aggressiveness coefficient corresponding to the target data identifier, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and a third mapping relationship. The third mapping relationship is used to indicate the correspondence between the target prefetch aggressiveness coefficient, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the current prefetch window size.

[0013] In this embodiment of the disclosure, by adding a prefetch aggressive coefficient to the cache prompt information, and then dynamically determining the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the third mapping relationship, the resource utilization rate is improved.

[0014] In some embodiments of this disclosure, the key information further includes the processing operator type of the data at the corresponding stage; when the processing operator type corresponding to the target data identifier is the shuffle type, the key information further includes first information; the target cache hint information is also used to indicate the shuffle data block location information, which is determined based on the first information; the data storage client is specifically configured to: determine an in-plan prefetch strategy based on the shuffle data block location information; determine a target prefetch strategy according to the target data prefetch strategy and the in-plan prefetch strategy; and prefetch the target data from the distributed storage layer based on the target prefetch strategy.

[0015] In this embodiment of the disclosure, by including first information corresponding to shuffle write in the key information, the target cache prompt information is further used to indicate the shuffle data block location information, so that the data storage client can determine the planned prefetch strategy based on the shuffle data block location information, and combine the target data prefetch strategy to determine a more suitable target prefetch strategy, thereby improving prefetch efficiency.

[0016] In some embodiments of this disclosure, the stage identifier is further used to indicate the processing operator type of the data in the corresponding stage; the data storage client is specifically configured to: obtain the target processing operator type corresponding to the target data identifier based on the target stage identifier corresponding to the target data identifier; if the target processing operator type is a first type, determine an in-plan prefetch strategy; determine a first prefetch strategy based on the target data prefetch strategy and the in-plan prefetch strategy; prefetch the target data from the distributed storage layer based on the first prefetch strategy; or, if the target processing operator type is a second type, determine a runtime prefetch strategy; determine a second prefetch strategy based on the target data prefetch strategy and the runtime prefetch strategy; prefetch the target data from the distributed storage layer based on the second prefetch strategy.

[0017] In some embodiments of this disclosure, when the target data prefetching strategy has been determined, it can be further determined whether it is planned prefetching or runtime prefetching based on the processing operator type. Combined with the target data prefetching strategy, a more accurate prefetching strategy can be determined, thereby improving prefetching efficiency.

[0018] In some embodiments of this disclosure, the data storage client is specifically configured to: prefetch target data indicated by the target data identifier from the distributed storage layer based on the target data prefetching strategy using a first thread and a first network bandwidth; wherein the first thread belongs to a thread pool outside the thread pool corresponding to the computing task, and the first network bandwidth belongs to a network bandwidth queue outside the network bandwidth queue corresponding to the computing task.

[0019] In this embodiment of the disclosure, the prefetch task and the compute task use different thread pools and network bandwidth queues, which can ensure that the prefetch task will not compete with the compute task (Spark's compute task) for critical resources and affect the critical path of the task.

[0020] In some embodiments of this disclosure, the data storage client has a metadata cache in its memory; specifically, the data storage client is configured to: when the data indicated by the target data identifier is not data in the metadata cache and is not data in the local cache, prefetch the target data from the distributed storage layer based on the target data identifier and the corresponding target data prefetching strategy.

[0021] In this embodiment of the disclosure, metadata stored in the distributed storage layer is pre-cached in the memory of the data storage client, resulting in a metadata cache. This significantly reduces the latency of accessing metadata in the distributed storage layer. Furthermore, when the data indicated by the target data identifier is neither in the metadata cache nor in the local cache, the target data is pre-fetched from the distributed storage layer based on the target data identifier and the corresponding target data pre-fetching strategy. This avoids repeatedly caching already cached data and prevents waste of storage resources.

[0022] Secondly, embodiments of this disclosure provide a data processing method applied to a data processing system, the data processing system including a data computing node and a data storage client, comprising: generating an execution plan based on a user-submitted data processing job through the data computing node, the execution plan including multiple stages; parsing the execution plan to generate multiple key information corresponding to each of the multiple stages, the key information including stage identifiers and data type information of the corresponding stages; generating target cache prompt information based on target key information corresponding to a target stage, the target cache prompt information indicating at least one data prefetching strategy corresponding to at least one data identifier included in the target stage, the data prefetching strategy being determined based on the data type information, the target stage being one of the multiple stages; sending the target cache prompt information to the data storage client; the data storage client being configured to: prefetch target data indicated by the target data identifier from a distributed storage layer based on the target data prefetching strategy corresponding to the target data identifier; storing the target data in a local cache of the data storage client, so that when the data computing node executes a target computing task corresponding to the target data, it reads the target data from the local cache and executes the computing task based on the target data.

[0023] Thirdly, embodiments of this disclosure provide a computer-readable storage medium, including: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the data processing method as described in the second aspect.

[0024] Fourthly, embodiments of this disclosure provide a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the data processing method as shown in the second aspect. Attached Figure Description

[0025] To more clearly illustrate the implementation methods in the embodiments of this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings.

[0026] Figure 1 One of the structural schematic diagrams of a data processing system according to some embodiments is shown; Figure 2 A second schematic diagram of the structure of a data processing system according to some embodiments is shown; Figure 3 One of the flowcharts of a data processing method according to some embodiments is shown; Figure 4 A second schematic flowchart of a data processing method according to some embodiments is shown; Figure 5 A third schematic flowchart of a data processing method according to some embodiments is shown; Figure 6 A fourth schematic flowchart of a data processing method according to some embodiments is shown; Figure 7 Fifth of a series of schematic flowcharts illustrating a data processing method according to some embodiments; Figure 8 A sixth schematic flowchart of a data processing method according to some embodiments is shown; Figure 9 A flowchart of a data processing method according to some embodiments is shown as diagram seven; Figure 10 Eighth schematic flowchart of a data processing method according to some embodiments is shown. Detailed Implementation

[0027] To make the objectives and implementation methods of this disclosure clearer, the exemplary embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this disclosure. Obviously, the exemplary embodiments described are only some embodiments of this disclosure, and not all embodiments.

[0028] It should be noted that the brief descriptions of terms in this disclosure are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this disclosure. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0029] The terms "first," "second," "third," etc., used in this disclosure, in the specification, claims, and accompanying drawings are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0030] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0031] In some embodiments of this disclosure, the data processing system may be a server device, a server cluster, or a data platform composed of multiple server devices or multiple server clusters. It may also be a data computing node and data storage system in a data platform. The data storage client is the front end of the data storage system, and the distributed storage layer may be the back end persistent storage of the data storage system. The specific details can be determined according to the actual situation and are not limited here.

[0032] The data computing nodes can be Apache Spark-based server nodes or other data computing nodes; no specific restrictions are imposed here. With the rapid growth in demand for big data analytics, Apache Spark has become the mainstream distributed data processing framework. Spark typically runs in a Yarn cluster environment, using distributed storage systems (such as HDFS and JuiceFS) for data reading and writing. Data access performance is a key factor affecting the efficiency of Spark tasks. File caching and prefetching techniques can effectively reduce data access latency and improve computational performance. The Spark community has made some optimization attempts, such as improving performance through RDD caching and DataFrame persistence, but these methods largely rely on memory and are subject to significant resource limitations. Moreover, these solutions still have shortcomings in terms of caching granularity and intelligent prefetching.

[0033] The data storage system can be JuiceFS, or other storage systems; no specific restrictions are imposed here. JuiceFS, as a POSIX-compliant distributed file system, supports file caching and prefetching, but its native capabilities have limitations in areas such as high concurrency, hot data identification, and intelligent prefetching.

[0034] In cloud-native and compute-storage separation architectures, Spark job data resides in remote object storage (such as S3 or OSS). Existing solutions (such as JuiceFS and Alluxio) employ simple LRU caching and conservative sequential prefetching, which have the following significant drawbacks: Cache blindness: It cannot distinguish between temporary intermediate data and frequently reused dimension tables / dictionaries, resulting in valuable cache space being occupied by one-time data; Prefetch isolation: Prefetch decisions are based solely on the current task's IO stream and cannot perceive the output data layout of the upstream stages in Spark's Directed Acyclic Graph (DAG), resulting in "untimely prefetching" or "prefetching useless data." Resource contention: The prefetching task and the computation task are not coordinated and may compete for the CPU, network and I / O bandwidth of the same node, interfering with the critical path computation.

[0035] There is a lack of a collaborative optimization framework in the relevant technologies that deeply couples the semantics of Spark applications with the caching behavior of the underlying storage system.

[0036] In some embodiments of this disclosure, a data processing system is provided, including: a data computing node configured to: generate an execution plan based on a user-submitted data processing job, the execution plan including multiple stages; parse the execution plan to generate multiple key information corresponding to each of the multiple stages, the key information including stage identifiers and data type information of the corresponding stages; generate target cache prompt information based on target key information corresponding to a target stage, the target cache prompt information being used to indicate at least one data prefetching strategy corresponding to at least one data identifier included in the target stage, the data prefetching strategy being determined based on the data type information, the target stage being one of the multiple stages; and send the target cache prompt information to a data storage client.

[0037] The data processing job can be an ETL job, which is a data processing task that performs the three core steps of extraction, transformation and loading, or other data processing jobs, which are not limited here.

[0038] In the Spark data processing architecture, Spark data processing jobs, execution plans, and DAGs (Directed Acyclic Graphs) form a three-layered nested relationship from the top-level task to the bottom-level scheduling basis. These three elements are decomposed layer by layer, support each other, and work together to realize the logical transformation and execution of distributed data processing. Each job corresponds to an execution plan, and the core representation of an execution plan is a DAG.

[0039] The specific process of generating an execution plan based on user-submitted data processing jobs can be found in relevant technologies and will not be elaborated here.

[0040] In some embodiments of this disclosure, a lightweight semantic analyzer can be embedded on the Spark side to parse the execution plan and generate multiple key pieces of information corresponding to the multiple stages.

[0041] In this context, multiple stages can correspond one-to-one with multiple key information, or some stages can correspond one-to-one with multiple key information. The specific correspondence can be determined based on the actual situation and is not limited here.

[0042] The stage identifier can be a stage ID or other identifier information; no limitation is made here.

[0043] The data type information may include fact tables (large), dimension tables (small), intermediate shuffle write files, etc., which are not limited here.

[0044] The data type information includes at least one data identifier indicating the data type.

[0045] A single stage can include one or more data points. Therefore, integrating and analyzing the key information of the target stage can generate target cache hint information (CacheHint).

[0046] The target stage can be any one of multiple stages, or any one of a subset of multiple stages; no limitation is made here.

[0047] In this context, at least one data identifier corresponds one-to-one with at least one data prefetching strategy, or at least a portion of the data identifiers may correspond one-to-one with at least one data prefetching strategy. The specific correspondence can be determined based on the actual situation and is not limited here.

[0048] In some embodiments of this disclosure, at least one data identifier may be a data identifier for all the data required for the target stage, or it may be a data identifier for a portion of the data required for the target stage; no limitation is made here.

[0049] The data identifier is used to indicate data. One data comes from one data, so one data identifier corresponds to one data.

[0050] The data prefetching strategy can be sequential prefetching, random prefetching, or full prefetching, and the specific strategy can be determined according to the actual situation. No limit is set here.

[0051] There is a correspondence between the data prefetching strategy and the data type information; what kind of data type information corresponds to what kind of data prefetching strategy.

[0052] In some embodiments of this disclosure, the data storage client is configured to: prefetch target data indicated by the target data identifier from the distributed storage layer based on the target data prefetching strategy corresponding to the target data identifier; store the target data in the local cache of the data storage client, so that when the data computing node executes the target computing task corresponding to the target data, it reads the target data from the local cache and executes the computing task based on the target data.

[0053] The target data identifier can be any one of at least one data identifier, or any one of some of the at least one data identifier; there is no limitation here.

[0054] The local cache is a disk (SSD / HDD) mounted locally on the data storage client, which can be used to store persistent data.

[0055] In some embodiments of this disclosure, such as Figure 1 As shown, the architecture of the data processing system 100 is divided into three layers: data computing nodes 101 (such as the Spark computing layer), data storage clients 102 (such as the enhanced JuiceFS client layer), which is the core for implementing intelligent caching and prefetching. It is embedded in each SparkExecutor process in the form of JNI or Hadoop Compatible File System; and distributed storage layer 103, which is the backend persistent storage of JuiceFS (such as object storage S3 / OSS / OBS), serving as a single trusted source of data.

[0056] In some embodiments of this disclosure, the JuiceFS Hadoop Client provides a standard HDFS interface, enabling Spark to access JuiceFS as transparently as it would access HDFS.

[0057] In some embodiments of this disclosure, such as Figure 2 As shown, data computing node 101 includes a Spark driver 101a and multiple executors (two are shown in the diagram, executors 101b1 and 101b2). The driver is responsible for task scheduling (DAG Scheduler), and the tasks within the executors are the actual data processing units. Figure 2 As shown, the data storage client 102 includes a cache manager 102a, a prefetch engine 102b, memory, and a local cache. The cache manager is the brain of the data storage client; it receives read requests from Spark, coordinates the metadata cache, local cache, and prefetch engine, employs a cache eviction management strategy (such as LRU), and makes prefetching decisions based on access patterns and historical information. The prefetch engine is an asynchronous data loader. According to the instructions of the prefetch controller, it pre-fetches potentially accessed data from the distributed storage layer to the local cache. The local cache uses local SSDs or memory as the caching medium to store frequently accessed data blocks. It employs chunked management for efficient caching and eviction.

[0058] In some embodiments of this disclosure, the Spark Driver's DAG Scheduler determines the data partition (e.g., File Partition part-00001) that each Task needs to process when partitioning Stages. Data storage clients can "eavesdrop" on or receive this task scheduling information. The cache manager combines this information (e.g., the file to be read, offset, Task location) and possible historical access patterns to predict the list of data blocks needed next. Based on this predicted list, the prefetch engine asynchronously prefetches data from the distributed storage layer into the local cache before the Task is actually executed.

[0059] In this embodiment, a data processing system consisting of a "data computing node - data storage client" is constructed. The data computing node parses the execution set plan corresponding to the data processing job to obtain key information for each stage, and generates cache prompts for the target stage based on this key information. The data storage client can then prefetch and cache the target data required for the calculation of the target stage based on these cache prompts. This allows the data computing node to read the target data from its local cache and execute the calculation task based on that target data when executing the target calculation task corresponding to the target data. By caching the prefetched data in the local cache of the data storage client, and considering the larger storage space required by both the data computing node and the data storage client, more data can be prefetched and cached, improving prefetching and caching efficiency.

[0060] In some embodiments of this disclosure, the key information further includes target information for the data at the corresponding stage. The target information includes at least one of the following: locality attribute, expected number of accesses, and processing operator type. The target cache prompt information is also used to indicate at least one priority information corresponding to the at least one piece of data. The priority information is determined based on the target information of the corresponding data and a first mapping relationship. The first mapping relationship is used to indicate the correspondence between the target information and the priority information. The priority information is used to indicate the cache priority of the corresponding data. The data storage client is further configured to delete the target data identifier from the local cache when the cache priority indicated by the target priority information corresponding to the target data identifier is lower than the priority threshold.

[0061] The target information may include one or more of the following: locality attributes, expected number of accesses, and processing operator type. The specific information can be determined based on the actual situation and is not limited here.

[0062] The priority information can be either cache priority or cache priority score.

[0063] In some embodiments of this disclosure, the first mapping relationship can be a mapping table, such as a pre-defined fixed mapping relationship between target information and priority information.

[0064] For example, the target information includes the processing operator type, and different cache priorities can be set according to the processing operator type.

[0065] In some embodiments of this disclosure, the first mapping relationship can also be a mapping function, such as calculating a cache priority score based on a function formula and target information.

[0066] For example, the target information includes locality attributes, expected number of accesses, and processing operator type, and the priority information is the cache priority score. The first mapping relationship is as follows: .

[0067] The LocalityScore is the data locality score, determined based on locality attributes. Locality attributes include PROCESS_LOCAL, NODE_LOCAL, RACK_LOCAL, and ANY. Different locality attributes correspond to different locality scores; for example, PROCESS_LOCAL=1.0, NODE_LOCAL=1.2, RACK_LOCAL=1.5, and ANY=2.0. A higher data locality score indicates a higher cost of retrieving data from a remote location, making caching more appropriate.

[0068] Where Freq_expected is the expected number of accesses based on the DAG. For example, if a dimension table is scanned in multiple tasks, the expected number of accesses is high.

[0069] Wherein, SemanticBoost is the semantic boosting factor determined based on the type of processing operator. Processing operator types can include BROADCAST, shuffle_READ, SCAN, BroadcastHashJoin, shuffleHashJoin, etc. For example, the semantic boosting factor for BROADCAST is 2.0, for shuffle_READ it is 1.5, and for SCAN it is 1.0.

[0070] in, , , These are adjustable weighting coefficients, and .

[0071] Among them, the processing operator type can also be called the operation type.

[0072] The execution plan includes a DAG, and the processing operator type is obtained from parsing the DAG. In this embodiment of the disclosure, through collaborative semantic awareness, the DAG is parsed on the Spark Driver side to extract data semantics and generate CacheHint to guide the underlying cache.

[0073] The priority threshold can be preset or determined according to the actual situation; no limitation is made here.

[0074] For example, if the priority information is cache priority, the priority threshold can be preset; if the priority information is cache priority score, the higher the cache priority score, the higher the cache priority. The priority threshold can be determined in real time based on the cache priority scores of multiple cached data. For example, the priority score of the nth cached data can be used as the priority threshold. The specific threshold can be determined according to the actual situation and is not limited here.

[0075] In some embodiments of this disclosure, low-value data (data with low cache priority) can be deleted periodically, or when the remaining cache in the local cache is less than the cache threshold, low-value data can be deleted.

[0076] The cache threshold can be a preset cache size or determined by multiplying the total local cache storage by a proportional threshold. The cache threshold can be determined based on actual circumstances and is not limited here.

[0077] The ratio threshold can be determined based on the actual situation, and is not limited here.

[0078] In this embodiment of the disclosure, based on the standard ARC (Adaptive Replacement Cache) algorithm, priority information is used instead of simple access counts. When evicting candidate blocks (low-value data), the group of data blocks with the lowest priority indicated by the priority information is selected for eviction, ensuring that high-value data resides for a long time.

[0079] In this embodiment of the disclosure, by adding priority information of the data to be cached to the cache prompt information and marking the data already cached in the local cache with the priority information, low-value data can be deleted periodically in order to improve cache space and cache utilization.

[0080] In some embodiments of this disclosure, the key information also includes the data processing operator type for the corresponding stage, and the data prefetching strategy is determined based on the data type information and the processing operator type.

[0081] The execution plan includes a DAG, and the processing operator type is obtained from parsing the DAG. In this embodiment of the disclosure, through collaborative semantic awareness, the DAG is parsed on the Spark Driver side to extract data semantics and generate CacheHint to guide the underlying cache.

[0082] In this embodiment of the disclosure, determining the data prefetching strategy based on the data type information and the processing operator type can make the data prefetching strategy more accurate.

[0083] In some embodiments of this disclosure, the size of the prefetch window can be fixed or dynamically determined, and is not limited here.

[0084] In some embodiments of this disclosure, the target cache hint information is further used to indicate at least one prefetch aggressive coefficient corresponding to the at least one data identifier, the prefetch aggressive coefficient being determined based on the data type information; the data storage client is further configured to: determine the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and a second mapping relationship, the second mapping relationship being used to indicate the correspondence between the target prefetch aggressive coefficient, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and the current prefetch window size; the data storage client is specifically configured to: prefetch the target data corresponding to the target data identifier from the distributed storage layer based on the target data prefetch strategy and the current prefetch window size.

[0085] There is a mapping relationship between the prefetch aggressiveness coefficient and the data type information, with one data type information corresponding to one prefetch aggressiveness coefficient.

[0086] The second mapping relationship can be determined according to the actual situation, and is not limited here.

[0087] In some embodiments of this disclosure, the current prefetch window size increases as the prefetch hit rate of the previous prefetch window increases. This allows the prefetch window to be appropriately increased when the recent hit rate is high and the window to be shrunk when the recent hit rate is low, thus avoiding resource waste.

[0088] For example, the second mapping relationship can be the following formula: .

[0089] in, This is the base prefetch window size, which can be set according to actual conditions and is not limited here. For example, 4MB.

[0090] in, To prefetch Aggressiveness, Indicates full-speed prefetching. This indicates that prefetching is disabled.

[0091] in, This is the prefetch hit rate of the previous prefetch window, which is the ratio of the number of data blocks that were hit (used for data processing job calculations) in the data blocks prefetched in the previous prefetch window to the total number of data blocks prefetched in the previous prefetch window.

[0092] in, An adjustment factor greater than 0 and less than or equal to 1, which can be a constant value or based on... The value of is a definite piecewise function, which can be determined according to the actual situation and is not limited here.

[0093] In this embodiment of the disclosure, by adding a prefetch aggressive coefficient to the cache prompt information, and then dynamically determining the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the basic prefetch window size, the prefetch hit rate of the previous prefetch window, and the second mapping relationship, adaptive prefetching is achieved by combining the hit rate feedback, which can improve resource utilization.

[0094] In some embodiments of this disclosure, the data storage client is specifically configured to: determine the current prefetch window size based on the target prefetch aggressiveness coefficient corresponding to the target data identifier, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and a third mapping relationship. The third mapping relationship is used to indicate the correspondence between the target prefetch aggressiveness coefficient, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the current prefetch window size.

[0095] In some embodiments of this disclosure, since the average prefetch hit rate increases with the increase of the prefetch hit rate of the previous prefetch window, and the increase in the average prefetch hit rate is less than the increase in the prefetch hit rate of the previous prefetch window, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate increases with the increase of the prefetch hit rate of the previous prefetch window. Consequently, the current prefetch window size increases with the increase of the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate. This allows for appropriately increasing the prefetch window when the recent hit rate is high and shrinking the window when the recent hit rate is low, thus avoiding resource waste.

[0096] For example, for example, the third mapping relationship can be the following formula: .

[0097] in, This is the base prefetch window size, which can be set according to actual conditions and is not limited here. For example, 4MB.

[0098] in, To prefetch Aggressiveness, Indicates full-speed prefetching. This indicates that prefetching is disabled.

[0099] in, This is the prefetch hit rate of the previous prefetch window, which is the ratio of the number of data blocks that were hit (used for data processing job calculations) in the data blocks prefetched in the previous prefetch window to the total number of data blocks prefetched in the previous prefetch window.

[0100] in, The average prefetch hit rate is the average of the prefetch hit rates across multiple prefetch windows.

[0101] In this embodiment of the disclosure, by adding a prefetch aggressive coefficient to the cache prompt information, and then dynamically determining the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the third mapping relationship, adaptive prefetching is achieved by combining the hit rate feedback, which can improve resource utilization.

[0102] In some embodiments of this disclosure, the key information further includes the processing operator type of the data at the corresponding stage; when the processing operator type corresponding to the target data identifier is the shuffle type, the key information further includes first information; the target cache hint information is also used to indicate the shuffle data block location information, which is determined based on the first information; the data storage client is specifically configured to: determine an in-plan prefetch strategy based on the shuffle data block location information; determine a target prefetch strategy according to the target data prefetch strategy and the in-plan prefetch strategy; and prefetch the target data from the distributed storage layer based on the target prefetch strategy.

[0103] The first piece of information includes the Stage's dependencies, shuffle identifier, partitioning rules, etc., which can be determined according to the actual situation and are not limited here.

[0104] The shuffle data block location information (shuffleLocationHints) indicates the location of the shuffled data blocks. ShuffleLocationHints are obtained from the Stage details and data dependencies, based on the operation type being shuffle. For example, shuffleLocationHints are generated based on Stage dependencies, shuffle identifiers, partitioning rules, etc.

[0105] The execution plan includes a Directed Acyclic Graph (DAG), and data dependencies are parsed from the DAG. In this embodiment, through collaborative semantic awareness, the DAG is parsed at the Spark Driver side to extract data semantics and generate CacheHints to guide the underlying caching. Thus, by perceiving the output data layout of the upstream Stage in the Spark DAG, prefetching timeliness and prefetch hit rate are improved, thereby increasing prefetching efficiency.

[0106] For example, if the target data prefetching strategy is a sequential prefetching strategy, then the target prefetching strategy is an in-plan sequential prefetching strategy.

[0107] In this embodiment of the disclosure, by including first information corresponding to shuffle write in the key information, the target cache prompt information is further used to indicate the shuffle data block location information, so that the data storage client can determine the planned prefetch strategy based on the shuffle data block location information, and combine the target data prefetch strategy to determine a more suitable target prefetch strategy, thereby improving prefetch efficiency.

[0108] In some embodiments of this disclosure, the stage identifier is further used to indicate the processing operator type of the data in the corresponding stage; the data storage client is specifically configured to: obtain the target processing operator type corresponding to the target data identifier based on the target stage identifier corresponding to the target data identifier; if the target processing operator type is a first type, determine an in-plan prefetch strategy; determine a first prefetch strategy based on the target data prefetch strategy and the in-plan prefetch strategy; prefetch the target data from the distributed storage layer based on the first prefetch strategy; or, if the target processing operator type is a second type, determine a runtime prefetch strategy; determine a second prefetch strategy based on the target data prefetch strategy and the runtime prefetch strategy; prefetch the target data from the distributed storage layer based on the second prefetch strategy.

[0109] The first type indicates the processing operator type corresponding to the planned prefetch, such as the shuffle type; the second type indicates the processing operator type corresponding to the runtime prefetch, such as the scan type.

[0110] The processing operator type exists within the Spark task process. Before caching data, the processing operator type can be obtained from the Spark task process using the stage identifier (stageId).

[0111] Among them, planned prefetching and runtime prefetching may be full prefetching, sequential prefetching, or random prefetching. The specific type can be determined according to the actual situation and is not limited here.

[0112] In some embodiments of this disclosure, when the target data prefetching strategy has been determined, it can be further determined whether it is planned prefetching or runtime prefetching based on the processing operator type. Combined with the target data prefetching strategy, a more accurate prefetching strategy can be determined, thereby improving prefetching efficiency.

[0113] In some embodiments of this disclosure, the data storage client is specifically configured to: prefetch target data indicated by the target data identifier from the distributed storage layer based on the target data prefetching strategy using a first thread and a first network bandwidth; wherein the first thread belongs to a thread pool outside the thread pool corresponding to the computing task, and the first network bandwidth belongs to a network bandwidth queue outside the network bandwidth queue corresponding to the computing task.

[0114] It is understandable that a first thread pool and a first network bandwidth queue are configured for prefetching tasks, and a second thread pool and a second network bandwidth queue are configured for computation tasks. The first thread pool and the second thread pool are different, and the first network bandwidth queue and the second network bandwidth queue are different. The first thread belongs to the first thread pool, and the first network bandwidth queue belongs to the first network bandwidth queue.

[0115] In some embodiments of this disclosure, the prefetch task uses a separate, configurable-capacity thread pool and a network bandwidth queue (utilizing cgroup or tc).

[0116] In this embodiment of the disclosure, the prefetch task and the compute task use different thread pools and network bandwidth queues, which can ensure that the prefetch task will not compete with the compute task (Spark's compute task) for critical resources and affect the critical path of the task.

[0117] In some embodiments of this disclosure, the data storage client has a metadata cache in its memory; specifically, the data storage client is configured to: when the data indicated by the target data identifier is not data in the metadata cache and is not data in the local cache, prefetch the target data from the distributed storage layer based on the target data identifier and the corresponding target data prefetching strategy.

[0118] Metadata caching includes database tables, partitions, fields, partition storage locations, etc., which can be referenced in relevant technologies and are not limited here.

[0119] In some embodiments of this disclosure, metadata (such as file attributes, block lists, and locations) of files and directories is cached in the memory of the data storage client. In this way, metadata stored in the distributed storage layer is pre-cached in the memory of the data storage client, resulting in metadata caching, which greatly reduces the latency of accessing metadata in the distributed storage layer.

[0120] It is understandable that the data storage client (including the cache manager) needs to first determine whether the data indicated by the target data identifier is data in the local cache or the metadata cache. If it is data in the local cache or the metadata cache, then prefetching is not required. In other words, even if the data compute node issues a prefetching task for the target data indicated by the target data identifier, the cache manager can ignore the prefetching task.

[0121] In some embodiments of this disclosure, the data computing node generates a cache prompt message for all the data required by the stage and sends it to the data storage client. The data storage client first checks whether there is corresponding data in the local cache and metadata cache. If there is no data in the local cache and metadata cache, it prefetches the data from the distributed storage layer.

[0122] In some embodiments of this disclosure, after receiving a cache notification message, the data storage client can first determine whether the data is in the metadata cache or the local cache based on the data type information or data identifier. If it is in either the metadata cache or the local cache, no prefetching is performed. For example, the client can sequentially query whether the data is included in the metadata cache and the local cache. If it is, no prefetching is performed. Alternatively, the client can determine whether the data is a metadata type. If it is, the data is in the metadata cache, and no prefetching is performed. If the data is a table or file type (not a metadata type), the client checks whether the data is in the local cache. If the data is in the local cache, no prefetching is performed. If the data is not in the local cache, prefetching is required.

[0123] In some embodiments of this disclosure, when it is determined that data needs to be prefetched, it can be prefetched directly from the distributed storage layer; alternatively, the metadata cache can be searched first to determine the storage location information of the data in the remote distributed system, and then the data can be prefetched from the distributed storage layer based on the storage location information.

[0124] In this embodiment of the disclosure, metadata stored in the distributed storage layer is pre-cached in the memory of the data storage client to obtain a metadata cache, which can greatly reduce the latency of accessing the metadata of the distributed storage layer. Furthermore, when the data indicated by the target data identifier is neither data in the metadata cache nor data in the local cache, the target data is pre-fetched from the distributed storage layer based on the target data identifier and the corresponding target data pre-fetching strategy. This avoids repeatedly caching already cached data and prevents waste of storage resources.

[0125] In this embodiment of the disclosure, a data processing job includes multiple stages. A CacheHint is generated for each stage. Then, based on a CacheHint, it is determined which subtasks in a stage will be prefetched and which cache eviction policy will be executed.

[0126] In some embodiments of this disclosure, when the key information includes stage data dependencies, the cache hint information is also used to indicate the file set identifier (fileSetId).

[0127] In some embodiments of this disclosure, the key information may also include other stage-related information, which is not limited here.

[0128] In some embodiments of this disclosure, the monitor monitors the issuance of cache prompt information before the task is executed, and performs prefetching and caching.

[0129] For example, a Hint model can be used: key information is encoded into a set of lightweight CacheHint metadata (caching hints), which is sent to the cache manager of the data storage client before the stage begins via a custom SparkListener. The CacheHint data structure is as follows: class CacheHint { / / Current Stage ID String stageId; / / File set identifier (e.g., RDD ID) String fileSetId; / / Cache priority CachePriority priority; / / Prefetch strategy (SEQUENTIAL - sequential prefetch, RANDOM - random prefetch, NONE - all prefetch) Prefetch Strategy; / / Pre-fetching aggressive coefficients double prefetchAggressiveness; / / Shuffle data block location information Map<String, String> shuffleLocationHints;} In some embodiments of this disclosure, consistency management of the metadata cache is supported, namely, a consistency check mechanism between the metadata cache and the metadata of the distributed storage layer. When the source file of the distributed storage layer is modified, the data storage client will ensure that the corresponding metadata cache is invalidated in a timely manner and update the latest metadata to the metadata cache, ensuring that the computing tasks of the data computing nodes will not read outdated data.

[0130] In some embodiments of this disclosure, consistency management of local cache is supported. That is, for local data cache, TTL (Time to Live) or callback-based invalidation mechanism is usually used to ensure strong consistency or eventual consistency, so as to ensure that the computing tasks of data computing nodes will not read outdated data.

[0131] For example, consider a typical Spark SQL query: SELECT * FROM fact_table f JOIN small_dimension_table d ON f.key =d.key.

[0132] 1. Spark Driver's semantic analyzer recognizes that `small_dimension_table` will be used for `BroadcastHashJoin` and generates a `CacheHint` (priority=HIGH, strategy=NONE, α=0) for it. In this case, asynchronous prefetching with shards is not required; a full prefetch can be performed directly.

[0133] 2. After receiving the Hint, each Executor node cache client calculates its priority score P as extremely high when reading the data of the small dimension table, stores it in the cache, and marks it as "resident" to avoid being evicted.

[0134] 3. For large-scale scans of fact_table, generate a CacheHint(priority=MEDIUM, strategy=SEQUENTIAL, α=0.8), and the client starts adaptive prefetching.

[0135] 4. If the job includes a shuffle phase, the Driver will encapsulate the location information of the Map Stage output files in a hint and send it to the corresponding Executor node of the Reduce Stage. These nodes can then prefetch the required shuffle data blocks in parallel before the Task starts.

[0136] This disclosure relates to the fields of big data computing and distributed storage technology, specifically to a method and system that significantly improves the performance of data analysis frameworks such as Apache Spark by integrating a compute-storage separation architecture (such as JuiceFS) through a collaboratively aware intelligent caching and prefetching mechanism. It significantly improves Spark performance, with TPC-DS tests showing an overall performance improvement of approximately 50.4%. By accurately allocating cache through a priority model, it avoids wasting bandwidth through invalid prefetching, making resource utilization more intelligent. As a plugin implementation, it has zero intrusion into Spark applications, allowing users to achieve automatic optimization without modifying their code.

[0137] The core of this disclosure lies in constructing a collaboratively aware intelligent caching and prefetching system involving both the Spark Driver and the cache client. This system enhances the execution of caching and prefetching strategies on the distributed cache client side by embedding a lightweight semantic analyzer on the Spark side, thereby achieving dynamic and precise optimization of caching and prefetching strategies.

[0138] To illustrate this solution in more detail, the following will use an exemplary approach. Figures 3 to 10 To explain, it is understandable that Figures 3 to 10 The steps involved may include more or fewer steps in actual implementation, and the order of these steps may also differ, as long as the data processing method provided in the embodiments of this disclosure can be achieved. A detailed description of the data processing method provided in the embodiments of this disclosure can be found in the relevant description of the above-described data processing system, which can achieve the same or similar technical effects, and will not be repeated here.

[0139] Figure 3 The present invention provides a flowchart of steps for implementing a data processing method according to one or more embodiments of the present disclosure. The data processing method is applied to a data processing system, which includes a data computing node and a data storage client. The data processing method may include the steps S301 to S306 described below.

[0140] S301, the data computing node generates an execution plan based on the data processing job submitted by the user.

[0141] The implementation plan includes multiple phases.

[0142] S302, The data computing node parses the execution plan and generates multiple key pieces of information corresponding to each of the multiple stages.

[0143] This key information includes the stage identifier and data type information for the corresponding stage.

[0144] S303, The data computing node generates target cache prompt information based on the target key information corresponding to the target stage.

[0145] The target cache hint information is used to indicate at least one data prefetching strategy corresponding to at least one data identifier included in the target stage. The data prefetching strategy is determined based on the data type information, and the target stage is one of the multiple stages.

[0146] S304. The data computing node sends the target cache prompt information to the data storage client.

[0147] S305. The data storage client prefetches the target data indicated by the target data identifier from the distributed storage layer based on the target data prefetching strategy corresponding to the target data identifier.

[0148] S306. The data storage client stores the target data in its local cache so that when the data computing node executes the target computing task corresponding to the target data, it reads the target data from the local cache and executes the computing task based on the target data.

[0149] In some embodiments of this disclosure, the key information further includes target information for the data at the corresponding stage. This target information includes at least one of the following: locality attribute, expected access count, and processing operator type. The target cache hint information is also used to indicate at least one priority information corresponding to each of the at least one data item. This priority information is determined based on the target information of the corresponding data and a first mapping relationship. The first mapping relationship indicates the correspondence between the target information and the priority information, and the priority information indicates the cache priority of the corresponding data. Figure 3 ,like Figure 4 As shown, after S306 above, the data processing method provided in this embodiment may further include S307 as described below.

[0150] S307. If the cache priority indicated by the target priority information corresponding to the target data identifier is lower than the priority threshold, the data storage client shall delete the target data identifier from the local cache.

[0151] In some embodiments of this disclosure, the key information also includes the data processing operator type for the corresponding stage, and the data prefetching strategy is determined based on the data type information and the processing operator type.

[0152] In some embodiments of this disclosure, the target cache hint information is further used to indicate at least one prefetch aggressive coefficient corresponding to each of the at least one data identifier, the prefetch aggressive coefficient being determined based on the data type information; combined with Figure 4 ,like Figure 5 As shown, prior to S305 above, the data processing method provided in this embodiment may further include S308 below, and S305 above can be specifically implemented by S305a below.

[0153] S308. The data storage client determines the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the basic prefetch window size, the prefetch hit rate of the previous prefetch window, and the second mapping relationship.

[0154] The second mapping relationship is used to indicate the correspondence between the target prefetch aggressiveness coefficient, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and the current prefetch window size.

[0155] S305a: The data storage client prefetches the target data corresponding to the target data identifier from the distributed storage layer based on the target data prefetching strategy and the current prefetch window size.

[0156] In some embodiments of this disclosure, combined with Figure 5 ,like Figure 6 As shown, the above S308 can be specifically implemented through the following S308a.

[0157] S308a The data storage client determines the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the third mapping relationship.

[0158] The third mapping relationship is used to indicate the correspondence between the target prefetch aggressiveness coefficient, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the current prefetch window size.

[0159] In some embodiments of this disclosure, the key information further includes the processing operator type of the data at the corresponding stage; when the processing operator type corresponding to the target data identifier is shuffle type, the key information further includes first information; the target cache hint information is also used to indicate shuffle data block location information, which is determined based on the first information; combined with Figure 4 ,like Figure 7 As shown, the above S305 can be specifically implemented through the following S305b to S305d.

[0160] S305b: The data storage client determines the planned prefetch strategy based on the location information of the shuffle data block.

[0161] S305c The data storage client determines the target prefetching strategy based on the target data prefetching strategy and the planned prefetching strategy.

[0162] S305d, the data storage client, prefetches the target data from the distributed storage layer based on the target prefetch strategy.

[0163] In some embodiments of this disclosure, the stage identifier is also used to indicate the type of data processing operator for the corresponding stage; combined with Figure 4 ,like Figure 8 As shown, the above S305 can be specifically implemented through the following S401 to S407.

[0164] S401. The data storage client obtains the target processing operator type corresponding to the target data identifier based on the target stage identifier corresponding to the target data identifier.

[0165] S402. When the target processing operator type is the first type, the data storage client determines the planned prefetching strategy.

[0166] S403. The data storage client determines the first prefetching strategy based on the target data prefetching strategy and the planned prefetching strategy.

[0167] S404. The data storage client prefetches the target data from the distributed storage layer based on the first prefetch strategy.

[0168] S405. When the target processing operator type is the second type, the data storage client determines the runtime prefetching strategy.

[0169] S406. The data storage client determines the second prefetching strategy based on the target data prefetching strategy and the runtime prefetching strategy.

[0170] S407. The data storage client prefetches the target data from the distributed storage layer based on the second prefetch strategy.

[0171] In some embodiments of this disclosure, combined with Figure 4 ,like Figure 9 As shown, the above S305 can be specifically implemented through the following S305e.

[0172] S305e, the data storage client, through the first thread and the first network bandwidth, prefetches the target data indicated by the target data identifier from the distributed storage layer based on the target data prefetching strategy.

[0173] Among them, the first thread belongs to a thread pool outside the thread pool corresponding to the computing task, and the first network bandwidth belongs to a network bandwidth queue outside the network bandwidth queue corresponding to the computing task.

[0174] In some embodiments of this disclosure, the data storage client caches metadata in its memory; combined with Figure 4 ,like Figure 10 As shown, the above S305 can be implemented by the following S305f.

[0175] S305f: If the data indicated by the target data identifier is not data in the metadata cache and is not data in the local cache, the data storage client prefetches the target data from the distributed storage layer based on the target data identifier and the corresponding target data prefetching strategy.

[0176] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data processing method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0177] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0178] The present invention provides a computer program product, comprising: when the computer program product is run on a computer, causing the computer to implement the above-described data processing method.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

[0180] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A data processing system, characterized in that, include: The data computing node is configured to generate an execution plan based on a user-submitted data processing job, the execution plan comprising multiple stages. The execution plan is parsed to generate multiple key pieces of information corresponding to the multiple stages, including the stage identifier and data type information of the corresponding stage; Target cache prompt information is generated based on the target key information corresponding to the target stage. The target cache prompt information is used to indicate at least one data prefetching strategy corresponding to at least one data identifier included in the target stage. The data prefetching strategy is determined based on the data type information. The target stage is one of the multiple stages. Send the target cache prompt information to the data storage client; The data storage client is configured to: prefetch target data corresponding to the target data identifier from the distributed storage layer based on the target data prefetching strategy corresponding to the target data identifier; The target data is stored in the local cache of the data storage client, so that when the data computing node executes the target computing task corresponding to the target data, it reads the target data from the local cache and executes the computing task based on the target data.

2. The data processing system according to claim 1, characterized in that, The key information also includes target information for the data at the corresponding stage. The target information includes at least one of the following: locality attribute, expected number of accesses, and processing operator type. The target cache hint information is also used to indicate at least one priority information corresponding to the at least one piece of data. The priority information is determined based on the target information of the corresponding data and a first mapping relationship. The first mapping relationship is used to indicate the correspondence between the target information and the priority information. The priority information is used to indicate the cache priority of the corresponding data. The data storage client is further configured to delete the target data from the local cache if the cache priority indicated by the target priority information corresponding to the target data identifier is lower than the priority threshold.

3. The data processing system according to claim 1, characterized in that, The key information also includes the data processing operator type for the corresponding stage, and the data prefetching strategy is determined based on the data type information and the processing operator type.

4. The data processing system according to claim 1, characterized in that, The target cache hint information is also used to indicate at least one prefetch aggressive coefficient corresponding to each of the at least one data identifier, wherein the prefetch aggressive coefficient is determined based on the data type information; The data storage client is further configured to: determine the current prefetch window size based on the target prefetch aggressive coefficient corresponding to the target data identifier, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and a second mapping relationship, wherein the second mapping relationship is used to indicate the correspondence between the target prefetch aggressive coefficient, the base prefetch window size, the prefetch hit rate of the previous prefetch window, and the current prefetch window size; The data storage client is specifically configured to: prefetch the target data corresponding to the target data identifier from the distributed storage layer based on the target data prefetching strategy and the current prefetching window size.

5. The data processing system according to claim 4, characterized in that, The data storage client is specifically configured as follows: Based on the target prefetch aggressiveness coefficient corresponding to the target data identifier, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and a third mapping relationship, the current prefetch window size is determined. The third mapping relationship is used to indicate the correspondence between the target prefetch aggressiveness coefficient, the base prefetch window size, the ratio of the prefetch hit rate of the previous prefetch window to the average prefetch hit rate of multiple prefetch windows, and the current prefetch window size.

6. The data processing system according to claim 1, characterized in that, The key information also includes the processing operator type of the data at the corresponding stage; when the processing operator type corresponding to the target data identifier is the shuffle type, the key information also includes first information; the target cache prompt information is also used to indicate the shuffle data block location information, and the shuffle data block location information is determined based on the first information; The data storage client is specifically configured to: determine the planned prefetch strategy based on the shuffle data block location information; Based on the target data prefetching strategy and the planned prefetching strategy, determine the target prefetching strategy; Based on the target prefetching strategy, the target data is prefetched from the distributed storage layer.

7. The data processing system according to claim 1, characterized in that, The stage identifier is also used to indicate the type of data processing operator for the corresponding stage; The data storage client is specifically configured to: obtain the target processing operator type corresponding to the target data identifier based on the target stage identifier corresponding to the target data identifier; When the target processing operator type is the first type, a prefetching strategy is determined. Based on the target data prefetching strategy and the planned prefetching strategy, a first prefetching strategy is determined; Based on the first prefetching strategy, the target data is prefetched from the distributed storage layer; or, When the target processing operator type is the second type, a runtime prefetching strategy is determined; Based on the target data prefetching strategy and the runtime prefetching strategy, a second prefetching strategy is determined; Based on the second prefetch strategy, the target data is prefetched from the distributed storage layer.

8. The data processing system according to claim 1, characterized in that, The data storage client is specifically configured as follows: Using the first thread and the first network bandwidth, the target data is prefetched from the distributed storage layer based on the target data prefetching strategy; Wherein, the first thread belongs to a thread pool outside the thread pool corresponding to the computing task, and the first network bandwidth belongs to a network bandwidth queue outside the network bandwidth queue corresponding to the computing task.

9. The data processing system according to claim 1, characterized in that, The data storage client has a metadata cache in its memory; The data storage client is specifically configured to: when the data indicated by the target data identifier is not data in the metadata cache and is not data in the local cache, prefetch the target data from the distributed storage layer based on the target data prefetching strategy corresponding to the target data identifier.

10. A data processing method, characterized in that, This is applied to a data processing system, which includes data computing nodes and data storage clients, including: The data computing node generates an execution plan based on the data processing job submitted by the user, and the execution plan includes multiple stages. The execution plan is parsed to generate multiple key pieces of information corresponding to the multiple stages, including the stage identifier and data type information of the corresponding stage; Target cache prompt information is generated based on the target key information corresponding to the target stage. The target cache prompt information is used to indicate at least one data prefetching strategy corresponding to at least one data identifier included in the target stage. The data prefetching strategy is determined based on the data type information. The target stage is one of the multiple stages. Send the target cache prompt information to the data storage client; The data storage client prefetches the target data from the distributed storage layer based on the target data identifier and the corresponding target data prefetching strategy. The target data is stored in the local cache of the data storage client, so that when the data computing node executes the target computing task corresponding to the target data, it reads the target data from the local cache and executes the computing task based on the target data.

Images