Method, related apparatus, device, and storage medium for processing shuffle data

Abstract

The application discloses a shuffle data processing method, which can be applied in the fields of big data, cloud computing, cloud storage and the like. The application comprises the following steps: sending a first registration request to a scheduling node, so that the scheduling node allocates a first service node for a target write task and generates first route matching information; receiving the first route matching information sent by the scheduling node; sending a first space allocation request to the first service node according to the first route matching information; receiving a first space allocation response sent by the first service node; and sending shuffle data to the first service node if the first space allocation response indicates that the space allocation is successful, so that the first service node writes the shuffle data into a storage space. The application further provides a device, equipment and storage medium. In the application, the failure of a shuffle process does not affect the stored shuffle data, thereby avoiding repeated generation of data. The shuffle data is stored in an external device, which is beneficial to improving the data storage capacity.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a method, related apparatus, equipment, and storage medium for processing mixed data. Background Technology

[0002] With the increasing scale and diversity of data, data analysis and processing technologies are constantly improving. Currently, various data analysis and processing architectures have emerged, such as the Hadoop / MapReduce (distributed computing / mapping and merging) architecture and Spark, a big data processing framework based on in-memory computing. In the Spark system, the data shuffle process is used to connect the upstream mapping (Map) tasks with the downstream reducing (Reduce) tasks.

[0003] Existing Spark data shuffle solutions mainly include hash-based shuffle and sort-based shuffle. Among them, sort-based shuffle writes the output data of each Map task to a file and generates an index file to record partition information, which can reduce the number of intermediate files compared to hash-based shuffle.

[0004] However, Spark's computational tasks employ a computation-storage coupled architecture, requiring shuffle data to be stored on executors. The disk storage capacity of executors is typically limited, thus restricting the amount of shuffle data that can be stored. Furthermore, if an executor fails, the data shuffle process will fail, leading to task retries and data regeneration. This not only wastes computational resources but also increases task runtime and reduces data processing efficiency. Summary of the Invention

[0005] This application provides a method, related apparatus, device, and storage medium for processing shuffled data. On one hand, the failure of the shuffle process does not affect the already stored shuffle data, thereby avoiding duplicate data generation and improving resource utilization. On the other hand, it breaks the limitation of executors storing shuffle data, allowing shuffle data to be stored on external devices, which helps to increase data storage capacity.

[0006] In view of this, this application provides a method for processing mixed data, including:

[0007] A first registration request is sent to the scheduling node so that the scheduling node allocates a first service node for the target write task and generates first routing matching information. The first registration request carries the task identifier of the target write task, and the first routing matching information includes the correspondence between the task identifier of the target write task and the first node identifier. The first node identifier is used to indicate the first service node.

[0008] Receive the first route matching information sent by the scheduling node;

[0009] According to the first route matching information, a first space allocation request is sent to the first service node. The first space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The first space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0010] Receive a first space allocation response sent by a first service node, wherein the first space allocation response is used to indicate the space allocation result;

[0011] If the first space allocation response indicates that the space allocation was successful, then shuffle data is sent to the first service node so that the first service node writes the shuffle data to the storage space.

[0012] This application also provides a method for processing mixed data, including:

[0013] The receiving driver node sends a space allocation request based on the routing matching information. The space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The routing matching information is generated by the scheduling node after allocating a first service node to the target writing task based on the registration request. The registration request carries the task identifier of the target writing task. The routing matching information includes the correspondence between the task identifier of the target writing task and the node identifier. The node identifier is used to indicate the first service node.

[0014] Allocate storage space for shuffle data based on the space allocation request;

[0015] Send a space allocation response to the driver node, so that the driver node sends a space allocation response to the execution node, wherein the space allocation response is used to indicate the space allocation result;

[0016] If the space allocation response indicates that the space allocation was successful, the shuffle data sent by the execution node is received and written to the storage space.

[0017] Another aspect of this application provides a mixed data processing apparatus, comprising:

[0018] The sending module is used to send a first registration request to the scheduling node so that the scheduling node allocates a first service node for the target write task and generates first routing matching information. The first registration request carries the task identifier of the target write task, and the first routing matching information includes the correspondence between the task identifier of the target write task and the first node identifier. The first node identifier is used to indicate the first service node.

[0019] The receiving module is used to receive the first route matching information sent by the scheduling node;

[0020] The sending module is also used to send a first space allocation request to the first service node according to the first routing matching information. The first space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The first space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0021] The receiving module is also configured to receive a first space allocation response sent by the first service node, wherein the first space allocation response is used to indicate the space allocation result;

[0022] The write module is used to send shuffle data to the first service node if the first space allocation response indicates that the space allocation was successful, so that the first service node writes the shuffle data to the storage space.

[0023] In one possible design, in another implementation of another aspect of the embodiments of this application, the mixed data processing apparatus further includes a waiting module;

[0024] The waiting module is used to enter a waiting state after receiving the first space allocation response sent by the first service node if the first space allocation response indicates that the space allocation has failed.

[0025] The sending module is also used to send a second space allocation request to the first service node according to the first routing matching information if the waiting time is greater than or equal to the time threshold. The second space allocation request carries the task identifier and the amount of shuffle data. The second space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0026] The receiving module is also used to receive the second space allocation response sent by the first service node;

[0027] The sending module is further configured to send shuffle data to the first service node if the second space allocation response indicates that the space allocation is successful, so that the first service node writes the shuffle data to the storage space, wherein the second space allocation response is used to indicate the space allocation result;

[0028] The waiting module is also used to re-enter the waiting state if the second space allocation response indicates that space allocation has failed.

[0029] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0030] The sending module is further configured to, after receiving the first space allocation response sent by the first service node, if the first space allocation response indicates that space allocation has failed, send a second registration request to the scheduling node so that the scheduling node allocates a second service node for the target writing task according to the second registration request and generates second routing matching information. The second registration request carries the task identifier of the target writing task, and the second routing matching information includes the correspondence between the task identifier of the target writing task and the second node identifier. The second node identifier is used to indicate the second service node.

[0031] The receiving module is also used to receive the second route matching information sent by the scheduling node;

[0032] The sending module is also used to send a second space allocation request to the second service node according to the second routing matching information. The second space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The second space allocation request is used to request the second service node to allocate storage space for the shuffle data.

[0033] The receiving module is also used to receive a second space allocation response sent by the second service node, wherein the second space allocation response is used to indicate the space allocation result;

[0034] The sending module is also used to send shuffle data to the second service node if the second space allocation response indicates that the space allocation was successful, so that the second service node can store the shuffle data.

[0035] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0036] The sending module is specifically used to write shuffle data into the buffer in the execution node in the form of data blocks;

[0037] If the data storage volume in the buffer reaches the storage threshold, then the event data is added to the data queue corresponding to the execution node;

[0038] Based on the event data, the sending thread in the execution node is invoked to send the data block in the buffer to the first service node, so that the first service node writes the data block to the storage space.

[0039] In one possible design, in another implementation of another aspect of the embodiments of this application, the mixed data processing apparatus further includes an acquisition module;

[0040] The acquisition module is used to acquire a set of task identifiers after sending shuffle data to the first service node so that the first service node writes the shuffle data to the storage space. The set of task identifiers includes at least one written task identifier, and each written task identifier is used to indicate a write task that has been written to the storage space.

[0041] The acquisition module is also used to acquire a set of data block identifiers for the target reading task from the first service node based on the task identifier set, wherein the set of data block identifiers includes at least one data block identifier, and each data block identifier indicates a data block;

[0042] The acquisition module is also used to acquire an index file, wherein the index file includes a mapping relationship between data block identifiers and data block read addresses;

[0043] The acquisition module is also used to read the target shuffle data according to the index file if a set of data block identifiers exists in the index file, wherein the target shuffle data includes at least one data block.

[0044] In one possible design, in another implementation of another aspect of the embodiments of this application, the mixed data processing device further includes a verification module;

[0045] The acquisition module is also used to read the target shuffle data from the index file and, for each data block in the target shuffle data, obtain the first verification value corresponding to each data block from the index file based on the data block identifier corresponding to each data block in the target shuffle data.

[0046] The verification module is used to perform verification calculations on each data block in the target shuffle data to obtain the second verification value corresponding to each data block.

[0047] The verification module is also used to compare the first verification value and the second verification value of each data block;

[0048] The verification module is also used to determine that the data block has been successfully verified if the first verification value and the second verification value corresponding to the data block are consistent.

[0049] The verification module is also used to determine that the data block verification has failed if the first verification value and the second verification value corresponding to the data block are inconsistent.

[0050] In one possible design, in another implementation of another aspect of the embodiments of this application, the shuffling data processing device further includes a stop module;

[0051] The sending module is also used to periodically send a first heartbeat message to the scheduling node during the lifecycle of the target application, so that the scheduling node retains the data associated with the target application. The first heartbeat message carries application information of the target application, including a timestamp and an application identifier. The target application includes at least one of a write task or a read task.

[0052] The stop module is used to stop sending the first heartbeat information to the scheduling node if the lifecycle of the target application has ended, so that the scheduling node can delete the data associated with the target application.

[0053] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0054] The sending module is also configured to send a first timestamp to the first service node when a communication connection is established with the first service node, so that the first service node updates the application information of the target application according to the first timestamp, and determines the survival status of the target application according to the first timestamp and the second timestamp, wherein the second timestamp is derived from the application information of the target application, and the target application includes at least one of a write task or a read task.

[0055] Another aspect of this application provides a mixed data processing apparatus, comprising:

[0056] The receiving module is used to receive the space allocation request sent by the driver node according to the routing matching information. The space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The routing matching information is generated by the scheduling node after allocating the first service node to the target writing task according to the registration request. The registration request carries the task identifier of the target writing task. The routing matching information includes the correspondence between the task identifier of the target writing task and the node identifier. The node identifier is used to indicate the first service node.

[0057] The allocation module is used to allocate storage space for shuffle data based on space allocation requests.

[0058] The sending module is used to send a space allocation response to the driver node, so that the driver node sends a space allocation response to the execution node, wherein the space allocation response is used to indicate the space allocation result;

[0059] The write module is used to receive shuffle data sent by the execution node and write the shuffle data to the storage space if the space allocation response indicates that the space allocation was successful.

[0060] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0061] The write module is specifically used to receive memory allocation requests sent by the execution node;

[0062] Allocate a buffer in the first service node for the execution node according to the memory allocation request;

[0063] Write the shuffle data sent by the execution node in the form of data blocks to the buffer;

[0064] If the data storage volume in the buffer reaches the storage threshold, then the event data is added to the data queue corresponding to the first service node.

[0065] Based on the event data, the write thread in the first service node is invoked to write the data blocks in the buffer to the storage space.

[0066] In one possible design, in another implementation of another aspect of the embodiments of this application, the execution node includes a first execution node and a second execution node;

[0067] The write module is specifically used to send a storage success message to the first execution node and release the space occupied by the shuffle data sent by the second execution node if the shuffle data sent by the first execution node has been written to the storage space and the shuffle data sent by the second execution node has not been written.

[0068] If the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, then a storage success message is sent to the second execution node, and the space occupied by the shuffle data sent by the first execution node is released.

[0069] In one possible design, in another implementation of another aspect of the embodiments of this application, the shuffling data processing apparatus further includes a deletion module;

[0070] The sending module is also used to periodically send heartbeat information to the scheduling node. The heartbeat information carries the node identifier, or the heartbeat information carries the node identifier and the memory usage of the first service node. The memory usage is used by the scheduling node to allocate service nodes for write tasks, and the heartbeat information is used to request the list of online applications.

[0071] The receiving module is also used to receive a list of online applications sent by the scheduling node, wherein the list of online applications includes the application identifiers corresponding to the applications within their lifecycle;

[0072] The deletion module is used to delete data associated with the target application if the application identifier corresponding to the target application is not included in the online application list. The target application includes at least one of a write task or a read task.

[0073] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0074] The memory is used to store programs;

[0075] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0076] Bus systems are used to connect memory and processor to enable communication between them.

[0077] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0078] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0079] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0080] This application provides a method for processing shuffled data. First, the driver node sends a registration request to the scheduling node. The scheduling node then assigns a service node to the target writing task and generates routing matching information. After receiving the routing matching information from the scheduling node, the driver node sends a space allocation request to the service node based on the routing matching information. If the service node's response indicates successful space allocation, the driver node sends shuffle data to the service node, which then writes the shuffle data to the storage space. Through this method, based on the service node external to Spark provided by the remote shuffle service, the scheduling node can request storage space for shuffle data from the service node. This storage space is used to store the shuffle data generated during the shuffle process, achieving separation of computation and storage. Therefore, on the one hand, the failure of the shuffle process will not affect the already stored shuffle data, thus avoiding duplicate data generation and improving resource utilization. On the other hand, it breaks the limitation of executors storing shuffle data, storing shuffle data on external devices, which is beneficial for increasing data storage capacity. Furthermore, it reduces the number of network requests accessing the executor, lowering the probability of shuffle errors. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the architecture of the mixed data processing system in the embodiments of this application;

[0082] Figure 2 This is a schematic diagram illustrating the component relationships during the application execution process in this embodiment of the application;

[0083] Figure 3 This is a flowchart illustrating the data shuffling process in an embodiment of this application.

[0084] Figure 4 This is a schematic diagram of the execution flow of the mixed data processing system in an embodiment of this application;

[0085] Figure 5 This is a schematic diagram of the mixed washing and writing process in an embodiment of this application;

[0086] Figure 6 This is a schematic diagram of the shuffling and reading process in an embodiment of this application;

[0087] Figure 7 This is a schematic diagram of the data verification mechanism in an embodiment of this application;

[0088] Figure 8 This is a schematic diagram of the heartbeat mechanism in an embodiment of this application;

[0089] Figure 9 This is another schematic diagram of the heartbeat mechanism in the embodiments of this application;

[0090] Figure 10 This is another flowchart illustrating the data shuffling process in the embodiments of this application;

[0091] Figure 11 This is another schematic diagram of the mixed washing and writing process in the embodiments of this application;

[0092] Figure 12 This is another schematic diagram of the heartbeat mechanism in the embodiments of this application;

[0093] Figure 13 This is a schematic diagram of a data shuffling processing apparatus in an embodiment of this application;

[0094] Figure 14 This is another schematic diagram of the data shuffling processing apparatus in the embodiments of this application;

[0095] Figure 15 This is a schematic diagram of the server structure in an embodiment of this application;

[0096] Figure 16 This is a schematic diagram of the structure of a terminal device in an embodiment of this application. Detailed Implementation

[0097] This application provides a method, related apparatus, device, and storage medium for processing shuffled data. On one hand, the failure of the shuffle process does not affect the already stored shuffle data, thereby avoiding duplicate data generation and improving resource utilization. On the other hand, it breaks the limitation of executors storing shuffle data, allowing shuffle data to be stored on external devices, which helps to increase data storage capacity.

[0098] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0099] With the development of computing technology, some applications require enormous computing power to complete, which would take a considerable amount of time to finish using centralized computing. To improve data processing efficiency, distributed computing has emerged. Distributed computing breaks down applications into many smaller parts and distributes them to multiple computers for processing. This saves overall computing time and significantly improves computing efficiency. For example, Spark, a distributed computing engine, can handle large datasets and machine learning tasks.

[0100] Cloud computing falls under the category of distributed computing, specifically a form of distributed computing geared towards providing external services. Cloud computing refers to the delivery and usage model of information technology (IT) infrastructure, meaning obtaining necessary resources through a network in an on-demand and easily scalable manner. In a broader sense, cloud computing refers to the delivery and usage model of services, meaning obtaining necessary services through a network in an on-demand and easily scalable manner. These services can be IT and software-related, internet-related, or other services. Cloud computing is a product of the convergence and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0101] With the development of the internet, real-time data streams, and the diversification of connected devices, as well as the demands for search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel and distributed computing, cloud computing will fundamentally revolutionize the entire internet model and enterprise management model.

[0102] In Spark systems, the data shuffle process serves as a bridge connecting upstream Map tasks and downstream Reduce tasks. To address the issue of data regeneration due to shuffle failures, which impacts fault tolerance and performance, this application provides a data shuffling method. This method treats the Spark shuffle operation as an external service, caching the data generated during the shuffle process on a service node before flushing it to a remote storage system (e.g., cloud storage). For further understanding, please refer to [link to relevant documentation]. Figure 1 . Figure 1This is a schematic diagram of the architecture of the shuffle data processing system in this application embodiment. As shown in the figure, the shuffle client is deployed on a terminal device and can run on the terminal device through a browser or as a standalone application. The specific form of the client is not limited here. The shuffle server node shown in the figure can be deployed on a server. The server involved in this application can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, PDA, personal computer, smart TV, smartwatch, in-vehicle device, wearable device, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, which is not limited here. The number of servers and terminal devices is also not limited.

[0103] based on Figure 1 The mixed data processing system shown below will be described in detail below.

[0104] The coordinator, typically deployed on a server and employing a High Availability (HA) architecture, is the centralized node in the remote shuffle service (RSS) system. Its primary function is to manage the shuffle service nodes, including but not limited to registration, heartbeats, metadata management, and data block management. The coordinator's HA implementation can be based on the Atomic Broadcast Protocol (Raft) or a distributed coordination service (ZooKeeper).

[0105] Shuffle client: This is divided into shuffle write and shuffle read. Shuffle write writes data to the shuffle server. Shuffle read reads data from persistent storage, which stores data indefinitely unless manually deleted.

[0106] Shuffle server: Its main job is to collect data sent from upstream and downstream tasks, aggregate the data, and write it to storage.

[0107] Storage: This can be RSS storage. Specifically, storage can be the disk of the shuffle server, or the Hadoop Distributed File System (HDFS), or cloud object storage (COS), or a distributed cloud storage system such as Ceph, used to store shuffle data.

[0108] Before introducing the embodiments of this application, for ease of understanding, the relevant terminology involved in this application will be explained below. Please refer to... Figure 2 , Figure 2 This is a schematic diagram illustrating the component relationships during application execution in this embodiment of the application. As shown in the figure, the concepts of each component during Spark application execution are introduced below.

[0109] (1) Task: A partition in a Resilient Distributed Dataset (RDD) corresponds to a task. A task is the smallest unit of processing on a single partition.

[0110] (2) Task set: A collection of related tasks that do not have shuffle dependencies on each other.

[0111] (3) Scheduling stage: Represents the stage corresponding to a taskset. Each job is divided into many stages according to the wide dependency relationship of the RDD. Each stage contains a taskset.

[0112] (4) job: A computational job consisting of one or more stages, generated by an action operator.

[0113] (5) Application: refers to a Spark application written by the user, consisting of one or more jobs. After being submitted to Spark, Spark allocates resources to the application, transforms the program and executes it.

[0114] (6) Shuffle write: The end that provides shuffle data is called the Map end, and the shuffle process of the Map end is called shuffle write.

[0115] (7) Shuffle read: The end that receives shuffle data is called the reduce end, and the shuffle process at the reduce end is called shuffle read.

[0116] (8) Driver: The application is the program that publishes the application and can assign tasks to various execution nodes (executors) for execution.

[0117] (9) executor: is the computing resource that executes tasks and is used to assist in the actual computation of various operators in Spark.

[0118] Based on the above introduction, the following section will describe the data shuffling processing method in this application from the perspective of the driver and executor. Please refer to [link / reference needed]. Figure 3 One embodiment of the mixed data shuffling processing method in this application includes:

[0119] 110. Send a first registration request to the scheduling node so that the scheduling node allocates a first service node for the target write task and generates first routing matching information. The first registration request carries the task identifier of the target write task, and the first routing matching information includes the correspondence between the task identifier of the target write task and the first node identifier. The first node identifier is used to indicate the first service node.

[0120] In one or more embodiments, the data shuffling processing device sends a first registration request to the coordinator, the first registration request carrying the task identifier (identity document, ID) of the target write task. It should be noted that the data shuffling processing device can be deployed on a server or terminal device, without limitation here. The data shuffling processing device includes a driver and an executor.

[0121] Specifically, the driver sends a first registration request to the coordinator. Based on the taskID carried in the first registration request, the coordinator allocates a first service node to the target write task. After allocation, the coordinator generates the first route matching information. For easier understanding, please refer to Table 1, which provides an illustration of the first route matching information.

[0122] Table 1

[0123] taskID First node ID 10000 001

[0124] As can be seen, the first route matching information includes the correspondence between the target write task's taskID and the first node ID. Based on this, assuming the target write task's taskID is "10000", then the first node ID corresponding to the first service node is "001". It can be understood that the "service node" in this application is the shuffle server described above.

[0125] Understandably, the coordinator's allocation strategy includes, but is not limited to, periodically obtaining the memory usage of different shuffle servers and prioritizing the shuffle server with lower memory usage as the first shuffle server. Alternatively, the coordinator may randomly assign a shuffle server as the first shuffle server. Furthermore, the coordinator can send first registration requests to multiple first shuffle servers to achieve redundant configuration, thereby improving the reliability of data processing.

[0126] 120. Receive the first route matching information sent by the scheduling node;

[0127] In one or more embodiments, the driver receives first route matching information sent by the coordinator.

[0128] 130. Based on the first routing matching information, send a first space allocation request to the first service node, wherein the first space allocation request carries the task identifier of the target writing task and the amount of shuffle data, and the first space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0129] In one or more embodiments, the driver can determine that it has completed registration with the first shuffle server based on the first routing matching information. Based on this, the driver sends a first space allocation request to the first shuffle server. The first shuffle server, according to the first space allocation request, attempts to allocate storage space for the shuffle data of the target write task. The first space allocation request carries the taskID of the target write task and the amount of shuffle data. The first shuffle server can determine whether it can allocate the corresponding storage space based on the amount of shuffle data.

[0130] For example, the first shuffle server can allocate local storage space; for instance, shuffle data can be stored on the first shuffle server's disk. Alternatively, the first shuffle server can request space allocation from a distributed cloud storage system such as HDFS, COS, or Ceph; if a match is found, the first shuffle server will receive a successful allocation response.

[0131] 140. Receive a first space allocation response sent by a first service node, wherein the first space allocation response is used to indicate the space allocation result;

[0132] In one or more embodiments, the driver receives a first space allocation response sent by a first shuffle server, wherein the first space allocation response may carry an identifier for indicating the space allocation result, for example, an identifier "1" indicates successful allocation and an identifier "0" indicates allocation failure.

[0133] 150. If the first space allocation response indicates that the space allocation was successful, then shuffle data is sent to the first service node so that the first service node writes the shuffle data to the storage space.

[0134] In one or more embodiments, if the first space allocation response indicates that the first shuffle server has completed the allocation of storage space, then the target write task in the executor sends the shuffle data to the first shuffle server indicated by the first node identifier based on the first routing matching information, and the shuffle server writes the shuffle data to the pre-allocated storage space.

[0135] For easier understanding, please refer to Figure 4 , Figure 4This is a schematic diagram of the execution flow of the shuffle data processing system in this application embodiment. As shown in the figure, specifically, in step A1, the driver sends a registration request to the coordinator, thereby the coordinator sends back the routing matching information of writing shuffle data to one or more shuffle servers to the driver. In step A2, the driver sends a space allocation request to the shuffle server according to the routing matching information, and the shuffle server sends back a space allocation response. In step A3, if the space allocation response indicates that the space allocation is successful, the target write task in the executor sends shuffle data to the shuffle server according to the routing matching information, specifically transmitting data in the form of blocks. In step A4, the shuffle server writes the shuffle data to storage. In step A5, after the target write task finishes, the executor can report a successful transmission notification message to the driver. In step A6, the target read task in the executor obtains the successful write task information (e.g., taskID) from the driver. Based on this, in step A7, the target read task in the executor obtains the blockID from the shuffle server, and in step A8, the target read task in the executor reads the corresponding shuffle data from the storage based on the blockID.

[0136] This application provides a method for processing shuffled data. Using this method, based on an external Spark service node provided by a remote shuffle service, the scheduling node can request storage space for shuffle data from the service node. This storage space is used to store the shuffle data generated during the shuffle process, achieving separation of computation and storage. Therefore, on the one hand, the failure of the shuffle process will not affect the already stored shuffle data, thereby avoiding duplicate data generation and improving resource utilization. On the other hand, it breaks the limitation of executors storing shuffle data, storing shuffle data on external devices, which is beneficial to increasing data storage capacity. Furthermore, it reduces the number of network requests accessing the executor, lowering the probability of shuffle errors.

[0137] Optionally, in the above Figure 3Based on the corresponding embodiments, in another optional embodiment provided by this application, after receiving the first space allocation response sent by the first service node, it may further include:

[0138] If the first space allocation response indicates that space allocation has failed, then enter a waiting state;

[0139] If the waiting time is greater than or equal to the time threshold, a second space allocation request is sent to the first service node according to the first routing matching information. The second space allocation request carries the task identifier and the amount of shuffle data. The second space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0140] Receive the second space allocation response sent by the first service node;

[0141] If the second space allocation response indicates that the space allocation was successful, then shuffle data is sent to the first service node so that the first service node writes the shuffle data to the storage space. The second space allocation response is used to indicate the space allocation result.

[0142] If the second space allocation response indicates that space allocation has failed, then the system will enter the waiting state again.

[0143] In one or more embodiments, a method for handling write failures is described. As described in the foregoing embodiments, if the first space allocation response indicates space allocation failure, it means that the executor is currently unable to send shuffle data to the first shuffle server. Based on this, the executor enters a waiting state. After waiting for a certain period of time, the driver will send a space allocation request to the first shuffle server again. If the shuffle data cannot be successfully written after several attempts (e.g., three), it means that the target write task has failed.

[0144] Specifically, taking a 90-second time threshold as an example, if the first space allocation response indicates space allocation failure, the executor enters a waiting state. When the waiting time is greater than or equal to 90 seconds, the driver, based on the first routing matching information, sends a second space allocation request to the first shuffle server again. Similarly, the second space allocation request also carries the taskID of the target write task and the amount of shuffle data. The first shuffle server can then determine whether the corresponding storage space can be allocated based on the amount of shuffle data. Thus, the driver receives the second space allocation response sent by the first shuffle server, where the second space allocation response may carry an identifier to indicate the space allocation result; for example, identifier "1" indicates successful allocation, and identifier "0" indicates allocation failure.

[0145] If the second space allocation response indicates that the storage space allocation was successful, then the target writetask in the executor sends the shuffle data to the first shuffle server indicated by the first node identifier based on the first routing matching information, and the shuffle server writes the shuffle data to the pre-allocated storage space.

[0146] If the second space allocation response indicates that storage space allocation failed, it means that the executor is currently unable to send shuffle data to the first shuffle server. Based on this, the executor enters a waiting state again.

[0147] Secondly, this application embodiment provides a method for handling write failures. Using this method, if the shuffle server's space is full, or the shuffle server is unable to write data due to hardware problems, or the shuffle server does not receive a space allocation response from the distributed cloud storage system for an extended period, a retry strategy can be initiated. That is, the driver requests storage space for the shuffle data from the shuffle server again. This improves the success rate of storage space allocation and avoids the driver making multiple registration requests to the coordinator, saving signaling transmission resources.

[0148] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, after receiving the first space allocation response sent by the first service node, it may further include:

[0149] If the first space allocation response indicates that the space allocation failed, a second registration request is sent to the scheduling node so that the scheduling node allocates a second service node for the target writing task according to the second registration request and generates second routing matching information. The second registration request carries the task identifier of the target writing task, and the second routing matching information includes the correspondence between the task identifier of the target writing task and the second node identifier. The second node identifier is used to indicate the second service node.

[0150] Receive the second route matching information sent by the scheduling node;

[0151] Based on the second routing matching information, a second space allocation request is sent to the second service node. The second space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The second space allocation request is used to request the second service node to allocate storage space for the shuffle data.

[0152] Receive a second space allocation response sent by the second service node, wherein the second space allocation response is used to indicate the space allocation result;

[0153] If the second space allocation response indicates that the space allocation was successful, then shuffle data is sent to the second service node so that the second service node can store the shuffle data.

[0154] In one or more embodiments, another method for handling write failures is described. As described in the foregoing embodiments, if the first space allocation response indicates space allocation failure, it means that the executor is currently unable to send shuffle data to the first shuffle server. Based on this, the driver sends a second registration request to the coordinator, which carries the taskID of the target write task. The coordinator allocates a second shuffle server for the target write task based on the taskID carried in the second registration request. After allocation is completed, the coordinator generates second routing matching information. For ease of understanding, please refer to Table 2, which is a schematic diagram of the second routing matching information.

[0155] Table 2

[0156] taskID Second node ID 10000 002

[0157] As can be seen, the second route matching information includes the correspondence between the target write task's taskID and the second node ID. Based on this, assuming the target write task's taskID is "10000", then the second node ID is "002".

[0158] The coordinator sends second route matching information to the driver. Based on this information, the driver determines that it has completed registration with the second shuffle server. Therefore, the driver sends a second space allocation request to the second shuffle server. The second shuffle server, based on this request, attempts to allocate storage space for the target write task's shuffle data. This second space allocation request carries the target write task's taskID and the amount of shuffle data. The second shuffle server determines whether it can allocate the corresponding storage space based on the amount of shuffle data.

[0159] For example, the second shuffle server can allocate local storage space; for instance, shuffle data can be stored on the second shuffle server's disk. Alternatively, the second shuffle server can request space allocation from a distributed cloud storage system such as HDFS, COS, or Ceph; if a match is found, the second shuffle server will receive a successful allocation response.

[0160] Therefore, the driver receives a second space allocation response from the second shuffle server, which may carry an identifier indicating the space allocation result. If the second space allocation response indicates that the second shuffle server has completed the allocation of storage space, then the target write task in the executor sends the shuffle data to the second shuffle server indicated by the second node identifier based on the second routing matching information, and the shuffle server writes the shuffle data to the pre-allocated storage space.

[0161] Secondly, this application embodiment provides another method for handling write failures. Using the above method, if the shuffle server's space is full, or the shuffle server is unable to write data due to hardware problems, or the shuffle server does not receive a space allocation response from the distributed cloud storage system for an extended period, a reallocation strategy can be initiated. That is, the driver requests the coordinator to register other shuffle servers again. This improves the success rate of storage space allocation.

[0162] Optionally, in the above Figure 3Based on the corresponding embodiments, in another optional embodiment provided by this application, shuffle data is sent to the first service node so that the first service node writes the shuffle data to the storage space, which may specifically include:

[0163] Shuffle data is written to the buffer in the execution node in the form of data blocks;

[0164] If the data storage volume in the buffer reaches the storage threshold, then the event data is added to the data queue corresponding to the execution node;

[0165] Based on the event data, the sending thread in the execution node is invoked to send the data block in the buffer to the first service node, so that the first service node writes the data block to the storage space.

[0166] In one or more embodiments, a method for a target write task in an executor to process shuffle data is described. As can be seen from the foregoing embodiments, the description takes the shuffle server as the first shuffle server.

[0167] Specifically, the target write task in the executor writes shuffle data into a buffer in the form of blocks. If the data storage volume of a buffer reaches a storage threshold (e.g., 90%), event data is added to the data queue corresponding to the executor. The event data includes the address corresponding to the block stored in the buffer. Based on this, the executor, according to the event data, sends the block in the buffer to the first shuffle server by calling the sender thread in the execution node. The first shuffle server then writes the block into the pre-allocated storage space.

[0168] For easier understanding, please refer to Figure 5 , Figure 5This is a schematic diagram of the shuffle write process in an embodiment of this application. As shown in the figure, in step B1, the target write task in the executor can send each record data in the shuffle data to the buffer corresponding to the target write task based on the partitionID, and store it in the buffer in the form of a block. In step B2, if the data storage volume of the buffer reaches the storage threshold, event data is added to the data queue. In step B3, the executor continuously retrieves event data from the data queue and submits it to the sender in the thread pool. In step B4, the sender in the executor first requests memory space from the first shuffle server, and then sends the block to the buffer of the first shuffle server, which then writes the block to the storage space.

[0169] It should be noted that, Figure 5 The number of buffers, event data, and senders shown are for illustrative purposes only and should not be construed as limiting the scope of this application.

[0170] Secondly, this application provides a method for a target write task in an executor to process shuffle data. Through this method, the target write task in the executor sends data in block form to the shuffle server. The shuffle server can then send the shuffle data to storage, thereby reducing network access for the executor and improving the success rate of large-scale job execution. Furthermore, in practical applications, multiple sending threads can simultaneously send blocks to the shuffle server, thereby improving data transmission efficiency.

[0171] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, after sending shuffle data to the first service node so that the first service node writes the shuffle data to the storage space, it may further include:

[0172] Obtain a set of task identifiers, wherein the set of task identifiers includes at least one written task identifier, and each written task identifier is used to indicate a write task that has been written to the storage space;

[0173] Based on the task identifier set, a data block identifier set for the target reading task is obtained from the first service node, wherein the data block identifier set includes at least one data block identifier, and each data block identifier is used to indicate a data block;

[0174] Obtain the index file, which includes a mapping between data block identifiers and data block read addresses;

[0175] If a set of data block identifiers exists in the index file, the target shuffle data is read from the index file, wherein the target shuffle data includes at least one data block.

[0176] In one or more embodiments, a method for a target read task in an executor to read target shuffle data is described. As can be seen from the foregoing embodiments, the shuffle server is taken as the first shuffle server, and the description is based on the target read task.

[0177] Specifically, the target read task in the executor obtains a set of taskIDs from the driver. This set includes at least one taskID that has been successfully written to storage. Therefore, the target read task in the executor can read the shuffle data corresponding to each taskID in the taskID set. The shuffle data is stored in storage in the form of blocks. After determining all the taskIDs to be read, the target read task in the executor obtains a set of blockIDs corresponding to each taskID from the first shuffle server, where one blockID indicates one block. The target read task in the executor needs to read the index file first. This index file is usually stored in storage; optionally, it can also be stored on the first shuffle server. Based on the index file, the target read task in the executor can determine the offset information corresponding to each block. This offset information indicates the read address of the block. Therefore, after the target read task in the executor reads the data file, it can read the corresponding block from the data file based on the offset information corresponding to each block. Finally, the target shuffle data composed of these blocks is obtained.

[0178] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of the shuffle read process in an embodiment of this application. As shown in the figure, specifically, in step C1, the target read task in the executor reads all the task IDs that were successfully written from the driver. In step C2, based on these task IDs, the target read task in the executor can obtain all the block IDs from the first shuffle server. In step C3, the target read task in the executor first reads the index file to verify whether these block IDs all exist, and then, based on the offset information in the index file, reads the data file, thereby obtaining the target shuffle data.

[0179] More specifically, there are two main implementation methods for reading data. The first method is for remote storage such as HDFS or COS. Based on this, in step C3.1, the target read task in the executor can directly read the target shuffle data from the storage. The second method is for the case where the storage is local storage. Based on this, in step C3.2, the target read task in the executor needs to read the target shuffle data through the first shuffle server.

[0180] Furthermore, this application provides a method for a target read task in an executor to read target shuffle data. Through this method, the executor can read target shuffle data not only stored on local storage but also stored on source storage, thereby improving the flexibility and reliability of the solution.

[0181] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, after reading the target shuffle data according to the index file, it may further include:

[0182] For each data block in the target shuffle data, the first checksum corresponding to each data block is obtained from the index file.

[0183] Perform verification calculations on each data block in the target shuffle data to obtain the second verification value corresponding to each data block;

[0184] Compare the first checksum with the second checksum for each data block;

[0185] If the first checksum and the second checksum of the data block match, then the data block has been successfully checked.

[0186] If the first checksum and the second checksum corresponding to the data block do not match, the data block checksum is determined to have failed.

[0187] In one or more embodiments, a method for verifying the integrity of blocks based on a verification mechanism is described. As described in the foregoing embodiments, the shuffle server is used as the first shuffle server in the illustration. It is understood that data is subject to various interferences during transmission, such as impulse interference, random noise interference, and human interference. Furthermore, data is stored on disk, which is susceptible to silent errors, leading to data inaccuracies. Therefore, a verification mechanism can be used to verify the blocks. It should be noted that since the target shuffle data often includes multiple blocks, for ease of explanation, the following description will focus on verifying one block. The verification method for other blocks is similar and will not be elaborated upon here.

[0188] Specifically, after the target read task in the executor reads the target shuffle data, it retrieves the first checksum corresponding to that block from the index file based on the blockID. Furthermore, it needs to perform a checksum calculation on the block to obtain a second checksum. The first checksum is obtained by the first shuffle server after performing the checksum calculation on the block, and the first shuffle server records the mapping relationship between the block ID and the first checksum in the index file. Based on this, the first checksum and the second checksum corresponding to the same block are compared. If they match, it means the block has been successfully checked, i.e., the data is complete. Conversely, if they do not match, it means the block has failed to be checked, i.e., the data is incomplete.

[0189] It should be noted that the verification algorithm involved here can be parity check, that is, in data storage and transmission, an extra bit is added to the byte as a check value to detect errors. This check bit can be calculated by XORing the data bits. The verification algorithm can also be Longitudinal Redundancy Check (LRC), Cyclic Redundancy Check (CRC), or digest algorithms, etc. The following will use the CRC algorithm as an example, that is, the first check value and the second check value are both CRC check values, but this should not be construed as a limitation of this application.

[0190] For easier understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of the data verification mechanism in an embodiment of this application. As shown in the figure, specifically, in step D1, the target write task in the executor calculates a CRC check value (e.g., a first CRC check value) for each block. While sending shuffle data to the shuffle server (e.g., a first shuffle server), it also sends the CRC check value corresponding to each block. Each successful write task saves the "partition->blockID" mapping relationship generated by the write task to the shuffle server's data structure. <partition,<taskID,blockID[]> In step D2, the shuffle server stores the CRC checksum in the index file, while the data is stored in the data file. In step D3, the successfully written task ID is stored in the driver to avoid data inconsistency issues caused by Spark Speculation. In step D4, the target read task in the executor reads the successfully written task ID from the driver. In step D5, the target read task in the executor obtains all valid block IDs in the corresponding partition from the shuffle server. In step D6, the executor reads the index file and verifies whether all valid block IDs exist in the index file. In step D7, the target read task in the executor calculates the corresponding CRC checksum (i.e., the second CRC checksum) for each read block and compares it with the CRC checksum stored in the index file (i.e., the first CRC checksum).

[0191] Secondly, this application provides a method for verifying the integrity of a block based on a verification mechanism. In this method, to ensure the integrity of the block, a specified algorithm is selected to calculate a verification value for the written block. The receiver then calculates the verification value for the read block using the same algorithm. If the two calculated verification values ​​are the same, it indicates that the data is complete. Therefore, by adding a verification mechanism to the shuffle data, the correctness of the shuffle data is improved.

[0192] Optionally, in the above Figure 3 Based on the corresponding embodiments, another optional embodiment provided in this application may further include:

[0193] If the target application is within its lifecycle, the first heartbeat information is periodically sent to the scheduling node so that the scheduling node retains the data associated with the target application. The first heartbeat information carries the application information of the target application, including a timestamp and an application identifier. The target application includes at least one of a write task or a read task.

[0194] If the target application's lifecycle has ended, stop sending the first heartbeat message to the scheduling node so that the scheduling node can delete the data associated with the target application.

[0195] In one or more embodiments, a method for a coordinator to clean up application-related data based on a heartbeat mechanism is described. As can be seen from the foregoing embodiments, the target write task or target read task is executed in the target application, wherein the lifecycle of the target application is usually the same as the lifecycle of the driver.

[0196] Specifically, for ease of understanding, please refer to Figure 8 , Figure 8 The figure shows a schematic diagram of the heartbeat mechanism in an embodiment of this application. In step E1, the driver can periodically (e.g., every 10 seconds) send the first heartbeat information to the coordinator. Based on the first heartbeat information, the coordinator can determine whether the target application is alive. If the target application is within its lifecycle, the coordinator continues to retain the data associated with the target application. Otherwise, if the target application is not within its lifecycle, the driver will no longer send the first heartbeat information to the coordinator, and the coordinator will delete the data associated with the target application.

[0197] Understandably, the coordinator records application information corresponding to each application. Therefore, the coordinator retrieves the relevant application information based on the application identifier (applicationID) of the target application. For example, suppose the driver sends a first heartbeat to the coordinator every 10 seconds, carrying the application information of the target application, including a timestamp and the applicationID. The coordinator can then determine the difference between the timestamps based on two consecutive received application messages. Therefore, if the difference between the timestamps exceeds a time difference threshold (e.g., 1 minute), it is considered that there may be a network communication failure between the driver and the coordinator, or other problems.

[0198] Secondly, this application embodiment provides a method for a coordinator to clean up application-related data based on a heartbeat mechanism. Using this method, if the application is still alive, the driver will periodically send heartbeat information to the coordinator, thus informing the coordinator that the application is still alive. Conversely, if the application is not alive, the driver will not send heartbeat information to the coordinator. Therefore, the coordinator can delete application-related information and release space, avoiding the occupation of internal space within the coordinator and maintaining the normal operation of the entire system.

[0199] Optionally, in the above Figure 3 Based on the corresponding embodiments, another optional embodiment provided in this application may further include:

[0200] When a communication connection is established with the first service node, a first timestamp is sent to the first service node so that the first service node updates the application information of the target application according to the first timestamp and determines the survival status of the target application according to the first timestamp and the second timestamp. The second timestamp comes from the application information of the target application, and the target application includes at least one of a write task or a read task.

[0201] In one or more embodiments, a method for an executor to report application information to a server is described. As can be seen from the foregoing embodiments, the example uses a shuffle server as the first shuffle server. The target write task or target read task is executed within the target application.

[0202] Specifically, for ease of understanding, please refer to Figure 9 , Figure 9 As shown in the figure, another schematic diagram of the heartbeat mechanism in this embodiment of the application illustrates that in step F1, each time the executor's write task (e.g., the target write task) or read task (e.g., the target read task) establishes a communication connection with the first shuffle server, it updates the timestamp of the target application (e.g., the first timestamp). This allows the first shuffle server to understand the currently active applications in more real-time.

[0203] Since the application information in the target application stored by the first shuffle server includes the latest timestamp (i.e., the second timestamp), after receiving the first timestamp sent by the executor, the difference between the first and second timestamps can be compared. If the difference is greater than a threshold (e.g., 90 seconds), it indicates that the target application is no longer alive. Conversely, if the difference is less than or equal to the threshold, it indicates that the target application is still alive. Therefore, the first shuffle server will update the timestamp in the application information, that is, update the second timestamp to the first timestamp.

[0204] Secondly, in this embodiment of the application, a method is provided for the executor to feed back application information to the server. In this way, the executor's task updates the application timestamp every time it communicates with the shuffle server, so that the shuffle server can obtain the information of the currently active applications in more real time, thereby improving the feasibility and operability of the solution.

[0205] Based on the above introduction, the following section will describe the data shuffling process in this application from the perspective of the shuffle server. Please refer to [link / reference needed]. Figure 10 Another embodiment of the data shuffling processing method in this application includes:

[0206] 210. Receive the space allocation request sent by the driver node according to the routing matching information. The space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The routing matching information is generated by the scheduling node after allocating the first service node to the target writing task according to the registration request. The registration request carries the task identifier of the target writing task. The routing matching information includes the correspondence between the task identifier of the target writing task and the node identifier. The node identifier is used to indicate the first service node.

[0207] In one or more embodiments, the driver can determine that it has completed registration with the first shuffle server based on the routing matching information. Based on this, the driver sends a space allocation request to the first shuffle server. The first shuffle server receives the space allocation request sent by the driver. The space allocation request carries the taskID of the target write task and the amount of shuffle data.

[0208] Specifically, the driver first sends a registration request to the coordinator, which carries the taskID of the target write task. Based on the taskID in the registration request, the coordinator assigns a first shuffle server to the target write task. After allocation, the coordinator generates routing matching information. This information includes the mapping between the target write task's taskID and the node ID of the first shuffle server. For example, assuming the target write task's taskID is "10000", the node ID corresponding to the first shuffle server would be "001".

[0209] Understandably, the coordinator's allocation strategy includes, but is not limited to, periodically obtaining the memory usage of different shuffle servers and prioritizing the shuffle server with lower memory usage as the first shuffle server. Alternatively, the coordinator may randomly assign a shuffle server as the first shuffle server. Furthermore, the coordinator can send registration requests to multiple different shuffle servers to achieve redundant configuration, thereby improving the reliability of data processing.

[0210] 220. Allocate storage space for shuffle data according to the space allocation request;

[0211] In one or more embodiments, the first shuffle server determines whether it can allocate corresponding storage space based on the amount of shuffle data. The first shuffle server then attempts to allocate storage space for the shuffle data of the target writetask based on a space allocation request.

[0212] For example, the first shuffle server may allocate local storage space, such as storing shuffle data on its own disk. Alternatively, the first shuffle server may request space allocation from a distributed cloud storage system such as HDFS, COS, or Ceph; if a match is found, the first shuffle server will receive a successful allocation response.

[0213] 230. Send a space allocation response to the driver node so that the driver node sends a space allocation response to the execution node, wherein the space allocation response is used to indicate the space allocation result;

[0214] In one or more embodiments, the first shuffle server sends a space allocation response to the driver, wherein the space allocation response may carry an identifier to indicate the result of the space allocation, for example, an identifier "1" indicating successful allocation and an identifier "0" indicating failed allocation.

[0215] 240. If the space allocation response indicates that the space allocation was successful, then receive the shuffle data sent by the execution node and write the shuffle data to the storage space.

[0216] In one or more embodiments, if the space allocation response indicates that the first shuffle server has completed the allocation of storage space, then the target write task in the executor sends the shuffle data to the first shuffle server indicated by the node identifier based on the routing matching information, and the first shuffle server writes the shuffle data to the pre-allocated storage space.

[0217] This application provides a method for processing shuffled data. Using this method, based on an external Spark service node provided by a remote Shuffle service, the scheduling node can request storage space for shuffle data from the service node. This storage space is used to store the shuffle data generated during the shuffle process, achieving separation of computation and storage. Therefore, even if a compute node fails, it will not affect the shuffle data stored on the service node, thereby avoiding the repeated generation of shuffle data and improving resource utilization.

[0218] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another optional embodiment provided by this application, receiving shuffle data sent by the execution node and writing the shuffle data to the storage space may specifically include:

[0219] Receive memory allocation requests sent by the execution node;

[0220] Allocate a buffer in the first service node for the execution node according to the memory allocation request;

[0221] Write the shuffle data sent by the execution node in the form of data blocks to the buffer;

[0222] If the data storage volume in the buffer reaches the storage threshold, then the event data is added to the data queue corresponding to the first service node.

[0223] Based on the event data, the write thread in the first service node is invoked to write the data blocks in the buffer to the storage space.

[0224] In one or more embodiments, a method for a shuffle server to write shuffle data to storage space is described. As can be seen from the foregoing embodiments, the description takes the first shuffle server as an example.

[0225] Specifically, the first shuffle server receives a memory allocation request from the executor and allocates a corresponding buffer for that executor. Based on this, the first shuffle server receives shuffle data sent by the executor in block form and writes the blocks into the allocated buffer. If the data storage capacity of a buffer reaches a storage threshold (e.g., 90%), event data is added to the data queue corresponding to the first shuffle server. This event data includes the addresses corresponding to the blocks stored in the buffer. Based on this, the first shuffle server, according to the event data, calls the writer thread to send the blocks in the buffer to the storage space provided by the storage service.

[0226] For easier understanding, please refer to Figure 11 , Figure 11 This is another schematic diagram of the shuffle write process in an embodiment of this application. As shown in the figure, in step G1, if the data storage volume of the buffer reaches the storage threshold, event data is added to the data queue. In step G2, the first shuffle server continuously retrieves event data from the data queue and submits it to the writer in the thread pool. In step G3, based on the shuffle data information (applicationID, shuffleID, partitionID), the first shuffle server can obtain the path where the shuffle data is stored in storage, and then writes the shuffle data to the index file and data file. In step G4, after the shuffle data writing is completed, the executor informs the first shuffle server that the task has been completed and obtains the number of completed tasks. If the number of completed tasks is less than a preset number, step G5 is executed. If the number of completed tasks is greater than or equal to the preset number, the first shuffle server will synchronously execute the remaining tasks, thereby improving processing efficiency. In step G5, after the task is completed, the executor records the taskID in the mapping status (MapStatus) and sends it to the driver, thereby supporting subsequent speculative execution functions.

[0227] It should be noted that, Figure 11The number of buffers, event data, and senders shown are for illustrative purposes only and should not be construed as limiting the scope of this application. Specifically, the data block identification module in the first shuffle server is used to label blocks. The commit information module is used to commit information (Commit Info) to other modules and storage. The pre-allocation information (PreAllocation Info) is used to allocate storage space for the shuffle data.

[0228] Secondly, this application provides a method for a shuffle server to write shuffle data to storage space. Using this method, the buffer in the shuffle server can store data in block form. Whenever the data storage volume of a buffer reaches a storage threshold, the blocks in that buffer can be sent to the corresponding storage space via a write thread. This improves the feasibility and operability of the solution, allows for reasonable resource allocation, and in practical applications, multiple write threads can simultaneously send blocks to storage space, thereby improving data transmission efficiency.

[0229] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another optional embodiment provided by this application, the execution node includes a first execution node and a second execution node;

[0230] Receive shuffle data sent by the execution node and write the shuffle data to the storage space, which may specifically include:

[0231] If the shuffle data sent by the first execution node has been written to the storage space, and the shuffle data sent by the second execution node has not been written, then a storage success message is sent to the first execution node, and the space occupied by the shuffle data sent by the second execution node is released.

[0232] If the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, then a storage success message is sent to the second execution node, and the space occupied by the shuffle data sent by the first execution node is released.

[0233] In one or more embodiments, a method for implementing speculative execution functionality is described. As can be seen from the foregoing embodiments, the example uses a first shuffle server as the shuffle server, and the execution nodes include a first executor and a second executor.

[0234] Specifically, the first shuffle server receives shuffle data from the first and second executors, respectively, and writes the shuffle data to pre-allocated storage space. However, in reality, different executors may have different data transmission efficiency, or network instability may cause differences in the progress of writing to storage space. Based on this, the first shuffle server can determine which executor should complete the writing of shuffle data first based on the write response from storage.

[0235] For example, suppose the first executor has written the shuffle data to storage, but the second executor has not yet written all the shuffle data to storage. Then, the first shuffle server sends a storage success message to the first executor, and can also send a storage failure message to the second executor. Finally, the first shuffle server should release the space occupied by the shuffle data sent by the second executor.

[0236] For example, suppose the second executor has written the shuffle data to storage, but the first executor has not yet written all the shuffle data to storage. Then, the first shuffle server sends a storage success message to the second executor, and can also send a storage failure message to the first executor. Finally, the first shuffle server should release the space occupied by the shuffle data sent by the first executor.

[0237] Secondly, this application provides a method for implementing speculative execution. This method takes into account potential hardware aging or software vulnerabilities that could cause some tasks to run very slowly. Therefore, repeatedly running tasks helps improve the success rate of task execution. Furthermore, it eliminates the need to wait for a task to encounter a problem before re-running it, thus saving processing time and improving processing efficiency.

[0238] Optionally, in the above Figure 10 Based on the corresponding embodiments, another optional embodiment provided in this application may further include:

[0239] Periodically send heartbeat information to the scheduling node. The heartbeat information carries the node identifier, or the heartbeat information carries the node identifier and the memory usage of the first service node. The memory usage is used by the scheduling node to allocate service nodes for write tasks, and the heartbeat information is used to request the list of online applications.

[0240] Receive the list of online applications sent by the scheduling node, wherein the list of online applications includes the application identifiers corresponding to the applications within their lifecycle;

[0241] If the application identifier corresponding to the target application is not included in the online application list, the data associated with the target application will be deleted. The target application includes at least one of a write task or a read task.

[0242] In one or more embodiments, a method for a shuffle server to clean up application-related data based on a heartbeat mechanism is described. As can be seen from the foregoing embodiments, the shuffle server is used as the first shuffle server for illustration, and the target write task or target read task is executed in the target application, wherein the lifecycle of the target application is usually the same as the lifecycle of the driver.

[0243] Specifically, for ease of understanding, please refer to Figure 12 , Figure 12 As another schematic diagram of the heartbeat mechanism in this embodiment, in step H1, the first shuffle server periodically (e.g., every 10 seconds) sends heartbeat information to the coordinator. The heartbeat information carries at least the node ID of the first shuffle server. Based on this, the coordinator returns a list of running online applications to the first shuffle server. This list includes the applicationIDs corresponding to applications still within their lifecycle. If the applicationID corresponding to the target application exists in the online application list, it indicates that the target application is still running; therefore, the first shuffle server continues to retain the data associated with the target application. Conversely, if the applicationID corresponding to the target application is not in the online application list, it indicates that the lifecycle of the target application has ended, and the first shuffle server deletes the data associated with the target application.

[0244] The heartbeat information can also carry the memory usage of the first shuffle server. As a result, the coordinator can know the memory usage of the first shuffle server at different times, and combine it with the memory usage reported by other shuffle servers to comprehensively evaluate the shuffle servers that can be registered. In this way, the coordinator can write tasks to allocate more suitable shuffle servers.

[0245] Secondly, in this embodiment of the application, a method is provided for a shuffle server to clean up application-related data based on a heartbeat mechanism. The shuffle server receives an online application list sent by the coordinator. The shuffle server can then compare whether the application ID exists in the online application list. If it does not exist, the data associated with the target application is deleted and the space is released, thereby avoiding occupying the storage space of the shuffle server and maintaining the normal operation of the entire system.

[0246] The mixed data processing apparatus of this application is described in detail below. Please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of one embodiment of the data shuffling processing apparatus in this application. The data shuffling processing apparatus 30 includes:

[0247] The sending module 310 is used to send a first registration request to the scheduling node so that the scheduling node allocates a first service node for the target writing task and generates first routing matching information. The first registration request carries the task identifier of the target writing task, and the first routing matching information includes the correspondence between the task identifier of the target writing task and the first node identifier. The first node identifier is used to indicate the first service node.

[0248] The receiving module 320 is used to receive the first route matching information sent by the scheduling node;

[0249] The sending module 310 is also used to send a first space allocation request to the first service node according to the first routing matching information. The first space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The first space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0250] The receiving module 320 is also configured to receive a first space allocation response sent by the first service node, wherein the first space allocation response is used to indicate the space allocation result;

[0251] The writing module 330 is used to send shuffle data to the first service node if the first space allocation response indicates that the space allocation is successful, so that the first service node writes the shuffle data to the storage space.

[0252] This application provides a data shuffling processing apparatus. Using this apparatus, based on an external Spark service node provided by a remote shuffle service, the scheduling node can request storage space for shuffle data from the service node. This storage space is used to store shuffle data generated during the shuffle process, achieving separation of computation and storage. Therefore, on the one hand, the failure of the shuffle process will not affect the already stored shuffle data, thereby avoiding duplicate data generation and improving resource utilization. On the other hand, it breaks the limitation of executors storing shuffle data, storing shuffle data on external devices, which is beneficial to increasing data storage capacity. Furthermore, it can reduce the number of network requests accessing the executor, lowering the probability of shuffle errors.

[0253] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed washing data processing device 30 provided in this application, the mixed washing data processing device 30 further includes a waiting module 340;

[0254] The waiting module 340 is used to enter a waiting state after receiving the first space allocation response sent by the first service node if the first space allocation response indicates that space allocation has failed.

[0255] The sending module 310 is also used to send a second space allocation request to the first service node according to the first routing matching information if the waiting time is greater than or equal to the time threshold. The second space allocation request carries the task identifier and the amount of shuffle data. The second space allocation request is used to request the first service node to allocate storage space for the shuffle data.

[0256] The receiving module 320 is also used to receive the second space allocation response sent by the first service node;

[0257] The sending module 310 is further configured to send shuffle data to the first service node if the second space allocation response indicates that the space allocation is successful, so that the first service node writes the shuffle data to the storage space, wherein the second space allocation response is used to indicate the space allocation result.

[0258] The waiting module 340 is also used to re-enter the waiting state if the second space allocation response indicates that the space allocation has failed.

[0259] This application provides a data shuffling processing apparatus. Using this apparatus, if the shuffle server's space is full, or if the shuffle server is unable to write data due to hardware problems, or if the shuffle server does not receive a space allocation response from the distributed cloud storage system for an extended period, a retry strategy can be initiated. That is, the driver can request storage space for the shuffle data from the shuffle server again. This improves the success rate of storage space allocation and avoids the driver making multiple registration requests to the coordinator, saving signaling transmission resources.

[0260] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed washing data processing apparatus 30 provided in this application,

[0261] The sending module 310 is further configured to, after receiving the first space allocation response sent by the first service node, if the first space allocation response indicates that space allocation has failed, send a second registration request to the scheduling node so that the scheduling node allocates a second service node for the target writing task according to the second registration request and generates second routing matching information, wherein the second registration request carries the task identifier of the target writing task, and the second routing matching information includes the correspondence between the task identifier of the target writing task and the second node identifier, and the second node identifier is used to indicate the second service node;

[0262] The receiving module 320 is also used to receive the second route matching information sent by the scheduling node;

[0263] The sending module 310 is also used to send a second space allocation request to the second service node according to the second routing matching information. The second space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The second space allocation request is used to request the second service node to allocate storage space for the shuffle data.

[0264] The receiving module 320 is also configured to receive a second space allocation response sent by the second service node, wherein the second space allocation response is used to indicate the space allocation result;

[0265] The sending module 310 is also configured to send shuffle data to the second service node if the second space allocation response indicates that the space allocation was successful, so that the second service node can store the shuffle data.

[0266] This application provides a data shuffling processing apparatus. Using this apparatus, if the shuffle server's space is full, or if the shuffle server is unable to write data due to hardware problems, or if the shuffle server does not receive a space allocation response from the distributed cloud storage system for an extended period, a reallocation strategy can be initiated. That is, the driver requests the coordinator to register other shuffle servers again. This improves the success rate of storage space allocation.

[0267] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed washing data processing apparatus 30 provided in this application,

[0268] The sending module 310 is specifically used to write shuffle data into the buffer in the execution node in the form of data blocks;

[0269] If the data storage volume in the buffer reaches the storage threshold, then the event data is added to the data queue corresponding to the execution node;

[0270] Based on the event data, the sending thread in the execution node is invoked to send the data block in the buffer to the first service node, so that the first service node writes the data block to the storage space.

[0271] This application provides a data shuffling processing apparatus. Using this apparatus, the target write task in the executor sends data in block format to the shuffle server. The shuffle server can then send the shuffle data to storage, thereby reducing network access for the executor and improving the success rate of large-scale job execution. Furthermore, in practical applications, multiple sending threads can simultaneously send blocks to the shuffle server, further improving data transmission efficiency.

[0272] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed washing data processing device 30 provided in this application, the mixed washing data processing device 30 further includes an acquisition module 350;

[0273] The acquisition module 350 is used to acquire a set of task identifiers after sending shuffle data to the first service node so that the first service node writes the shuffle data to the storage space. The set of task identifiers includes at least one written task identifier, and each written task identifier is used to indicate a write task that has been written to the storage space.

[0274] The acquisition module 350 is also used to acquire a set of data block identifiers for the target reading task from the first service node based on the task identifier set, wherein the set of data block identifiers includes at least one data block identifier, and each data block identifier indicates a data block;

[0275] The acquisition module 350 is also used to acquire an index file, wherein the index file includes a mapping relationship between data block identifiers and data block read addresses;

[0276] The acquisition module 350 is also used to read the target shuffle data according to the index file if a set of data block identifiers exists in the index file, wherein the target shuffle data includes at least one data block.

[0277] This application provides a mixed shuffle data processing apparatus. Using this apparatus, the executor can read not only the target shuffle data stored on the local storage, but also the target shuffle data stored on the source storage, thereby improving the flexibility and reliability of the solution.

[0278] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed data processing device 30 provided in this application, the mixed data processing device 30 further includes a verification module 360.

[0279] The acquisition module 350 is also used to, after reading the target shuffle data according to the index file, obtain the first verification value corresponding to each data block from the index file for the data block identifier corresponding to each data block in the target shuffle data;

[0280] The verification module 360 ​​is used to perform verification calculations on each data block in the target shuffle data to obtain the second verification value corresponding to each data block.

[0281] The verification module 360 ​​is also used to compare the first verification value and the second verification value of each data block;

[0282] The verification module 360 ​​is also used to determine that the data block has been successfully verified if the first verification value and the second verification value corresponding to the data block are consistent.

[0283] The verification module 360 ​​is also used to determine that the data block verification has failed if the first verification value and the second verification value corresponding to the data block are inconsistent.

[0284] This application provides a shuffle data processing apparatus. Using this apparatus, to ensure the integrity of the blocks, a specified algorithm can be selected to calculate a checksum for the written block. The receiver then calculates a checksum for the read block using the same algorithm. If the two calculated checksums are the same, the data is considered complete. Thus, by adding a checksum verification mechanism to the shuffle data, the correctness of the shuffle data is improved.

[0285] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed washing data processing device 30 provided in this application, the mixed washing data processing device 30 further includes a stop module 370;

[0286] The sending module 310 is also used to periodically send a first heartbeat message to the scheduling node during the lifecycle of the target application, so that the scheduling node retains the data associated with the target application. The first heartbeat message carries application information of the target application, including a timestamp and an application identifier. The target application includes at least one of a write task or a read task.

[0287] The stop module 370 is used to stop sending the first heartbeat information to the scheduling node if the lifecycle of the target application has ended, so that the scheduling node can delete the data associated with the target application.

[0288] This application provides a data shuffling processing apparatus. Using this apparatus, if the application is still alive, the driver periodically sends heartbeat information to the coordinator, thus informing the coordinator that the application is still alive. Conversely, if the application is not alive, the driver does not send heartbeat information to the coordinator. Therefore, the coordinator can delete application-related information and free up space, avoiding occupying internal space within the coordinator and maintaining the normal operation of the entire system.

[0289] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another embodiment of the mixed washing data processing apparatus 30 provided in this application,

[0290] The sending module 310 is further configured to send a first timestamp to the first service node when a communication connection is established with the first service node, so that the first service node updates the application information of the target application according to the first timestamp, and determines the survival status of the target application according to the first timestamp and the second timestamp, wherein the second timestamp is derived from the application information of the target application, and the target application includes at least one of a write task or a read task.

[0291] This application provides a data shuffling processing apparatus. Using this apparatus, each time an executor task communicates with the shuffle server, it updates the application's timestamp, enabling the shuffle server to obtain information about currently active applications more realistically, thereby improving the feasibility and operability of the solution.

[0292] The mixed data processing apparatus of this application is described in detail below. Please refer to [link / reference]. Figure 14 , Figure 14 This is a schematic diagram of another embodiment of the data shuffling processing apparatus in this application. The data shuffling processing apparatus 40 includes:

[0293] The receiving module 410 is used to receive a space allocation request sent by the driving node according to the routing matching information. The space allocation request carries the task identifier of the target writing task and the amount of shuffle data. The routing matching information is generated by the scheduling node after allocating a first service node to the target writing task according to the registration request. The registration request carries the task identifier of the target writing task. The routing matching information includes the correspondence between the task identifier of the target writing task and the node identifier. The node identifier is used to indicate the first service node.

[0294] The allocation module 420 is used to allocate storage space for shuffle data according to the space allocation request;

[0295] The sending module 430 is used to send a space allocation response to the driving node, so that the driving node sends a space allocation response to the execution node, wherein the space allocation response is used to indicate the space allocation result;

[0296] The write module 440 is used to receive the shuffle data sent by the execution node and write the shuffle data to the storage space if the space allocation response indicates that the space allocation is successful.

[0297] This application provides a data shuffling processing apparatus. Using this apparatus, based on an external Spark service node provided by a remote Shuffle service, the scheduling node can request storage space for shuffle data from the service node. This storage space is used to store the shuffle data generated during the shuffle process, achieving separation of computation and storage. Therefore, even if a compute node fails, the shuffle data stored on the service node will not be affected, thus avoiding the repeated generation of shuffle data and improving resource utilization.

[0298] Optionally, in the above Figure 14 Based on the corresponding embodiments, in another embodiment of the mixed data processing apparatus 40 provided in this application,

[0299] The write module 440 is specifically used to receive memory allocation requests sent by the execution node;

[0300] Allocate a buffer in the first service node for the execution node according to the memory allocation request;

[0301] Write the shuffle data sent by the execution node in the form of data blocks to the buffer;

[0302] If the data storage volume in the buffer reaches the storage threshold, then the event data is added to the data queue corresponding to the first service node.

[0303] Based on the event data, the write thread in the first service node is invoked to write the data blocks in the buffer to the storage space.

[0304] This application provides a data shuffling processing apparatus. Using this apparatus, the buffer in the shuffle server can store data in block format. Whenever the data storage capacity of a buffer reaches a storage threshold, the blocks in that buffer can be sent to the corresponding storage space via a write thread. This improves the feasibility and operability of the solution, allows for reasonable resource allocation, and in practical applications, multiple write threads can simultaneously send blocks to the storage space, thereby improving data transmission efficiency.

[0305] Optionally, in the above Figure 14 Based on the corresponding embodiments, in another embodiment of the mixed data processing device 40 provided in this application, the execution nodes include a first execution node and a second execution node;

[0306] The writing module 440 is specifically used to send a storage success message to the first execution node and release the space occupied by the shuffle data sent by the second execution node if the shuffle data sent by the first execution node has been written to the storage space and the shuffle data sent by the second execution node has not been written.

[0307] If the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, then a storage success message is sent to the second execution node, and the space occupied by the shuffle data sent by the first execution node is released.

[0308] This application provides a data shuffling processing apparatus. Using this apparatus, considering potential hardware aging or software vulnerabilities that could cause some tasks to run very slowly, repeatedly running tasks helps improve the success rate of task execution. Furthermore, it eliminates the need to wait for a task to encounter a problem before re-running it, thereby saving processing time and improving processing efficiency.

[0309] Optionally, in the above Figure 14 Based on the corresponding embodiments, in another embodiment of the mixed data processing device 40 provided in this application, the mixed data processing device 40 further includes a deletion module 450;

[0310] The sending module 430 is also used to periodically send heartbeat information to the scheduling node, wherein the heartbeat information carries a node identifier, or the heartbeat information carries a node identifier and the memory usage of the first service node, the memory usage is used by the scheduling node to allocate service nodes for the write task, and the heartbeat information is used to request the list of online applications;

[0311] The receiving module 410 is also used to receive a list of online applications sent by the scheduling node, wherein the list of online applications includes application identifiers corresponding to applications within their lifecycles;

[0312] The deletion module 450 is used to delete data associated with the target application if the application identifier corresponding to the target application is not included in the online application list, wherein the target application includes at least one of a write task or a read task.

[0313] This application provides a data shuffling processing apparatus. Using this apparatus, the shuffle server receives a list of online applications from the coordinator. The shuffle server can then compare the application ID to the list. If the application ID is not present, the data associated with the target application is deleted, and space is released. This avoids occupying the shuffle server's storage space and maintains the normal operation of the entire system.

[0314] The mixed-wash data processing device provided in this application can be deployed on a server. Please refer to [link / reference]. Figure 15 , Figure 15 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 522 (e.g., one or more processors) and memory 532, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 542 or data 544. The memory 532 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 522 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the server 500.

[0315] Server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input / output interfaces 558, and / or one or more operating systems 541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0316] The steps performed by the server in the above embodiments can be based on this Figure 15 The server structure shown.

[0317] The data processing device for shuffling provided in this application can be deployed on terminal devices. Figure 16As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0318] Figure 16 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 16 The smartphone includes components such as a radio frequency (RF) circuit 610, a memory 620, an input unit 630, a display unit 640, a sensor 650, an audio circuit 660, a wireless fidelity (WiFi) module 670, a processor 680, and a power supply 690. Those skilled in the art will understand that... Figure 16 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0319] The following is combined Figure 16 A detailed introduction to the various components of a smartphone:

[0320] RF circuit 610 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 680; additionally, it transmits uplink data to the base station. Typically, RF circuit 610 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, RF circuit 610 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0321] The memory 620 can be used to store software programs and modules. The processor 680 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 620. The memory 620 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0322] The input unit 630 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 630 may include a touch panel 631 and other input devices 632. The touch panel 631, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 631), and drive corresponding connected devices according to a pre-set program. Optionally, the touch panel 631 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 680, and can also receive and execute commands sent by the processor 680. In addition, the touch panel 631 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 631, the input unit 630 may also include other input devices 632. Specifically, other input devices 632 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0323] Display unit 640 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 640 may include display panel 641, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, touch panel 631 may cover display panel 641. When touch panel 631 detects a touch operation on or near it, it transmits the information to processor 680 to determine the type of touch event. Subsequently, processor 680 provides corresponding visual output on display panel 641 based on the type of touch event. Although in Figure 16 In this embodiment, the touch panel 631 and the display panel 641 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 631 and the display panel 641 can be integrated to realize the input and output functions of the smartphone.

[0324] The smartphone may also include at least one sensor 650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 641 according to the ambient light level, and the proximity sensor can turn off the display panel 641 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0325] Audio circuit 660, speaker 661, and microphone 662 provide an audio interface between the user and the smartphone. Audio circuit 660 converts received audio data into electrical signals and transmits them to speaker 661, where speaker 661 converts them into sound signals for output. On the other hand, microphone 662 converts collected sound signals into electrical signals, which are received by audio circuit 660, converted into audio data, and then processed by processor 680 before being transmitted via RF circuit 610 to, for example, another smartphone, or the audio data can be output to memory 620 for further processing.

[0326] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi module 670, can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 16 WiFi module 670 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0327] The processor 680 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 620, and by calling data stored in the memory 620. Optionally, the processor 680 may include one or more processing units; optionally, the processor 680 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 680.

[0328] The smartphone also includes a power supply 690 (such as a battery) that powers various components. Optionally, the power supply can be logically connected to the processor 680 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0329] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0330] The steps performed by the terminal device in the above embodiments can be based on this Figure 16 The terminal device structure is shown.

[0331] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0332] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0333] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0334] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0335] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0336] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0337] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0338] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing mixed shuffled data, characterized in that, include: A first registration request is sent to the scheduling node so that the scheduling node allocates a first service node for the target write task and generates first routing matching information. The first registration request carries the task identifier of the target write task, and the first routing matching information includes the correspondence between the task identifier of the target write task and a first node identifier. The first node identifier is used to indicate the first service node. Receive the first route matching information sent by the scheduling node; Based on the first routing matching information, a first space allocation request is sent to the first service node. The first space allocation request carries the task identifier of the target write task and the amount of shuffle data. The first space allocation request is used to request the first service node to allocate storage space for the shuffle data. Receive a first space allocation response sent by the first service node, wherein the first space allocation response is used to indicate the space allocation result; If the first space allocation response indicates successful space allocation, the shuffle data is sent to the first service node so that the first service node writes the shuffle data to the storage space. The shuffle data includes shuffle data sent from the first execution node and the second execution node. The first service node writing the shuffle data to the storage space includes: if the shuffle data sent by the first execution node has been written to the storage space, and the shuffle data sent by the second execution node has not been written, a storage success message is sent to the first execution node, and the space occupied by the shuffle data sent by the second execution node is released; if the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, a storage success message is sent to the second execution node, and the space occupied by the shuffle data sent by the first execution node is released.

2. The processing method according to claim 1, characterized in that, After receiving the first space allocation response sent by the first service node, the method further includes: If the first space allocation response indicates that space allocation has failed, then enter a waiting state; If the waiting time is greater than or equal to the time threshold, a second space allocation request is sent to the first service node according to the first routing matching information. The second space allocation request carries the task identifier and the amount of data of the shuffle data. The second space allocation request is used to request the first service node to allocate storage space for the shuffle data. Receive the second space allocation response sent by the first service node; If the second space allocation response indicates that the space allocation was successful, then the shuffle data is sent to the first service node so that the first service node writes the shuffle data to the storage space, wherein the second space allocation response is used to indicate the space allocation result; If the second space allocation response indicates that space allocation has failed, then the system will enter the waiting state again.

3. The processing method according to claim 1, characterized in that, After receiving the first space allocation response sent by the first service node, the method further includes: If the first space allocation response indicates that space allocation has failed, a second registration request is sent to the scheduling node so that the scheduling node allocates a second service node for the target writing task according to the second registration request and generates second routing matching information. The second registration request carries the task identifier of the target writing task, and the second routing matching information includes the correspondence between the task identifier of the target writing task and the second node identifier. The second node identifier is used to indicate the second service node. Receive the second route matching information sent by the scheduling node; According to the second routing matching information, a second space allocation request is sent to the second service node. The second space allocation request carries the task identifier of the target write task and the data volume of the shuffle data. The second space allocation request is used to request the second service node to allocate storage space for the shuffle data. Receive a second space allocation response sent by the second service node, wherein the second space allocation response is used to indicate the space allocation result; If the second space allocation response indicates that the space allocation was successful, the shuffle data is sent to the second service node so that the second service node can store the shuffle data.

4. The processing method according to claim 1, characterized in that, Sending the shuffle data to the first service node so that the first service node writes the shuffle data to the storage space includes: The shuffle data is written to the buffer in the execution node in the form of data blocks; If the data storage volume of the buffer reaches the storage threshold, then event data is added to the data queue corresponding to the execution node; Based on the event data, the sending thread in the execution node is invoked to send the data block in the buffer to the first service node, so that the first service node writes the data block into the storage space.

5. The processing method according to claim 1, characterized in that, After sending the shuffle data to the first service node so that the first service node writes the shuffle data to the storage space, the method further includes: Obtain a set of task identifiers, wherein the set of task identifiers includes at least one written task identifier, and each written task identifier is used to indicate a write task that has been written to the storage space; Based on the task identifier set, a data block identifier set for the target reading task is obtained from the first service node, wherein the data block identifier set includes at least one data block identifier, and each data block identifier is used to indicate a data block; Obtain an index file, wherein the index file includes a mapping relationship between data block identifiers and data block read addresses; If the data block identifier set exists in the index file, then the target shuffle data is read according to the index file, wherein the target shuffle data includes at least one data block.

6. The processing method according to claim 5, characterized in that, After reading the target shuffle data according to the index file, the method further includes: For each data block in the target shuffle data, a first checksum corresponding to each data block is obtained from the index file. A verification calculation is performed on each data block in the target shuffle data to obtain a second verification value corresponding to each data block; The first check value and the second check value of each data block are compared. If the first check value corresponding to the data block matches the second check value, then the data block has been successfully checked. If the first check value corresponding to the data block does not match the second check value, then the data block check is determined to have failed.

7. The processing method according to claim 1, characterized in that, The method further includes: If, during the lifecycle of the target application, a first heartbeat message is periodically sent to the scheduling node so that the scheduling node retains the data associated with the target application, wherein the first heartbeat message carries the application information of the target application, the application information including a timestamp and an application identifier, and the target application includes at least one of a write task or a read task; If the lifecycle of the target application has ended, then the sending of the first heartbeat information to the scheduling node is stopped, so that the scheduling node deletes the data associated with the target application.

8. The processing method according to claim 1, characterized in that, The method further includes: When a communication connection is established with the first service node, a first timestamp is sent to the first service node so that the first service node updates the application information of the target application according to the first timestamp, and determines the survival status of the target application according to the first timestamp and the second timestamp, wherein the second timestamp comes from the application information of the target application, and the target application includes at least one of a write task or a read task.

9. A method for processing mixed data, characterized in that, include: The system receives a space allocation request sent by the driver node based on routing matching information. The space allocation request carries the task identifier of the target write task and the amount of shuffle data. The routing matching information is generated by the scheduling node after allocating a first service node to the target write task based on the registration request. The registration request carries the task identifier of the target write task. The routing matching information includes the correspondence between the task identifier of the target write task and the node identifier. The node identifier is used to indicate the first service node. Allocate storage space for the shuffle data according to the space allocation request; A space allocation response is sent to the driver node, so that the driver node sends the space allocation response to the execution node, wherein the space allocation response is used to indicate the space allocation result; If the space allocation response indicates successful space allocation, then the shuffle data sent by the execution node is received, and the shuffle data is written to the storage space; the execution node includes a first execution node and a second execution node; receiving the shuffle data sent by the execution node and writing the shuffle data to the storage space includes: If the shuffle data sent by the first execution node has been written to the storage space, and the shuffle data sent by the second execution node has not been written, then a storage success message is sent to the first execution node, and the space occupied by the shuffle data sent by the second execution node is released. If the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, then a storage success message is sent to the second execution node, and the space occupied by the shuffle data sent by the first execution node is released.

10. The processing method according to claim 9, characterized in that, The step of receiving the shuffle data sent by the execution node and writing the shuffle data to the storage space includes: Receive the memory allocation request sent by the execution node; The execution node is allocated a buffer from the first service node according to the memory allocation request; The shuffle data sent by the execution node in the form of data blocks is written to the buffer; If the data storage volume of the buffer reaches the storage threshold, then event data is added to the data queue corresponding to the first service node; Based on the event data, the write thread in the first service node is invoked to write the data block in the buffer to the storage space.

11. The processing method according to any one of claims 9 to 10, characterized in that, The method further includes: The system periodically sends heartbeat information to the scheduling node, wherein the heartbeat information carries the node identifier, or the heartbeat information carries the node identifier and the memory usage of the first service node, the memory usage is used by the scheduling node to allocate service nodes for write tasks, and the heartbeat information is used to request the online application list; Receive the online application list sent by the scheduling node, wherein the online application list includes application identifiers corresponding to applications within their lifecycle; If the online application list does not include the application identifier corresponding to the target application, then the data associated with the target application is deleted, wherein the target application includes at least one of a write task or a read task.

12. A mixed data processing device, characterized in that, include: The sending module is used to send a first registration request to the scheduling node so that the scheduling node allocates a first service node for the target write task and generates first routing matching information. The first registration request carries the task identifier of the target write task, and the first routing matching information includes the correspondence between the task identifier of the target write task and a first node identifier. The first node identifier is used to indicate the first service node. A receiving module is used to receive the first route matching information sent by the scheduling node; The sending module is further configured to send a first space allocation request to the first service node according to the first routing matching information, wherein the first space allocation request carries the task identifier of the target writing task and the amount of shuffle data, and the first space allocation request is used to request the first service node to allocate storage space for the shuffle data. The receiving module is further configured to receive a first space allocation response sent by the first service node, wherein the first space allocation response is used to indicate the space allocation result; The writing module is configured to, if the first space allocation response indicates successful space allocation, send the shuffle data to the first service node, so that the first service node writes the shuffle data to the storage space; the shuffle data includes shuffle data sent from the first execution node and the second execution node; the first service node writing the shuffle data to the storage space includes: if the shuffle data sent by the first execution node has been written to the storage space, and the shuffle data sent by the second execution node has not been written, then sending a storage success message to the first execution node and releasing the space occupied by the shuffle data sent by the second execution node; if the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, then sending a storage success message to the second execution node and releasing the space occupied by the shuffle data sent by the first execution node.

13. The processing apparatus according to claim 12, characterized in that, Sending the shuffle data to the first service node so that the first service node writes the shuffle data to the storage space includes: The shuffle data is written to the buffer in the execution node in the form of data blocks; If the data storage volume of the buffer reaches the storage threshold, then event data is added to the data queue corresponding to the execution node; Based on the event data, the sending thread in the execution node is invoked to send the data block in the buffer to the first service node, so that the first service node writes the data block into the storage space.

14. A mixed data processing device, characterized in that, include: The receiving module is used to receive a space allocation request sent by the driver node according to the routing matching information. The space allocation request carries the task identifier of the target write task and the amount of shuffle data. The routing matching information is generated by the scheduling node after allocating a first service node to the target write task according to the registration request. The registration request carries the task identifier of the target write task. The routing matching information includes the correspondence between the task identifier of the target write task and the node identifier. The node identifier is used to indicate the first service node. The allocation module is used to allocate storage space for the shuffle data according to the space allocation request; A sending module is configured to send a space allocation response to the driving node, so that the driving node sends the space allocation response to the execution node, wherein the space allocation response is used to indicate the space allocation result; The writing module is configured to receive the shuffle data sent by the execution node and write the shuffle data to the storage space if the space allocation response indicates successful space allocation; the execution node includes a first execution node and a second execution node; the writing module is specifically configured to: If the shuffle data sent by the first execution node has been written to the storage space, and the shuffle data sent by the second execution node has not been written, then a storage success message is sent to the first execution node, and the space occupied by the shuffle data sent by the second execution node is released. If the shuffle data sent by the first execution node has not been written, and the shuffle data sent by the second execution node has been written to the storage space, then a storage success message is sent to the second execution node, and the space occupied by the shuffle data sent by the first execution node is released.

15. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the processing method as described in any one of claims 1 to 8 according to the instructions in the program code, or to execute the processing method as described in any one of claims 9 to 11; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

16. A computer-readable storage medium comprising instructions, which, when executed on a computer, cause the computer to perform the processing method as claimed in any one of claims 1 to 8, or to perform the processing method as claimed in any one of claims 9 to 11.

17. A computer program product, comprising a computer program and instructions, characterized in that, The computer program and instructions are executed by a processor using the processing method as described in any one of claims 1 to 8, or using the processing method as described in any one of claims 9 to 11.

Images