Task processing method and device, electronic equipment and storage medium

Abstract

The application provides a task processing method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining M first tuples to be processed; processing the M first tuples by using a Spark task function of a current node to obtain N second tuples; generating a tuple set corresponding to the current node according to the N second tuples and a preset tuple number of a single tuple set, wherein the tuple set comprises at least one virtual tuple in addition to the second tuples; and the M and the N are integers greater than or equal to 1. The method improves the confidentiality in the Spark task execution process.

Description

Technical Field

[0001] This application relates to task processing technology, and more particularly to a task processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Spark is an open-source big data computing engine based on in-memory iteration. In the era of big data, more and more users are choosing to deploy Spark tasks on third-party cloud services. While providing convenient services, cloud service providers must also ensure data security.

[0003] In existing technologies, to ensure data security during Spark task execution, cloud service providers choose to run Spark tasks in a trusted execution environment and encrypt data outside the trusted execution environment to protect data confidentiality. However, if the attacker is an administrator of a third-party cloud service, they can combine prior knowledge to observe the data flow between different nodes to obtain partial information about the user's dataset, thereby completing a side-channel attack and compromising the confidentiality of Spark tasks. Summary of the Invention

[0004] This application provides a task processing method, apparatus, electronic device, and storage medium to solve the problem of poor confidentiality during Spark task execution.

[0005] Firstly, this application provides a task processing method, the method comprising:

[0006] Obtain the M first tuples to be processed;

[0007] The M first tuples are processed using the Spark task function of the current node to obtain N second tuples;

[0008] Based on the N second tuples and the preset number of tuples in a single tuple set, a tuple set corresponding to the current node is generated. The tuple set includes at least one virtual tuple in addition to the second tuples. M and N are both integers greater than or equal to 1.

[0009] Optionally, where N≥2, tuples are represented using key-value pairs, and generating the tuple set corresponding to the current node based on the N second tuples and a preset number of tuples in a single tuple set includes:

[0010] The N second tuples are merged to combine second tuples with the same key into one second tuple, resulting in the merged second tuple;

[0011] Based on the number of the merged second tuples and the preset number of tuples in a single tuple set, at least one virtual tuple is generated.

[0012] Based on the merged second tuple and the at least one virtual tuple, a set of tuples corresponding to the current node is generated, wherein the sum of the number of the merged second tuple and the number of virtual tuples is equal to the preset number of tuples.

[0013] Optionally, generating the set of tuples corresponding to the current node based on the merged second tuple and the at least one virtual tuple includes:

[0014] The aggregated second tuple and the virtual tuple are padded according to the preset maximum padding length, so that the length of the padded second tuple and the padded virtual tuple are both the maximum padding length.

[0015] Based on the filled second tuple and the filled virtual tuple, generate the tuple set corresponding to the current node.

[0016] Optionally, generating the set of tuples corresponding to the current node based on the padded second tuple and the padded virtual tuple includes:

[0017] The padded second tuple and the padded virtual tuple are compressed.

[0018] The compressed second tuple and the compressed virtual tuple are used to construct the tuple set corresponding to the current node.

[0019] Optionally, if the current node is the last node of a Spark task, after generating the set of tuples corresponding to the current node, the method further includes:

[0020] The set of tuples corresponding to the current node is encrypted and stored in the file system.

[0021] Optionally, if the current node is not the last node of the Spark task, after generating the set of tuples corresponding to the current node, the process further includes:

[0022] Perform a hash function operation on the tuples in the tuple set corresponding to the current node to generate the hash value of the tuple set corresponding to the current node;

[0023] Based on the directed acyclic graph information of the Spark task, obtain the identifier of the next execution stage after the current node and the identity information of the next node;

[0024] The hash value, the identifier of the next execution stage after the current node, and the identity information of the next node are embedded into the tuple set corresponding to the current node;

[0025] Encrypt the set of tuples corresponding to the current node;

[0026] Send the encrypted set of tuples corresponding to the current node to the next node.

[0027] Optionally, the current node is not the first node of the Spark task, and obtaining the M first tuples to be processed includes:

[0028] Receive the encrypted set of tuples corresponding to the previous node;

[0029] Decrypt the encrypted tuple set corresponding to the previous node to obtain the tuple set corresponding to the previous node;

[0030] The integrity of the tuple set corresponding to the previous node is verified based on the hash value carried by the tuple set corresponding to the previous node, the identifier of the next execution stage of the previous node, and the identity information of the next node.

[0031] After the integrity verification is passed, the M first tuples to be processed are recovered from the tuple set corresponding to the previous node.

[0032] Optionally, the current node is the first node of the Spark task, and obtaining the M first tuples to be processed includes:

[0033] Receive the M first tuples input by the user.

[0034] Secondly, this application provides a task processing apparatus, the apparatus comprising:

[0035] The acquisition module is used to acquire the M first tuples to be processed;

[0036] The processing module is used to process the M first tuples using the Spark task function of the current node to obtain N second tuples;

[0037] The generation module is used to generate a tuple set corresponding to the current node based on the N second tuples and the preset number of tuples in a single tuple set. The tuple set includes at least one virtual tuple in addition to the second tuples. M and N are both integers greater than or equal to 1.

[0038] Thirdly, this application provides an electronic device, the electronic device comprising: a processor, and a memory communicatively connected to the processor;

[0039] The memory stores computer-executed instructions;

[0040] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0041] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the task processing method as described in any one of the first aspects.

[0042] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0043] The task processing method, apparatus, electronic device, and storage medium provided in this application allow the current node to generate a tuple set corresponding to the current node based on N second tuples obtained by processing M first tuples using the Spark task function of the current node, and a preset number of tuples in a single tuple set. This tuple set includes at least one virtual tuple in addition to the second tuples. Because the tuple set includes virtual tuples, attackers cannot accurately obtain the actual number of tuples in the tuple set. This interferes with attackers' ability to obtain user information, preventing attackers from launching side-channel attacks and compromising the confidentiality of Spark tasks, thereby improving the data security of Spark tasks. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] Figure 1 This is a schematic diagram of a Spark architecture;

[0046] Figure 2 This is a schematic diagram of a Spark task processing flow;

[0047] Figure 3 A flowchart illustrating the first task processing method provided in this application;

[0048] Figure 4 A flowchart illustrating the second task processing method provided in this application;

[0049] Figure 5 A flowchart illustrating the third task processing method provided in this application;

[0050] Figure 6 A flowchart illustrating the fourth task processing method provided in this application;

[0051] Figure 7 A flowchart illustrating the fifth task processing method provided in this application;

[0052] Figure 8 A schematic diagram of the structure of a task processing device provided in this application;

[0053] Figure 9 This is a schematic diagram of the structure of an electronic device 900 provided in this application.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] Spark is an open-source big data computing engine based on in-memory iteration. It supports various data processing tasks, such as batch processing, stream processing, structured query language (SQL) queries, machine learning, and graph computation, by sharing abstract resilient distributed datasets (RDDs) through data sharing. Due to its high speed, high throughput, and distributed scalability, Spark has become the most popular big data computing engine and is widely used in big data systems across various industries.

[0057] Figure 1 This is a schematic diagram of a Spark architecture, such as... Figure 1As shown, the Spark runtime architecture includes a Driver node for each application, a Cluster Manager, Worker Nodes (hereinafter referred to as nodes) that run jobs, and Executors deployed on the worker nodes that are responsible for executing specific tasks. The application mentioned above is the Spark application written by the user. The Spark architecture performs operations on the dataset included in the Spark task by running this application. A Spark task includes the application and the dataset to be processed, which consists of multiple tuples.

[0058] Tuples are a fundamental concept in relational databases. A relation can be understood as a table, and each row in the table (i.e., each record in the database) is a tuple. Tuples can be represented as key-value pairs, where a key in a tuple can correspond to one or more values.

[0059] Figure 2 This is a schematic diagram of a Spark task processing flow, such as... Figure 2 As shown, after a Spark task is submitted to the Spark architecture for execution, the task control node receives the Spark task and creates a Spark Context based on the application to request computing resources, allocate tasks, and monitor performance. The Spark Context includes a Directed Acyclic Graph (DAG). The task control node uses this DAG to divide the application into one or more job tasks, and further divides each job task into one or more execution stages based on whether a shuffle is required. Each execution stage includes one or more tasks.

[0060] Shuffle is a data distribution process across nodes and processes within a cluster in a distributed computing scenario. If a Spark task requires redistribution of the dataset, a shuffle is necessary. The shuffle process includes shuffle write and shuffle read, which reside on different nodes. During Spark task execution, a node can use the shuffle write process to generate an intermediate file from the dataset to be sent to the next node. After obtaining this intermediate file, the next node can use the shuffle read process to extract the data required for the task execution on that node.

[0061] Spark tasks are divided into aggregated Spark tasks and non-aggregated Spark tasks. Aggregated Spark tasks perform aggregation operations during the Shuffle-Write phase, such as the WordCount task. These tasks do not need to retain the original values ​​of tuples in the dataset during the tuple merging process. Non-aggregated Spark tasks do not perform aggregation operations during the Shuffle-Write phase, such as the SortByKey task. These tasks need to retain the original values ​​of tuples in the dataset during the tuple merging process.

[0062] Continue to refer to Figure 2 After the task control node divides each job task into one or more execution stages, it requests resources from the cluster resource manager. This allows the resource manager to allocate nodes for the execution processes running the tasks and to start those processes. Then, the nodes request tasks from the task control node through their execution processes. The task control node sends the application code and dataset for each task to the corresponding node. Subsequently, each node runs the task through its execution process to process the Spark task's dataset, obtains the results, and either sends the results back to the task control node or stores them in a distributed file system.

[0063] In the era of big data, more and more users are choosing to deploy Spark tasks on the Spark architecture within third-party cloud services. While providing convenient services, cloud service providers must also ensure the confidentiality of Spark task data. If data from user-run Spark tasks within cloud services is stolen by attackers, it will not only leak user data but also cause irreparable damage to the reputation of the third-party cloud service provider.

[0064] However, since Spark tasks operate as a black box in cloud services, users cannot know about security issues such as data tampering, malicious operations, or eavesdropping during data transmission, thus posing significant security risks.

[0065] A Trusted Execution Environment (TEE) is a secure area within the host processor. It aims to create a secure execution environment within the central processing unit (CPU) through software or hardware, providing security for programs and data running within it. Current solutions typically place Spark tasks within a TEE and encrypt data outside the TEE to protect data confidentiality.

[0066] However, the above methods cannot guarantee data security during Spark task execution. Attackers can still combine prior knowledge to observe the data flow during the shuffle process between different nodes in each Spark task, as well as the data characteristics of each node's dataset, such as data flow direction, length, and data volume, to obtain partial information about the user's dataset, thereby completing a side-channel attack and compromising the confidentiality of Spark tasks.

[0067] The side-channel attack mentioned here, also known as a hidden channel attack, refers to an attack method in which attackers can obtain private information in user datasets by observing side-channel data, such as the data flow direction mentioned above.

[0068] The inventors considered that if attackers could not know the true data characteristics of the datasets of each node, they would not be able to obtain user information based on the data characteristics of the datasets, and thus would not be able to launch side-channel attacks, thereby compromising the confidentiality of Spark tasks.

[0069] In view of this, this application provides a task processing method that generates virtual tuples during Spark task processing, making it impossible for attackers to accurately know the data characteristics of the dataset, thereby preventing attackers from launching side-channel attacks by obtaining user information and ensuring the confidentiality of Spark tasks.

[0070] The task processing method of this application can be applied to any Spark architecture, which can be deployed locally or in the cloud, and can run in a TEE environment or a non-TEE environment. When running in a TEE environment, the execution security of the task processing method of this application can be further improved.

[0071] Specifically, the task processing method in this application describes how to process a task within a stage of a Spark task execution. The execution entity of this task processing method is the node where the task is deployed, which can be, for example, a computer or a server. This application does not limit whether the node is also used to process other tasks in the current execution stage, or whether it is used to process tasks in other execution stages.

[0072] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0073] Figure 3A flowchart illustrating the first task processing method provided in this application is shown below. Figure 3 As shown, the method includes:

[0074] S101. Obtain the M first tuples to be processed.

[0075] M is an integer greater than or equal to 1. This application does not limit the specific value of M. The first tuple mentioned above can be any tuple to be processed, and this application does not limit the specific data content of the first tuple.

[0076] The way the current node obtains the M first tuples to be processed depends on the execution stage of the task being processed by the current node. For example, if the execution stage is the first execution stage of a Spark task, the current node could receive M first tuples input by the user. Alternatively, the task control node might obtain the M first tuples sent by the user terminal, which are input by the user through the user terminal. The user terminal could be, for example, a computer or mobile phone. Then, the current node obtains the M first tuples input by the user from the task control node.

[0077] If the execution phase is not the first execution phase of a Spark task, the current node can, for example, obtain M first tuples from the previous node. It should be understood that the previous node refers to the node where the tasks included in the previous execution phase were deployed. This previous node can be one node or multiple nodes, depending on the number and deployment method of the tasks included in the previous execution phase; this application does not limit it.

[0078] S102. Use the Spark task function of the current node to process the M first tuples to obtain N second tuples.

[0079] N is an integer greater than or equal to 1. This application does not limit the specific value of N. The aforementioned Spark task function is the application code used to logically process M first tuples. The task deployed on the current node includes the Spark task function obtained from the task control node. This application does not limit the specific form of the Spark task function; those skilled in the art can set it according to the actual situation.

[0080] In this step, the current node uses its Spark task function to process M first tuples to obtain N second tuples.

[0081] S103. Based on N second tuples and the preset number of tuples in a single tuple set, generate a tuple set corresponding to the current node. This tuple set includes at least one virtual tuple in addition to the second tuples.

[0082] The current node can either merge the N second tuples first, and then generate the tuple set corresponding to the current node based on the merged second tuples and the preset number of tuples in a single tuple set; or it can directly generate the tuple set corresponding to the current node based on the N second tuples and the preset number of tuples in a single tuple set without merging the N second tuples.

[0083] The above describes merging N second tuples, which involves combining tuples of the same type from the N tuples into a single tuple. The type of tuple can be determined, for example, by its key; if multiple tuples share the same key, they are considered to belong to the same type. When the current node generates its own set of tuples based on the merged second tuples, the amount of temporary data generated during task processing can be reduced, thus improving the data transfer rate between nodes.

[0084] The preset number of tuples in a single tuple set is related to whether N second tuples are merged. For example, if the current node merges N second tuples, the preset number of tuples in a single tuple set can be greater than the number of tuple types in a Spark task.

[0085] If the current node does not merge the N second tuples, the preset number of tuples in a single tuple set can be greater than or equal to the total number of tuples in the Spark task. Specifically, those skilled in the art can set this according to the actual situation.

[0086] The aforementioned virtual tuple is a tuple containing virtual data. This virtual data is data that does not belong to any of the N second tuples. A virtual tuple can be entirely virtual. For example, when the second tuple is a key-value pair, the virtual tuple can be a preset key-value pair, such as <#, #>. In this case, the virtual tuple does not contain any real data. The tuple set must include at least one of the aforementioned virtual tuples.

[0087] Based on this, this application does not limit whether the current node performs further processing on the second tuple and the aforementioned virtual tuple. For example, the current node may also include virtual tuples generated by padding real tuples with virtual values. For example, the real second tuple may be <1, 3>, and the current node may generate a virtual tuple, such as <1, 3##>, by padding with the virtual value "#". When the tuple set includes multiple virtual tuples, these multiple virtual tuples may be the same, different, partially the same, and partially different; this application does not limit this.

[0088] The length of the dummy tuples in the tuple set is related to whether the current Spark task is an aggregate Spark task, i.e., whether the current Spark task needs to retain the original data. The length of the dummy tuples here refers to the number of integers included in the dummy tuple's value. If the current Spark task is an aggregate Spark task, i.e., the current Spark task does not need to retain the original data, then the length of the dummy tuples in the tuple set can be greater than or equal to the number of tuples of the most common tuple type in the Spark task.

[0089] Optionally, when the current Spark task is an aggregate Spark task, the current node can first compress the second tuple, and then generate the corresponding tuple set for the current node based on the compressed second tuple and the preset number of tuples in a single tuple set. The compression mentioned here refers to shortening the length of each second tuple. This application does not limit the specific compression method used in actual applications; for example, it could be the snappy compression algorithm or the zip compression algorithm. The above method can reduce the amount of data during Spark task execution, improve the transmission efficiency of data exchange between nodes, and improve the performance of Spark task implementation.

[0090] In this embodiment, the current node generates a tuple set corresponding to the current node based on N second tuples obtained by processing M first tuples using the Spark task function of the current node, and a preset number of tuples in a single tuple set. This tuple set includes at least one virtual tuple in addition to the second tuples. Because the tuple set includes virtual tuples, attackers cannot accurately obtain the actual number of tuples in the tuple set. This interferes with attackers' ability to obtain user information, preventing attackers from launching side-channel attacks and compromising the confidentiality of Spark tasks, thus improving the data security of Spark tasks.

[0091] Optionally, if the current node is the last node of the Spark task, i.e., the node in the last execution stage, after generating the tuple set corresponding to the current node, the current node can store the tuple set to the file system for retrieval by other applications or file systems; alternatively, it can send the tuple set to the task control node. Optionally, the current node can also encrypt the tuple set and store it in the file system or send it to the task control node to further improve the security of tuple set transmission and storage.

[0092] Optionally, after the current node generates its corresponding tuple set, it can further embed one or more of the following into the tuple set: the hash value of the current node's corresponding tuple set, the identifier of the next execution stage following the current node, and the identity information of the next node. This allows the next node to verify the integrity of the tuple set based on the aforementioned information. Through these operations, attackers can be prevented from compromising data integrity by discarding data, retransmitting data, or creating fake data, thereby interfering with the execution results of Spark tasks.

[0093] The following describes how, when N≥2 and tuples are represented as key-value pairs, the current node generates the tuple set corresponding to the current node based on N second tuples and the preset number of tuples in a single tuple set, i.e., step S103 in the above embodiment. Figure 4 A flowchart illustrating the second task processing method provided in this application is shown below. Figure 4 As shown, step S103 may include the following steps:

[0094] S201. Merge N second tuples to combine second tuples with the same key into one second tuple, and obtain the merged second tuple.

[0095] In this step, the current node merges N second tuples, which can reduce the amount of temporary data generated during task execution, improve the data transfer efficiency between execution stages, and improve Spark task processing performance.

[0096] In this step, the method for merging the N second tuples is related to whether the Spark task is an aggregation Spark task, i.e., whether the original data needs to be retained. Therefore, in this step, for different situations, there are the following two merging methods:

[0097] Scenario 1: The Spark task is an aggregated Spark task

[0098] In this scenario, the current node can use the key of the second tuple with the same key corresponding to each task as the key of the merged second tuple. The values ​​of the second tuples with the same key are summed to obtain the value of the merged second tuple. The length of each merged second tuple is 1, meaning that each key in the merged second tuple corresponds to only one value. For example, assuming there are two second tuples with the same key, <2, 1> and <2, 2>, the merged second tuple could be <2, 3>.

[0099] Scenario 2: The Spark task is a non-aggregate Spark task

[0100] In this scenario, the current node can use the key of the second tuple with the same key as the key of the merged second tuple, and the value of the second tuple with the same key as the value of the merged second tuple. For example, assuming there are two second tuples with the same key, <2, 1> and <2, 2>, merging these two tuples would result in a merged second tuple of <2, 1_2>. This merging process ensures that the key of the merged second tuple is unique among all merged second tuples.

[0101] S202. Generate at least one virtual tuple based on the number of merged second tuples and the preset number of tuples in a single tuple set.

[0102] In this embodiment, since N second tuples are merged, the preset number of tuples in a single tuple set can be greater than or equal to the number of types of second tuples.

[0103] In this step, the way the current node generates at least one virtual tuple based on the number of merged second tuples and the preset number of tuples in a single tuple set is related to whether the Spark task is an aggregated Spark task or a non-aggregated Spark task. Specifically, there are two cases:

[0104] Scenario 1: The Spark task is an aggregated Spark task

[0105] In this scenario, since the original data does not need to be retained, the length of the merged second tuple is always 1. Attackers cannot analyze the information contained in the dataset based on the length of the second tuple, and therefore cannot launch a side-channel attack. At this point, the current node can generate one or more virtual tuples of length 1, so that the preset number of tuples in a single tuple set equals the sum of the number of aggregated second tuples and the number of virtual tuples.

[0106] For example, if the number of the aggregated second tuples is 2 and the preset number of tuples in a single tuple set is 5, then 3 virtual tuples of length 1 are generated.

[0107] Scenario 2: The Spark task is a non-aggregate Spark task

[0108] In this scenario, since the merged second tuple retains the original data, and the number of second tuples of the same type varies, the number of values ​​corresponding to each key in the merged second tuple may differ, meaning the length of the merged second tuple may vary. Therefore, at least one virtual tuple can be generated in this step using the following implementation method.

[0109] In implementation method 1, the current node can generate one or more virtual tuples such that the preset number of tuples in a single tuple set is equal to the sum of the number of aggregated second tuples and the number of virtual tuples. In this case, the application does not limit the length of the virtual tuple; for example, it can be 0 or 1.

[0110] Implementation Method 2: The current node can generate at least one virtual tuple based on the number of aggregated second tuples, the preset number of tuples in a single tuple set, and the preset maximum padding length.

[0111] For example, the current node can generate at least one virtual tuple based on the aggregated second tuple, a preset number of tuples in a single tuple set, and a preset maximum padding length. In this case, the preset maximum padding length is greater than or equal to the length of the longest aggregated second tuple. For example, the preset maximum padding length could be (V... length +1)×Z-1, where V length Z is the length of the longest tuple in the first tuple, and Z is the number of times the first tuple appears in the Spark task.

[0112] For example, the current node can generate virtual tuples with a preset maximum padding length, such that the preset number of tuples in a single tuple set is equal to the sum of the number of aggregated second tuples and the number of virtual tuples.

[0113] Alternatively, the current node can first compress the aggregated second tuple, and then generate at least one virtual tuple based on the compressed aggregated second tuple, the preset number of tuples in a single tuple set, and the preset maximum padding length. As mentioned above, the compression here refers to compressing the length of each second tuple.

[0114] In this case, the preset maximum padding length can be greater than or equal to the length of the longest compressed second tuple. This avoids the situation where, when the data size of a Spark task is large, the preset maximum padding length is too large, resulting in an excessively large amount of virtual tuple data and consuming too many computing resources, thereby improving the efficiency of Spark task execution and data transfer.

[0115] S203. Generate a set of tuples corresponding to the current node based on the merged second tuple and at least one virtual tuple.

[0116] In this step, the way the current node generates at least one virtual tuple based on the number of aggregated second tuples and the preset number of tuples in a single tuple set is related to the way virtual tuples are generated in step S202, and there are two specific cases:

[0117] Scenario 1: The Spark task is an aggregated Spark task

[0118] In this scenario, the current node can directly utilize aggregated second tuples of length 1, along with at least one virtual tuple, to form the tuple set corresponding to the current node. The sum of the number of aggregated second tuples and the number of virtual tuples in the tuple set equals the preset number of tuples.

[0119] Scenario 2: The Spark task is a non-aggregate Spark task

[0120] In this case, depending on the implementation method of step S202, this step can be implemented in different ways.

[0121] For example, if implementation method 1 is used in step S202, the current node can pad the aggregated second tuple and virtual tuple according to the preset maximum padding length, so that the lengths of the padded second tuple and the padded virtual tuple are both the maximum padding length. Then, the current node generates a tuple set corresponding to the current node based on the padded second tuple and the padded virtual tuple. The preset number of tuples in a single tuple set is equal to the sum of the number of padded second tuples and the number of padded virtual tuples in the tuple set.

[0122] For example, if the aggregated second tuple is <1, 2_3>, and the length of the second tuple is 3, and the preset maximum padding length is 5, then the padded second tuple could be, for example, <1, 2_3##>, where # represents the padded virtual data. Alternatively, if the virtual tuple is <#, #>, then the padded virtual tuple would be <#, #####>.

[0123] Optionally, after obtaining the padded second tuple and the padded virtual tuple, the current node can directly use these tuples to construct its own tuple set. Alternatively, the padded second tuple and the padded virtual tuple can be compressed; then, the compressed second tuple and the compressed virtual tuple can be used to construct the current node's tuple set, thereby reducing the data size of the tuple set, improving the transmission rate of the tuple set between nodes, and enhancing the overall execution performance of the Spark task.

[0124] If step S202 uses implementation method 2 to generate virtual tuples and does not compress the aggregated second tuples, then in this step, the current node can pad the aggregated second tuples according to the preset maximum padding length, so that the length of the padded second tuples is the preset maximum padding length. Then, the current node uses the padded second tuples and at least one virtual tuple generated in step S202 to form the tuple set corresponding to the current node. The sum of the number of padded second tuples and the number of virtual tuples in the tuple set is equal to the preset number of tuples.

[0125] If the current node compressed the aggregated second tuple in step 2, then in this step, the current node can fill the compressed aggregated second tuple with a preset maximum fill length so that the length of the filled second tuple is the preset maximum fill length. Then, the current node uses the filled second tuples and at least one virtual tuple generated in step S202 to construct the tuple set corresponding to the current node. The sum of the number of filled second tuples and the number of virtual tuples in the tuple set equals the preset number of tuples.

[0126] In this embodiment, the current node first aggregates the N second tuples to obtain aggregated second tuples. Then, based on the number of aggregated second tuples and the preset number of tuples in a single tuple set, at least one virtual tuple is generated, and a tuple set corresponding to the current node is generated accordingly. The sum of the number of aggregated second tuples and the number of virtual tuples equals the preset number of tuples. By aggregating the N second tuples, the current node can reduce the temporary data generated during Spark task execution, reduce the amount of data transfer between nodes, and thus improve the data transfer efficiency between nodes.

[0127] By making the sum of the number of aggregated second tuples and the number of virtual tuples equal to the preset number of tuples, and generating a tuple set accordingly, the true number of tuples in the tuple set can be hidden. This prevents attackers from analyzing the number of tuples in the tuple set to obtain user information and launch side-channel attacks, thereby improving the confidentiality of Spark task execution and ensuring data security.

[0128] Figure 5 A flowchart illustrating the third task processing method provided in this application is shown below. Figure 5 As shown, optionally, if the current node is not the last node of the Spark task, after generating the set of tuples corresponding to the current node, the following steps may also be included. It should be understood that the last node referred to here is the node used to execute the task in the last execution phase.

[0129] S301. Perform a hash function operation on the tuples in the tuple set corresponding to the current node to generate the hash value of the tuple set corresponding to the current node.

[0130] This application does not limit the specific form of the hash function described above; those skilled in the art can determine it according to the actual situation. When the tuples in the tuple set are determined, the hash value generated accordingly is unique. In this step, the current node performs a hash function operation on the tuples in its corresponding tuple set to generate a hash value for the current node's corresponding tuple set, which serves as a verification identifier for the integrity of the current node's corresponding tuple set. Subsequently, if it is necessary to verify whether the tuple set corresponding to the current node is complete before sending it to the next node, integrity verification can be performed based on this hash value to prevent attackers from interfering with the smooth execution of Spark tasks by using the integrity of the tuple set.

[0131] S302. Based on the directed acyclic graph information of the Spark task, obtain the identifier of the next execution stage after the current node and the identity information of the next node.

[0132] The aforementioned "next node" refers to the node where the tasks in the next execution phase are deployed. It should be understood that the "next node" can be one or more, depending on the number of tasks included in the next execution phase. This application does not limit the specific form of the identifier of the next execution phase and the identity information of the next node. For example, the identifier of the next execution phase could be, for instance, the number of the next execution phase, and the identity information of the next node could be, for instance, the number of the next node.

[0133] Since the directed acyclic graph (DAG) information of a Spark task includes the identifiers of all execution stages and the identity information of all nodes, in this step, the current node can obtain the identifier of the next execution stage and the identity information of the next node based on the DAG information of the Spark task.

[0134] It should be understood that the execution order of the above steps S301 and S302 is not important.

[0135] S303. Embed the hash value, the identifier of the next execution stage after the current node, and the identity information of the next node into the tuple set corresponding to the current node.

[0136] For example, the current node can add the hash value to a tuple set, establishing a mapping relationship between the tuple set and the hash value, so as to embed the hash value into the tuple set corresponding to the current node. Establishing this mapping relationship between the tuple set and the hash value can be achieved by adding the hash value as an attribute parameter of the tuple set.

[0137] For example, the current node can embed the identifier of the next execution stage after the current node into a tuple set, establishing a mapping relationship between each tuple in the tuple set and the identifier of the next execution stage. For example, the identifier of the next execution stage can be added to the tuple set as an attribute parameter of each tuple.

[0138] For example, the current node embeds the identity information of the next node into the tuples in the set of tuples that need to be processed by the next node. This establishes a mapping relationship between the tuples in the set that need to be processed by the next node and the corresponding identity information of the next node. This mapping relationship could, for example, involve generating a mapping table between each padded virtual tuple and its padded second tuple and the identifier of the next execution stage. The current node can then use this mapping table, along with the padded virtual tuple and its padded second tuple, to find the identity information of the corresponding next node.

[0139] In this step, the current node embeds the hash value, the identifier of the next execution stage after the current node, and the identity information of the next node into the tuple set corresponding to the current node. This allows the next node to verify the complete row of the tuple set based on the above content, preventing attackers from compromising the integrity of the Spark task by discarding data, resending data, or constructing fake data, thus preventing users from obtaining the correct execution results.

[0140] S304. Encrypt the set of tuples corresponding to the current node.

[0141] This application does not limit the encryption method. In this step, the current node encrypts the set of tuples corresponding to the current node to further enhance data security. It should be understood that in some embodiments, the current node may not encrypt the set of tuples corresponding to the current node.

[0142] This step can further improve the confidentiality and security of data transmission between nodes.

[0143] S305. Send the encrypted set of tuples corresponding to the current node to the next node.

[0144] In this step, the current node sends the encrypted tuple set corresponding to the current node to the next node, so that the next node can read the encrypted tuple set and process it using the Spark task function corresponding to the next node.

[0145] In this embodiment, the current node embeds the hash value, the identifier of the next execution stage after the current node, and the identity information of the next node into the tuple set corresponding to the current node. This allows the next node to verify the integrity of the tuple set based on the above content, preventing attackers from destroying the integrity of Spark tasks by discarding data, resending data, or constructing fake data, thus preventing users from obtaining the correct execution results and further improving the security of Spark tasks.

[0146] The following describes how to obtain the M first tuples to be processed when the current node is not the first node of a Spark task, i.e., step S101 in the above embodiment. It should be understood that the first node referred to here is the node used to execute the task in the first execution phase. This embodiment is an example of Spark task processing using the task processing method of the previous embodiment on the node preceding the current node. Figure 6 A flowchart illustrating the fourth task processing method provided in this application is shown below. Figure 6 As shown, step S101 may include:

[0147] S401: Receive the encrypted set of tuples corresponding to the previous node.

[0148] It should be understood that the previous node mentioned here can be one node or multiple nodes, depending on the number of tasks included in the previous execution phase.

[0149] Based on the above implementation, if the previous node encrypts the encrypted tuple set corresponding to the previous node and sends it to the current node, then the current node receives the encrypted tuple set corresponding to the previous node.

[0150] S402. Decrypt the encrypted tuple set corresponding to the previous node to obtain the tuple set corresponding to the previous node.

[0151] In this step, the current node may pre-store a key for the encrypted tuple set. The current node uses this key to decrypt the encrypted tuple set corresponding to the previous node, and obtains the tuple set corresponding to the previous node.

[0152] S403. Perform integrity verification on the tuple set corresponding to the previous node based on the hash value carried by the tuple set corresponding to the previous node, the identifier of the next execution stage of the previous node, and the identity information of the next node.

[0153] In this step, the current node performs integrity verification on the tuple set corresponding to the previous node to prevent attackers from interfering with the smooth execution of Spark tasks by compromising data integrity.

[0154] One possible implementation is that the current node first performs a hash function operation on the tuples belonging to the same tuple set using the hash function used by the previous node to obtain the hash value corresponding to that tuple set. Then, it compares this hash value with the hash values ​​embedded in the tuple set. If the two hash values ​​are not equal, it indicates that the tuples belonging to the same tuple set obtained by the current node are not the same as the tuple set sent by the previous node, suggesting the possibility that an attacker may have corrupted the tuple set, and the integrity verification fails.

[0155] If the two hash values ​​are equal, it means that the tuples obtained by the current node that belong to the same tuple set are the same as the tuples included in the tuple set sent by the previous node. In this case, the current node will then perform integrity verification through the identity information of the next node.

[0156] For example, the current node stores its own identity information. The current node determines whether a tuple in the tuple set contains its own identity information. If it does, the tuple is retained. If it does not, i.e., T... identity ≠T id (T identity T represents the identity information of the node embedded in this tuple. id If the tuple is the identity information of the current Task, it indicates that the tuple does not need to be processed by the current node, so the current node removes the tuple from the current node.

[0157] Subsequently, the current node can verify whether the tuples in its received tuple set are in the correct execution phase. For example, the current node stores an identifier for the current execution phase. The current node determines whether the identifier of the next execution phase of the previous node embedded in the tuple set matches the identifier of the current execution phase. If they match, it indicates that the tuples in the current node do not belong to other execution phases, and the integrity verification passes; if they do not match, i.e., T... stage ≠T current (T stage T serves as the identifier for the next execution phase of the previous node. current If the current execution phase is not specified, it indicates that there are tuples in the current node that should belong to other execution phases. In other words, an attacker could send tuples that should have been executed by other execution phases to the current execution phase, causing the integrity verification to fail.

[0158] Optionally, if integrity verification fails, the current node can terminate the execution of this Spark task and output a message indicating that integrity may have been compromised.

[0159] S404. After the integrity verification is passed, recover the M first tuples to be processed from the tuple set corresponding to the previous node.

[0160] In this step, the method by which the current node recovers the M first tuples to be processed from the tuple set corresponding to the previous node is related to whether the Spark task is an aggregate Spark task or a non-aggregate Spark task. For example, it can include the following two cases:

[0161] Case 1: The Spark task is an aggregated Spark task.

[0162] In this scenario, the current node identifies whether a tuple in the tuple set corresponding to the previous node is a virtual tuple. For example, the current node might store the form of a virtual tuple. The current node compares the form of the stored virtual tuple with that of a tuple in the tuple set. If they match, the tuple is considered a virtual tuple. If the tuple is determined to be a virtual tuple, the current node deletes it from its current node list.

[0163] Since if the Spark task is an aggregated Spark task, the previous node does not need to fill the aggregated second tuple and dummy tuple according to the preset maximum tuple length. Therefore, the current node can restore the M first tuples to be processed by deleting the dummy tuple from the current node.

[0164] Scenario 2: The Spark task is a non-aggregate Spark task.

[0165] In this case, after the current node deletes the virtual tuples that do not contain real data from the current node, the current node can identify whether there is any virtual data in the remaining tuples based on the form of the pre-stored filled virtual data. If so, the data is removed.

[0166] Then, the current node restores the original data, that is, it restores the aggregated tuples to their unaggregated state. For example, if the aggregated second tuple is <1, 2_3>, then the tuple is restored to <1, 2>, <1, 3>. In this way, the M first tuples to be processed can be obtained.

[0167] In this embodiment, the current node performs integrity verification on the tuple set corresponding to the previous node based on the hash value carried by the tuple set corresponding to the previous node, the identifier of the next execution stage of the previous node, and the identity information of the next node. If the integrity verification passes, the M first tuples to be processed are recovered from the tuple set corresponding to the previous node.

[0168] By verifying the hash value, the current node can determine whether the previous node sent a complete set of tuples to it, preventing attackers from compromising data integrity by discarding them. By verifying the identifier of the next execution stage of the previous node, the current node can determine if any tuples originally intended for processing in other execution stages were sent to it, preventing attackers from compromising data integrity in this way. By verifying the identity information carried in the previous node's tuple set, tuples deployed for processing on the current node can be removed, avoiding errors in the execution result due to processing tuples originally intended for other nodes. Overall, these methods ensure data integrity during Spark task processing, further guaranteeing the accuracy of Spark task execution results.

[0169] The task processing method provided in this application will be described below with reference to specific embodiments. Figure 7 This is a flowchart illustrating the fifth task processing method provided in this application. The diagram illustrates an example of a Spark task comprising two execution stages. The first execution stage includes three tasks, and the second execution stage includes two tasks. Content in this embodiment that is the same as or similar to the above embodiments can be referred to the above embodiments, and will not be repeated here. Figure 7 As shown, this embodiment may include, for example, the following steps:

[0170] Step 1: The nodes deployed by the tasks included in the first execution phase (hereinafter referred to as the first node) obtain the corresponding encrypted datasets.

[0171] Reference Figure 7 In this step, the nodes deployed for the three tasks included in the first execution phase each obtain their corresponding encrypted datasets. Specifically, the node deployed for Task 1 obtains four first tuples, the node deployed for Task 2 obtains three first tuples, and the node deployed for Task 3 obtains one first tuple.

[0172] Step 2: The first node processes the first tuple in the acquired encrypted dataset using the corresponding Spark task function to obtain the corresponding processed second tuple.

[0173] Step 3: The first node performs a merging process on the corresponding second tuples to merge second tuples with the same key into a single second tuple, resulting in the merged second tuple.

[0174] As described in the above embodiments, different merging methods are used to merge the second tuple for different Spark task types.

[0175] For example, continue to refer to Figure 7When a Spark task is a non-aggregate Spark task, the nodes deployed by the tasks included in the first execution phase will use the key of the second tuple with the same key as the key of the merged second tuple, and use the value of the second tuple with the same key as the value of the merged second tuple, thus obtaining the merged second tuple.

[0176] For example, the node deployed in Task 1 will merge <2,1>, <2,2>, and <2,3> in the second tuple into <2,1_2_3>.

[0177] Step 4: The first node generates at least one virtual tuple based on the number of merged second tuples and the preset number of tuples in a single tuple set.

[0178] For example, refer to Figure 7 The preset number of tuples in a single tuple set is 4. The number of merged second tuples corresponding to the node deployed in Task 1 is 2. Therefore, this node generates 2 virtual tuples.

[0179] Step 5: The first node generates a corresponding set of tuples based on the merged second tuple and at least one virtual tuple. The sum of the number of merged second tuples and the number of virtual tuples for each task is equal to the preset number of tuples.

[0180] For example, continue to refer to Figure 7 The node deployed in Task 2 fills the merged second tuple and virtual tuple with padding, ensuring that the lengths of both the filled second tuple and the filled virtual tuple are the preset maximum padding length of 5. Then, the node generates a corresponding set of tuples based on the filled second tuple and the filled virtual tuple.

[0181] Step 6: The first node obtains the hash value of the corresponding tuple set, the identifier of the next execution stage, and the identity information of the next node.

[0182] Step 7: The first node embeds the corresponding hash value, the identifier of the next execution stage, and the identity information of the next node into the corresponding tuple set.

[0183] Step 8: Encrypt the set of tuples corresponding to the first node.

[0184] Step 9: The first node sends the corresponding encrypted tuple set to the next node.

[0185] Correspondingly, the nodes deployed by the tasks included in the second execution phase (hereinafter referred to as the second nodes) each receive the encrypted tuple set corresponding to the previous node.

[0186] Step 10: The second node decrypts the encrypted tuple set corresponding to the previous node to obtain the tuple set corresponding to the previous node.

[0187] Step 11: The second node performs integrity verification on the tuple set corresponding to the previous node based on the hash value carried by the tuple set corresponding to the previous node, the identifier of the next execution stage of the previous node, and the identity information of the next node.

[0188] Step 12: After the integrity verification is passed, the second node recovers the second tuple to be processed from the tuple set corresponding to the previous node.

[0189] For example, continue to refer to Figure 7 When the Spark task is a non-aggregate Spark task, the second node removes the virtual tuple from the tuple set corresponding to the previous node and restores the combined second tuple to its state before combination, thus obtaining the second tuple to be processed.

[0190] Step 13: The second node processes the second tuple to be processed using the corresponding Spark task function to obtain the corresponding processed third tuple.

[0191] Step 14: The second node performs merging processing on the corresponding third tuples to merge third tuples with the same key into a single third tuple, resulting in the merged third tuple.

[0192] Step 15: The second node generates at least one virtual tuple based on the number of merged third tuples and the preset number of tuples in a single tuple set.

[0193] Step 16: The second node generates corresponding tuple sets based on the merged third tuple and at least one virtual tuple, wherein the sum of the number of merged third tuples and the number of virtual tuples for each task is equal to the preset number of tuples.

[0194] Step 17: The second node encrypts the corresponding set of tuples and stores it in the file system.

[0195] The task processing method provided in this embodiment can both add virtual tuples to the tuple set transmitted between nodes to prevent attackers from obtaining user information based on prior knowledge, data flow, and data characteristics, thereby launching side-channel attacks; and embed hash values, next-stage identifiers, and the identity information of the next node into the tuple set to verify the integrity of the dataset in the tuple set, preventing users from compromising data integrity by discarding data, retransmitting data, or creating fake data, thus interfering with the execution results of Spark tasks. The method of this application improves the confidentiality of Spark tasks from multiple aspects, including resisting side-channel attacks and ensuring data integrity.

[0196] Figure 8 A schematic diagram of the structure of a task processing device provided in this application is shown below. Figure 8 As shown, the task processing device includes: an acquisition module 11, a processing module 12, and a generation module 13. Optionally, the device may also include one or more of the following modules: a storage module 14 and a verification module 15.

[0197] Module 11 is used to obtain the M first tuples to be processed;

[0198] Processing module 12 is used to process the M first tuples using the Spark task function of the current node to obtain N second tuples;

[0199] The generation module 13 is used to generate a tuple set corresponding to the current node based on the N second tuples and the preset number of tuples in a single tuple set. The tuple set includes at least one virtual tuple in addition to the second tuples. M and N are both integers greater than or equal to 1.

[0200] One possible implementation method is as follows: N ≥ 2, tuples are represented using key-value pairs, and generation module 13 is specifically used to merge the N second tuples to combine second tuples with the same key into one second tuple, resulting in a merged second tuple; based on the number of merged second tuples and a preset number of tuples in a single tuple set, at least one virtual tuple is generated; based on the merged second tuples and the at least one virtual tuple, a tuple set corresponding to the current node is generated, wherein the sum of the number of merged second tuples and the number of virtual tuples is equal to the preset number of tuples.

[0201] For example, the generation module 13 is specifically used to fill the aggregated second tuple and the virtual tuple according to the preset maximum fill length, so that the length of the filled second tuple and the filled virtual tuple is the maximum fill length; and to generate the tuple set corresponding to the current node according to the filled second tuple and the filled virtual tuple.

[0202] For example, the generation module 13 is specifically used to compress the filled second tuple and the filled virtual tuple; and to use the compressed second tuple and the compressed virtual tuple to form the tuple set corresponding to the current node.

[0203] One possible implementation is that if the current node is the last node of the Spark task, the storage module 14 is used to encrypt the tuple set corresponding to the current node and store it in the file system after the generation module 13 generates the tuple set corresponding to the current node.

[0204] One possible implementation is as follows: If the current node is not the last node of the Spark task, the verification module 15 is used to perform a hash function operation on the tuples in the tuple set corresponding to the current node after the generation module 13 generates the tuple set corresponding to the current node, generating a hash value of the tuple set corresponding to the current node; based on the directed acyclic graph information of the Spark task, obtain the identifier of the next execution stage after the current node and the identity information of the next node; embed the hash value, the identifier of the next execution stage after the current node, and the identity information of the next node into the tuple set corresponding to the current node; encrypt the tuple set corresponding to the current node; and send the encrypted tuple set corresponding to the current node to the next node.

[0205] In one possible implementation of the above method, where the current node is not the first node of the Spark task, the acquisition module 11 is specifically used to receive the encrypted tuple set corresponding to the previous node; decrypt the encrypted tuple set corresponding to the previous node to obtain the tuple set corresponding to the previous node; perform integrity verification on the tuple set corresponding to the previous node based on the hash value carried by the tuple set corresponding to the previous node, the identifier of the next execution stage of the previous node, and the identity information of the next node; after the integrity verification is passed, recover the M first tuples to be processed from the tuple set corresponding to the previous node.

[0206] In one possible implementation, the current node is the first node of the Spark task, and the acquisition module 11 is specifically used to receive the M first tuples input by the user.

[0207] The task processing apparatus provided in this application embodiment can execute the task processing method in the above method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here. It should be noted that the above... Figure 8 The division of modules shown is merely illustrative. This application does not limit the division of modules or the naming of modules.

[0208] Figure 9 This is a schematic diagram of the structure of an electronic device 900 provided in this application. Figure 9 As shown, the electronic device 900 may include at least one processor 901 and a memory 902.

[0209] The memory 902 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0210] The memory 902 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0211] The processor 901 is used to execute computer execution instructions stored in the memory 902 to implement the task processing method described in the foregoing method embodiments. The processor 901 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0212] The electronic device 900 may also include a communication interface 903, through which it can communicate and interact with external devices, such as user terminals (e.g., computers, tablets). In specific implementations, if the communication interface 903, memory 902, and processor 901 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0213] Optionally, in a specific implementation, if the communication interface 903, memory 902, and processor 901 are integrated on a single chip, then the communication interface 903, memory 902, and processor 901 can communicate through an internal interface.

[0214] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used in the task processing method described in the above embodiments.

[0215] This application also provides a program product including executable instructions stored in a readable storage medium. At least one processor of an electronic device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the electronic device to perform the task processing methods provided in the various embodiments described above.

[0216] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0217] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A task processing method, characterized in that, The method includes: Obtain the M first tuples to be processed; The M first tuples are processed using the Spark task function of the current node to obtain N second tuples; wherein N≥2, and the tuples are represented by key-value pairs; Based on the N second tuples and the preset number of tuples in a single tuple set, a tuple set corresponding to the current node is generated. The tuple set includes at least one virtual tuple in addition to the second tuples. M and N are both integers greater than or equal to 1. The step of generating the tuple set corresponding to the current node based on the N second tuples and a preset number of tuples in a single tuple set includes: The N second tuples are merged to combine second tuples with the same key into a single second tuple, resulting in a merged second tuple. The merging process includes: when the Spark task is an aggregated Spark task, using the key of the second tuples with the same key as the key of the merged second tuple, summing the values ​​of the second tuples with the same key, and using the sum as the value of the merged second tuple; the length of the merged second tuple is always 1. When the Spark task is a non-aggregated Spark task, using the key of the second tuples with the same key as the key of the merged second tuple, and using the values ​​of the second tuples with the same key as the value of the merged second tuple. Based on the number of the merged second tuples and the preset number of tuples in a single tuple set, at least one virtual tuple is generated. The merged second tuple and the virtual tuple are padded according to the preset maximum padding length, so that the length of the padded second tuple and the padded virtual tuple are both the maximum padding length. Based on the filled second tuple and the filled virtual tuple, a set of tuples corresponding to the current node is generated, wherein the sum of the number of the merged second tuple and the number of virtual tuples is equal to the preset number of tuples.

2. The method according to claim 1, characterized in that, The step of generating the tuple set corresponding to the current node based on the padded second tuple and the padded virtual tuple includes: The padded second tuple and the padded virtual tuple are compressed. The compressed second tuple and the compressed virtual tuple are used to construct the tuple set corresponding to the current node.

3. The method according to any one of claims 1-2, characterized in that, If the current node is the last node of the Spark task, after generating the set of tuples corresponding to the current node, the method further includes: The set of tuples corresponding to the current node is encrypted and stored in the file system.

4. The method according to any one of claims 1-2, characterized in that, If the current node is not the last node of the Spark task, after generating the set of tuples corresponding to the current node, the process further includes: Perform a hash function operation on the tuples in the tuple set corresponding to the current node to generate the hash value of the tuple set corresponding to the current node; Based on the directed acyclic graph information of the Spark task, obtain the identifier of the next execution stage after the current node and the identity information of the next node; The hash value, the identifier of the next execution stage after the current node, and the identity information of the next node are embedded into the tuple set corresponding to the current node; Encrypt the set of tuples corresponding to the current node; Send the encrypted set of tuples corresponding to the current node to the next node.

5. The method according to claim 4, characterized in that, The current node is not the first node of the Spark task, and the step of obtaining the M first tuples to be processed includes: Receive the encrypted set of tuples corresponding to the previous node; Decrypt the encrypted tuple set corresponding to the previous node to obtain the tuple set corresponding to the previous node; The integrity of the tuple set corresponding to the previous node is verified based on the hash value carried by the tuple set corresponding to the previous node, the identifier of the next execution stage of the previous node, and the identity information of the next node. After the integrity verification is passed, the M first tuples to be processed are recovered from the tuple set corresponding to the previous node.

6. The method according to any one of claims 1-2, characterized in that, The current node is the first node of the Spark task, and obtaining the M first tuples to be processed includes: Receive the M first tuples input by the user.

7. A task processing device, characterized in that, The device includes: The acquisition module is used to acquire the M first tuples to be processed; The processing module is used to process the M first tuples using the Spark task function of the current node to obtain N second tuples; wherein N≥2, and the tuples are represented by key-value pairs; The generation module is used to generate a tuple set corresponding to the current node based on the N second tuples and a preset number of tuples in a single tuple set. The tuple set includes at least one virtual tuple in addition to the second tuples. M and N are both integers greater than or equal to 1. The generation module is specifically used to merge the N second tuples to combine second tuples with the same key into a single second tuple, resulting in a merged second tuple. The merging process includes: when the Spark task is an aggregated Spark task, using the key of the second tuples with the same key as the key of the merged second tuple, and summing the values ​​of the second tuples with the same key to obtain the value of the merged second tuple; the length of the merged second tuple is always 1. When the Spark task is a non-aggregated Spark task, using the key of the second tuples with the same key as the key of the merged second tuple, and summing the values ​​of the second tuples with the same key to obtain the value of the merged second tuple. Based on the number of the merged second tuples and the preset number of tuples in a single tuple set, at least one virtual tuple is generated. The merged second tuple and the virtual tuple are padded according to the preset maximum padding length, so that the length of the padded second tuple and the padded virtual tuple are both the maximum padding length. Based on the filled second tuple and the filled virtual tuple, a set of tuples corresponding to the current node is generated, wherein the sum of the number of the merged second tuple and the number of virtual tuples is equal to the preset number of tuples.

8. An electronic device, characterized in that, The electronic device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the task processing method as described in any one of claims 1 to 6.

Images