Processing method of data processing platform, computer device and storage apparatus

Abstract

The application discloses a processing method of a data processing platform, a computer device and a storage device. The method comprises the following steps: receiving a task processing request for to-be-processed data; determining a management and control execution node satisfying the task processing request; performing affinity scheduling on a first task for executing the task processing request based on the management and control execution node, so as to allocate a first execution node; executing the first task on the to-be-processed data by using the first execution node, and obtaining intermediate processing data; performing affinity scheduling on a second task for executing the task processing request based on the management and control execution node, so as to allocate a second execution node; and executing the second task on the intermediate processing data by using the second execution node, and obtaining a processing result. The above scheme can improve the execution efficiency of the task.

Description

Technical Field

[0001] This application relates to the field of data processing platform technology, and in particular to a data processing platform processing method, computer equipment, and storage device. Background Technology

[0002] With the continuous development of information science, data has entered an era of massive growth, and there are more and more distributed computing platforms for data processing and analysis.

[0003] Existing big data processing platforms all provide data stream processing models, enabling functions such as data processing, task scheduling, data caching, and data communication. Distributed data processing systems, such as MapReduce and Spark, experience increased network transmission pressure when dealing with large amounts of computation, as MapReduce requires the transmission of numerous intermediate results over the network. This can reduce the efficiency of the data processing platform in handling tasks. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a data processing platform, a computer device, and a storage device that can improve the efficiency of task execution.

[0005] To address the aforementioned problems, the first aspect of this application provides a data processing platform processing method, comprising: receiving a task processing request for data to be processed; determining a control execution node that satisfies the task processing request; performing affinity scheduling on a first task of executing the task processing request based on the control execution node to allocate a first execution node; using the first execution node to execute the first task on the data to be processed to obtain intermediate processed data; performing affinity scheduling on a second task of executing the task processing request based on the control execution node to allocate a second execution node; and using the second execution node to execute a second task on the intermediate processed data to obtain a processing result.

[0006] To address the aforementioned problems, a second aspect of this application provides a computer device comprising a memory and a processor coupled to each other, wherein the memory stores program data and the processor executes the program data to implement any step of the processing method of the aforementioned data processing platform.

[0007] To address the aforementioned problems, a third aspect of this application provides a storage device that stores program data executable by a processor, the program data being used to implement any step of the processing method of the aforementioned data processing platform.

[0008] The above scheme receives task processing requests for data to be processed and determines the control execution node that satisfies the request. Based on the control execution node, it performs affinity scheduling on the first task to execute the task processing request, and allocates it to the first execution node. The first execution node is used to execute the first task on the data to be processed, obtaining intermediate processing data. Based on the control execution node, it performs affinity scheduling on the second task to execute the task processing request, and allocates it to the second execution node. The second execution node is used to execute the second task on the intermediate processing data, obtaining the processing result. Because the first and second tasks are scheduled on the control execution node when they are satisfied, they can be executed on the same node, thereby reducing the transmission of intermediate processing data over the network and reducing disk read / write operations on intermediate processing data, thus improving task execution efficiency. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0010] Figure 1 This is a flowchart illustrating an embodiment of the data processing method of the data processing platform of this application;

[0011] Figure 2 This application Figure 1 A flowchart illustrating an embodiment of step S12;

[0012] Figure 3 This application Figure 1 A flowchart illustrating an embodiment of step S13;

[0013] Figure 4 This application Figure 1 A flowchart illustrating an embodiment of step S15;

[0014] Figure 5 This application Figure 1 A flowchart illustrating an embodiment of step S16;

[0015] Figure 6 This is a schematic diagram of the structure of a processing device of the data processing platform of this application;

[0016] Figure 7 This is a schematic diagram of the structure of an embodiment of the computer device of this application;

[0017] Figure 8 This is a schematic diagram of the structure of an embodiment of the storage device of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0020] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Through long-term research, the inventors of this application have discovered that, taking the data processing platform MapReduce as an example, MapReduce is a computing model, framework, and platform for parallel processing of big data. It is used for parallel computation of large-scale datasets (greater than 1TB) and can be divided into two stages: Map and Reduce. MapReduce can abstract the complex computational process of data processing into multiple sets of Map and Reduce steps. Map and Reduce are each executed by multiple mapper and reducer functions. Different mappers and reducers are allocated to different computing nodes, realizing efficient distributed computing processing on large computing clusters.

[0022] When MapReduce processes data, after the Map task is completed, the intermediate processing data is stored in the local storage space of the compute node. When the Reduce task is executed, the compute node at this stage copies the intermediate processing data of each Map task to the local storage space of the node via HTTP (Hypertext Transfer Protocol), and then executes the Reduce task to perform calculations on the intermediate processing data. When the amount of data being processed is large, it will be limited by the network bandwidth and disk I / O (input / output) performance of the environment, thereby reducing the data processing efficiency of the data processing platform.

[0023] To address the aforementioned problems, this application provides the following embodiments, which are described in detail below.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data processing method of the data processing platform of this application. The method may include the following steps:

[0025] S11: Receive task processing requests for data to be processed.

[0026] The client can send a task processing request for the data to be processed to the data processing platform. When the data processing platform receives the task processing request, it can process the data to be processed.

[0027] In some implementations, the data processing platform may include a programming model platform for data processing or a parallel computing model. The data processing platform may divide the data processing process into a first task and a second task, such as MapReduce, Spark, etc., which may include Map tasks and Reduce tasks. The data processing platform of this application may also be other parallel computing data processing platforms, and this application does not limit it.

[0028] In some implementations, a task processing request may include multiple tasks computed in parallel, such as a first task and a second task. The first task includes mapping processing, and the second task includes reduction computation. For example, the first task is a Map task, and the second task is a Reduce task. The first and second tasks in this application may also be other parallel processing tasks, and this application does not impose any limitations on them.

[0029] S12: Determine the control execution node that satisfies the task processing request.

[0030] The client can send a task processing request for the data to be processed to the data processing platform. After receiving the task processing request from the client, the data processing platform can determine the control execution node that satisfies the task processing request. The control execution node can control the operation of MapReduce.

[0031] S13: Based on the control and execution node, perform affinity scheduling on the first task of the task processing request to allocate the first execution node.

[0032] After starting the first task, a first task request can be initiated. Based on the control execution node, the first task request for executing the task can be scheduled with affinity, thereby allocating a first execution node for executing the first task.

[0033] In some implementations, affinity scheduling can be performed on the first task of the task processing request based on affinity information of the control execution node.

[0034] In some implementations, if the control execution node satisfies the first task, then the control execution node is designated as the first execution node. Satisfying the first task can mean that the resources of the control execution node meet the computing resources required for the first task, or that the affinity information of the control execution node meets the processing requirements of the first task. In such cases, the first task can be scheduled on the control execution node, i.e., the control execution node is designated as the first execution node.

[0035] In some implementations, if the control execution node does not meet the first task, the first task can be scheduled to other nodes, and the other nodes can be used as the first execution node.

[0036] In some implementations, if the control execution node includes multiple nodes, and at least some of the control execution nodes satisfy the first task, then at least some of the control execution nodes can be used as the first execution node. If the resources of at least some of the control execution nodes do not satisfy the first task, then at least some of the control execution nodes can be used together with other nodes as the first execution node.

[0037] S14: Utilize the first execution node to perform the first task on the data to be processed, and obtain intermediate processing data.

[0038] The first execution node is used to execute the first task on the data to be processed. For example, if the first task is a Map task, the data to be processed can be mapped to obtain intermediate processed data.

[0039] In some implementations, a first execution node is used to execute a first task to obtain intermediate processing data, and the intermediate processing data is stored in the local path of the first execution node. The local path includes the local path of the first execution node; for example, the first execution node can be used to map the data to be processed, and the resulting intermediate processing data can be stored in the local storage space of the first execution node.

[0040] S15: Based on the control execution node, perform affinity scheduling on the second task of the execution task processing request to allocate the second execution node.

[0041] After starting the second task, a second task request can be initiated. Based on the control execution node, the second task request for executing the task can be scheduled with affinity, thereby allocating a second execution node for executing the second task.

[0042] In some implementations, affinity scheduling can be performed on the second task of the execution task processing request based on affinity information of the control execution node.

[0043] In some implementations, if the control execution node satisfies the second task, then the control execution node is designated as the second execution node. Satisfying the second task can mean that the resources of the control execution node meet the computing resources required for the second task, or that the affinity information of the control execution node meets the processing requirements of the second task. In such cases, the second task can be scheduled on the control execution node, i.e., the control execution node is designated as the second execution node.

[0044] In some implementations, if the control execution node does not meet the requirements of the second task, the second task can be scheduled to other nodes, and the other nodes can be used as the second execution nodes.

[0045] In some implementations, if the control execution node includes multiple nodes, and at least some of the control execution nodes satisfy the second task, then at least some of the control execution nodes can be used as the second execution node. If the resources of at least some of the control execution nodes do not satisfy the second task, then at least some of the control execution nodes can be used together with other nodes as the second execution node.

[0046] S16: Utilize the second execution node to perform the second task on the intermediate processing data and obtain the processing result.

[0047] By using the second execution node to perform a second task on the data to be processed, such as a Reduce task, the data to be processed can be reduced to obtain the processing result.

[0048] In some implementations, the second execution node can obtain intermediate processing data required to execute the second task, wherein the intermediate processing data is obtained by the first execution node through mapping processing of the data to be processed.

[0049] In some implementations, when the control execution node satisfies the first task and the second task, the first task and the second task can be scheduled on the control execution node, so that the first task and the second task can be executed on the same node. At this time, the second execution node can be used to directly read intermediate processing data to execute the second task.

[0050] In this embodiment, by receiving a task processing request for the data to be processed, a control execution node that satisfies the task processing request is determined. Based on the control execution node, a first task for executing the task processing request is performed using affinity scheduling to allocate a first execution node. The first execution node is used to execute the first task for the data to be processed to obtain intermediate processing data. Based on the control execution node, a second task for executing the task processing request is performed using affinity scheduling to allocate a second execution node. The second execution node is used to execute the second task on the intermediate processing data to obtain the processing result. Since the first and second tasks are scheduled on the control execution node when they are satisfied, the first and second tasks can be executed on the same node, thereby reducing the transmission of intermediate processing data over the network and reducing the read and write operations on the disk for intermediate processing data, which improves the execution efficiency of the task.

[0051] In some embodiments, please refer to Figure 2 Step S12 above, determining the control execution node that satisfies the task processing request, may include the following steps:

[0052] S121: Use the task manager to call the management container corresponding to the task processing request and set the node affinity information for the management container.

[0053] In some implementations, MapReduce can be run on Kubernetes using Yarn, where Yarn is a cluster resource management system for Hadoop and Kubernetes is a container orchestration framework, also known as a container cluster management system.

[0054] The client can send task processing requests for the data to be processed to the data processing platform. After receiving the task processing request from the client, the data processing platform can use the NodeManager to call the Kubernetes interface to start the task, that is, start the management container of the Task Manager (MRAppMaster). The Task Manager is a role in the MapReduce process, used to manage the execution of MapReduce. The NodeManager manages the execution of tasks; it acts as an agent for each machine (or node), monitoring the application's resource usage and reporting it to the MRAppMaster.

[0055] In some implementations, affinity information for nodes can be set on the management container, that is, the affinity information for nodes is added to the management container.

[0056] S122: Utilize the affinity information of nodes to call nodes that meet preset conditions as control and execution nodes that fulfill task processing requests.

[0057] Node affinity information can be used to select nodes that meet preset conditions as the management execution nodes for task processing requests. Since there is no container with this affinity information on Kubernetes at this time, this management container / task manager can be scheduled on any node that meets the preset conditions.

[0058] In some implementations, the preset conditions include the same node satisfying affinity scheduling conditions and / or different nodes satisfying resource scheduling conditions. For example, the task manager can schedule nodes that satisfy preset conditions. If a node satisfies resource scheduling conditions, that is, the node's resources meet the task processing request, and if a node satisfies affinity scheduling conditions, etc., the node can be used as the control execution node. The nodes satisfying affinity scheduling conditions can be the same node or multiple nodes, multiple different nodes satisfying affinity scheduling conditions, multiple different nodes satisfying resource scheduling conditions, etc. This application does not impose any limitations on this.

[0059] In some implementations, the task manager can be scheduled onto the management execution node to launch the task manager / management container, etc., using the management execution node.

[0060] In some embodiments, please refer to Figure 3 In step S13 above, if the control execution node satisfies the first task, then the control execution node is designated as the first execution node, which may include the following steps:

[0061] S131: Use the task manager to initiate the first task, call the first task container, and set the affinity information of the nodes in the first task container.

[0062] Once the Task Manager is started, it can send a request to the Node Manager to run the first task, which is to initiate a MapTask (the Map task in MapReduce). After receiving the request, the Node Manager can call the Kubernetes interface to start the first task container. The first task container is a MapTask container, and node affinity information can be added to the first task container.

[0063] S132: If the management execution node corresponding to the task manager satisfies the first task, then the first task will be scheduled to the management execution node so that the management execution node is used as the first execution node.

[0064] At this point, since there are already affinity-enabled containers running on Kubernetes (i.e., management containers with affinity information), when the management execution node corresponding to the task manager meets the conditions for the first task—that is, when the node's affinity information satisfies affinity scheduling—the resources of the management execution node can execute the first task. Alternatively, if at least some management execution nodes meet the conditions for the first task, the management execution node that meets the conditions can be designated as the first execution node, and the first task can be scheduled to that node. In other words, the first task can be scheduled to the node where MRAppMaster resides, allowing the node where MRAppMaster resides to execute the first task. The node where MRAppMaster resides can be multiple nodes, and these multiple nodes can be all nodes that satisfy affinity scheduling.

[0065] In some implementations, if the control execution node corresponding to the task controller does not meet the requirements of the first task, the first task can be scheduled to other nodes so that other nodes can be used as the first execution nodes to execute the first task.

[0066] In some implementations, after the first execution node completes the first task on the data to be processed, it can output the obtained intermediate processing data to the local storage space / disk of the first execution node. At the same time, the first execution node can also send the storage path and node information of the intermediate processing data to the task manager, wherein the node information is the node information of the first execution node.

[0067] In some embodiments, please refer to Figure 4 In step S15 above, if the control execution node satisfies the second task, then the control execution node is designated as the second execution node, which may include the following steps:

[0068] S151: Use the task manager to initiate a second task, which calls the second task container and sets the affinity information of the nodes in the second task container.

[0069] After the first task finishes running, the task manager can send a request to the node manager to run the second task, that is, to send a ReduceTask (the Reduce task in MapReduce) to the node manager. After receiving the request, the node manager can call the Kubernetes interface to start the second task container. The second task container is a ReduceTask container, and node affinity information can be added to the second task container.

[0070] S152: If the control execution node corresponding to the task manager satisfies the second task, then the second task is scheduled to the control execution node so that the control execution node is used as the second execution node.

[0071] If the managing execution node corresponding to the task manager meets the conditions for the second task, that is, when the node's affinity information satisfies affinity scheduling, the resources of the managing execution node can execute the second task, or at least some of the managing execution nodes meet the conditions for the second task, then the managing execution node that meets the conditions for the second task can be used as the second execution node, and the second task can be scheduled to the managing execution node, that is, scheduled to the node where MRAppMaster resides, so that the second task can be executed using the node where MRAppMaster resides. The node where MRAppMaster resides can be multiple nodes, and these multiple nodes can be all nodes that satisfy affinity scheduling.

[0072] In some implementations, if the control execution node corresponding to the task controller does not meet the requirements of the second task, the second task can be scheduled to other nodes so that other nodes can be used as the second execution nodes to execute the second task.

[0073] In some implementations, since there are already affinity information containers running on Kubernetes, that is, there are affinity information management containers and first task containers running, if the management execution node satisfies the second task, the second task is scheduled to the management execution node so that the management execution node is used as the second execution node. If the first execution node satisfies the second task, the second task is scheduled to the first execution node so that the first execution node is used as the second execution node to execute the second task.

[0074] In some implementations, after the second execution node starts the ReduceTask, it can send its node information, which may include the host machine information of the second execution node, to the MRAppMaster. It can also request the storage path for the intermediate processing data of the corresponding MapTask from the MRAppMaster.

[0075] In this embodiment, by utilizing the affinity information of Kubernetes nodes, the first and second tasks can be scheduled based on affinity, thereby avoiding data transmission, improving the execution speed of tasks, and enhancing the processing efficiency of the data processing platform.

[0076] In some embodiments, please refer to Figure 5 Step S16 above, which uses the second execution node to perform a second task on the intermediate processing data to obtain the processing result, includes the following steps:

[0077] S161: Determine whether the first execution node and the second execution node are the same node.

[0078] After the first task is completed using the first execution node, the intermediate processing data is stored in the local path of the first execution node. The first execution node then sends the storage path of the intermediate processing data and node information to the task manager.

[0079] When the second task is started, the second execution node will send its node information to the task manager and obtain the storage information of the intermediate processing data from the task manager. The storage information includes the node information of the first execution node that stores the intermediate processing data and the storage path of the intermediate processing data, so as to obtain the intermediate processing data through the storage information.

[0080] In some implementations, the task manager can use the node information of the second execution node and the storage information of intermediate processing data to determine whether the second execution node and the first execution node are the same node. Specifically, if the node information of the first execution node in the storage information matches that of the second execution node, then the second execution node and the first execution node are determined to be the same node. If the node information of the first execution node in the storage information does not match that of the second execution node, then the second execution node and the first execution node are determined to be different nodes. Other methods can also be used to determine whether the first execution node and the second execution node are the same node, and this application does not limit this method.

[0081] If it is determined that they are the same node, then proceed to step S162.

[0082] If it is determined that they are not the same node, then proceed to step S163.

[0083] S162: Return the local path of the intermediate processing data to the second execution node so that the second execution node can access the local path, perform the second task on the intermediate processing data, and obtain the processing result.

[0084] Since the second execution node is the same node as the first execution node, the task manager can return the local path of the intermediate processing data to the second node, so that the second execution node can access the local path, read the intermediate processing data to execute the second task, and obtain the processing result.

[0085] In some implementations, after receiving the local path, the second execution node can read the intermediate processing data stored in the local path of the second execution node, that is, read the intermediate processing data stored in the local storage space, use the second execution node to perform a second task on the intermediate processing data, obtain the processing result, and output the processing result of the task processing request.

[0086] S163: Return the network path of the intermediate processing data to the second execution node so that the second execution node can access the network path, perform the second task on the intermediate processing data, and obtain the processing result.

[0087] Because the second execution node is not the same as the first execution node, the Task Manager returns the network path of the intermediate processing data to the second execution node. This network path may contain storage information for the intermediate processing data. This storage information includes the node information of the first execution node storing the intermediate processing data and the local path of the intermediate processing data on the first execution node.

[0088] In some implementations, the network path may be an HTTP address for accessing intermediate processing data, and the HTTP address may indicate the node information of the first execution node storing the intermediate processing data and the local path of the first execution node storing the intermediate processing data.

[0089] In some implementations, the second execution node can use a network path to obtain intermediate processing data stored in the local path of the first execution node, and store the intermediate processing data in the local path of the second execution node. That is, the intermediate processing data is copied to the local path of the second execution node by accessing the network path.

[0090] The second execution node reads the intermediate processing data stored in the local path, and uses the second execution node to execute the second task on the intermediate processing data, that is, to execute the Reduce task, obtain the processing result, and output the processing result of the task processing request.

[0091] In this embodiment, by determining whether the first execution node and the second execution node are the same node, if so, the local path of the intermediate processing data is returned to the second execution node, allowing it to access the local path and execute the second task on the intermediate processing data. If not, the network path of the intermediate processing data is returned to the second execution node, allowing it to access the network path and execute the second task on the intermediate processing data. This reduces the transmission of intermediate processing data over the network and the pressure on the network environment during task processing request execution. Furthermore, reducing the copying of intermediate processing data by the second execution node from the first execution node lowers disk read / write pressure and improves task execution speed, thereby enhancing the data processing platform's task processing efficiency.

[0092] In conjunction with the above embodiments, this application also provides a processing apparatus for a data processing platform. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the structure of a processing device of the data processing platform of this application. The processing device 20 of the data processing platform includes a receiving module 21, a determining module 22, a scheduling module 23, a first execution module 24, and a second execution module 25.

[0093] The receiving module 21 is used to receive task processing requests for the data to be processed.

[0094] The determination module 22 is used to determine the control execution node that satisfies the task processing request.

[0095] The scheduling module 23 is used to perform affinity scheduling on a first task requesting task processing based on the control and management execution node, so as to allocate a first execution node. It is also used to perform affinity scheduling on a second task requesting task processing based on the control and management execution node, so as to allocate a second execution node.

[0096] The first execution module 24 is used to execute the first task on the data to be processed using the first execution node to obtain intermediate processing data.

[0097] The second execution module 25 is used to perform a second task on the intermediate processing data using the second execution node to obtain the processing result.

[0098] In some implementations, the data processing platform includes a programming model platform for data processing; the first task includes performing mapping processing; and the second task includes performing reduction calculations.

[0099] In some implementations, the determining module 22 is used to determine the control execution node that satisfies the task processing request, including: using the task manager to call the management container corresponding to the task processing request, setting the node affinity information of the management container; and using the node affinity information to call the node that meets the preset conditions as the control execution node that satisfies the task processing request.

[0100] In some implementations, the scheduling module 23 is used to perform affinity scheduling on the first task of the execution task processing request based on the control execution node, so as to allocate the first execution node, including: if the control execution node satisfies the first task, then the control execution node is used as the first execution node.

[0101] In some implementations, a first task is initiated using a task manager to invoke a first task container and set node affinity information for the first task container; if the management execution node corresponding to the task manager satisfies the first task, the first task is scheduled to the management execution node so that the management execution node is used as the first execution node.

[0102] In some implementations, the scheduling module 23 is used to perform affinity scheduling on the second task of the execution task processing request based on the control execution node, so as to allocate the second execution node, including: if the control execution node satisfies the second task, then the control execution node is used as the second execution node.

[0103] In some implementations, the second task is initiated using a task manager to invoke a second task container and set node affinity information for the second task container; if the management execution node corresponding to the task manager satisfies the second task, the second task is scheduled to the management execution node so that the management execution node is used as the second execution node.

[0104] In some implementations, the first execution module 24 is used to perform a first task on the data to be processed using the first execution node to obtain intermediate processing data, including: performing the first task using the first execution node to obtain intermediate processing data, and storing the intermediate processing data in the local path of the first execution node.

[0105] In some implementations, the second execution module 25 is used to perform a second task on the intermediate processing data using the second execution node to obtain a processing result, including: determining whether the first execution node and the second execution node are the same node; if so, returning the local path of the intermediate processing data to the second execution node so that the second execution node can access the local path and perform the second task on the intermediate processing data; if not, returning the network path of the intermediate processing data to the second execution node so that the second execution node can access the network path and perform the second task on the intermediate processing data. The local path includes the local path of the first execution node.

[0106] In some implementations, the second execution module 25 is used to read intermediate processing data stored in the local path of the second execution node; and to use the second execution node to perform a second task on the intermediate processing data to obtain the processing result.

[0107] In some implementations, the second execution module 25 is used to obtain intermediate processing data stored in the local path of the first execution node using a network path, and store the intermediate processing data in the local path of the second execution node; and use the second execution node to perform a second task on the intermediate processing data to obtain a processing result.

[0108] In some implementations, the second execution module 25 is used to determine whether the first execution node and the second execution node are the same node, including: sending the node information of the second execution node to the task manager; and obtaining the storage information of intermediate processing data from the task manager, wherein the storage information includes the node information of the first execution node storing the intermediate processing data; using the node information of the second execution node and the storage information of the intermediate processing data, determining whether the second execution node and the first execution node are the same node; wherein, if the node information of the first execution node in the storage information is consistent with the node information of the second execution node, then it is determined that the second execution node and the first execution node are the same node.

[0109] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0110] Regarding the above embodiments, this application provides a computer device; please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present application. The computer device 30 includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled to each other. The memory 31 stores program data, and the processor 32 is used to execute the program data to implement the steps of any embodiment of the processing method of the data processing platform described above.

[0111] In this embodiment, processor 32 can also be referred to as CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0112] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0113] The methods described in the above embodiments can be implemented as computer programs; therefore, this application proposes a storage device. Please refer to [link to relevant documentation]. Figure 8 , Figure 8This is a schematic diagram of the structure of a storage device according to an embodiment of the present application. The storage device 40 stores program data 41 that can be executed by a processor. The program data 41 can be executed by the processor to implement the steps of any embodiment of the processing method of the data processing platform described above.

[0114] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0115] In this embodiment, the storage device 40 can be a medium that can store program data 41, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Alternatively, it can be a server that stores the program data 41. The server can send the stored program data 41 to other devices for execution, or it can run the stored program data 41 itself.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage device, which is a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0120] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0121] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A processing method for a data processing platform, characterized in that, The method includes: Receive task processing requests for data to be processed; Determine the control execution node that satisfies the task processing request; Based on the control and execution node, the first task executing the task processing request is subject to affinity scheduling to allocate a first execution node, including: initiating the first task using a task manager to invoke a first task container and setting the affinity information of the node for the first task container; if the control and execution node corresponding to the task manager satisfies the first task, then the first task is scheduled to the control and execution node to use the control and execution node as the first execution node. The first task is executed on the data to be processed using the first execution node to obtain intermediate processed data; Based on the control execution node, the second task that executes the task processing request is subject to affinity scheduling to allocate a second execution node, including: initiating the second task using a task manager to call a second task container and setting the affinity information of the node for the second task container; if the control execution node corresponding to the task manager satisfies the second task, then the second task is scheduled to the control execution node, so that the control execution node is used as the second execution node; The second task is performed on the intermediate processing data using the second execution node to obtain the processing result.

2. The processing method according to claim 1, characterized in that, The step of using the second execution node to perform the second task on the intermediate processing data includes: Determine whether the first execution node and the second execution node are the same node; If so, the local path of the intermediate processing data is returned to the second execution node, so that the second execution node can access the local path and perform the second task on the intermediate processing data; If not, the network path of the intermediate processing data is returned to the second execution node so that the second execution node can access the network path and perform the second task on the intermediate processing data.

3. The processing method according to claim 2, characterized in that, The local path includes the local path of the first execution node; the step of using the first execution node to execute the first task on the data to be processed to obtain intermediate processing data includes: The first task is executed using the first execution node to obtain the intermediate processing data, and the intermediate processing data is stored in the local path of the first execution node.

4. The method according to claim 3, characterized in that, The second execution node accesses the local path and performs the second task on the intermediate processing data, including: Read the intermediate processing data stored in the local path of the second execution node; The second task is performed on the intermediate processing data using the second execution node to obtain the processing result.

5. The method according to claim 3, characterized in that, The second execution node accesses the network path and performs the second task on the intermediate processing data, including: The intermediate processing data stored in the local path of the first execution node is obtained using the network path, and the intermediate processing data is stored in the local path of the second execution node; The second task is performed on the intermediate processing data using the second execution node to obtain the processing result.

6. The processing method according to claim 2, characterized in that, The process of determining the control execution node that satisfies the task processing request includes: The task manager is used to call the management container corresponding to the task processing request, and the affinity information of the node is set for the management container. Using the affinity information of the nodes, nodes that meet preset conditions are invoked as control and execution nodes to fulfill the task processing request.

7. The method according to claim 6, characterized in that, The step of determining whether the first execution node and the second execution node are the same node includes: Send the node information of the second execution node to the task manager; and Obtain the storage information of the intermediate processing data from the task manager, wherein the storage information includes node information of the first execution node storing the intermediate processing data; Using the node information of the second execution node and the storage information of the intermediate processing data, it is determined whether the second execution node and the first execution node are the same node; wherein, if the node information of the first execution node in the storage information is consistent with the node information of the second execution node, it is determined that the second execution node and the first execution node are the same node.

8. The method according to claim 1, characterized in that, The data processing platform includes a programming model platform for data processing; The first task includes performing mapping processing; the second task includes performing reduction calculations.

9. A computer device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program data and the processor executing the program data to implement the steps of the method according to any one of claims 1 to 8.

10. A storage device, characterized in that, The system stores program data that can be executed by a processor, the program data being used to implement the steps of the method according to any one of claims 1 to 8.

Images