Cross-cluster task execution method and device, computer equipment and storage medium

Abstract

The invention provides a cross-cluster task execution method and device, computer equipment and a storage medium. The method comprises the steps of obtaining task data of a to-be-run target task and a running instruction of the target task; determining a task submission environment and an initial code topology of the target task running on the initial cluster according to a submission class in the task data; converting the task submission environment into environment information of the target task running on the target cluster according to the parameter information; converting the initial code topology into a target code topology of the target task running on a target cluster according to the parameter information; and packaging the environment information and the target code topology to obtain a target code of the target task running on the target cluster. By adopting the method, the operation which needs to be modified on the user level originally is sunk into the platform layer, so that the workload and the risk caused by the fact that a user code needs to be modified and a code package needs to be printed again are avoided.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a cross-cluster task execution method, device, computer equipment and storage medium. Background Art

[0002] JStorm is a distributed real-time computing engine. JStorm is a system similar to Hadoop MapReduce. Users implement a task according to the specified interface, and then hand over the task to the JStorm system. JStorm runs the task and runs it 24 / 7. If a worker fails unexpectedly, the scheduler immediately assigns a new worker to replace the failed worker. For user code, commonly used coding interfaces include topology, spout, bolt, etc.

[0003] Apache Flink is a distributed processing engine and framework for stateful computation of unbounded and bounded data streams. It can run in common cluster environments and execute at memory speed and at any scale. Common interfaces for users include source, transform, sink / operator, etc.

[0004] JStorm and Flink are popular technologies in the field of real-time computing. If users want to use Flink technology for business codes that were originally implemented with JStorm, they will have to face the need to re-implement them with Flink technology, which involves various R&D, testing and other issues, requiring a lot of manpower and material resources, and is inefficient. Summary of the invention

[0005] Based on this, it is necessary to provide a cross-cluster task execution method, device, computer equipment and storage medium to address the above technical problems, so that the operations that originally need to be modified at the user level are sunk to the platform layer, avoiding the workload and risks brought by the need to modify user code and re-package the code.

[0006] In a first aspect, the present application provides a cross-cluster task execution method, comprising: obtaining task data of a target task to be run and a running instruction of the target task, the task data being code data running on an initial cluster, and the running instruction including parameter information of a target cluster running the target task; determining a task submission environment and an initial code topology for running the target task on the initial cluster according to a submission class in the task data; converting the task submission environment into environment information for running the target task on the target cluster according to the parameter information; converting the initial code topology into a target code topology for running the target task on the target cluster according to the parameter information; encapsulating the environment information and the target code topology to obtain a target code for running the target task on the target cluster.

[0007] In some embodiments of the present application, converting the initial code topology into a target code topology on which the target task runs on the target cluster according to the parameter information includes: parsing the initial code topology to obtain a first node operator and directed edges of the target task; converting the first node operator of the initial cluster into a second node operator of the target cluster according to the parameter information; and connecting the second node operators according to the parameter information and the directed edges to obtain the target code topology on which the target task runs on the target cluster.

[0008] In some embodiments of the present application, converting the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information includes: parsing the data processing logic module of the first node operator of the initial cluster; determining the data input module and the data output module of the second node operator according to the parameter information; and obtaining the second node operator of the target cluster according to the data processing logic module, the data input module and the data output module.

[0009] In some embodiments of the present application, the node operator includes a data source operator and a data processing operator, and the converting of the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information includes: converting the first data source operator of the initial cluster into the second data source operator of the target cluster according to the parameter information; converting the first data processing operator of the initial cluster into the second data processing operator of the target cluster according to the parameter information.

[0010] In some embodiments of the present application, converting the task submission environment into the environment information for the target task to run on the target cluster according to the parameter information includes: if the task submission environment is to run locally, converting it into the environment information for the target task to run locally on the target cluster according to the parameter information; if the task submission environment is to be submitted remotely, converting it into the environment information for the target task to be submitted remotely on the target cluster according to the parameter information.

[0011] In some embodiments of the present application, the environmental information includes task execution environment information and cluster interface environment information, and the encapsulation of the environmental information and the target code topology to obtain the target code for the target task running on the target cluster includes: encapsulating the target code topology to obtain the target code according to the task execution environment information and the cluster interface environment information; and submitting the target code to the target cluster for execution based on the cluster interface environment information.

[0012] In the second aspect, the present application provides a cross-cluster task execution device, including: an information acquisition module, used to obtain task data of a target task to be run and an operating instruction of the target task, the task data being code data running on an initial cluster, and the operating instruction including parameter information of a target cluster running the target task; a code analysis module, which is communicated and connected with the information acquisition module, and is used to determine a task submission environment and an initial code topology for running the target task on the initial cluster according to a submission class in the task data; a code conversion module, which is communicated and connected with the code analysis module, and is used to convert the task submission environment into environment information for running the target task on the target cluster according to the parameter information; convert the initial code topology into a target code topology for running the target task on the target cluster according to the parameter information; encapsulate the environment information and the target code topology to obtain a target code for running the target task on the target cluster.

[0013] In some embodiments of the present application, the code conversion module is also used to parse the initial code topology to obtain the first node operator and directed edge of the target task; convert the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information; connect the second node operator according to the parameter information and the directed edge to obtain the target code topology for the target task running on the target cluster.

[0014] In some embodiments of the present application, the code conversion module is also used to parse the data processing logic module of the first node operator of the initial cluster; determine the data input module and data output module of the second node operator according to the parameter information; and obtain the second node operator of the target cluster according to the data processing logic module, the data input module and the data output module.

[0015] In some embodiments of the present application, the node operator includes a data source operator and a data processing operator, and the code conversion module is also used to convert the first data source operator of the initial cluster into the second data source operator of the target cluster according to the parameter information; and convert the first data processing operator of the initial cluster into the second data processing operator of the target cluster according to the parameter information.

[0016] In some embodiments of the present application, the code conversion module is also used to convert the task submission environment into environment information for the target task to run locally in the target cluster according to the parameter information if the task submission environment is local operation; and to convert the task submission environment into environment information for the target task to be submitted remotely in the target cluster according to the parameter information if the task submission environment is remote submission.

[0017] In some embodiments of the present application, the environmental information includes task execution environment information and cluster interface environment information, and the code conversion module is also used to encapsulate the target code topology according to the task execution environment information and the cluster interface environment information to obtain the target code; and submit the target code to the target cluster for execution based on the cluster interface environment information.

[0018] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0019] one or more processors;

[0020] A memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the cross-cluster task execution method.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in the cross-cluster task execution method.

[0022] The above-mentioned cross-cluster task execution method, device, computer equipment and storage medium determine the task submission environment and initial code topology through the submission class in the task data of the target task running on the initial cluster, and then convert them into the environment information and target code topology of the target cluster, and finally encapsulate and generate the target code that enables the target task to run on the target cluster, sinking the operations that originally need to be modified at the user level to the platform layer, avoiding the workload and risks brought by the need to modify user code and re-package the code, and also avoiding the workload of users needing to urgently re-modify the code due to interceptor upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a flowchart of a cross-cluster task execution method in an embodiment of the present application;

[0025] Figure 2 is a flowchart of a cross-cluster task execution method in another embodiment of the present application;

[0026] Figure 3 is a flowchart of a cross-cluster task execution method in another embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of the structure of a cross-cluster task execution device in an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0030] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0031] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0032] JStorm and Flink use different technical architectures. To implement the same business functions, different API interfaces are required for coding and calling. The industry has also explored repackaging JStorm code using Flink technology. Users modify the original JStorm code and submit it to the Flink cluster. In this solution, users need to make two changes:

[0033] 1. Users need to introduce the code package encapsulated by Flink technology;

[0034] 2. Users need to modify the existing jstorm code and repackage it;

[0035] Since the user needs to modify the code, it brings a certain amount of work to the user. On the other hand, if the code encapsulated by Flink technology has bugs or needs to be updated for other reasons, the user needs to introduce a new code package. Therefore, it is not very user-friendly.

[0036] See also Figure 1 , the embodiment of the present application provides a cross-cluster task execution method, the method includes steps S101 to S105, which are as follows:

[0037] S101, acquiring task data of a target task to be executed and an execution instruction of the target task, wherein the task data is code data in a format to be executed on an initial cluster, and the execution instruction includes parameter information of a target cluster for executing the target task.

[0038] Specifically, the task data of the target task to be run is obtained. The task data is code data in a corresponding format that can be directly run on the initial cluster. The running instructions of the target task input by the user are obtained. The running instructions include parameter information of the target cluster for running the target task, that is, the running instructions control the target task to run in the target cluster. The parameter information includes but is not limited to the submission class environment information of the target cluster, cluster interface information, etc. The target cluster and the initial cluster are different clusters. For example, the initial cluster is a jstorm cluster, the initial cluster is a flink cluster, and the task data is the user code of the jstorm cluster.

[0039] S102: Determine, according to the submission class in the task data, a task submission environment and an initial code topology for running the target task on the initial cluster.

[0040] Specifically, the task data of the target task is code data in a corresponding format that can be directly run on the initial cluster. However, if the target task is to be run on the initial cluster, the task data needs to be submitted to the initial cluster. The task submission environment and initial code topology for the target task to be run on the initial cluster are determined based on the submission class in the task data. The task submission environment includes the target task's operating environment requirements, parameter settings, etc. For example, through the Submitter API of the task submission, the jstorm Submitter java class in the jstorm environment can be called. The initial code topology includes the node operators and directed edges (data flow relationships) executed by the target task. The initial code topology contains a variety of operators, and the operators contain the business processing logic of the user code.

[0041] S103: Convert the task submission environment into environment information for running the target task on the target cluster according to the parameter information.

[0042] Specifically, the task submission environment is converted into the environment information for the target task to run on the target cluster based on the parameter information. According to the user's use of jstorm's native submission class, it is converted into flink's submission class. The purpose is to convert the current task submission environment from jstorm to flink. During the flink environment construction process, the current fink parameter information will be loaded and the flink task submission client will be built.

[0043] In one embodiment, this step includes: S201, if the task submission environment is local operation, converting it into the environment information of the target task running locally in the target cluster according to the parameter information; S202, if the task submission environment is remote submission, converting it into the environment information of the target task remotely submitted in the target cluster according to the parameter information.

[0044] Specifically, the present embodiment submits the task data of the target task to the target cluster for operation through the JVM virtual machine, and a code submission interceptor is embedded in the JVM virtual machine. Before the JVM virtual machine starts to run the task data, the code class of the task data will be loaded through the class loader of the JVM virtual machine. When loading the user's code class, the user's code class will be first transmitted to the code interceptor. The interceptor will determine whether the class passed in is the LocalCluster (local operation) or the jstorm Submitter class (remote submission) of jstorm. Local operation means running on the target cluster of the local client, and remote submission is not running on the target cluster of the local client. The operating environments of the two are different. Therefore, if the task submission environment is local operation (Local Cluster), it is converted into the environment information (Local Stream Environment) of the target task running locally in the target cluster according to the parameter information. If the task submission environment is remote submission, it is converted into the environment information (Remont Stream Environment) of the target task remotely submitted in the target cluster according to the parameter information. The present embodiment provides different submission methods, which is convenient for user operation, and the characteristics of the operating environment based on different submission methods are respectively targeted for conversion, so that the conversion accuracy is higher.

[0045] S104: Convert the initial code topology into a target code topology for running the target task on the target cluster according to the parameter information.

[0046] Specifically, different clusters have different topological architectures and different business processing logics of operators. The initial code topology is converted into the target code topology for the target task to run on the target cluster according to the parameter information. For example, the topology structure of jstorm is parsed and encapsulated in the flink framework to be converted into the flink topology.

[0047] In one embodiment, this step includes: S301, parsing the initial code topology to obtain the first node operator and directed edge of the target task; S302, converting the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information; S303, connecting the second node operator according to the parameter information and the directed edge to obtain the target code topology for the target task to run on the target cluster.

[0048] Specifically, the initial code topology is parsed, and the nodes and data flow relationships therein are disassembled and transformed to obtain the first node operators and directed edges of the target task, where the directed edges are the connection relationships between the first node operators. The first node operators of the initial cluster are converted into the second node operators of the target cluster according to the parameter information of the target cluster, and then the second node operators are connected in combination with the directed edges of the previously parsed initial code topology to obtain the target code topology for the target task to run on the target cluster. Based on the cluster topology structure, the operators, i.e., the connection relationships between operators, are converted accordingly so that the second cluster after the conversion has the same topological logic as the first cluster.

[0049] In one embodiment, step S302, converting the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information, includes: S401, parsing the data processing logic module of the first node operator of the initial cluster; S402, determining the data input module and the data output module of the second node operator according to the parameter information; S403, obtaining the second node operator of the target cluster according to the data processing logic module, the data input module and the data output module.

[0050] Specifically, the data processing logic module of the first node operator of the initial code topology of the initial cluster is parsed, and the data input module and data output module of the second node operator of the target cluster are determined according to the parameter information. Then, the data processing logic module of the first node operator is connected to the data input module and the data output module to obtain the second node operator of the target cluster. That is to say, the interior of the second node operator is still the business processing logic of the first node operator, and the actual data processing method has not changed. That is, the processing logic of the target cluster is used as the outer packaging of the second node operator to wrap the business processing logic of the first node operator inside, thereby reducing the amount of code modification during the code conversion process. The actual data processing method has not changed, which also reduces the probability of data processing errors.

[0051] In one embodiment, the node operator includes a data source operator and a data processing operator. Step S302, converting the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information, includes: S501, converting the first data source operator of the initial cluster into the second data source operator of the target cluster according to the parameter information; S502, converting the first data processing operator of the initial cluster into the second data processing operator of the target cluster according to the parameter information.

[0052] Specifically, the node operator includes a data source operator and a data processing operator. The data source operator and the data processing operator have different positions and functions in the code topology. The data source operator is used to obtain data streams, and the data processing operator is used to process data streams. The first node operator of the initial cluster includes a first data source operator and a first data processing operator, and the second node operator of the target cluster includes a second data source operator and a second data processing operator. For example, jstorm Topology includes a spout operator and a bolt operator, and flink Topology includes a source operator and an operator operator. The conversion of the node operator needs to be based on the function of the operator and converted into an operator with the same function. Therefore, according to the parameter information, the first data source operator of the initial cluster is converted into the second data source operator of the target cluster, and the first data processing operator of the initial cluster is converted into the second data processing operator of the target cluster. For example, the spout operator of jstorm Topology is converted into the source operator of flink Topology, and the bolt operator of jstorm Topology is converted into the operator operator of flink Topology. Different types of operators are converted accordingly based on the cluster topology structure, so that the second cluster after conversion has the same topological logic as the first cluster.

[0053] S105 , encapsulating the environment information and the target code topology to obtain the target code for the target task to run on the target cluster.

[0054] Specifically, the target code topology is encapsulated according to the environmental information, that is, the target cluster framework is encapsulated on the converted target code topology to obtain the target code for the target task running on the target cluster. The target code can be run directly on the target cluster, thereby achieving the goal of cross-cluster execution of the target task.

[0055] In one embodiment, the environmental information includes task execution environment information and cluster interface environment information. This step includes: S601, encapsulating the target code topology to obtain the target code according to the task execution environment information and the cluster interface environment information; S602, submitting the target code to the target cluster for execution based on the cluster interface environment information.

[0056] Specifically, the environment information includes task execution environment information and cluster interface environment information. The task execution environment information refers to the operating environment of the target code topology of the target task, so that the target code topology can run in the target cluster. The cluster interface environment information refers to the interface information that connects with the target cluster when the target code topology is packaged and submitted to the target cluster for execution, so that the packaged target code can be submitted to the target cluster. Therefore, according to the task execution environment information and the cluster interface environment information, the target code topology is packaged to obtain the target code, and then the target code is submitted to the target cluster for operation based on the cluster interface environment information.

[0057] In one embodiment, step S105, after encapsulating the environment information and the target code topology to obtain the target code for the target task to run on the target cluster, also includes: S701, when the target task is run on the target cluster, the data input module of the target second node operator receives the first data sent by the previous second node operator, the data processing logic module of the target second node operator processes the first data to obtain second data, and the data output module of the target second node operator sends the second data to the next second node operator.

[0058] Specifically, when the target task is run on the target cluster, the second data source operator obtains the data stream and sends the data stream to the corresponding second data processing operator for processing according to the data flow of the target code topology until the data stream is finally processed and stored. The internal business processing logic of any second node operator, including the second data source operator and the second data processing operator, is still the business processing logic of the first node operator of the initial cluster.

[0059] Therefore, for any target second-node operator, when running the target task, the data input module of the target second-node operator receives the first data sent by the data output module of the previous second-node operator, the data processing logic module of the target second-node operator (the business processing logic of the first-node operator of the initial cluster) processes the first data to obtain the second data, and the data output module of the target second-node operator sends the second data to the data input module of the next second-node operator.

[0060] That is to say, when the target task runs on the target cluster, the second node operator actually wraps the business processing logic of the first node operator of the initial cluster. The data transmission between different second node operators of the target code topology is processed according to the framework logic of the target cluster, but the processing of the data flow inside the second node operator still maintains the business processing logic of the first node operator of the initial cluster unchanged.

[0061] This embodiment determines the task submission environment and the initial code topology through the submission class in the task data of the target task running on the initial cluster, and then converts them into the environment information and target code topology of the target cluster. Finally, it encapsulates and generates the target code that enables the target task to run on the target cluster, and sinks the operations that originally needed to be modified at the user level to the platform layer, avoiding the workload and risks brought by the need to modify user code and re-package the code, and also avoids the workload of users having to urgently re-modify code due to interceptor upgrades.

[0062] like Figure 2 As shown, the embodiment of the present application provides a cross-cluster task execution method. Figure 2 The left half is the jstorm code and the related information of the jstorm cluster, and the right half is the related information of the flink code converter and the flink cluster. When the underlying bytecode is not adjusted by the java agent technology in step 3, the target task of jstorm will be submitted to the jstorm cluster for execution through step 3.

[0063] Java Agent (also called Java instrumentation technology) is an advanced feature provided by the Java virtual machine, which allows dynamic modification of bytecode and operation of Java applications while the program is running. Java Agent is usually used in the following scenarios: 1. Perform code performance analysis and debugging. 2. Dynamically modify Java bytecode to implement functions such as AOP (Aspect-Oriented Programming). 3. Dynamically modify the behavior of an application while the application is running. Java Agent technology was introduced in Java SE5. Before Java SE5, the functions of Java Agent could only be implemented by modifying class files. Java Agent is implemented through an agent mechanism, that is, by loading the agent class into the Java virtual machine, the application class can be monitored and controlled.

[0064] The cross-cluster task execution method includes the following steps:

[0065] Step 1: The user code (i.e. task data) of the jstorm cluster of the target task written by the user will contain various first-node operators, which contain the user's business processing logic;

[0066] Step 2: Through the Submitter API in the task data submitted by the target task, you can call the stormSubmitter java class in the jstorm cluster environment;

[0067] Step 3: Use Java agent instrumentation to modify the behavior of the jstormSubmitter class so that it proxies to the jstorm->Flink converter.

[0068] Step 4: The jstorm->Flink converter prepares the context for program execution, that is, converts the task submission environment of the jstorm cluster in the task data into the environment information for the target task to run on the flink cluster;

[0069] Step 5: Combine the context in step 4 to create a converter from jstorm topology to flink topology;

[0070] Step 6: The topology converter in step 5 parses the topology structure of the jstorm cluster, encapsulates the second node operator and directed edge in the jstorm topology structure (i.e., the initial code topology) with the flink framework, and converts it into the flink topology (i.e., the target code topology);

[0071] Step 7: Call the FlinkSubmitter API to submit the task.

[0072] Step 8: The API in step 7 calls the "task submission environment" FlinkSubmitter to submit the converted target code topology of the target task to the flink cluster for execution.

[0073] Through these 8 steps, users can convert jstorm tasks into flink tasks without changing the code.

[0074] Steps 3 to 6 are the key steps to achieve the goal of not changing the code by the user. The submitted class in the user code will be found and repackaged. In this way, the work of "users changing the code" is transferred to the "platform layer" to avoid the workload of users changing the code. At the same time, in the future, technical upgrades do not require changes to the user layer, but only require updates to the platform layer.

[0075] like Figure 3As shown, when the target task is submitted, a code submission interceptor is embedded in the JVM virtual machine. This interceptor will be loaded into the JVM virtual machine before the user's code runs. Before the JVM virtual machine starts running the user's code, the class in the user's code will be loaded through the JVM class loader. When loading the user's code class, the user's code class will be first transmitted to the code interceptor. The interceptor will determine whether the incoming class is Storm's LocalCluster (local running) or StormSubmitter class (remote submission). If it is one of these two classes, the code of the submitTopology method of this class will be rewoven. The code reweaving process mainly includes the following steps:

[0076] First, when users use the native submission class of Storm, they convert it to the submission class of Flink. The purpose is to convert the current submission environment from Storm to Flink. During the construction of the Flink environment, the current Flink environment configuration information will be loaded and the client for Flink submission tasks will be built. Among them, the Local Cluster in the Storm submission class is converted to the Local Stream Environment in the Flink submission class, and the StormSubmitter in the Storm submission class is converted to the Remont Stream Environment in the Flink submission class.

[0077] Second, extract the original initial topology structure stormTopology of Storm from submitTopology("topoloygName", stormTopology) of Storm submission class, disassemble and transform the nodes and data flow relationship in stormTopology, transform the spout operator in stormTopology into flink's source operator, transform the bolt operator in stormTopology into flink's operator operator, and then transform the data flow relationship between nodes, thus completing the transformation of stormTopoloyg (initial topology structure) to flink topology (target topology structure).

[0078] Third, submit the flink topology parameters in the second step as the parameters of the flink client in the first step, completing the conversion and submission process.

[0079] This embodiment intercepts the user code and reweaves the code, sinking the operations that originally need to be modified at the user level into the code interceptor, thus avoiding the workload and risk of changing the user code and repackaging the code, and also avoiding the workload of users needing to urgently re-change the code due to interceptor upgrades. This solution allows users to run jstorm tasks on the flink cluster without feeling, and the user level does not need to participate too much, so this solution greatly reduces the workload during the user switching process.

[0080] like Figure 4 As shown, the embodiment of the present application provides a cross-cluster task execution device 900, including:

[0081] The information acquisition module 910 is used to acquire task data of the target task to be executed and an execution instruction of the target task, wherein the task data is code data executed on the initial cluster, and the execution instruction includes parameter information of the target cluster for executing the target task;

[0082] A code analysis module 920, which is in communication with the information acquisition module 910, and is used to determine the task submission environment and initial code topology of the target task running on the initial cluster according to the submission class in the task data;

[0083] The code conversion module 930 is communicatively connected with the code analysis module 920, and is used to convert the task submission environment into the environment information for the target task to run on the target cluster according to the parameter information; convert the initial code topology into the target code topology for the target task to run on the target cluster according to the parameter information; and encapsulate the environment information and the target code topology to obtain the target code for the target task to run on the target cluster.

[0084] In some embodiments of the present application, the code conversion module 930 is also used to parse the initial code topology to obtain the first node operator and directed edge of the target task; convert the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information; connect the second node operator according to the parameter information and the directed edge to obtain the target code topology for the target task running on the target cluster.

[0085] In some embodiments of the present application, the code conversion module 930 is also used to parse the data processing logic module of the first node operator of the initial cluster; determine the data input module and data output module of the second node operator according to the parameter information; and obtain the second node operator of the target cluster according to the data processing logic module, the data input module and the data output module.

[0086] In some embodiments of the present application, the node operator includes a data source operator and a data processing operator, and the code conversion module 930 is also used to convert the first data source operator of the initial cluster into the second data source operator of the target cluster according to the parameter information; and convert the first data processing operator of the initial cluster into the second data processing operator of the target cluster according to the parameter information.

[0087] In some embodiments of the present application, the code conversion module 930 is also used to, when the target task is run on the target cluster, the data input module of the target second node operator receives the first data sent by the previous second node operator, the data processing logic module of the target second node operator processes the first data to obtain second data, and the data output module of the target second node operator sends the second data to the next second node operator.

[0088] In some embodiments of the present application, the code conversion module 930 is also used to convert the task submission environment into environment information for the target task to run locally in the target cluster according to the parameter information if the task submission environment is local operation; and to convert the task submission environment into environment information for the target task to be submitted remotely in the target cluster according to the parameter information if the task submission environment is remote submission.

[0089] In some embodiments of the present application, the environmental information includes task execution environment information and cluster interface environment information, and the code conversion module 930 is also used to encapsulate the target code topology according to the task execution environment information and the cluster interface environment information to obtain the target code; and submit the target code to the target cluster for execution based on the cluster interface environment information.

[0090] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0091] For example, Figure 5 The computer device shown can be Figure 4The information acquisition module 910 in the cross-cluster task execution device 900 shown executes step S101. The computer device can execute step S102 through the code analysis module 920. The computer device can execute steps S103, S104 and S105 through the code conversion module 930. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, a cross-cluster task execution method is implemented.

[0092] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] In some embodiments of the present application, a computer device is provided, comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor in the steps of the cross-cluster task execution method. The steps of the cross-cluster task execution method here may be the steps of the cross-cluster task execution method in the above-mentioned embodiments.

[0094] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program, and the computer program is loaded by a processor, so that the processor executes the steps of the above-mentioned cross-cluster task execution method. The steps of the cross-cluster task execution method here can be the steps of the cross-cluster task execution method in each of the above-mentioned embodiments.

[0095] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0096] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above is a detailed introduction to a cross-cluster task execution method, device, computer equipment and storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A cross-cluster task execution method, characterized in that: include: Acquire task data of a target task to be run and a running instruction of the target task, wherein the task data is code data running on an initial cluster, and the running instruction includes parameter information of a target cluster running the target task; Determine, according to the submission class in the task data, a task submission environment and an initial code topology for running the target task on the initial cluster; Converting the task submission environment into environment information for running the target task on the target cluster according to the parameter information; Converting the initial code topology into a target code topology for the target task to run on the target cluster according to the parameter information; The environment information and the target code topology are encapsulated to obtain the target code for the target task to run on the target cluster.

2. The cross-cluster task execution method according to claim 1, characterized in that: The converting the initial code topology into a target code topology for the target task to run on the target cluster according to the parameter information includes: Parsing the initial code topology to obtain the first node operator and directed edge of the target task; Converting the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information; The second node operators are connected according to the parameter information and the directed edges to obtain a target code topology for the target task running on the target cluster.

3. The cross-cluster task execution method according to claim 2, characterized in that: The converting the first node operator of the initial cluster into the second node operator of the target cluster according to the parameter information includes: Parsing a data processing logic module of a first node operator of the initial cluster; Determine a data input module and a data output module of the second node operator according to the parameter information; A second node operator of the target cluster is obtained according to the data processing logic module, the data input module and the data output module.

4. The cross-cluster task execution method according to claim 2 or 3, characterized in that: The node operator includes a data source operator and a data processing operator, and the first node operator of the initial cluster is converted into the second node operator of the target cluster according to the parameter information, including: Converting the first data source operator of the initial cluster into the second data source operator of the target cluster according to the parameter information; The first data processing operator of the initial cluster is converted into the second data processing operator of the target cluster according to the parameter information.

5. The cross-cluster task execution method according to claim 3, characterized in that: After encapsulating the environment information and the target code topology to obtain the target code for the target task to run on the target cluster, the method further includes: When the target task is run on the target cluster, the data input module of the target second node operator receives the first data sent by the previous second node operator, the data processing logic module of the target second node operator processes the first data to obtain second data, and the data output module of the target second node operator sends the second data to the next second node operator.

6. The cross-cluster task execution method according to claim 1, characterized in that: The step of converting the task submission environment into environment information for running the target task on the target cluster according to the parameter information includes: If the task submission environment is local operation, converting the parameter information into environment information for the target task to run locally in the target cluster; If the task submission environment is remote submission, the parameter information is converted into environment information for remote submission of the target task in the target cluster.

7. The cross-cluster task execution method according to claim 4, characterized in that: The environment information includes task execution environment information and cluster interface environment information. The encapsulating the environment information and the target code topology to obtain the target code for the target task to run on the target cluster includes: According to the task execution environment information and the cluster interface environment information, encapsulating the target code topology to obtain the target code; The target code is submitted to the target cluster for execution based on the cluster interface environment information.

8. A cross-cluster task execution device, characterized in that: include: An information acquisition module, used to acquire task data of a target task to be run and an operation instruction of the target task, wherein the task data is code data running on an initial cluster, and the operation instruction includes parameter information of a target cluster running the target task; A code analysis module, which is in communication with the information acquisition module and is used to determine the task submission environment and initial code topology of the target task running on the initial cluster according to the submission class in the task data; A code conversion module is communicatively connected with the code analysis module, and is used to convert the task submission environment into the environment information of the target task running on the target cluster according to the parameter information; convert the initial code topology into the target code topology of the target task running on the target cluster according to the parameter information; and encapsulate the environment information and the target code topology to obtain the target code of the target task running on the target cluster.

9. A computer device, characterized in that: The computer device comprises: one or more processors; A memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the cross-cluster task execution method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the steps in the cross-cluster task execution method according to any one of claims 1 to 7.

Images