A method, apparatus, device and medium for creating a data processing task

Abstract

The application relates to the computer technical field, in particular to a data processing task creation method and device, equipment and medium. Embodiments of the application receive at least one execution task sent by a client; for any one execution task, a Flink execution operator corresponding to the execution task is generated according to task parameters of the execution task and task dependency information corresponding to the execution task; a directed acyclic graph is generated according to the Flink execution operators corresponding to the execution tasks; the generated directed acyclic graph is configured into a Flink cluster to create a data processing task. Embodiments of the application build real-time data processing tasks based on Flink components and perform unified task management on a large number of different execution tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data processing task creation method and device, equipment and medium. BACKGROUND

[0002] With rapid development of company business, a large amount of data is generated, and a real-time platform department needs to provide easy-to-use, stable and efficient real-time data services for a business department. More and more business jobs based on big data real-time computing are put into online use, and play an important role in the Internet field, such as real-time recommendation, double eleven real-time large screen statistics and the like.

[0003] Flink, as a mainstream real-time data processing engine, plays a crucial role in the field of real-time big data. How to build a real-time data processing task based on Flink components becomes a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a data processing task creation method, device, apparatus and computer storage medium, to provide a building scheme of a real-time data processing task based on Flink components.

[0005] In a first aspect, the embodiments of the present application provide a data processing task creation method, the data processing task is based on Flink components to process data, comprising:

[0006] Receiving at least one execution task sent by a client;

[0007] For any one execution task, according to the task parameters of the execution task and the task dependency information corresponding to the execution task, a Flink execution operator corresponding to the execution task is generated;

[0008] According to the Flink execution operator corresponding to each execution task, a directed acyclic graph is generated;

[0009] The generated directed acyclic graph is configured into a Flink cluster to create the data processing task.

[0010] Optionally, the Flink execution operator corresponding to the execution task is generated according to the task parameters of the execution task and the task dependency information corresponding to the execution task, and specifically comprises:

[0011] The task parameters of the execution task and the task dependency information corresponding to the execution task are loaded into the session corresponding to the execution task through an execution engine;

[0012] Based on the session comprising the task parameter of the execution task and the task dependency information corresponding to the execution task, a runstatement method is called to generate the Flink execution operator corresponding to the execution task.

[0013] Optionally, before loading the task parameter of the execution task and the task dependency information corresponding to the execution task to the session corresponding to the execution task through the execution engine, the method further comprises:

[0014] Based on the asynchronous message processing object corresponding to the execution task, the task parameter of the execution task is acquired.

[0015] According to the task parameter of the execution task and the corresponding relationship between the plurality of task parameters and the session stored in the session manager, the session corresponding to the execution task is determined.

[0016] Optionally, the method further comprises:

[0017] If it is determined that the session manager does not exist the session corresponding to the execution task, the session corresponding to the execution task is created according to the task parameter of the execution task.

[0018] In the session manager, the corresponding relationship between the task parameter of the execution task and the created session is stored.

[0019] Optionally, the generating the directed acyclic graph according to the Flink execution operator corresponding to each execution task comprises:

[0020] According to the Flink execution operator corresponding to each execution task, the directed acyclic graph is generated by triggering the execute method.

[0021] Optionally, before the data processing task is created, the method further comprises:

[0022] According to the preset resource manager running mode, a YARN client is constructed.

[0023] According to the constructed YARN client, the generated directed acyclic graph is configured to YARN, and the YARN is run.

[0024] In a second aspect, a data processing task creation method is provided, and the data processing task is based on a Flink component to perform data processing, which comprises:

[0025] In response to the user submitting at least one task execution command, the at least one task execution command is respectively subjected to syntax normalization processing to obtain at least one execution task;

[0026] The at least one execution task is sent to the server through a task gateway, so that the server creates the data processing task according to the at least one execution task.

[0027] In a third aspect, an embodiment of the present application provides a device for creating a data processing task, comprising:

[0028] A receiving module is configured to receive at least one execution task sent by a client;

[0029] A generating module is configured to, for any one execution task, generate a Flink execution operator corresponding to the execution task according to a task parameter of the execution task and task dependency information corresponding to the execution task;

[0030] A determining module is configured to generate a directed acyclic graph according to the Flink execution operator corresponding to each execution task;

[0031] A creating module is configured to configure the generated directed acyclic graph into a Flink cluster to create the data processing task.

[0032] In a fourth aspect, an embodiment of the present application provides a device for creating a data processing task, comprising:

[0033] A processing module is configured to, in response to a user submitting at least one task execution command, respectively subject the at least one task execution command to syntax normalization processing to obtain at least one execution task;

[0034] A sending module is configured to send the at least one execution task to a server through a task gateway, so that the server creates the data processing task according to the at least one execution task.

[0035] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising:

[0036] A processor;

[0037] A memory for storing instructions executable by the processor;

[0038] The processor is configured to execute the instructions to implement the method for creating a data processing task according to any one of the first aspect or the second aspect.

[0039] Sixthly, embodiments of this application provide a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a method for creating a data processing task as described in either the first or second aspect above.

[0040] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that is executed by a processor to implement a method for creating a data processing task as described in either the first or second aspect above.

[0041] In this embodiment, after receiving at least one execution task sent by the client, for any given execution task, a Flink execution operator corresponding to the execution task is generated based on the task parameters and corresponding task dependency information. A directed acyclic graph (DAG) is generated based on the Flink execution operators for each execution task. The generated DAG is then configured into the Flink cluster to create a data processing task. This embodiment provides a method for creating data processing tasks, enabling unified task management for a large number of different execution tasks, a unified submission entry point for execution task execution, and improved task execution speed. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic diagram illustrating an application scenario for an optional data processing task creation method provided in an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a method for creating a client-side data processing task, as provided in this application embodiment;

[0045] Figure 3 A schematic diagram illustrating a task execution command input interface displayed in a client's display interface, provided in an embodiment of this application;

[0046] Figure 4 This application provides a schematic diagram illustrating the submission of a task execution command in an embodiment of the present application.

[0047] Figure 5 A schematic diagram illustrating a task submission and execution based on a task gateway, as provided in this application embodiment;

[0048] Figure 6A whole flow chart of a data processing task creation method provided by an embodiment of the present application is provided on the client side;

[0049] Figure 7 A flow chart of a data processing task creation method provided by an embodiment of the present application is provided on the server side;

[0050] Figure 8 A task gateway, an asynchronous message processing object, a session manager, and a session relationship diagram based on a Netty server provided by an embodiment of the present application are shown;

[0051] Figure 9 A whole flow chart of a data processing task creation method provided by an embodiment of the present application is provided on the client side;

[0052] Figure 10 A device diagram for creating a data processing task provided by an embodiment of the present application is shown;

[0053] Figure 11 Another device diagram for creating a data processing task provided by an embodiment of the present application is shown;

[0054] Figure 12 A structure diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0056] The application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art can know that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. In the description of the present application, unless otherwise specified, the meaning of “multiple” is two or more.

[0057] With the rapid development of company business, the data generated is growing rapidly, and the real-time platform department needs to provide easy-to-use, stable and efficient real-time data services for the business department. Based on the real-time computing of big data, more and more business operations are put into online use. Many companies will use Flink engine as the main component of real-time data processing. Therefore, many companies will take Flink component as the core to build or develop a set of real-time processing platform suitable for the company. However, the business needs of different departments often change, and different users in each department write and execute tasks, which causes scattered code management, difficulty in locating problems, and unreasonable allocation of task execution resources. Therefore, it is necessary to manage and submit a large number of different execution tasks uniformly.

[0058] To solve the above problems, the application provides a data processing task creation method, receiving at least one execution task sent by a client; for any one execution task, generating a Flink execution operator corresponding to the execution task according to the task parameters of the execution task and the task dependency information corresponding to the execution task; generating a directed acyclic graph according to the Flink execution operator corresponding to each execution task; configuring the generated directed acyclic graph into a Flink cluster to create a data processing task.

[0059] The user of the embodiment of the application can trigger the creation of a task through the client, and create a data processing task on the server side.

[0060] As shown in Figure 1 An application scenario diagram of an optional data processing task creation method of the embodiment of the application, including a server 100 and a terminal 101. The server 100 and the terminal 101 can be communicatively connected through a network to implement the data processing task creation method of the application.

[0061] The user can use the server 100 to interact with the terminal 101 through the network, such as receiving or sending messages. The terminal 101 can be installed with various client application programs, such as program writing application, web browser application, search application, etc. The terminal 101 can be various electronic devices with display screen and supporting web browsing, including but not limited to smart phones, tablet computers, desktop computers, etc.

[0062] The client installed on the terminal 101 is used to respond to at least one task execution command submitted by the user, and perform syntax normalization processing on the at least one task execution command to obtain at least one execution task; send the at least one execution task to the server through a task gateway, so that the server creates a data processing task according to the at least one execution task.

[0063] Server 100 receives at least one execution task sent by the client; for any given execution task, it generates a Flink execution operator corresponding to the task based on the task parameters and corresponding task dependencies; it generates a directed acyclic graph (DAG) based on the Flink execution operators for each execution task; and it configures the generated DAG into the Flink cluster to create data processing tasks. Server 100 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0064] like Figure 2 As shown in the flowchart of a data processing task creation method provided in this application embodiment, applied to the client side, it may specifically include the following steps:

[0065] Step S201: Respond to at least one task execution command submitted by the user, and perform syntax normalization processing on the at least one task execution command to obtain at least one execution task;

[0066] Step S202: Send at least one execution task to the server through the task gateway so that the server can create a data processing task based on at least one execution task.

[0067] When a user needs to create a data processing task, the user can input the task execution command through the client used to create the data processing task. The client performs syntax normalization on at least one task execution command submitted by the user to obtain the execution task. The client then sends the at least one execution task obtained after normalization to the server through the task gateway, and the server executes the data processing task to create the task.

[0068] In practice, when a data processing task needs to be created, the user launches the client through the terminal's interface; for example, the user can double-click the client's icon to launch the client.

[0069] The client responds to the user's startup action, loads and registers the driver, and creates a Connection object and a Statement object;

[0070] After the client starts up, the task execution command input interface is displayed on the client's screen; for example... Figure 3 As shown, users can enter task execution commands on this interface;

[0071] like Figure 4 As shown, after entering the task execution command in the interface, the user clicks the "Submit" option to submit the task execution command.

[0072] The user-submitted task execution command in this application embodiment may include one or more execution tasks; for example, the execution tasks in the user-submitted task execution command may include: task creation and task query, etc.

[0073] For example, based on their own business needs, users can enter task execution commands in the client as follows:

[0074] create temporary table's_orders'(

[0075] order_id BIGINT COMMENT 'Order ID, auto-incrementing',

[0076] auction_id BIGINT COMMENT'commodity id',

[0077] user_id BIGINT COMMENT'user_id',

[0078] tran_amount INT COMMENT'Transaction Amount',

[0079] cate_id INT'category id',

[0080] gmt_create as CURRENT_TIMESTAMP'Order Creation Time',)

[0081] with(

[0082] 'connector' = 'datagen');

[0083] create temporary table'order_sink'(

[0084] order_cnt BIGINT,

[0085] gmv BIGINT)

[0086] with(

[0087] 'connector' = 'kafka'

[0088] 'properties' = '...');

[0089] INSERT INTO 'order_sink'

[0090] select

[0091] COUNT(*) AS order_cnt -- total orders

[0092] SUM(tran_amount) AS gmv -- total gross money value

[0093] from

[0094] s_orders

[0095] GROUP BY auction_id, TUMBLE(gmt_create, INTERVAL '1' MINUTES).

[0096] When the user inputs a task execution command on the client side, the user does not need to be proficient in programming syntaxes such as Java and Flink features, and uses the Flink SQL language to write the task execution command, so that the user can quickly start with only simple SQL syntax, improve the business efficiency of the user in writing the task execution command, and unify the development specification input by the user; the Flink SQL language is a set of development languages that meet the standard SQL semantics and are designed to reduce the threshold of users using real-time computing.

[0097] In step S201, the client obtains the task execution command submitted by the user, and performs syntax normalization processing on the task execution command by using the Calcite tool.

[0098] The syntax normalization processing includes, but is not limited to, syntax checking and syntax optimization.

[0099] The embodiment of the present application performs syntax normalization processing on the task execution command submitted by the user, reduces the problem of non-standard syntax of the user input task execution command, so that the execution tasks sent to the task gateway are all after syntax normalization processing, and the running efficiency of the task gateway can be improved. In step S202, the client sends at least one execution task obtained after the syntax normalization processing to the server through the task gateway; since the client of the embodiment of the present application sends at least one execution task obtained after the syntax normalization processing to the server through the task gateway, the submission path of the task submitted through the task gateway can be unified.

[0100] After the client sends at least one execution task obtained after the syntax normalization processing to the server through the task gateway through the RESTful API, the client obtains a ResultSet object, and obtains running state query data through the ResultSet object; the client is completed after obtaining the running state query data, and closes the ResultSet object, and closes the Statement object and the Connection object.

[0101] In the embodiments of the present application, a task gateway is constructed based on a Netty server; the Netty server is implemented based on a RestServerEndpoint in a Flink-runtime dependency; a dependency is a kind of relationship, indicating a connection between classes, indicating a definition that one class depends on another class, and the dependency relationship is unidirectional. In order to facilitate understanding of the dependency relationship, an example is given: assuming that there are two classes, class A and class B, and a parameter of a member method of class A has class B, which indicates that class A uses class B, class A depends on class B, and the dependency relationship is a uses-a relationship. It can be understood that, in addition to being used as a method parameter of a dependent party, the dependency relationship can also exist as a method return value of a dependent party, that is, the dependency relationship has multiple forms of expression.

[0102] As shown in Figure 5 , the embodiments of the present application provide a schematic diagram of submitting and executing a task based on a task gateway, a user accesses his own application through a RESTful API or a JDBC development package integrated with a RESTful API, Figure 5 , Beeline, Java application, Zeppelin and the like in the application represent development tools required according to actual business needs; Beeline is a tool of an open source community; Java application represents codes developed by a user according to different business needs; Zeppelin is a notebook providing interactive data analysis and based on Web, which is convenient for making data-driven, interactive and collaborative beautiful documents, and supports multiple languages, including Scala (using Apache Spark), Python (Apache Spark), SparkSQL, Hive, Markdown, Shell and the like; JDBC (Java Database Connectivity) is a Java API for executing SQL statements, which can provide a unified access interface for multiple relational databases.

[0103] The client submits a task through an IP and a port exposed by the task gateway. The IP and the port exposed by the task gateway are a RESTful API or a JDBC development package integrated with the RESTful API. The JDBC development package integrated with the RESTful API provides a Flink-JDBC dependency and internally integrates the RESTful API, so that the same DriverManager as Mysql can be used for connection. The connection in the JDBC connection mode can be integrated into third-party visual development tools, such as multiple data analysis tools, a beeline command line tool in the data analysis tool, a hue visual tool, and the like. The RESTful API (Representational State Transfer) is a protocol for standardizing the interaction between multiple forms of front ends and the same back end.

[0104] As shown in Figure 6 A flowchart of a method for creating a data processing task on a client side is provided in the embodiments of the application, and the method can specifically include the following steps:

[0105] In step S601, in response to a start operation triggered by a user, a client starts to load and register a driver.

[0106] In step S602, the client creates a Connection object.

[0107] In step S603, the client creates a Statement object.

[0108] In step S604, the client obtains a task execution command input by a user.

[0109] In step S605, the client performs syntax normalization processing on the task execution command through a Calcite tool.

[0110] In step S606, the client sends at least one execution task obtained through the syntax normalization processing to a server through a task gateway through a RESTful API.

[0111] In step S607, the client obtains a ResultSet object and obtains running state query data through the ResultSet object.

[0112] In step S608, the client closes the ResultSet object.

[0113] In step S609, the client closes the Statement object.

[0114] In step S610, the client closes the Connection object.

[0115] As Figure 7 shown, the embodiment of the present application provides a data processing task creation method flow chart, which is applied to the server side and can specifically include the following steps:

[0116] Step S701, receiving at least one execution task sent by the client;

[0117] Step S702, for any one execution task, generating a Flink execution operator corresponding to the execution task according to the task parameters of the execution task and the task dependency information corresponding to the execution task;

[0118] Step S703, generating a directed acyclic graph according to the Flink execution operators corresponding to each execution task;

[0119] Step S704, configuring the generated directed acyclic graph into a Flink cluster to create a data processing task.

[0120] In step S701, the server receives at least one execution task sent by the client through the task gateway; wherein the execution task is obtained after the client performs syntax standardization processing on the task execution command submitted by the user in response to the task execution command.

[0121] In an optional implementation, after the server receives the execution task, the server converts the at least one execution task submitted by the client according to the execution task optimization rules set by the server itself, to obtain the execution task after optimization; and / or, after the server receives the execution task, the server adjusts the parameters in the execution task according to the parameters in the received execution task and the actual situation of the user's business requirements, and uniformly manages and controls the resources corresponding to the execution task, to prevent the problem of insufficient resources and avoid resource waste.

[0122] After the server receives at least one execution task, the server generates a Flink execution operator corresponding to each execution task;

[0123] Optionally, for any one execution task, the embodiment of the present application generates a Flink execution operator corresponding to the execution task according to the following manner:

[0124] The execution engine loads the task parameters of the execution task and the task dependency information corresponding to the execution task into the session corresponding to the execution task; based on the session containing the task parameters of the execution task and the task dependency information corresponding to the execution task, the runstatement method is called to generate a Flink execution operator corresponding to the execution task.

[0125] In the implementation, the task parameter of the execution task is obtained based on an asynchronous message processing object (handler) corresponding to the execution task;

[0126] The task parameter of the execution task includes, but is not limited to, a session ID and a task name corresponding to the execution task.

[0127] It should be noted that the session manager stores a plurality of sessions. Figure 8 As shown in the figure, the session manager stores n sessions, each session includes a session ID, shared resources and exclusive resources, and each session is created based on an asynchronous message processing object (handler). The exclusive resources of the sessions created based on different types of handlers are different. For example, the handler includes a select handler, a status handler, a cancel handler and the like.

[0128] The shared resources of the session are shared resources that can be relied on by all sessions. For example, it can be an execution default parameter, a class loader and the like. The exclusive resources of the session are exclusive resources relied on by the session according to business needs when a user develops a task execution command. For example, it can be tEnv, a catalogManager and the like.

[0129] It should be noted that the shared resources of the n sessions stored in the session manager can be the same resources. Figure 8

[0130] After the asynchronous message processing object (handler) obtains the task parameter of the execution task, the session corresponding to the execution task is searched from the session manager based on the session ID contained in the task parameter.

[0131] It should be noted that if the session corresponding to the execution task is not found in the session manager, the session corresponding to the execution task is created in the session manager through the asynchronous message processing object corresponding to the execution task.

[0132] ​The embodiment of the application determines the session corresponding to the execution task, and then loads the task parameters of the execution task and the task dependency information corresponding to the execution task into the session corresponding to the execution task through the execution engine, runs the corresponding runstatement method, and generates the Flink execution operator corresponding to the execution task.

[0133] It should be noted that the execution engine is a shared execution engine for all sessions, and the Flink execution operator corresponding to the execution task can be a Flink operation, and the task dependency information corresponding to the execution task can be shared resources and exclusive resources in the session corresponding to the execution task.

[0134] In step S703, a directed acyclic graph is generated by triggering the execute method according to the Flink execution operator corresponding to each execution task; wherein the directed acyclic graph represents the business logic of at least one execution task.

[0135] For example, the execution task in the session includes a creation task and a query task, and a directed acyclic graph is generated by triggering the execute method according to the corresponding Flink execution operator.

[0136] After the directed acyclic graph is generated, before the data processing task is created, a YARN client is constructed according to a preset resource manager running mode; the generated directed acyclic graph is configured to YARN according to the constructed YARN client, and YARN is run. YARN (Yet Another Resource Negotiator) is a new Hadoop resource manager, which is a general resource management system and can provide unified resource management and scheduling for upper-layer applications.

[0137] In step S704, the server creates ProgramDeployer, deploys the generated directed acyclic graph into the Flink cluster through ProgramDeployer, so that the execution task can run normally, and the data processing task is created; wherein the Flink cluster is composed of multiple Flink components.

[0138] After the data processing task is obtained, when the execution task is run, the execution operation is added to the execution task of the session; all execution tasks in the same session ID run in the same session.

[0139] For example, tasks performed in a session include creation tasks and query tasks. Creation tasks and query tasks are bound using the same session ID, and during session runtime, creation tasks and query tasks are executed in the same session.

[0140] It should be noted that there is no fixed order in which the steps of configuring the generated directed acyclic graph (DAG) onto YARN and running YARN are performed, and the steps of configuring the generated DAG onto a Flink cluster and creating a data processing task are performed.

[0141] like Figure 9 As shown in the figure, this application embodiment provides an overall flowchart of a method for creating a data processing task.

[0142] Step S901: Respond to at least one task execution command submitted by the user. The client performs syntax normalization processing on the at least one task execution command to obtain at least one execution task.

[0143] Step S902: The client sends at least one execution task to the server through the task gateway;

[0144] Step S903: The server receives at least one execution task sent by the client;

[0145] Step S904: The server obtains the task parameters of the task based on the asynchronous message processing object corresponding to the task.

[0146] Step S905: The server determines the session corresponding to the task to be executed based on the task parameters of the task to be executed and the correspondence between multiple task parameters and sessions stored in the session manager.

[0147] Step S906: For any given execution task, the server loads the task parameters and corresponding task dependency information of the execution task into the session corresponding to the execution task through the execution engine.

[0148] Step S907: The server calls the runstatement method based on the session containing the task parameters and the task dependency information corresponding to the task to generate the Flink execution operator corresponding to the task.

[0149] Step S908: The server generates a directed acyclic graph by triggering the execute method based on the Flink execution operator corresponding to each execution task;

[0150] Step S909: The server constructs a YARN client according to the preset resource manager running mode;

[0151] Step S910, the service end configures the generated directed acyclic graph to YARN according to the constructed YARN client, and runs YARN;

[0152] Step S911, the service end configures the generated directed acyclic graph to the Flink cluster, and creates a data processing task.

[0153] The embodiment of the application provides a device for creating a data processing task, as shown in the figure, the device comprises a receiving module 1001, a generating module 1002, a determining module 1003, and a creating module 1004, wherein: Figure 10

[0154] The receiving module 1001 is configured to receive at least one execution task sent by a client, wherein the at least one execution task is obtained by performing syntax normalization processing on at least one task execution command submitted by the client in response to a user.

[0155] The generating module 1002 is configured to generate a Flink execution operator corresponding to any execution task according to task parameters of the execution task and task dependency information corresponding to the execution task.

[0156] The determining module 1003 is configured to generate a directed acyclic graph according to the Flink execution operator corresponding to each execution task.

[0157] The creating module 1004 is configured to configure the generated directed acyclic graph to a Flink cluster to create the data processing task.

[0158] Optionally, the generating module 1002 is specifically configured to:

[0159] Load the task parameters of the execution task and the task dependency information corresponding to the execution task to a session corresponding to the execution task through an execution engine.

[0160] Invoke a runstatement method based on the session containing the task parameters of the execution task and the task dependency information corresponding to the execution task to generate the Flink execution operator corresponding to the execution task.

[0161] Optionally, after receiving the at least one execution task sent by the client, before loading the task parameters of the execution task and the task dependency information corresponding to the execution task to the session corresponding to the execution task through the execution engine, the generating module 1002 is further configured to:

[0162] ​obtain the task parameter of the execution task based on the asynchronous message processing object corresponding to the execution task;

[0163] determine the session corresponding to the execution task according to the task parameter of the execution task and a correspondence between a plurality of task parameters and sessions stored in the session manager.

[0164] Optionally, the generating module 1002 is further configured to:

[0165] if it is determined that the session manager does not have the session corresponding to the execution task, create the session corresponding to the execution task according to the task parameter of the execution task;

[0166] store the correspondence between the task parameter of the execution task and the created session in the session manager, so as to:

[0167] Optionally, the determining module 1003 is specifically configured to:

[0168] generate a directed acyclic graph by triggering an execute method according to the Flink execution operator corresponding to each execution task.

[0169] Optionally, before the creating module 1004 generates the directed acyclic graph representing the business logic between the at least one execution task and creates the data processing task, the creating module 1004 is further configured to:

[0170] construct a YARN client according to a preset resource manager running mode;

[0171] configure the generated directed acyclic graph to the YARN according to the constructed YARN client, and run the YARN.

[0172] The embodiment of the application further provides another apparatus for creating a data processing task, as shown in Figure 11 The apparatus comprises a processing module 1101 and a sending module 1102, wherein:

[0173] The processing module 1101 performs syntax normalization processing on at least one task execution command submitted by a user to obtain at least one execution task.

[0174] The sending module 1102 sends the at least one execution task to a server through a task gateway, so that the server creates the data processing task according to the at least one execution task.

[0175] The embodiment of the application further provides an electronic device. The following refers to Figure 12To describe an electronic device 120 according to this embodiment of the present application. Figure 12 The electronic device 120 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0176] like Figure 12 As shown, the electronic device 120 is presented in the form of a general electronic device. The components of the electronic device 120 may include, but are not limited to: at least one processor 121, at least one memory 122, and a bus 123 connecting different system components (including memory 122 and processor 121).

[0177] Bus 123 represents one or more of several types of bus structures, including memory bus or memory controller, peripheral bus, processor, or local bus using any of the multiple bus structures.

[0178] The memory 122 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1221 and / or cache memory 1222, and may further include read-only memory (ROM) 1223.

[0179] The memory 122 may also include a program / utility 1225 having a set (at least one) of program modules 1224, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0180] Electronic device 120 can also communicate with one or more external devices 124 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 120, and / or with any device that enables electronic device 120 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 125. Furthermore, electronic device 120 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 126. As shown, network adapter 126 communicates with other modules used in electronic device 120 via bus 123. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 120, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0181] In an example embodiment, a computer readable storage medium, for example, a memory 122 including instructions, is also provided, which can be executed by the processor 121 to complete the creation method of the data processing task as described above. Optionally, the storage medium can be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0182] In an example embodiment, a computer program product is also provided, which includes a computer program, and the computer program is executed by the processor 121 to implement any of the creation methods of the data processing task as provided by the present application.

[0183] In an example embodiment, each aspect of the creation method of the data processing task provided by the present application can also be implemented as a program product in the form of a program code, which is used to make the computer device execute the steps of the creation method of the data processing task according to various example embodiments of the present application described above in the specification when the program product is run on the computer device.

[0184] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0185] The program product for the creation method of the data processing task of the embodiments of the present application can adopt a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used or combined with an instruction execution system, device or apparatus.

[0186] A readable signal medium can be any available medium or media that can be accessed by a general purpose or special purpose electronic computer or other machine including a computer readable medium. By way of example, and not limitation, such computer readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium or media that can be used to carry or store desired computer readable code in the form of computer readable instructions, data structures, program modules or other data, alone or in combination with one another. In addition, the computer readable medium can be readable by the computing device 1002. The computer readable medium can be presented to the computing device 1002. The computer readable medium can be external to the computing device 1002. The computer readable medium can include a removable computer readable storage medium or media.

[0187] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0188] Program code, used by or in connection with the routines described herein, can be written in any suitable programming language such as object oriented programming languages, such as Java, C++, and the like, conventional procedural programming languages, such as the "C" programming language, or the like, and / or combined programming languages. Program code can execute entirely on a user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user's electronic device through any suitable network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device, such as through the Internet using an Internet Service Provider (ISP).

[0189] It should be noted that although the above detailed description refers to several units or sub-units of a device, such division into units or sub-units is merely exemplary and not mandatory. In fact, according to an implementation of the present application, features and functions of two or more units described above can be embodied in one unit. Conversely, a unit described above can be divided into several sub-units to embody its features and functions.

[0190] Furthermore, although the operations of the method according to the application are described in a particular, sequential order, this order is not necessarily a mandatory order, and the operations can be performed in different orders or at the same time. Additionally or alternatively, certain steps can be omitted, combined, or further divided into further steps.

[0191] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0192] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0193] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0195] While preferred embodiments of the application have been described, modifications and variations can be effected to such embodiments by those of ordinary skill in the art once the nature of the

[0196] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for creating a data processing task, characterized in that, The data processing task is based on Flink components for data processing, and the method includes: The client receives at least one execution task sent by the client, wherein the at least one execution task is obtained by the client responding to at least one task execution command submitted by the user and then performing syntactic normalization on the at least one execution task command; For any given execution task, the execution engine loads the task parameters and corresponding task dependency information of the execution task into the session corresponding to the execution task. Based on the session containing the task parameters and corresponding task dependency information, the `runStatement` method is called to generate the Flink execution operator corresponding to the execution task. The task dependency information includes shared resources and unique resources in the session corresponding to the execution task. The shared resources of the session are shared resources that all sessions can depend on. The unique resources of sessions created based on different types of asynchronous message processing objects are different. Generate a directed acyclic graph based on the Flink execution operators corresponding to each execution task; Configure the generated directed acyclic graph into the Flink cluster to create the data processing task.

2. The method as described in claim 1, characterized in that, After receiving at least one execution task sent by the client, before loading the task parameters and corresponding task dependency information of the execution task into the session corresponding to the execution task through the execution engine, the process further includes: The task parameters of the execution task are obtained based on the asynchronous message processing object corresponding to the execution task. The session corresponding to the task being executed is determined based on the task parameters of the task being executed and the correspondence between multiple task parameters and sessions stored in the session manager.

3. The method as described in claim 2, characterized in that, The method further includes: If it is determined that the session manager does not have a session corresponding to the task being executed, then the session corresponding to the task being executed is created according to the task parameters of the task being executed. The session manager stores the correspondence between the task parameters of the executed task and the created session.

4. The method according to any one of claims 1 to 3, characterized in that, The process of generating a directed acyclic graph based on the Flink execution operators corresponding to each execution task specifically includes: Based on the Flink execution operators corresponding to each task, a directed acyclic graph is generated by triggering the execute method.

5. The method according to any one of claims 1 to 3, characterized in that, After generating the directed acyclic graph and before creating the data processing task, the method further includes: The YARN client is constructed based on the preset resource manager running mode; Based on the constructed YARN client, the generated directed acyclic graph is configured onto YARN, and YARN is run.

6. A method for creating a data processing task, characterized in that, The data processing task is based on Flink components for data processing, and the method includes: In response to at least one task execution command submitted by the user, the at least one task execution command is subjected to syntax normalization processing to obtain at least one execution task; The at least one execution task is sent to the server through a task gateway, so that the server creates the data processing task based on the at least one execution task. The data processing task is created by the server after configuring a directed acyclic graph (DAG) into the Flink cluster. The DAG is generated by the server based on the Flink execution operators corresponding to each execution task. The Flink execution operator corresponding to any one of the execution tasks is generated by the server by calling the runstatement method based on a session containing the task parameters and task dependency information of the execution task. The session is the session corresponding to the execution task loaded by the server through the execution engine, which contains the task parameters and task dependency information of the execution task. The task dependency information includes shared resources and exclusive resources in the session corresponding to the execution task. The shared resources of the session are shared resources that all sessions can depend on. The exclusive resources of sessions created based on different types of asynchronous message processing objects are different.

7. An apparatus for creating data processing tasks, characterized in that, include: The receiving module is used to receive at least one execution task sent by the client, wherein the at least one execution task is obtained by the client responding to at least one task execution command submitted by the user and performing syntax normalization on the at least one execution task command; The generation module is used to load the task parameters and corresponding task dependency information of any given execution task into the session corresponding to the execution task through the execution engine; based on the session containing the task parameters and corresponding task dependency information, the runStatement method is called to generate the Flink execution operator corresponding to the execution task. The task dependency information includes shared resources and unique resources in the session corresponding to the execution task. The shared resources of the session are shared resources that all sessions can depend on, while the unique resources of sessions created based on different types of asynchronous message processing objects are different. The determination module is used to generate a directed acyclic graph based on the Flink execution operators corresponding to each execution task; A module is created to configure the generated directed acyclic graph into the Flink cluster, thereby creating the data processing task.

8. An apparatus for creating data processing tasks, characterized in that, include: The processing module responds to at least one task execution command submitted by the user, and performs syntax normalization processing on the at least one task execution command to obtain at least one execution task; The sending module sends the at least one execution task to the server through the task gateway, so that the server can create the data processing task based on the at least one execution task. The data processing task is created by the server after configuring the directed acyclic graph (DAG) into the Flink cluster. The DAG is generated by the server based on the Flink execution operators corresponding to each execution task. The Flink execution operator corresponding to any one of the execution tasks is generated by the server by calling the runstatement method based on the session containing the task parameters and task dependency information of the execution task. The session is the session corresponding to the execution task loaded by the server through the execution engine, which contains the task parameters and task dependency information of the execution task. The task dependency information includes shared resources and exclusive resources in the session corresponding to the execution task. The shared resources of the session are shared resources that all sessions can depend on. The exclusive resources of sessions created based on different types of asynchronous message processing objects are different.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for creating a data processing task as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method for creating a data processing task as described in any one of claims 1-6.

Images