Distributed task generation method and device, equipment and storage medium

Abstract

The present disclosure relates to the technical field of computers, and particularly relates to a distributed task generation method and device, equipment and a storage medium. The method comprises the following steps: obtaining a first configuration instruction input through a table editing interface, configuring at least one table model according to the first configuration instruction; obtaining a second configuration instruction input through a flow editing interface, configuring a flow arrangement model according to the second configuration instruction and the table model; obtaining a third configuration instruction input through a task editing interface, configuring a task model according to the third configuration instruction and the table model; and parsing the flow arrangement model, the task model and the at least one table model, and converting them into a task data packet corresponding to a target distributed task, wherein the task data packet is used to support running the target distributed task in a target distributed environment. The present disclosure is used to solve the defect of low efficiency in manually creating a distributed task in the prior art, and improve the efficiency in generating a distributed task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a distributed task generation method and device, equipment and storage medium. BACKGROUND

[0002] In the prior art, distributed technology is gradually applied in various industries. For example, Flink is an open source stream processing framework designed for distributed, high-performance, always-available and accurate stream processing applications. In recent years, Flink has been increasingly used in real-time data processing, such as Extract-Transform-Load (ETL) processing and real-time computing processing based on Flink. At present, when creating a distributed task such as a Flink task, a technical personnel needs to pre-master the usage and process arrangement of distributed technology and other related professional knowledge, and then manually constructs a distributed task according to specific needs. This manual construction of a distributed task requires a high level of professional skills of the technical personnel, occupies a large amount of human and time resources, and leads to low efficiency in generating a distributed task. SUMMARY

[0003] The present disclosure provides a distributed task generation method, device, equipment and storage medium to solve the defect of low efficiency in manually creating a distributed task in the prior art and improve the efficiency in generating a distributed task.

[0004] The present disclosure provides a distributed task generation method, comprising: obtaining a first configuration instruction input through a table editing interface, configuring at least one table model according to the first configuration instruction, wherein the table model is used to represent the logic of preprocessing source data; obtaining a second configuration instruction input through a process editing interface, configuring a process arrangement model according to the second configuration instruction and the table model, wherein the process arrangement model is used to represent the process arrangement of the target distributed task processing logic; obtaining a third configuration instruction input through a task editing interface, configuring a task model according to the third configuration instruction and the table model, wherein the task model is used to represent the overall description of the target distributed task; and parsing the process arrangement model, the task model and at least one table model, and converting them into a task data packet corresponding to the target distributed task, wherein the task data packet is used to support running the target distributed task in a target distributed environment.

[0005] According to the distributed task generation method provided by the present disclosure, a first configuration instruction input through a table editing interface is acquired, and at least one table model is configured according to the first configuration instruction, including: the configuration process of each table model is as follows: one or more table attribute components are displayed through the table editing interface, wherein each table attribute component corresponds to a table attribute of the table model; the first configuration instruction is acquired, wherein the first configuration instruction is generated according to a selection operation or a filling operation of at least one table attribute component input through the table editing interface; at least one table attribute is configured according to the first configuration instruction; when first confirmation information input through the table editing interface is acquired, the table model is generated based on the configured table attribute, wherein the first confirmation information is generated according to a click operation of a first logical button, and the first logical button is displayed on the table editing interface.

[0006] According to the distributed task generation method provided by the present disclosure, the table attribute component includes at least one of an identification component, a source component and a destination component; the identification component is used for configuring an identity of the table model; the source component is used for configuring an input attribute corresponding to the table model, and the input attribute is used for acquiring a data source; and the destination component is used for configuring an output attribute corresponding to the table model, and the output attribute is used for outputting a corresponding data result.

[0007] According to the distributed task generation method provided by the present disclosure, the second configuration instruction includes a flow framework configuration instruction and a component attribute configuration instruction; a second configuration instruction input through a flow editing interface is acquired, and a flow arrangement model is configured according to the second configuration instruction and the table model, including: one or more dragable components are displayed in a component display area in the flow editing interface; the flow framework configuration instruction is acquired, wherein the flow framework configuration instruction is generated according to a drag operation of at least one dragable component input in an editing area of the flow editing interface; a flow framework of the target distributed task is generated according to the flow framework configuration instruction; the component attribute configuration instruction is acquired, wherein the component attribute configuration instruction is generated according to a selection operation or a filling operation of at least one dragable component in the editing area input in an attribute configuration area of the flow editing interface; each dragable component in the flow framework is configured with attributes according to the component attribute configuration instruction, wherein the attribute configuration includes configuration of a connection relationship between the table model and the flow framework; when second confirmation information input through the flow editing interface is acquired, the flow arrangement model is generated based on the flow framework configured with attributes, wherein the second confirmation information is generated according to a click operation of a second logical button, and the second logical button is displayed on the flow editing interface.

[0008] According to the distributed task generation method provided by the present disclosure, the dragable component includes at least one of an input component, an output component and a calculation column component; the input component is used to configure the connection relationship between the table model and the process framework, and the input parameter of the table model is configured through a corresponding source component; the output component is used to output the data result after logical processing of data according to the input attribute of the table model configured by the input component; and the calculation column component is used to configure the logical processing of data.

[0009] According to the distributed task generation method provided by the present disclosure, a third configuration instruction input through a task editing interface is acquired, and a task model is configured according to the third configuration instruction and the table model, including: displaying one or more task attribute components through the task editing interface, wherein each task attribute component corresponds to a task attribute of the task model; acquiring the third configuration instruction, wherein the third configuration instruction is generated according to the selection operation or the filling operation of at least one task attribute component input through the task editing interface, and at least one task attribute component is used to configure the table model required by the target distributed task; configuring at least one task attribute according to the third configuration instruction; and when third confirmation information input through the task editing interface is acquired, generating the task model based on the configured task attribute, wherein the third confirmation information is generated according to the clicking operation of a third logic button, and the task editing interface displays the third logic button.

[0010] According to the distributed task generation method provided by the present disclosure, the process arrangement model, the task model and at least one table model are analyzed and converted into a task data packet corresponding to the target distributed task, including: analyzing the task model to determine at least one table model required by the target distributed task; analyzing the process arrangement model and at least one table model, converting the configured input component in the process arrangement model into a source end of the target distributed task; converting the configured output component in the process arrangement model into an output end of the target distributed task; converting the configured calculation column component in the process arrangement model into a basic operator of the target distributed task; arranging a logical processing process for the source end, the output end and the basic operator based on the process arrangement model, and generating the task data packet.

[0011] The disclosure further provides a distributed task generation apparatus, comprising: a table configuration module configured to obtain a first configuration instruction input via a table editing interface, and to configure at least one table model according to the first configuration instruction, wherein the table model is configured to represent a logic of preprocessing source data; a flow configuration module configured to obtain a second configuration instruction input via a flow editing interface, and to configure a flow orchestration model according to the second configuration instruction and the table model, wherein the flow orchestration model is configured to represent a flow orchestration of the target distributed task processing logic; a task configuration module configured to obtain a third configuration instruction input via a task editing interface, and to configure a task model according to the third configuration instruction and the table model, wherein the task model is configured to represent an overall description of the target distributed task; and a data conversion module configured to parse the flow orchestration model, the task model and at least one table model, and to convert them into a task data packet corresponding to the target distributed task, wherein the task data packet is configured to support running the target distributed task in a target distributed environment.

[0012] The disclosure further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the distributed task method according to any one of the above when executing the program.

[0013] The disclosure further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the distributed task method according to any one of the above.

[0014] The distributed task generation method, apparatus, device and storage medium provided by the disclosure are configured to input a first configuration instruction via a table editing interface to configure a table model, input a second configuration instruction via a flow editing interface to configure a flow orchestration model, input a third configuration instruction via a task editing interface to configure a task model, and then generate a task data packet of a target distributed task based on the flow orchestration model, the task model and at least one table model. That is, the configuration process of generating the target distributed task is visualized via the table editing interface, the flow editing interface and the task editing interface, the configuration difficulty of the target distributed task is reduced, the professional skill level requirement of technical personnel is lowered, and the efficiency of generating the target distributed task is improved. Meanwhile, the modular configuration of the target distributed task is formed by the table model, the flow orchestration model and the task model, which can save a large amount of human resources and time cost compared with the manual creation of the distributed task, and further improve the efficiency of generating the target distributed task. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0016] Figure 1 is one of the flow schematic diagrams of the distributed task generation method provided by the present disclosure;

[0017] Figure 2 is an entity-contact diagram of the distributed task provided by the present disclosure;

[0018] Figure 3 is an example diagram of the table editing interface provided by the present disclosure;

[0019] Figure 4 is an example diagram of the flow editing interface provided by the present disclosure;

[0020] Figure 5 is an example diagram of the task editing interface provided by the present disclosure;

[0021] Figure 6 is one of the task engine conversion process schematic diagrams provided by the present disclosure;

[0022] Figure 7 is the second one of the task engine conversion process schematic diagrams provided by the present disclosure;

[0023] Figure 8 is a structural schematic diagram of the distributed task generation device provided by the present disclosure;

[0024] Figure 9 is a structural schematic diagram of the electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in the following with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the embodiments of the present disclosure.

[0026] The distributed task generation method provided by the present disclosure is realized through a preset logical algorithm, which can be implemented in a processing device with logical processing capability, such as a central processing unit (CPU), a microcontroller unit (MCU), a computer, or any other device with processing capability.

[0027] The distributed task generation method provided by the present disclosure is realized through a preset logical algorithm, which can be implemented in a processing device with logical processing capability, such as a central processing unit (CPU), a microcontroller unit (MCU), a computer, or any other device with processing capability. Figures 1-7 The distributed task generation method provided by the present disclosure is realized through a preset logical algorithm, which can be implemented in a processing device with logical processing capability, such as a central processing unit (CPU), a microcontroller unit (MCU), a computer, or any other device with processing capability.

[0028] In one embodiment, as shown in Figure 1 The flow steps of the distributed task generation method are as follows:

[0029] Step 101, obtaining a first configuration instruction input through a table editing interface, and configuring at least one table model according to the first configuration instruction, wherein the table model is used to represent the logic of preprocessing the source data.

[0030] In this embodiment, a distributed task refers to a task running in a distributed environment, for example, a Flink task running in a Flink environment. A Flink task essentially refers to a processing framework that can logically process source data in a Flink environment. Different Flink tasks have different processing logic configurations. Similarly, any distributed task is a processing framework implemented in a corresponding distributed environment. The target distributed task refers to any distributed task that needs to be generated.

[0031] The preprocessing logic includes but is not limited to the processing logic of extracting, transforming, and loading the source data.

[0032] In this embodiment, as shown in Figure 2 The entity-relationship diagram (E-R diagram) of the distributed task, taking a table model included in the target distributed task as an example, is shown in the figure. The table model is the basic unit of the target distributed task and is used to identify the preprocessing of the source data. The E-R diagram corresponds to the architecture logic of a distributed task. Through the table editing interface, the flow editing interface, and the task editing interface, the target distributed task can be configured from different levels.

[0033] When configuring the task level, that is, configuring the overall description of the distributed task, each task attribute of the distributed task can be configured. Specifically, through the task editing interface, any one or several of the task identity document (id or ID), task type, task publishing platform configuration, and start-up parameter can be configured.

[0034] When the table model layer is configured, that is, the pre-processing process of the required source data of the distributed task is configured, each table attribute of the distributed task can be configured. Specifically, any one or several table attributes of the mapping field, table id, belonging cluster id, input and output of the table model are configured through the table editing interface. More specifically, each table attribute can further include corresponding preset sub-attributes. For example, the mapping field includes source field name, destination field name, field type, source table, source field and / or source table alias. The input attribute can select a message queue (mq or MQ) message or a data interface (D-type data interface connector, db or DB) connection. Similarly, the output attribute can select a mq message or a db connection. The mq message needs to configure parameters such as client ID (clilentid), topic, username, password, etc. The db connection needs to configure parameters such as password, username, table name, connection address, etc.

[0035] When the process arrangement layer is configured, that is, the process arrangement of the target distributed task is configured, specifically, the table model, input and output are regarded as calculation columns respectively. Then, the output field list, input field list and parameters such as sharding selection, user identification (UID), name, operator type and / or parallelism of the operator layer corresponding to each calculation column are configured through the process editing interface. For example, the input calculation column (input attribute) can be configured through the process editing interface. When the input attribute selects the mq message form, the specific parameters such as password and clilentid can also be configured. The output calculation column (output attribute) can be configured through the process editing interface. When the output attribute selects the db connection form, the specific parameters such as password and connection address can also be configured.

[0036] In this embodiment, the table model is the basic unit of the target distributed task. The source data is the original data that has not been processed. For example, the data in the enterprise is scattered, disordered and not standardized. In order to facilitate the subsequent data processing process, the source data needs to be pre-processed. Different table models are used to pre-process source data from different sources. The table model includes the processing logic of the pre-processing. Preferably, the pre-processing adopts the Extract, Transform and Load (ETL) mode. Specifically, ETL is a process of loading the data (i.e. source data) of the business system to the data warehouse after extraction, cleaning and conversion. The purpose is to integrate the scattered, disordered and non-standardized data in the enterprise together to provide analysis basis for subsequent decision-making. That is, the basic attribute information of ETL can be configured through the table model.

[0037] In this embodiment, the table editing interface refers to a visual interface capable of providing human-computer interaction. A technical person can configure various table attributes of a table model through the table editing interface. The table editing interface can be an interface integrated on a processing device. For example, when the processing device is a smart phone with a touch screen, the table editing interface can be displayed on the touch screen. The table editing interface can also be displayed on a separate visual device. The first configuration instruction input through the table editing interface can be transmitted to the processing device in a wired or wireless manner to generate a task data packet of a final target distributed task.

[0038] In one embodiment, one target distributed task can include one or more table models. Each table model can be configured through a table editing interface. Specifically, a first configuration instruction input through the table editing interface is obtained, and at least one table model is configured according to the first configuration instruction. The implementation process is as follows: the configuration process of each table model is as follows: one or more table attribute components are displayed through the table editing interface, wherein each table attribute component corresponds to one table attribute of the table model; the first configuration instruction is obtained, wherein the first configuration instruction is generated according to a selection operation or a filling operation of at least one table attribute component input through the table editing interface; at least one table attribute is configured according to the first configuration instruction; when first confirmation information input through the table editing interface is obtained, a table model is generated based on the configured table attribute, wherein the first confirmation information is generated according to a click operation of a first logical button, and the table editing interface displays the first logical button.

[0039] In this embodiment, the table attribute components displayed on the table editing interface can be pre-set according to actual conditions and needs. Each table attribute of a table model can be configured through each table attribute component. The table editing interface sets a first logical button. When each table attribute of a table model is configured, a technical person can click the first logical button to generate first confirmation information, so as to generate a table model according to the configured table attributes.

[0040] In one embodiment, the table attribute component includes at least one of an identification component, a source component, and a destination component; the identification component is used to configure the identity of the table model; the source component is used to configure the input attribute corresponding to the table model, and the input attribute is used to obtain a data source; and the destination component is used to configure the output attribute corresponding to the table model, and the output attribute is used to output a corresponding data result.

[0041] In this embodiment, since the table model is used to represent the processing logic of extracting, transforming, and loading the source data, the table model relates to how the source data is input and output. Therefore, the table editing interface sets the source component and the destination component to configure the input and output of the table model. Meanwhile, the identification component is used to configure the id of the table model, and the identity is used to represent the corresponding table model.

[0042] In one specific embodiment, the table model represents basic information of ETL of a table, including field mapping, field type, and extended information of table fields, etc. Figure 3 An example diagram of a table editing interface is shown in FIG. 1. In the table editing interface, a configuration mode component is provided, i.e., a Data Definition Language (DDL) parsing and manual configuration. The DDL parsing mode can automatically fill each table attribute based on data such as a Structured Query Language (SQL) table. The manual configuration can be configured by a technician for each table attribute. The two configuration modes make the table model configuration process more flexible and diversified.

[0043] In the table editing interface, the identification component includes a table id, a table name, and a table Chinese name, which are used to generate various identities of the table model. For example, the table id is 7, which can be automatically coded according to the generation order of the table model; the table name is pla_source_tabble_columr, which can be manually written; and the table Chinese name is source table structure, which can be manually written. Each identity can represent the table model, and which identity is used in the subsequent processing stage can be selected according to actual conditions.

[0044] In the table editing interface, the following table attribute components are also provided: a state, a table belonging cluster, a table type, a target table name, a parameter list, a table description, an added column, and attributes of each column. Specifically, the attributes of each column include a column id, a state, a Chinese name, a source column name, a target column name, a field type, a column attribute, a description, and an operation. The state can be selected as invalid or valid, indicating that the table model is invalid or valid. The table belonging cluster indicates a cluster required for storing the data result output by the table model, for example, test-es (demo cluster 1). The table type is a preset type, for example, a business table or a basic table. The target table name indicates the name of the result data output by the table model, for example, pla_source_tabble_columr, which can be manually written or changed. The parameter list indicates preset required logical processing parameters, for example, filtering, camel case, and filtering update column, each parameter corresponding to a processing mode of the source data. The table description can manually write some table description information, for example, source table structure.

[0045] The source component and the destination component are specifically set in the form of columns, and specifically include a source column name and a destination column name. The source column name represents information of source data input in the form of columns, such as id, table_id, seq, field_name, source_field_name or field_comment. The destination column name represents information of data results output in the form of columns, such as id, table_id, seq, field_name, source_field_name or field_comment. Each column is provided with a corresponding column id and a Chinese name representing the identity of the column, for example, the column id can be 71, 72, 73, 74, 75 or 76 generated by the system in sequence, and the Chinese name can be the ID of the table, the sequence of the field column, the field name, the source field name or the column name. The state represents the valid state or the invalid state of the column. The field type can be selected from any one of the preset types, such as a large integer number or a string. The column attribute can be selected from a preset column attribute, such as a primary key, an update column or a shard. The description can write information to be described in the column. The operation can display the corresponding operation processing of the column.

[0046] Of course, the above table editing interface is only an example, and more table attribute components can be set according to actual conditions and needs to better complete the configuration of the table model. The protection scope of the present application is not limited by the specific number and display form of the table attribute components.

[0047] In step 102, a second configuration instruction input through the flow editing interface is obtained, and a flow arrangement model is arranged according to the second configuration instruction and the table model, wherein the flow arrangement model is used to represent the flow arrangement of the target distributed task processing logic.

[0048] In the embodiment, the flow arrangement model is used to represent the flow arrangement of the target distributed task processing logic, that is, after the source data is acquired and preliminarily processed by the table model, how to logically process the preliminarily processed data to obtain the final data results, and output the data results according to the configuration of the table model.

[0049] In one embodiment, the second configuration instruction includes a flow framework configuration instruction and a component attribute configuration instruction. The second configuration instruction input through the flow editing interface is acquired, and the flow orchestration model is configured according to the second configuration instruction and the table model. The specific implementation process is as follows: one or more than one dragable component is displayed through the component display area in the flow editing interface; the flow framework configuration instruction is acquired, which is generated according to the drag operation of at least one dragable component input in the editing area of the flow editing interface; the flow framework of the target distributed task is generated according to the flow framework configuration instruction; the component attribute configuration instruction is acquired, which is generated according to the selection operation or the filling operation of at least one dragable component in the editing area input in the attribute configuration area of the flow editing interface; each dragable component in the flow framework is configured with attributes according to the component attribute configuration instruction, wherein the attribute configuration includes the configuration of the connection relationship between the table model and the flow framework; when the second confirmation information input by the flow editing interface is acquired, the flow orchestration model is generated based on the flow framework after the attribute configuration, wherein the second confirmation information is generated according to the click operation of the second logical button, and the second logical button is displayed on the flow editing interface.

[0050] In this embodiment, the flow editing interface refers to a visual interface capable of providing human-computer interaction. The technical personnel can configure each table attribute of the table model through the flow editing interface. Similar to the table editing interface, the flow editing interface can be an interface integrated on the processing device. For example, when the processing device is a smart phone with a touch screen, the flow editing interface can be displayed on the touch screen. The flow editing interface can also be displayed on a separate visualization device. The first configuration instruction input through the flow editing interface can be transmitted to the processing device in a wired or wireless manner to generate the task data packet of the final target distributed task.

[0051] In this embodiment, the dragable components displayed on the flow editing interface can be pre-set according to actual conditions and needs. The flow framework of the flow orchestration model and the attributes of each dragable component can be configured through each dragable component. The flow editing interface is provided with a second logical button. When the flow orchestration configuration of the target distributed task is completed, the technical personnel can generate the second confirmation information by clicking the second logical button, so as to generate the flow orchestration model corresponding to the target distributed task according to the configured flow framework and the attributes of each dragable component.

[0052] In one embodiment, the dragable component includes at least one of an input component, an output component, and a calculation column component; wherein the input component is configured to configure a connection relationship between a table model and a process framework, the table model is configured with an input parameter of the table model through a corresponding source component; the output component is configured to output a data result after logical processing of data with an output attribute of the table model configured by the input component; and the calculation column component is configured to configure the logical processing of the data.

[0053] In one specific embodiment, as shown in the example diagram of a process editing interface. The process editing interface is divided into a component display area, an editing area, and an attribute configuration area. Among them, the component display area displays various basic nodes and combined nodes. The basic nodes mainly include input components and output components, such as input MQ, output MQ, input DB, and / or output DB, wherein MQ and DB correspond to different data transmission forms. The combined nodes mainly include input calculation columns and table-level calculation columns. The calculation column refers to a calculation logic unit. The input calculation column is used for normalizing the data to facilitate subsequent logical calculation of the data by the table-level calculation column. Figure 4

[0054] The editing area is used to edit the connection relationship of each dragable component. As shown in the test input node (input MQ or input DB) in Figure 3 , denoted as fink_single_tese(lv), connected to the first input calculation column (test input calculation column 1, testMqProcess1), the first input calculation column connected to the second input calculation column (test input calculation column 2, testMqProcess2), the second input calculation column connected to the first table-level calculation column (table-level calculation column 1, tableProcess1), the first table-level calculation column connected to the second table-level calculation column (table-level calculation column 2, tableProcess2), and the second table-level calculation column connected to the test output node, denoted as tese-es(es). The output node corresponds to the input node.

[0055] When each dragable component in the editing area is selected, the component attribute configuration can be performed in the attribute configuration area. For example, after selecting the first input calculation column, the parallelism of the first input calculation column can be configured. For another example, after selecting the input node, the required table model can be configured.

[0056] Of course, the process editing interface can also set a component search area, as well as delete, undo, forward, copy, cut, move, png export, svg export, print, link to JSON, save, and other buttons according to actual conditions and needs. ​

[0057] The flow arrangement model is for flow arrangement of table processing. Each component can be configured with different attributes, such as MQ can select corresponding MQ information DB, corresponding DB connection can be selected, and calculation column is selected corresponding calculation column. All the components can be configured with parallelism.

[0058] Of course, the above flow editing interface is only an example, and more table attribute components can be set according to actual conditions and needs to better complete the configuration of the flow arrangement model. The protection scope of the present application is not limited to the specific display form of the flow editing interface.

[0059] In step 103, the third configuration instruction input through the task editing interface is obtained, and the task model is configured according to the third configuration instruction and the table model, wherein the task model is used to represent the overall description of the target distributed task.

[0060] In the embodiment, the task model is the overall description of the target distributed task, for example, the configuration of the identity information of a task. The task model configures the target distributed task at the level of the task.

[0061] In one embodiment, the third configuration instruction input through the task editing interface is obtained, and the task model is configured according to the third configuration instruction and the table model, and the specific implementation form is as follows: one or more than one task attribute component is displayed through the task editing interface, wherein each task attribute component corresponds to a task attribute of the task model; the third configuration instruction is obtained, wherein the third configuration instruction is generated according to the selection operation or the filling operation of at least one task attribute component input through the task editing interface, wherein the at least one task attribute component is used to configure the table model required by the target distributed task; at least one task attribute is configured according to the third configuration instruction; when the third confirmation information input by the task editing interface is obtained, the task model is generated based on the configured task attribute, wherein the third confirmation information is generated according to the clicking operation of the third logical button, and the task editing interface displays the third logical button.

[0062] In the embodiment, the task editing interface refers to a visual interface capable of providing human-computer interaction, and the technical personnel can specifically configure each table attribute of the table model through the task editing interface. Similar to the table editing interface, the task editing interface can be an interface integrated on the processing device, for example, when the processing device is a smart phone with a touch screen, the task editing interface can be displayed on the touch screen. The task editing interface can also be displayed on a separate visual device, and the first configuration instruction input through the task editing interface can be transmitted to the processing device in a wired or wireless manner to generate the task data packet of the final target distributed task.

[0063] In this embodiment, the task attribute components displayed on the task editing interface can be pre-set according to actual conditions and needs. Each task attribute can be configured through each task attribute component. The task editing interface sets a third logical button. When the configuration of each task attribute of the target distributed task is completed, the technician can generate a third confirmation information by clicking the third logical button, so as to generate a task model corresponding to the target distributed task according to the configured task attribute.

[0064] In one specific embodiment, an example diagram of the task editing interface is shown in FIG. 2. Figure 5 The task attribute components in the task editing interface include ID, task name, task type, state, publishing platform, platform task code, task start parameter and publishing table. The ID and the task name represent the identity of the target distributed task. The ID can be automatically generated by the system, for example, 339, and the task name can be written or modified by the technician, for example, testtask. The task type is used to select any one of the preset types, for example, single table, wide table stage one or wide table stage two. The publishing platform can select the platform on which the distributed task needs to be published, for example, JRC (a platform name). The state is used to configure the validity or invalidity of the distributed task. The platform task code is used to represent the code based on the configured platform setting, for example, 12345. The task start parameter is used for the technician to flexibly write the parameter value of the specific parameter in the distributed task according to needs, for example, the value of the parameter key is 1, that is, {“key”: 1}. The publishing table is used to select the table model required by the target distributed task.

[0065] In one embodiment, the table editing interface, the process editing interface and the task editing interface are used to configure the target distributed task from different levels. The table editing interface is used to configure the preprocessing process of the source data required by the target distributed task. The process editing interface is used to configure the process arrangement of the target distributed task. The task editing interface is used to configure the overall description of the target distributed task. When the table editing interface, the process editing interface and the task editing interface are configured respectively, man-machine interaction can be realized. In order to facilitate the configuration operation of the user, the table editing interface, the process editing interface and the task editing interface can be displayed through the display screen of the same device. More specifically, the table editing interface, the process editing interface and the task editing interface can be switched or jumped through operation instructions.

[0066] For example, the logical buttons corresponding to the task editing interface and the process editing interface are set in advance on the table editing interface. When the user clicks the logical button corresponding to the task editing interface on the table editing interface, the table editing interface can jump to the task editing interface. When the user clicks the logical button corresponding to the process editing interface on the table editing interface, the table editing interface can jump to the process editing interface. Correspondingly, the logical buttons corresponding to the table editing interface and the task editing interface are set in advance on the process editing interface. The logical buttons corresponding to the table editing interface and the process editing interface are set in advance on the task editing interface.

[0067] For example, the jump logic between the table editing interface, the process editing interface and the task editing interface is set in advance. Specifically, according to the order of the table editing interface, the process editing interface and the task editing interface, when a sliding instruction from right to left is obtained once through the touch display screen, the table editing interface jumps to the process editing interface. When a sliding instruction from right to left is obtained again through the touch display screen, the process editing interface jumps to the task editing interface. When a sliding instruction from right to left is obtained again through the touch display screen, the task editing interface jumps to the table editing interface.

[0068] Of course, the jump or switching mode between the table editing interface, the process editing interface and the task editing interface can also be set to other modes according to actual conditions and needs.

[0069] In step 104, the process arrangement model, the task model and the at least one table model are parsed and converted into a task data packet corresponding to the target distributed task, wherein the task data packet is used to support running the target distributed task in the target distributed environment.

[0070] In this embodiment, after the process arrangement model, the task model and the at least one table model are generated respectively, the process arrangement model, the task model and the at least one table model need to be parsed and converted into a task data packet corresponding to the target distributed task, so as to run the target distributed task in the target distributed environment.

[0071] In one embodiment, the process arrangement model, the task model and the at least one table model are parsed and converted into a task data packet corresponding to the target distributed task. The specific implementation process is as follows: the task model is parsed to determine at least one table model required by the target distributed task; the process arrangement model and the at least one table model are parsed, the configured input component in the process arrangement model is converted into a source end of the target distributed task; the configured output component in the process arrangement model is converted into an output end of the target distributed task; the configured calculation column component in the process arrangement model is converted into a basic operator of the target distributed task; based on the process arrangement model, the source end, the output end and the basic operator are arranged in a logical processing process, and a task data packet is generated.

[0072] In this embodiment, the process orchestration model, task model, and at least one table model generated through the table editing interface, process editing interface, and task editing interface cannot directly implement the processing logic of distributed tasks in a distributed environment. In this case, it is necessary to perform topological transformation on the process orchestration model, task model, and at least one table model.

[0073] In one specific embodiment, the conversion process is completed through a pre-configured task engine, such as... Figure 6 As shown, the configuration center provides table editing, process editing, and task editing interfaces, which respectively configure and generate process orchestration models, task models, and at least one table model. After exporting each model, the data of each model is cached, and then transformed by the interpreter, transformer, and process orchestration configured by the task engine, ultimately enabling the execution of the target distributed task in the executor. When running this target distributed task, both the input of source data and the input of data results are based on DB or MQ formats.

[0074] More specifically, such as Figure 7 As shown, taking the Flink environment as an example, the specific process of transformation by the interpreter, transformer, and orchestration is as follows: The task engine reads the corresponding orchestration model, task model, and at least one table model based on the task ID in the task model. Based on the read table models, the input nodes in the orchestration model are transformed into Flink Sources, the output nodes into Flink Sinks, and the computation columns can be transformed into Flink transactions (basic operators). More specifically, the computation columns can be transformed into corresponding Flink map, flatmap, process, and window operators depending on the actual situation. After the transformation is completed, the transformed Sources, Sinks, and transactions are orchestrated based on the orchestration model. The parallelism of each transaction operator is taken from the parallelism configuration in the orchestration model. Finally, the task data package corresponding to the target distributed task is generated. This task data package is published on the pre-defined JRC platform for execution, realizing the normal operation of the target distributed task.

[0075] The distributed task generation method provided by the disclosure is configured by inputting a first configuration instruction through a table editing interface to configure a table model, inputting a second configuration instruction through a flow editing interface to configure a flow arrangement model, inputting a third configuration instruction through a task editing interface to configure a task model, and then generating a task data packet of a target distributed task based on the flow arrangement model, the task model and at least one table model. That is, the configuration process of the target distributed task is visualized through the table editing interface, the flow editing interface and the task editing interface, the configuration difficulty of the target distributed task is reduced, the professional skill level requirement of the technical personnel is reduced, and the efficiency of generating the target distributed task is improved. At the same time, the modular configuration of the target distributed task is formed by the table model, the flow arrangement model and the task model, which can save a large amount of human resources and time cost compared with the manual creation of the distributed task, and further improve the efficiency of generating the target distributed task.

[0076] More specifically, by providing a configuration platform, a table model to be processed can be configured, and then the data processing flow is arranged through page by combining a flow arrangement tool. Then, the flow arrangement result is converted into a corresponding Flink topology by a set of analysis engines at the bottom layer, and then the topology is submitted to the Flink platform for execution. This greatly reduces the complexity of Flink processing and greatly improves the development efficiency. Moreover, the flow arrangement combined with the configuration of the table model is visualized, which is easier to expand.

[0077] The distributed task generation device provided by the embodiments of the disclosure is described below. The distributed task generation device described below can be correspondingly referred to the distributed task generation method described above. As shown in Figure 8 The distributed task generation device includes:

[0078] The table configuration module 801 is configured to obtain a first configuration instruction input through a table editing interface, and configure at least one table model according to the first configuration instruction, wherein the table model is used to represent the logic of preprocessing the source data.

[0079] The flow configuration module 802 is configured to obtain a second configuration instruction input through a flow editing interface, and configure a flow arrangement model according to the second configuration instruction and the table model, wherein the flow arrangement model is used to represent the flow arrangement of the processing logic of the target distributed task.

[0080] The task configuration module 803 is configured to obtain a third configuration instruction input through a task editing interface, and configure a task model according to the third configuration instruction and the table model, wherein the task model is used to represent the overall description of the target distributed task.

[0081] The data conversion module 804 is configured to parse the process orchestration model, the task model and the at least one table model, and convert into a task data package corresponding to the target distributed task, where the task data package is used to support running the target distributed task in the target distributed environment.

[0082] Figure 9 An example of a schematic diagram of an entity structure of an electronic device is shown in Figure 9 As shown, the electronic device can include a processor 901, a communications interface 902, a memory 903 and a communications bus 904, wherein the processor 901, the communications interface 902 and the memory 903 complete mutual communication through the communications bus 904. The processor 901 can invoke a logical instruction in the memory 903 to execute a distributed task generation method, which includes: obtaining a first configuration instruction input through a table editing interface, and configuring at least one table model according to the first configuration instruction, where the table model is used to represent a logic of preprocessing source data; obtaining a second configuration instruction input through a process editing interface, and configuring a process orchestration model according to the second configuration instruction and the table model, where the process orchestration model is used to represent a process orchestration of a target distributed task processing logic; obtaining a third configuration instruction input through a task editing interface, and configuring a task model according to the third configuration instruction and the table model, where the task model is used to represent an overall description of the target distributed task; and parsing the process orchestration model, the task model and the at least one table model, and converting into a task data package corresponding to the target distributed task, where the task data package is used to support running the target distributed task in the target distributed environment.

[0083] In addition, the logical instruction in the memory 903 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present disclosure essentially or partly, or part of the technical solutions, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0084] In another aspect, the present disclosure also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, enable the computer to perform the distributed task generation method provided by any of the above methods, the method comprising: obtaining a first configuration instruction input through a table editing interface, and configuring at least one table model according to the first configuration instruction, wherein the table model is used to represent the logic of preprocessing source data; obtaining a second configuration instruction input through a flow editing interface, and configuring a flow orchestration model according to the second configuration instruction and the table model, wherein the flow orchestration model is used to represent the flow orchestration of the target distributed task processing logic; obtaining a third configuration instruction input through a task editing interface, and configuring a task model according to the third configuration instruction and the table model, wherein the task model is used to represent the overall description of the target distributed task; and parsing the flow orchestration model, the task model and the at least one table model, and converting them into a task data packet corresponding to the target distributed task, wherein the task data packet is used to support running the target distributed task in a target distributed environment.

[0085] In another aspect, the present disclosure also provides a non-transitory computer-readable storage medium, which stores a computer program, the computer program being executed by a processor to implement the distributed task generation method provided by any of the above methods, the method comprising: obtaining a first configuration instruction input through a table editing interface, and configuring at least one table model according to the first configuration instruction, wherein the table model is used to represent the logic of preprocessing source data; obtaining a second configuration instruction input through a flow editing interface, and configuring a flow orchestration model according to the second configuration instruction and the table model, wherein the flow orchestration model is used to represent the flow orchestration of the target distributed task processing logic; obtaining a third configuration instruction input through a task editing interface, and configuring a task model according to the third configuration instruction and the table model, wherein the task model is used to represent the overall description of the target distributed task; and parsing the flow orchestration model, the task model and the at least one table model, and converting them into a task data packet corresponding to the target distributed task, wherein the task data packet is used to support running the target distributed task in a target distributed environment.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0087] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform from the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present disclosure, and not to limit them; although the present disclosure has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A method for distributed task generation, the method comprising: The method comprises the following steps: obtaining a first configuration instruction input through a table editing interface, and configuring at least one table model according to the first configuration instruction, wherein the table model is used to represent the logic of preprocessing source data; obtaining a second configuration instruction input through a flow editing interface, and configuring a flow orchestration model according to the second configuration instruction and the table model, wherein the flow orchestration model is used to represent the flow orchestration of a target distributed task processing logic; obtaining a third configuration instruction input through a task editing interface, and configuring a task model according to the third configuration instruction and the table model, wherein the task model is used to represent the overall description of the target distributed task; parsing the flow orchestration model, the task model and at least one table model, and converting them into a task data packet corresponding to the target distributed task, wherein the task data packet is used to support running the target distributed task in a target distributed environment.

2. The method of claim 1, wherein, The method of obtaining a first configuration instruction input through a table editing interface and configuring at least one table model according to the first configuration instruction comprises the following steps: The configuration process of each table model is as follows: displaying one or more table attribute components through the table editing interface, wherein each table attribute component corresponds to a table attribute of the table model; obtaining the first configuration instruction, wherein the first configuration instruction is generated according to the selection operation or the filling operation of at least one table attribute component input through the table editing interface; configuring at least one table attribute according to the first configuration instruction; generating the table model based on the configured table attribute when the first confirmation information input through the table editing interface is obtained, wherein the first confirmation information is generated according to the clicking operation of a first logic button, and the first logic button is displayed on the table editing interface.

3. The method of claim 2, wherein, The table attribute component comprises at least one of an identification component, a source component and a destination component; The identification component is used to configure the identity of the table model; The source component is used to configure the input attribute corresponding to the table model, and the input attribute is used to obtain a data source; The destination component is used to configure the output attribute corresponding to the table model, and the output attribute is used to output a corresponding data result.

4. The method of claim 1, wherein, The second configuration instruction comprises a flow framework configuration instruction and a component attribute configuration instruction; The method of obtaining a second configuration instruction input through a flow editing interface and configuring a flow orchestration model according to the second configuration instruction and the table model comprises the following steps: displaying one or more dragable components in a component display area in the flow editing interface; obtaining the flow framework configuration instruction, wherein the flow framework configuration instruction is generated according to the drag operation of at least one dragable component input through the editing area of the flow editing interface; generating the flow framework of the target distributed task according to the flow framework configuration instruction; obtaining the component attribute configuration instruction, wherein the component attribute configuration instruction is generated according to the selection operation or the filling operation of at least one dragable component in the editing area input through the attribute configuration area of the flow editing interface. According to the component attribute configuration instruction, attribute configuration is performed on each of the drag-able components in the flow framework, and the attribute configuration includes configuration of a connection relationship between the table model and the flow framework; When second confirmation information input by the flow editing interface is acquired, the flow orchestration model is generated based on the flow framework after attribute configuration, and the second confirmation information is generated according to a click operation of a second logic button, and the flow editing interface displays the second logic button.

5. The method of claim 4, wherein, The drag-able component includes at least one of an input component, an output component and a calculation column component; The input component is configured to configure a connection relationship between the table model and the flow framework, and the table model is configured with an input parameter of the table model through a corresponding source component; The output component is configured to output a data result after logical processing of data according to an input attribute of the table model configured by the input component; The calculation column component is configured to configure logical processing of data.

6. The method of claim 1, wherein, Third configuration instruction input by a task editing interface is acquired, and a task model is configured according to the third configuration instruction and the table model, including: One or more task attribute components are displayed through the task editing interface, and each of the task attribute components corresponds to a task attribute of the task model; The third configuration instruction is acquired, and the third configuration instruction is generated according to selection operation or filling operation of at least one of the task attribute components input by the task editing interface, and at least one of the task attribute components is configured to configure the table model required by the target distributed task; According to the third configuration instruction, at least one of the task attributes is configured; When third confirmation information input by the task editing interface is acquired, the task model is generated based on the configured task attribute, and the third confirmation information is generated according to a click operation of a third logic button, and the task editing interface displays the third logic button.

7. The method of claim 1, wherein, The flow orchestration model, the task model and at least one of the table models are analyzed and converted into a task data packet corresponding to the target distributed task, including: The task model is analyzed to determine at least one of the table models required by the target distributed task; The flow orchestration model and at least one of the table models are analyzed, the input component configured in the flow orchestration model is converted into a source end of the target distributed task, the output component configured in the flow orchestration model is converted into an output end of the target distributed task, and the calculation column component configured in the flow orchestration model is converted into a basic operator of the target distributed task; Based on the flow orchestration model, the source end, the output end and the basic operator are arranged in a logical processing flow, and the task data packet is generated.

8. A distributed task generation apparatus characterized by comprising: The table configuration module is configured to acquire first configuration instruction input by a table editing interface, and configure at least one table model according to the first configuration instruction, and the table model is configured to represent logical processing of source data. ​ A flow configuration module is configured to acquire a second configuration instruction input through a flow editing interface, and configure a flow orchestration model according to the second configuration instruction and the table model, wherein the flow orchestration model is used to represent flow orchestration of a target distributed task processing logic; A task configuration module is configured to acquire a third configuration instruction input through a task editing interface, and configure a task model according to the third configuration instruction and the table model, wherein the task model is used to represent overall description of the target distributed task; A data conversion module is configured to parse the flow orchestration model, the task model and at least one table model, and convert them into a task data packet corresponding to the target distributed task, wherein the task data packet is used to support running of the target distributed task in a target distributed environment.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the distributed task generation method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the distributed task generation method according to any one of claims 1 to 7.

Images