Task generation method and device, equipment, storage medium and product

Abstract

The invention discloses a task generation method and device, equipment, a storage medium and a product. The invention relates to the technical field of data processing. The method comprises the steps of obtaining a change information type and a database type corresponding to a real-time job task to be created; selecting a matched target real-time job task template from a pre-created real-time task template set based on the change information type and the database type; in response to a loading operation of the target real-time job task template, obtaining and adjusting task configuration parameters in the target real-time job task template; and generating a corresponding real-time operation task based on the task configuration parameter. The target real-time operation task template is selected, the task configuration parameters in the target real-time operation task template are adjusted, and the corresponding real-time operation task is generated according to the task configuration parameters, so that the real-time operation task can be generated only by adjusting the task configuration parameters without manually writing a real-time task running script, and the real-time operation efficiency is improved. And the efficiency of generating the real-time operation task is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a task generation method, device, equipment, storage medium and product. Background Art

[0002] Many business scenarios require rapid processing and response to instant events or data. Real-time job tasks can be built to process time and data in real time.

[0003] At present, in the process of creating real-time job tasks, it is necessary to manually write the running script and configure the task running parameters in the script code. The writing of the script and the adjustment of the parameters during the task running are both cumbersome. At the same time, business personnel are required to have strong programming knowledge. Therefore, how to conveniently generate real-time job tasks and improve the efficiency of writing real-time job tasks has become a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a task generation method, device, equipment, storage medium and product, which are used to solve the problem in the prior art that generating real-time operation tasks is cumbersome and requires business personnel to have strong programming knowledge.

[0005] In a first aspect, an embodiment of the present invention provides a task generation method, which includes:

[0006] Obtain the change information type and database type corresponding to the real-time job task to be created;

[0007] Selecting a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type;

[0008] In response to the loading operation of the target real-time job task template, acquiring and adjusting the task configuration parameters in the target real-time job task template;

[0009] Generate a corresponding real-time job task based on the task configuration parameters.

[0010] In a second aspect, an embodiment of the present invention further provides a task generation device, which includes:

[0011] The type acquisition module is used to obtain the change information type and database type corresponding to the real-time job task to be created;

[0012] A template determination module, configured to select a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type;

[0013] A parameter adjustment module, configured to obtain and adjust task configuration parameters in the target real-time job task template in response to a loading operation of the target real-time job task template;

[0014] The task generation module is used to generate corresponding real-time job tasks based on the task configuration parameters.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a task generation method as described in any one of the embodiments of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a task generation method as described in any one of the embodiments of the present invention.

[0017] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a task generation method as described in any one of the embodiments of the present invention.

[0018] In an embodiment of the present invention, by obtaining the change information type and database type corresponding to the real-time job task to be created, a matching target real-time job task template is selected from a pre-created real-time task template set based on the change information type and the database type, thereby realizing on-demand selection of the target real-time job task template and ensuring the matching degree of the target real-time job task template, without the need to manually write a real-time task running script; at the same time, by responding to the loading operation of the target real-time job task template, the task configuration parameters in the target real-time job task template are obtained and adjusted, and then the corresponding real-time job task is generated based on the task configuration parameters, thereby realizing that the real-time job task can be generated by only adjusting the task configuration parameters, thereby improving the efficiency of generating real-time job tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of a task generation method provided by an embodiment of the present invention;

[0021] Figure 2 An example diagram of an information suspension layer provided by an embodiment of the present invention;

[0022] Figure 3 A flowchart of another task generation method provided by an embodiment of the present invention;

[0023] Figure 4 A flowchart of another task generation method provided by an embodiment of the present invention;

[0024] Figure 5 An example diagram of task configuration items for a Flink real-time job task provided by an embodiment of the present invention;

[0025] Figure 6 A flow chart showing task operation parameters matching a real-time job task provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the structure of a task generation device provided by an embodiment of the present invention;

[0027] Figure 8 A structural block diagram of an electronic device for a task generation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0031] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0032] In one embodiment, Figure 1 This is a flowchart of a task generation method provided by an embodiment of the present invention. This embodiment is applicable to the case of generating real-time job tasks. The method can be executed by a task generation device. The task generation device can be implemented in the form of hardware and / or software. The task generation device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110: Obtain the change information type and database type corresponding to the real-time job task to be created.

[0034] The change information type can be understood as the type of change information generated by business processing. The change information type can be related to specific business needs and scenarios. For example, when accurate database change records or synchronization are required, the change information type can be a binlog message; when high-throughput message streams need to be processed, the change information type can be a kafka message.

[0035] In one embodiment, the change information type includes one of the following: binlog message and kafka message; the database type includes one of the following: HIVE database and Doris database.

[0036] Among them, Binlog message, that is, binary log message, is a record generated by the database when performing data change operations (such as insert, update, delete, etc.). Kafka message refers to the data unit transmitted and stored in the distributed message queue system. HIVE database refers to a data warehouse tool based on Hadoop. Doris database is a distributed analytical database.

[0037] In an embodiment, the change information type and database type of the real-time job task to be created can be extracted respectively. In the actual operation process, the change information type of the real-time job task can be determined according to the required change information, and the database type can be determined according to the size of the data volume, or it can be preset.

[0038] S120 : Select a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type.

[0039] Among them, the real-time task template set can be understood as a task with strict time limits and immediacy requirements. The real-time task template set can include multiple real-time job task templates. Generally speaking, the real-time job task template can be created in advance according to the task attribute information of the real-time job task. For each task attribute information, the corresponding target real-time job task template can be queried in the real-time task template set. There may be task configuration items in each real-time task template to adjust and generate the corresponding real-time job task. The real-time job task to be created can be understood as a real-time job task waiting to be created, that is, a task that needs to be processed and completed at a specific time point or within a very short time interval. Exemplarily, the real-time job task to be created may include real-time monitoring of the operating status, real-time tracking of location information and other job tasks. The target real-time job task template can be understood as a real-time job task template that matches the task attribute information of the real-time job task to be created in the real-time task template set.

[0040] In an embodiment, after determining the change information type and the database type, the corresponding real-time job task template can be matched in the real-time task template set according to the task attribute information as the target real-time job task template. In the actual operation process, the matching target real-time job task template can be selected in the real-time task template set according to the change information type and the database type. In one implementation, a matching real-time job task template can be first selected in the real-time task template set according to the change information type as a candidate real-time job task template, and then a matching candidate real-time job task template can be selected in the candidate real-time job task template according to the database type as the target real-time job task template; or, a matching real-time job task template can be first selected in the real-time task template set according to the database type as a candidate real-time job task template, and then a matching candidate real-time job task template can be selected in the candidate real-time job task template according to the change information type as the target real-time job task template.

[0041] S130 . In response to the loading operation of the target real-time job task template, obtain and adjust the task configuration parameters in the target real-time job task template.

[0042] Among them, the loading operation can be understood as the operation of reading the target real-time job task template into the run wake-up, and the loading operation can be an operation triggered by a user action. Exemplarily, the loading operation can be triggered by the user clicking on the target real-time job task template. The task configuration parameters can be understood as the configuration parameters corresponding to each task configuration item in the target real-time job task template. Generally speaking, the task configuration items can include data information such as execution cluster, task parallelism, task manager memory, job manager memory, execution mode and dependent configuration information, and the task configuration parameters of each task configuration item can be adjusted according to the real-time job task to be created.

[0043] In an embodiment, after receiving a load operation of a target real-time job task template, the task configuration parameters in the target real-time job task template can be extracted and adjusted. In an actual operation process, the task configuration items in the target real-time job task template can be extracted, and the task configuration parameters of the task configuration items can be dynamically adjusted according to the demand.

[0044] S140: Generate a corresponding real-time job task based on the task configuration parameters.

[0045] Among them, real-time job tasks can be understood as tasks that need to be processed and completed at a specific time point or within a very short time interval, and are job tasks generated according to task configuration parameters. In actual operation, real-time job tasks can be task execution commands, that is, instructions or statements used to start, control and manage the execution of specific real-time tasks.

[0046] In an embodiment, after determining the task configuration parameters, a corresponding task execution command can be generated according to the task configuration parameters, and the task execution command is used as the corresponding real-time job task. In the actual operation process, the task configuration parameters corresponding to the task configuration items such as the execution cluster, task parallelism, task manager memory, job manager memory, execution mode and dependent configuration information can be determined, and the corresponding task execution command is formed according to the execution cluster, task manager memory, job manager memory, execution mode, dependent configuration information and task parallelism, and the task execution command is used as the corresponding real-time job task.

[0047] According to an embodiment of the present invention, by obtaining the change information type and database type corresponding to the real-time job task to be created, a matching target real-time job task template is selected from a pre-created real-time task template set based on the change information type and the database type, thereby ensuring the matching degree of the target real-time job task template, without the need to manually write a real-time task running script; at the same time, by responding to the loading operation of the target real-time job task template, the task configuration parameters in the target real-time job task template are obtained and adjusted, and then the corresponding real-time job task is generated based on the task configuration parameters, so that the real-time job task can be generated by only adjusting the task configuration parameters, thereby improving the efficiency of generating real-time job tasks.

[0048] In one embodiment, the task generation method further includes:

[0049] In response to a task running instruction, start and run a real-time job task;

[0050] On the visual display interface, task running parameters matching the real-time job tasks are displayed.

[0051] Among them, the task running instruction can be understood as a command or instruction to instruct the task to run, which is used to control the operation of executing the running task. In the actual operation process, the task running instruction can be triggered by a user action, such as clicking a start button, touching a specified position in a visual interface, etc. The visual display interface can be understood as a user interface that images data through visual elements such as charts, images, and colors, and can be used to display task running parameters that match real-time job tasks. Task running parameters can be understood as the running parameters generated by running real-time job tasks.

[0052] In one embodiment, the task running parameters include at least one of the following: resource status; task status information;

[0053] Among them, resource status includes: resource usage, total available resources and total resources;

[0054] The task status information includes: task name, task operation mode, task operation time, task status, task end time, task last modification and job identification information; the task status includes: executing; waiting; submitted.

[0055] Among them, resource usage can be understood as the amount or degree of resources consumed by real-time job tasks. For example, resource usage can include but is not limited to memory usage, central processing unit (CPU) usage, and disk space usage; the total amount of available resources can be understood as the remaining amount of resources after running real-time job tasks. The total amount of available resources can include but is not limited to the total amount of available memory, the total amount of available CPU, and the total amount of available disk space; the total amount of resources can be understood as the number and scale of all resources in the space where real-time job tasks run. For example, the total amount of resources can include the total memory capacity and the total number of CPU cores and computing power. Task name refers to the name of the real-time job task, which can be composed of text and numbers. Task running mode can be understood as the method and rules adopted by the real-time job task during execution. For example, the task running mode can include but is not limited to serial model, parallel model, and distributed mode. Task running time can be understood as the length of time for running real-time job tasks. Task status can be understood as a description of different stages and situations in the execution of real-time job tasks. For example, task status can include executing, waiting, submitted, etc. The task end time can be understood as the time when the real-time job task is completed, or the time when the status is considered to be submitted; the task last modified can be understood as the time when the real-time job task was last modified; the job identification information can be understood as the identification information used to indicate the real-time job task. For example, the job identification information can include but is not limited to a number or an identity document (ID). In the actual operation process, the job identification information can be represented by data such as a string, a number, or text.

[0056] In the embodiment, after receiving the task running instruction, the real-time job task can be run in the Flink framework. At the same time, during the running of the real-time job task, the task running parameters such as resource status and task status information matching the real-time job task can be displayed on the visual interface. By summarizing and displaying the resource status and task status information, it is convenient to determine the execution problem when the real-time job task is executed incorrectly, reducing the difficulty of troubleshooting the problem.

[0057] In one embodiment, the real-time job task runs in the Flink framework, and the task running parameters matching the real-time job task are displayed, including:

[0058] Obtain distributed operation parameters that match real-time job tasks based on the Hadoop page interface;

[0059] Obtain the job running status that matches the real-time job task based on the Flink Dashboard interface;

[0060] Integrate the distributed operation parameters and job operation status to obtain and display the task operation parameters that match the real-time job tasks.

[0061] Among them, the Flink framework is an open source big data processing framework for processing streaming data, which can realize real-time data processing and batch processing. Real-time job tasks can be run in the Flink framework to achieve low-latency and high-throughput processing.

[0062] Hadoop page interface refers to a page interface of an open source distributed computing framework, which is used to obtain distributed operating parameters that match real-time job tasks. Distributed operating parameters can be understood as specific settings and values ​​used to control and adjust the performance and collaboration mode of each node or component in a distributed system. Exemplarily, distributed operating parameters may include at least cluster node indicators, task scheduling parameters, etc., such as task name, task operation mode, task end time, task last modification, and job identification information. The Flink Dashboard interface is an interface for monitoring and managing real-time job tasks running in the Flink framework. The Flink Dashboard interface can be used to obtain job operation status that matches real-time job tasks, such as executing, waiting, submitted, and other job operation status.

[0063] In the embodiment, the distributed operation parameters matching the real-time job task can be obtained through the Hadoop page interface, and the job operation status matching the real-time job task can be obtained through the Flink Dashboard interface, and the obtained distributed operation parameters and job operation status are integrated as task operation parameters matching the real-time job task, and displayed through the visual display interface. In this embodiment, parameter information matching the real-time job task is obtained through the Hadoop page interface and the Flink Dashboard interface respectively, and the parameter information is integrated as the task operation parameters for display, thereby improving the efficiency of obtaining the task operation parameters.

[0064] In one embodiment, the task generation method further includes:

[0065] Obtain the actual resource usage of real-time job tasks;

[0066] When the actual resource usage reaches the pre-configured available resource threshold, the corresponding maximum available resource usage is dynamically adjusted according to the task running parameters of the real-time job task.

[0067] Among them, the actual resource usage can be understood as the actual consumption and occupation of various resources by real-time job tasks. Generally speaking, the actual resource usage may include but is not limited to CPU usage, memory usage, etc. The available resource threshold can be understood as the critical value of the available resources. When it is determined that the actual resource usage reaches the available resource threshold, the maximum available resource usage can be adjusted. For each parameter in the actual resource usage, a corresponding resource threshold can be set. The maximum available resource usage can be understood as the maximum amount and degree of resources allowed to be used.

[0068] In an embodiment, the actual resource usage of the real-time job task can be extracted to determine whether the actual resource usage reaches a pre-configured threshold of available resources. If the actual resource usage reaches the pre-configured threshold of available resources, the corresponding maximum usage of available resources can be dynamically adjusted according to the task operation parameters of the real-time job task. Exemplarily, when the CPU usage rate in the actual resource usage reaches the corresponding threshold of available resources, the required CPU usage in the task operation parameters of the real-time job task can be determined, and the maximum usage of available CPU resources can be dynamically adjusted according to the CPU usage. In this embodiment, the probability of insufficient resources occurring during the operation of the real-time job task is reduced through the dynamic adjustment of resources, thereby ensuring the normal operation of the real-time job task.

[0069] In one embodiment, the task generation method further includes:

[0070] In response to the received information query instruction, displaying an information floating layer corresponding to the information to be queried on the visual display interface;

[0071] The information introduction corresponding to the information to be queried is displayed in the information floating layer.

[0072] Among them, the information query instruction can be understood as the execution information indicating the information query. In the actual operation process, the information query instruction can be triggered by a user action, such as clicking the information to be queried by the mouse on the visual display interface; or, placing the mouse above the information to be queried. The information to be queried can be understood as the information that needs to be queried, such as the task running parameters of real-time jobs. The information floating layer can be understood as a pop-up window that displays the information introduction corresponding to the information to be queried in a pop-up or floating manner when the user interacts with the visual interface. The information introduction can be understood as the introduction and configuration information of the information to be queried. In the actual operation process, the information introduction can be pre-stored in the knowledge base.

[0073] In an embodiment, after receiving an information query instruction, an information suspension layer corresponding to the information to be queried can be displayed on the visual display interface. At the same time, the information introduction corresponding to the information to be queried can be automatically queried in the knowledge base, and the information introduction corresponding to the information to be queried can be displayed in the information suspension layer. When the user has questions during the configuration of real-time job tasks, the user can be assisted in obtaining the information introduction corresponding to the information to be queried, thereby reducing the difficulty of creating real-time job tasks.

[0074] In one embodiment, Figure 2 An example diagram of an information suspension layer provided by an embodiment of the present invention, such as Figure 2 As shown, the mouse position is the information to be queried, and an information floating layer is displayed above the information to be queried. The information floating layer displays the information introduction corresponding to the information to be queried.

[0075] In one embodiment, Figure 3 A flowchart of another task generation method provided by an embodiment of the present invention. This embodiment is based on the above embodiment and further describes how to determine a target real-time job task template, adjust the task configuration parameters in the target real-time job task template, and generate a corresponding real-time job task based on the task configuration parameters. Figure 3 As shown, the method includes:

[0076] S210: Obtain the change information type and database type corresponding to the real-time job task to be created.

[0077] S220: Obtain the change information type and database type of each real-time job task template in the pre-created real-time task template set.

[0078] In an embodiment, the real-time job task templates in the real-time task template set may be extracted respectively to determine the change information type and database type of the real-time job task template.

[0079] S230: Based on the actual matching degree between the change information type and database type of each real-time job task template and the change information type and database type of the real-time job task to be created.

[0080] The actual matching degree can be understood as the degree of fit actually achieved in the actual comparison process. Generally speaking, when the change information type matches successfully or the database type matches successfully, the actual matching degree value can be increased, and the real-time job task template with the highest actual matching degree can be used as the corresponding target real-time job task template.

[0081] In an embodiment, the change information type and database type of each real-time job task template can be matched with the change information type and database type of the real-time job task to be created to determine the actual matching degree. In the actual operation process, the change information type of each real-time job task template can be matched with the change information type of the real-time job task to be created, and the real-time job task template corresponding to the successfully matched change information type can be increased in actual matching degree, and then the database type of each real-time job task template can be matched with the database type of the real-time job task to be created, and the real-time job task template corresponding to the successfully matched database type can be increased in actual matching degree, and finally the actual matching degree between the change information type and database type of the real-time job task to be created can be obtained.

[0082] S240: Using the real-time job task template with the highest actual matching degree as the corresponding target real-time job task template.

[0083] In an embodiment, the actual matching degree of each real-time job task template may be determined, and the real-time job task template with the highest actual matching degree may be finally used as the corresponding target real-time job task template.

[0084] S250: In response to the loading operation of the target real-time job task template, obtain the task configuration items in the target real-time job task template.

[0085] Among them, the task configuration items can be understood as specific configuration options for defining real-time job task parameters.

[0086] In one embodiment, the task configuration items include: execution cluster, task parallelism, task manager memory, job manager memory, execution mode and dependency configuration information. Among them, the execution cluster can be understood as a group of computing nodes or servers working together to jointly execute the workload of real-time job tasks. Task parallelism refers to the number of work units that can be processed simultaneously during the execution of real-time job tasks. Task manager memory refers to the information related to computer memory usage displayed in the task manager; job manager memory refers to the memory resources required for the job manager to run and manage jobs itself; execution mode refers to the specific method of completing real-time job tasks. Exemplarily, the execution mode may include sequential execution, parallel execution, and asynchronous execution. Dependency configuration information refers to the dependency information of the real-time job task operation. Exemplary dependency configuration information may include but is not limited to interface information.

[0087] In an embodiment, after receiving the load operation of the target real-time job task template, the task configuration items contained in the target real-time job task template can be extracted in response to the load operation of the target real-time job task template. In the actual operation process, the field information in the target real-time job task template can be extracted as the task configuration item.

[0088] S260: In response to the received adjustment operation, dynamically adjust the task configuration parameters corresponding to the task configuration item.

[0089] The adjustment operation may be understood as instruction information for adjusting task configuration parameters.

[0090] In an embodiment, after an adjustment operation is received, task configuration parameters corresponding to the task configuration items may be extracted according to the adjustment operation and dynamically adjusted.

[0091] S270: Generate a corresponding task execution command based on the task configuration parameters.

[0092] The task execution command is an instruction or command line statement used to start, control and manage the running of real-time job tasks.

[0093] In an embodiment, the corresponding task execution command can be generated through the task configuration parameters. In the actual operation process, the task configuration parameters can be parsed and the corresponding task execution command can be generated according to the pre-set rules.

[0094] S280: Use the task execution command as the corresponding real-time job task.

[0095] According to an embodiment of the present invention, by acquiring the change information type and database type corresponding to the real-time job task to be created, the change information type and database type of each real-time job task template in the pre-created real-time task template set are acquired, and based on the actual matching degree between the change information type and database type of each real-time job task template and the change information type and database type of the real-time job task to be created, the real-time job task template with the highest actual matching degree is used as the corresponding target real-time job task template, so as to determine the target real-time job task template according to the actual matching degree and ensure the matching degree of the target real-time job task template; at the same time, by responding to the loading operation of the target real-time job task template, the task configuration items in the target real-time job task template are acquired, and in response to the received adjustment operation, the task configuration parameters corresponding to the task configuration items are dynamically adjusted, and the corresponding task execution command is generated based on the task configuration parameters, and the task execution command is used as the corresponding real-time job task, so as to achieve that in the process of generating the real-time job task, only the task configuration parameters need to be dynamically adjusted, thereby improving the efficiency of generating the real-time job task.

[0096] In one embodiment, Figure 4 A flowchart of another task generation method provided by an embodiment of the present invention. Figure 4 As shown in the figure, taking the flink real-time job task as an example, a task generation method is further explained, as shown in Figure 4 As shown, the method includes:

[0097] Step 1: Extract the task attribute information of the Flink real-time job to form a separate task configuration item.

[0098] Step 2: Determine a target real-time job task template in the real-time job task template set according to the task attribute information.

[0099] Among them, the real-time job task template set is composed of real-time job task templates, and different types of real-time job task templates can be written according to business conditions. For example: real-time job task templates corresponding to binlog messages and HIVE databases, real-time job task templates corresponding to kafka messages and HIVE databases, real-time job task templates corresponding to kafka messages and Doris databases, etc. Configuration parameters that need to be adjusted due to different businesses, such as service addresses, can be configured in the task configuration parameters.

[0100] Step 3: Adjust the task configuration parameters in the target real-time job task template.

[0101] Furthermore, information introductions of real-time job tasks may be stored in the knowledge base as a knowledge base assistant. When a user has questions about the use of the information to be queried, information introductions corresponding to the information to be queried may be displayed in the information floating layer.

[0102] Step 4: Generate a corresponding task execution command based on the task configuration parameters, and use the task execution command as the corresponding real-time job task.

[0103] Furthermore, after the real-time job task is generated, click the start button, and the task running parameters matching the real-time job task will be displayed on the visual display interface. At the same time, the corresponding maximum usage of available resources can be dynamically adjusted according to the task running parameters of the real-time job task.

[0104] In one embodiment, Figure 5 An example diagram of a task configuration item of a Flink real-time job task provided by an embodiment of the present invention. In one embodiment, the task configuration item may include: execution cluster, task parallelism, task manager memory, job manager memory, execution mode, and dependency configuration information.

[0105] In one embodiment, Figure 6 A flowchart showing task operation parameters matching a real-time job task is provided in an embodiment of the present invention. Figure 6As shown, the method includes: starting the business log, obtaining the distributed operation parameters matching the real-time job task based on the Hadoop page interface, obtaining the job operation status matching the real-time job task based on the Flink Dashboard interface, integrating the distributed operation parameters and the job operation status, and obtaining and displaying the task operation parameters matching the real-time job task.

[0106] In one embodiment, Figure 7 A schematic diagram of the structure of a task generation device provided by an embodiment of the present invention. Figure 7 As shown, the device includes: a type acquisition module 71, a template determination module 72, a parameter adjustment module 73 and a task generation module 74.

[0107] The type acquisition module 71 is used to acquire the change information type and database type corresponding to the real-time job task to be created.

[0108] The template determination module 72 is used to select a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type.

[0109] The parameter adjustment module 73 is used to obtain and adjust the task configuration parameters in the target real-time job task template in response to the loading operation of the target real-time job task template.

[0110] The task generation module 74 is used to generate corresponding real-time operation tasks based on task configuration parameters.

[0111] According to an embodiment of the present invention, by obtaining the change information type and database type corresponding to the real-time job task to be created, a matching target real-time job task template is selected from a pre-created real-time task template set based on the change information type and the database type, thereby ensuring the matching degree of the target real-time job task template, without the need to manually write a real-time task running script; at the same time, by responding to the loading operation of the target real-time job task template, the task configuration parameters in the target real-time job task template are obtained and adjusted, and then the corresponding real-time job task is generated based on the task configuration parameters, so that the real-time job task can be generated by only adjusting the task configuration parameters, thereby improving the efficiency of generating real-time job tasks.

[0112] In one embodiment, a task generation device further includes:

[0113] A task starting module, used to start and run the real-time job task in response to the task running instruction;

[0114] The parameter display module is used to display the task running parameters matching the real-time job tasks on the visual display interface.

[0115] In one embodiment, the template selection unit is specifically used to:

[0116] Obtaining the change information type and database type of each real-time job task template in the pre-created real-time task template set;

[0117] Based on the actual matching degree between the change information type and database type of each real-time job task template and the change information type and database type of the real-time job task to be created;

[0118] The real-time job task template with the highest actual matching degree is used as the corresponding target real-time job task template.

[0119] In one embodiment, the change information type includes one of the following: binlog message and kafka message; the database type includes one of the following: HIVE database and Doris database.

[0120] In one embodiment, the parameter adjustment module 73 includes:

[0121] A configuration item determination unit, used to obtain a task configuration item in a target real-time job task template;

[0122] The parameter adjustment unit is used to dynamically adjust the task configuration parameter corresponding to the task configuration item in response to the received adjustment operation.

[0123] In one embodiment, the task configuration items include: execution cluster, task parallelism, task manager memory, job manager memory, execution mode, and dependency configuration information.

[0124] In one embodiment, the task generation module 74 includes:

[0125] A command generation unit, used to generate corresponding task execution commands based on task configuration parameters;

[0126] The task generation unit is used to use the task execution command as the corresponding real-time job task.

[0127] In one embodiment, the task running parameters include at least one of the following: resource status; task status information;

[0128] Among them, resource status includes: resource usage, total available resources and total resources;

[0129] Task status information includes: task name, task operation mode, task operation duration, task status, task end time, task last modification and job identification information; task status includes: executing; waiting; submitted.

[0130] In one embodiment, the real-time job task runs in the Flink framework; the parameter display module is specifically used for:

[0131] Obtain distributed operation parameters that match real-time job tasks based on the Hadoop page interface;

[0132] Obtain the job running status that matches the real-time job task based on the Flink Dashboard interface;

[0133] Integrate the distributed operation parameters and job operation status to obtain and display the task operation parameters that match the real-time job tasks.

[0134] In one embodiment, a task generation device further includes:

[0135] Resource determination module, used to obtain the actual resource usage of real-time job tasks;

[0136] The resource adjustment module is used to dynamically adjust the corresponding maximum usage of available resources according to the task running parameters of the real-time job task when the actual resource usage reaches the pre-configured available resource threshold.

[0137] In one embodiment, a task generation device further includes:

[0138] A suspension layer determination module, configured to display an information suspension layer corresponding to the information to be queried on a visual display interface in response to a received information query instruction;

[0139] The information display module is used to display the information introduction corresponding to the information to be queried in the information floating layer.

[0140] The task generation device provided in the embodiment of the present invention can execute the task generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0141] In one embodiment, Figure 8 A structural block diagram of an electronic device for a task generation method provided by an embodiment of the present invention, such as Figure 8 As shown, a schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0142] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a task generation method.

[0145] In some embodiments, the task generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the task generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the task generation method in any other appropriate manner (e.g., by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0152] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the task generation method provided in any embodiment of the present application.

[0153] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0154] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A task generation method, characterized in that: include: Obtain the change information type and database type corresponding to the real-time job task to be created; Selecting a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type; In response to the loading operation of the target real-time job task template, acquiring and adjusting the task configuration parameters in the target real-time job task template; Generate a corresponding real-time job task based on the task configuration parameters.

2. The method according to claim 1, characterized in that The method further comprises: In response to the task running instruction, starting and running the real-time job task; On the visual display interface, task running parameters matching the real-time job task are displayed.

3. The method according to claim 1, characterized in that The selecting a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type includes: Obtaining the change information type and database type of each real-time job task template in the pre-created real-time task template set; Based on the actual matching degree between the change information type and database type of each real-time job task template and the change information type and database type of the real-time job task to be created; The real-time job task template with the highest actual matching degree is used as the corresponding target real-time job task template.

4. The method according to claim 1, characterized in that: The change information type includes one of the following: binlog message and kafka message; the database type includes one of the following: HIVE database and Doris database.

5. The method according to claim 1, characterized in that The obtaining and adjusting the task configuration parameters in the target real-time job task template includes: Obtaining task configuration items in the target real-time job task template; In response to the received adjustment operation, the task configuration parameter corresponding to the task configuration item is dynamically adjusted.

6. The method according to claim 5, characterized in that The task configuration items include: execution cluster, task parallelism, task manager memory, job manager memory, execution mode and dependency configuration information.

7. The method according to claim 1, characterized in that The generating a corresponding real-time job task based on the task configuration parameters includes: Generate a corresponding task execution command based on the task configuration parameters; The task execution command is used as the corresponding real-time job task.

8. The method according to claim 2, characterized in that: The task operation parameters include at least one of the following: resource status; Task status information; The resource status includes: resource usage, total available resources and total resource amount; The task status information includes: task name, task running mode, task running time, task status, task end time, task last modification and job identification information; the task status includes: executing; waiting; submitted.

9. The method according to claim 2, characterized in that: The real-time job task runs in the Flink framework; the display of task running parameters matching the real-time job task includes: Acquire distributed operation parameters matching the real-time job task based on the Hadoop page interface; Obtain the job running status matching the real-time job task based on the Flink Dashboard interface; The distributed operation parameters and the job operation status are integrated to obtain and display task operation parameters that match the real-time job task.

10. The method according to claim 7, characterized in that The method further comprises: Obtaining actual resource usage of the real-time job task; When the actual resource usage reaches a pre-configured available resource threshold, the corresponding maximum available resource usage is dynamically adjusted according to the task running parameters of the real-time job task.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: In response to the received information query instruction, displaying an information floating layer corresponding to the information to be queried on the visual display interface; The information introduction corresponding to the information to be queried is displayed in the information floating layer.

12. A task generation device, characterized in that: include: The type acquisition module is used to obtain the change information type and database type corresponding to the real-time job task to be created; A template determination module, configured to select a matching target real-time job task template from a pre-created real-time task template set based on the change information type and the database type; A parameter adjustment module, configured to obtain and adjust task configuration parameters in the target real-time job task template in response to a loading operation of the target real-time job task template; The task generation module is used to generate corresponding real-time job tasks based on the task configuration parameters.

13. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the task generation method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the task generation method as described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the task generation method according to any one of claims 1 to 11.

Images