A task control method, device and electronic device based on cloud service

Abstract

The present application discloses a task control method, device and electronic device based on cloud service, which obtains a task configuration table predefined by a user, wherein the task configuration table includes multiple tasks, starts a corresponding cluster service module based on the task configuration table, controls the cluster service module to execute the tasks in the task configuration table, can start the corresponding cluster service module based on the task configuration table, and execute each task by the cluster service module, and automatically shut down the cluster service after all tasks are completed, thereby avoiding the user from manually executing each specific data task, shortening the overall time consuming of executing data tasks, and improving the efficiency of cluster configuration and the cost of cluster use.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a task control method, device and electronic device based on cloud services. Background Art

[0002] In the scenario of using a big data computing framework based on cloud services, many data tasks need to configure and use cluster services based on specific business needs and data task loads.

[0003] In some scenarios, when using cloud services to execute data tasks, you manually start the cluster by clicking on the product interface, and then manually execute each specific data task in the cluster after the cluster is started. This method results in a longer overall time consumption for executing data tasks, which in turn leads to low execution efficiency of data tasks. Summary of the invention

[0004] The embodiments of the present application provide a cloud service-based task control method, device, and electronic device, which solve the problems of long overall time consumption and low execution efficiency of data tasks.

[0005] In a first aspect, an embodiment of the present application further provides a cloud service-based task control method, comprising:

[0006] Obtain a task configuration table predefined by a user, the task configuration table including multiple tasks; start a corresponding cluster service module based on the task configuration table; and control the cluster service module to execute the tasks in the task configuration table.

[0007] In a second aspect, an embodiment of the present application further provides a task control device based on cloud services, including:

[0008] The acquisition module is used to obtain the task configuration table predefined by the user, which includes multiple tasks; the startup module is used to start the corresponding cluster service module based on the task configuration table; the control module is used to control the cluster service module to execute the tasks in the task configuration table.

[0009] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of any of the above methods.

[0010] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of any of the above methods are implemented.

[0011] The technical solution disclosed in the embodiment of the present application obtains a task configuration table predefined by the user, which includes multiple tasks. Based on the task configuration table, the corresponding cluster service module is started, and the cluster service module is controlled to execute the tasks in the task configuration table. The corresponding cluster service module can be started based on the task configuration table, and each task is executed by the cluster service module, thereby avoiding the user from manually executing each specific data task, shortening the overall time consumption of executing data tasks, and improving the efficiency of cluster configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1A A first flow chart of a cloud service-based task control method provided in an embodiment of the present application;

[0014] Figure 1B A second flow chart of the cloud service-based task control method provided in an embodiment of the present application;

[0015] Figure 2 A third flow chart of the cloud service-based task control method provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of the module composition of a cloud service-based task control device provided in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0019] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0020] like Figure 1AAs shown, the embodiment of the present application provides a task control method based on cloud service, the execution subject of the method can be a cloud server, wherein the cloud server can be an independent server or a server cluster composed of multiple servers, and the server can be a server capable of executing data tasks. The communication method can specifically include the following steps S101-S103:

[0021] S101: Obtain a task configuration table predefined by a user.

[0022] Specifically, the task configuration table includes multiple tasks. Users can customize the task configuration table by customizing the development data project, which can be any executable code block. The task configuration table includes but is not limited to: task execution dependency configuration file, gitci deployment scheduled task configuration information. Task execution dependency configuration file includes but is not limited to: task configuration information (such as conf.yml file), task scheduling method information (such as flow.project file), task dependency information (such as project.flow file). The gitci deployment scheduled task configuration information can be a gitlab-ci.yml file.

[0023] Task configuration information can be used to configure scheduled tasks. Cron has two configuration files. One is the global configuration file ( / etc / crontab), which is for system tasks; the other is the configuration file generated by the crontab command (files under / var / spool / cron), which is for user-defined scheduled tasks. Specifically, it configures the time to start the scheduled task and the mailbox used to receive the task execution status.

[0024] The task scheduling method information includes the task scheduling method, such as using a task scheduling tool (azkaban tool) for scheduling.

[0025] Task dependency information includes a series of interrelated tasks. For example, user-defined tasks include Task 1, Task 2, and Task 3. In the task dependency information, the user defines Task 1, Task 2, and Task 3 to manage each other and defines the order of task execution as Task 1, Task 2, and Task 3.

[0026] The main purpose of gitci deployment of scheduled task configuration information is to package the task execution dependency configuration files and transmit them to the scheduling tool (such as azkaban tool).

[0027] S102: Start the corresponding cluster service module based on the task configuration table.

[0028] Specifically, after obtaining the task configuration table to determine the specific task, start the SDK call module of the cluster in AWS and start the cluster service module that executes the above task.

[0029] S103: Control the cluster service module to execute tasks in the task configuration table.

[0030] Specifically, after the cluster service module is started, each task has a corresponding cluster service module, and the cluster service module executes the corresponding task. Specifically, the cluster service module automatically executes the task according to the task execution order in the task dependency relationship and the task execution time configured in the task configuration information until all related tasks in the task configuration table are completed.

[0031] In a possible implementation, controlling the cluster service module to execute tasks in the task configuration table includes: scheduling multiple tasks in the task dependency information through a scheduling tool in the task scheduling method information, where each task is related to each other; and controlling the cluster service module to execute tasks in the task configuration table based on the task dependency information.

[0032] Through the technical solution disclosed in the embodiment of the present application, by obtaining the task configuration table predefined by the user, based on the task configuration table, starting the corresponding cluster service module, and controlling the cluster service module to execute the tasks in the task configuration table, the corresponding cluster service module can be started based on the task configuration table, and each task can be executed by the cluster service module, and after all tasks are completed, the cluster service is automatically closed, avoiding the user to manually execute each specific data task, shortening the overall time consumption of executing data tasks, and improving the efficiency of cluster configuration and the cost of cluster use.

[0033] In one possible implementation, Figure 1B As shown, after step S103, the method further includes: step S104.

[0034] S104: When the shutdown condition is triggered, the cluster service module is shut down. Specifically, when all tasks in the task configuration table are executed and there are no other execution processes in the cluster service module, the cluster service module is controlled to be shut down, which has high reliability.

[0035] like Figure 2 As shown, the embodiment of the present application provides a task control method based on cloud service, the execution subject of the method can be a cloud server, wherein the cloud server can be an independent server or a server cluster composed of multiple servers, and the server can be a server capable of executing data tasks. The communication method can specifically include the following steps S200-S203:

[0036] S200: Pull the user's predefined task configuration information, task scheduling method information and task dependency information to the git repository, and deploy the anonymous function (lambda function) scheduled task module in the cloud service platform (aws) through the git repository according to the task configuration information, task scheduling method information and task dependency information.

[0037] Specifically, through the open source distributed version control system (git) repository, the Lambda function scheduled task module in AWS is deployed according to the task configuration information, task scheduling information, and task dependency information. The modules include:

[0038] If a lambda function scheduled task module exists in AWS, the lambda function scheduled task module is updated through the git repository according to the task configuration information, task scheduling method information and task dependency information; if a lambda function scheduled task module does not exist in AWS, a lambda function scheduled task module is created through the git repository according to the task configuration information, task scheduling method information and task dependency information.

[0039] That is to say, if there is a lambda function scheduled task module in AWS, the relevant information in the lambda function scheduled task module is updated according to the task configuration information, task scheduling information and task dependency information, and the public resources in the lambda function scheduled task module do not need to be updated. If there is no lambda function scheduled task module in AWS, it is necessary to re-create the lambda function scheduled task module according to the task configuration information, task scheduling information and task dependency information.

[0040] S201: Obtain a task configuration table predefined by a user through a lambda function timer task module.

[0041] S202: Start the corresponding cluster service module based on the task configuration table.

[0042] S203: Control the cluster service module to execute tasks in the task configuration table.

[0043] It is worth noting that S201 to S203 have the same or similar implementation methods as the above-mentioned S101 to S103, which can refer to each other, and the embodiments of the present application will not be repeated here.

[0044] Through the technical solution disclosed in the embodiment of the present application, by obtaining the task configuration table predefined by the user, based on the task configuration table, starting the corresponding cluster service module, and controlling the cluster service module to execute the tasks in the task configuration table, the corresponding cluster service module can be started based on the task configuration table, and each task can be executed by the cluster service module, and after all tasks are completed, the cluster service is automatically closed, avoiding the user to manually execute each specific data task, shortening the overall time consumption of executing data tasks, and improving the efficiency of cluster configuration and the cost of cluster use.

[0045] The technical solution disclosed in the embodiment of the present application is further explained below in combination with specific application scenarios. For example, taking the calculation of user tags in the long bridge data task as an example, the cloud service-based task control method provided in the embodiment of the present application is further explained.

[0046] First, configure the following:

[0047] Perform task configuration, specifically configure the conf.yml file. The conf.yml file includes the following content (task configuration information):

[0048] PROJECT: lb-user-profile (project name)

[0049] JOB_LANG: python (project development language)

[0050] CRON: cron (timer)

[0051] FLOW: user_profile (name of the task)

[0052] ENV: PROD (execution environment)

[0053] Configure the task scheduling method, specifically configure the flow.project file, such as the task scheduling method is azkaban-flow-version: 2.0.

[0054] Configure task dependencies, specifically customize the tasks to be executed, as well as the relationship and execution order between tasks.

[0055] Configure the task configuration table, which includes the name of the task to be executed (project); task scheduling method; task dependencies; the region served by AWS (region_name); the version of the cluster (releaseLabel); the type of service that requires a cluster (applications); the configuration of metadata information in the cluster (configurations); the permission-related settings of the cluster in AWS; the time for the cloud platform to complete initialization (wait_for_cluster); select the type of cluster (InstanceGroups); and the code block executed during the cluster initialization process (bootstrapActions), which mainly downloads third-party resource packages.

[0056] Among them, the service types (applications) that require clusters take the task of calculating user tags in the Changqiao data task as an example. The service types are data warehouse service (Hive) and computing engine service (Spark), and the code is expressed as "[{"Name": "Hive"}, {"Name": "Spark"}]".

[0057] The configuration code of metadata information in the cluster can be expressed as "[{"Classification": "hive-site","Properties":{"hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}},{"Classification":"spark-hive-site","Properties":{"hive.metastore.client.factory.class":“com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory”}}]".

[0058] After the above configuration, it is stored in the database task configuration, then pulled to the git repository, and deployed to the lambda function scheduled task module from the git repository. In the process of deploying the lambda function scheduled task module, if the lambda function scheduled task module exists in AWS, the lambda function scheduled task module is updated through the git repository according to the task configuration information, task scheduling method information, and task dependency information; if the lambda function scheduled task module does not exist in AWS, the lambda function scheduled task module is created through the git repository according to the task configuration information, task scheduling method information, and task dependency information.

[0059] After the above lambda function scheduled task module is deployed, first read the task configuration table predefined by the user. Reading the task configuration table can be implemented by the following code:

[0060] # read emr parameter from mysql

[0061] ####################

[0062] job_parameter = []

[0063] try:

[0064] job_parameter = simple_q.select(

[0065] "scheduler_job_configuration_info",

[0066] column="*",

[0067] condition="project = '" + PROJECT + "'" )

[0069] except:

[0070] print>>sys.stderr, "Query EMR parameter from MySQL Faild."

[0071] exit(1)

[0072] if job_parameter:

[0073] job_parameter = job_parameter[0]

[0074] else:

[0075] print>>sys.stderr, "EMR parameter Not Read."

[0076] exit(1)

[0077] if len(job_parameter) != 18:

[0078] print>>sys.stderr, "EMR parameter Not Match"

[0079] exit(1)

[0080] After reading the task configuration table, wait for the cluster service to be created successfully and obtain the cluster ID, and then start the cluster service module, which is implemented by the following code:

[0081] response = aws.run_job_flow_no_steps(clustername, emr_logUri,releaseLabel, applications, instances, configurations, bootstrapActions,serviceRole, jobFlowRole)

[0082] ##########################################

[0083] # wait for emr ready to get master ip

[0084] ##########################################

[0085] time.sleep(wait_for_cluster)

[0086] CLUSTER_ID = response['JobFlowId']

[0087] try:

[0088] instance_groups = aws.get_instance_groups(cluster_id=CLUSTER_ID)

[0089] except:

[0090] print>>sys.stderr, "GET CLUSTERID FAILED, TERMINATE EMR."

[0091] aws.terminate_emr(cluster_id=CLUSTER_ID)

[0092] exit(1)

[0093] try:

[0094] instance_fleet = aws.get_instance_fleet(cluster_id=CLUSTER_ID)

[0095] except:

[0096] print>>sys.stderr, "GET JOBFLOWID FAILED, TERMINATE EMR."

[0097] aws.terminate_emr(cluster_id=CLUSTER_ID)

[0098] exit(1)

[0099] try:

[0100] master_id = aws.get_master_id(instance_groups=instance_groups)

[0101] except:

[0102] print>>sys.stderr, "GET MASTER_ID FAILED, TERMINATE EMR."

[0103] aws.terminate_emr(cluster_id=CLUSTER_ID)

[0104] exit(1)

[0105] try:

[0106] master_ip = aws.get_master_ip(instance_fleet=instance_fleet, master_id=master_id)

[0107] except:

[0108] print>>sys.stderr, "GET MASTER_IP FAILED, TERMINATE EMR."

[0109] aws.terminate_emr(cluster_id=CLUSTER_ID)

[0110] exit(1)

[0111] steps = [{

[0112] 'Name': 'deploy_peoject',

[0113] 'ActionOnFailure': 'TERMINATE_CLUSTER',

[0114] 'HadoopJarStep': {

[0115] 'Jar': 's3: / / ' + region_name + '.elasticmapreduce / libs / script-runner / script-runner.jar',

[0116] # 'Jar': 'command-runner.jar',

[0117] 'Args': [

[0118] PROJECT_PATH + 'service / deploy-project.sh', ENV, PROJECT, master_ip,JOB_LANG, FLOW ]

[0120] }

[0121] } ]

[0123] After starting the corresponding cluster service module, add specific tasks (that is, pull the tasks to be executed to the scheduling tool), which is implemented through the following code:

[0124] aws.add_job_flow(cluster_id=CLUSTER_ID, steps=steps).

[0125] Then execute the task according to the task dependency in the task configuration table. The following code can be used to implement it:

[0126] aws.run_job_flow_with_steps(clustername, emr_logUri, releaseLabel, applications, instances, configurations, bootstrapActions, serviceRole, jobFlowRole, steps).

[0127] After all tasks are completed, shut down the corresponding cluster service module. This can be achieved in the following ways:

[0128] try:

[0129] from common import AWS

[0130] # TODO read from mysql

[0131] aws = AWS(region_name='ap-east-1')

[0132] cluster_info = aws.list_cluster()

[0133] except Exception, err:

[0134] print err

[0135] print>>sys.stderr, "INITAL AWS FAILED."

[0136] exit(1)

[0137] try:

[0138] target_cluster_id = get_cluster(cluster_info, CLUSTER_NAME)

[0139] except Exception, err:

[0140] print err

[0141] print>>sys.stderr, "GET TARGET EMR FAILED."

[0142] exit(1)

[0143] if len(target_cluster_id) == 0:

[0144] print "NO TARGET RUNING EMR"

[0145] exit(1)

[0146] for cluster_id in target_cluster_id:

[0147] try:

[0148] aws.terminate_emr(cluster_id=cluster_id)

[0149] except Exception, err:

[0150] print err

[0151] print>>sys.stderr, "TERMINATE {cluster_id} FAILED.".format(cluster_id=cluster_id).

[0152] Through the technical solution disclosed in the embodiment of the present application, by obtaining the task configuration table predefined by the user, based on the task configuration table, starting the corresponding cluster service module, and controlling the cluster service module to execute the tasks in the task configuration table, the corresponding cluster service module can be started based on the task configuration table, and each task can be executed by the cluster service module, and after all tasks are completed, the cluster service is automatically closed, avoiding the user to manually execute each specific data task, shortening the overall time consumption of executing data tasks, and improving the efficiency of cluster configuration and the cost of cluster use.

[0153] Corresponding to the cloud service-based task control method provided in the above embodiment, based on the same technical concept, the embodiment of the present application also provides a cloud service-based task control device, Figure 3 A schematic diagram of the module composition of a cloud service-based task control device provided in an embodiment of the present application, wherein the cloud service-based task control device is used to execute Figure 1A to Figure 2 The described cloud service-based mission control method, such as Figure 3 As shown, the cloud service-based task control device 300 includes: an acquisition module 301 , a start module 302 and a control module 303 .

[0154] The acquisition module 301 is used to obtain the task configuration table predefined by the user, which includes multiple tasks; the startup module 302 is used to start the corresponding cluster service module based on the task configuration table; the control module 303 is used to control the cluster service module to execute the tasks in the task configuration table.

[0155] Through the technical solution disclosed in the embodiment of the present application, the corresponding cluster service module can be started based on the task configuration table, and each task can be executed by the cluster service module, avoiding the user from manually executing each specific data task, shortening the overall time consumption of executing data tasks, and improving the efficiency of cluster configuration.

[0156] In a possible implementation, the method further includes:

[0157] The shutdown module is used to shut down the cluster service module when the shutdown condition is triggered.

[0158] In a possible implementation, the task configuration table includes: task configuration information, task scheduling method information and task dependency information, and also includes: a pull module, which is used to pull the user's predefined task configuration information, task scheduling method information and task dependency information to a git repository; a deployment module, which is used to deploy the lambda function scheduled task module in aws through the git repository according to the task configuration information, task scheduling method information and task dependency information; and an acquisition module, which is also used to obtain the user's predefined task configuration table through the lambda function scheduled task module.

[0159] In a possible implementation, the control module 303 is also used to schedule multiple tasks in the task dependency information through the scheduling tool in the task scheduling method information, and each task is related to each other; based on the task dependency information, the cluster service module is controlled to execute the tasks in the task configuration table.

[0160] In a possible implementation, the deployment module is further used to update the lambda function scheduled task module according to the task configuration information, the task scheduling mode information and the task dependency information through the git repository when the lambda function scheduled task module exists in the aws;

[0161] If there is no lambda function scheduled task module in AWS, create a lambda function scheduled task module through the git repository according to the task configuration information, task scheduling method information and task dependency information.

[0162] The cloud service-based task control device provided in the embodiment of the present application can implement each process in the embodiment corresponding to the above-mentioned cloud service-based task control method. To avoid repetition, it will not be repeated here.

[0163] It should be noted that the cloud service-based task control device provided in the embodiment of the present application and the cloud service-based task control method provided in the embodiment of the present application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned cloud service-based task control method, and the repeated parts will not be repeated.

[0164] Corresponding to the task control method based on cloud service provided in the above embodiment, based on the same technical concept, the embodiment of the present application further provides an electronic device, which is used to execute the task control method based on cloud service. Figure 4 A schematic diagram of the structure of an electronic device for implementing each embodiment of the present application is shown in FIG. Figure 4 As shown. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and memory 402, and the memory 402 may store one or more storage applications or data. Among them, the memory 402 may be a temporary storage or a permanent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device.

[0165] Furthermore, the processor 401 may be configured to communicate with the memory 402 to execute a series of computer executable instructions in the memory 402 on the electronic device. The electronic device may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input and output interfaces 405, and one or more keyboards 406.

[0166] Specifically in this embodiment, the electronic device includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory, and implements the following method steps: obtaining a task configuration table predefined by the user, the task configuration table including multiple tasks, based on the task configuration table, starting the corresponding cluster service module, and controlling the cluster service module to execute the tasks in the task configuration table.

[0167] It can be seen from the technical solution provided in the above embodiments of the present application that the corresponding cluster service module can be started based on the task configuration table, and each task can be executed by the cluster service module, thereby avoiding the user from manually executing each specific data task, shortening the overall time for executing data tasks, and improving the efficiency of cluster configuration.

[0168] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0169] Obtain a task configuration table predefined by the user, which includes multiple tasks. Based on the task configuration table, start the corresponding cluster service module and control the cluster service module to execute the tasks in the task configuration table.

[0170] It can be seen from the technical solution provided in the above embodiments of the present application that the corresponding cluster service module can be started based on the task configuration table, and each task can be executed by the cluster service module. After all tasks are completed, the cluster service is automatically shut down, avoiding users from manually executing each specific data task, shortening the overall time for executing data tasks, and improving the efficiency of cluster configuration and the cost of cluster use.

[0171] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0173] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0175] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0176] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0177] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0178] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0179] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0180] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A task control method based on cloud service, characterized in that: include: Obtaining a task configuration table predefined by a user, wherein the task configuration table includes a plurality of tasks; Based on the task configuration table, start the corresponding cluster service module; Controlling the cluster service module to execute the tasks in the task configuration table; The task configuration table includes a task dependency configuration file and gitci deployment scheduled task configuration information, the task dependency configuration file includes task configuration information, task scheduling mode information and task dependency information, and the gitci deployment scheduled task configuration information is used to package the task execution dependency configuration file and transmit it to the scheduling tool; Before obtaining the task configuration table predefined by the user, the method further includes: Pull the user's predefined task configuration information, task scheduling information, and task dependency information to the git repository; Deploy the lambda function scheduled task module in AWS according to the task configuration information, the task scheduling method information and the task dependency information through the git repository; The obtaining of the task configuration table predefined by the user comprises: Obtaining a user-predefined task configuration table through the lambda function timing task module; The starting of the corresponding cluster service module based on the task configuration table includes: The lambda function timing task module obtains a task configuration table predefined by the user; Wait for the cluster service to be created successfully and obtain the cluster ID; Start the corresponding cluster service module.

2. The cloud service-based task control method according to claim 1, characterized in that: After controlling the cluster service module to execute the task in the task configuration table, the method further includes: When a shutdown condition is triggered, the cluster service module is shut down.

3. The cloud service-based task control method according to claim 1, characterized in that: The controlling the cluster service module to execute the tasks in the task configuration table includes: Scheduling multiple tasks in the task dependency information by using a scheduling tool in the task scheduling mode information, wherein the tasks are mutually related; Based on the task dependency information, the cluster service module is controlled to execute the tasks in the task configuration table.

4. The cloud service-based task control method according to claim 1, characterized in that: The deploying of the lambda function scheduled task module in AWS according to the task configuration information, the task scheduling mode information and the task dependency information through the git repository includes: In the case where the lambda function scheduled task module exists in the aws, the lambda function scheduled task module is updated through the git repository according to the task configuration information, the task scheduling mode information and the task dependency information; In the case that the lambda function scheduled task module does not exist in the aws, the lambda function scheduled task module is created through the git repository according to the task configuration information, the task scheduling method information and the task dependency information.

5. A task control device based on cloud service, characterized in that: include: An acquisition module, used to acquire a task configuration table predefined by a user, wherein the task configuration table includes a plurality of tasks; A startup module, used to start the corresponding cluster service module based on the task configuration table; A control module, used to control the cluster service module to execute the tasks in the task configuration table; The task configuration table includes a task dependency configuration file and gitci deployment scheduled task configuration information, the task dependency configuration file includes task configuration information, task scheduling mode information and task dependency information, and the gitci deployment scheduled task configuration information is used to package the task execution dependency configuration file and transmit it to the scheduling tool; The pull module is used to pull the user's predefined task configuration information, task scheduling information and task dependency information to the git repository; A deployment module, used to deploy the lambda function scheduled task module in AWS through the git repository according to the task configuration information, the task scheduling mode information and the task dependency information; The acquisition module is further used to acquire a task configuration table predefined by a user through the lambda function timing task module; The startup module is specifically used to obtain a task configuration table predefined by a user through the lambda function timing task module; Wait for the cluster service to be created successfully and obtain the cluster ID; Start the corresponding cluster service module.

6. The cloud service-based task control device according to claim 5, characterized in that: Also includes: The closing module is used to close the cluster service module when a closing condition is triggered.

7. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1 to 4.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images