A task execution method, apparatus, device, and storage medium

Abstract

The embodiments of the present application disclose a task running method, device, equipment, and storage medium. The method includes: when the NodeManager is started, obtaining a target mirror file; when the NodeManager starts to execute a target task, starting an application container engine and mounting a configuration file corresponding to the target task into the target mirror file; after the application container engine is started, executing a container running script and executing a container process corresponding to the target task based on the configuration file in the target mirror file. This method can effectively limit the resource consumption when the NodeManager runs its Container.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a task execution method, apparatus, device, and storage medium. Background Art

[0002] Yet Another Resource Negotiator (Yarn) is a resource manager for the new distributed file system Hadoop. In essence, it is a general resource management system that can provide unified resource management and scheduling for upper-layer applications.

[0003] NodeManager is a single-node agent in Yarn, which is responsible for managing a single computing node in the Hadoop cluster. During actual operation, NodeManager needs to interact with ApplicationMaster and ResourceManager, receive commands (such as start commands, stop commands, etc.) for relevant containers from ApplicationMaster and execute them, and report the running status of each container and the node health status to ResourceManager.

[0004] In related technologies, when NodeManager in Yarn runs its containers, resource isolation usually cannot be achieved, that is, the configured upper limit of resource usage for a container cannot effectively limit the resources consumed by the container during runtime. For example, a container with a configured resource of 2GB and 1 core may actually consume more than 4GB of virtual machine memory and the utilization rate of the Central Processing Unit (CPU) may exceed 300% during runtime.

[0005] In summary, how to effectively limit the resource consumption of NodeManager when running its containers has become an urgent problem to be solved currently. Summary of the Invention

[0006] In view of this, embodiments of this application provide a task running method, apparatus, device, and storage medium, which can effectively limit the resource consumption of NodeManager when running its containers.

[0007] The first aspect of this application provides a task running method, and the method includes:

[0008] When NodeManager is started, obtain a target image file;

[0009] When the NodeManager starts to execute the target task, it starts the application container engine and mounts the configuration file corresponding to the target task into the target image file;

[0010] After the application container engine starts, it executes the container running script and executes the container process corresponding to the target task based on the configuration file in the target image file.

[0011] Optionally, the method further includes:

[0012] Detect the target user information corresponding to the submitter of the target task;

[0013] After the application container engine starts, according to the target user information, generate a target virtual user and the permissions corresponding to the target virtual user;

[0014] Then, after the application container engine starts, executing the container running script and executing the container process corresponding to the target task based on the configuration file in the target image file includes:

[0015] After the application container engine starts, it executes the container running script, and through the target virtual user according to its corresponding permissions, executes the container process corresponding to the target task based on the configuration file in the target image file.

[0016] Optionally, the configuration file includes at least one of the following:

[0017] Resource configuration file of the distributed file system Hadoop, resource configuration file of the computing engine Spark, target software package file.

[0018] Optionally, when the NodeManager starts, the method further includes:

[0019] Detect whether a target user account is currently logged in to the application container engine;

[0020] If so, execute the obtaining of the target image file;

[0021] If not, obtain the target user login information, log in to the target user account on the application container engine based on the target user login information, and after detecting successful login, execute the obtaining of the target image file.

[0022] Optionally, the obtaining of the target image file includes:

[0023] Detect whether there is a mirror file configured by the target user pre-stored in the container mirror file library;

[0024] If so, obtain the mirror file configured by the target user as the target mirror file;

[0025] If not, obtain the default mirror file preset by the operating system as the target mirror file.

[0026] Optionally, the method further includes:

[0027] Receive a task submission request; the task submission request is generated in response to a task submission operation triggered by the user;

[0028] The NodeManager defaults to running the container process corresponding to the task in the application container engine.

[0029] Optionally, the application container engine includes any one of the following: Docker, HeroKu, and NodeJitsu.

[0030] The second aspect of this application provides a task running device, and the device includes:

[0031] A mirror file acquisition module, configured to obtain a target mirror file when the node manager NodeManager starts;

[0032] A configuration file mounting module, configured to start the application container engine and mount the configuration file corresponding to the target task into the target mirror file when the NodeManager starts to execute the target task;

[0033] A task execution module, configured to execute a container running script after the application container engine starts, and execute the container process corresponding to the target task based on the configuration file in the target mirror file.

[0034] Optionally, the device further includes:

[0035] An information detection module, configured to detect target user information corresponding to the submitter of the target task;

[0036] A permission generation module, configured to generate a target virtual user and the permissions corresponding to the target virtual user according to the target user information after the application container engine starts;

[0037] Then the task execution module is specifically configured to:

[0038] After the application container engine starts, execute the container running script, and execute the container process corresponding to the target task based on the configuration file in the target mirror file by the target virtual user according to its corresponding permissions.

[0039] Optionally, the configuration file includes at least one of the following:

[0040] Resource configuration files of the distributed file system Hadoop, resource configuration files of the computing engine Spark, and target software package files.

[0041] Optionally, the device further includes:

[0042] A login detection module, configured to detect whether a target user account is currently logged in to the application container engine when the NodeManager starts; if so, trigger the mirror file acquisition module to execute the acquisition of the target mirror file; if not, obtain target user login information, log in the target user account on the application container engine based on the target user login information, and after detecting successful login, trigger the mirror file to execute the acquisition of the target mirror file.

[0043] Optionally, the mirror file acquisition module is specifically configured to:

[0044] Detect whether a mirror file configured by the target user is pre-stored in the container image file library;

[0045] If so, obtain the mirror file configured by the target user as the target mirror file;

[0046] If not, obtain the default mirror file preset by the operating system as the target mirror file.

[0047] Optionally, the device further includes:

[0048] A request receiving module, configured to receive a task submission request; the task submission request is generated in response to a task submission operation triggered by a user;

[0049] The task execution module is specifically configured to:

[0050] The NodeManager defaults to running the container process corresponding to the task in the application container engine.

[0051] Optionally, the application container engine includes any one of the following: Docker, HeroKu, and NodeJitsu.

[0052] A third aspect of this application provides an electronic device, which includes:

[0053] A processor and a memory:

[0054] The memory is used to store a computer program and transmit the computer program to the processor;

[0055] The processor is configured to execute the task running method described in the first aspect according to the computer program.

[0056] In the fourth aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the task running method described in the first aspect above.

[0057] In the fifth aspect of the present application, a computer program product including instructions is provided. When it runs on a computer, it causes the computer to execute the task running method described in the first aspect above.

[0058] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0059] The embodiment of the present application provides a task running method. This method runs the Container process of NodeManager in the application container engine, which can effectively limit the resources consumed during the runtime of the Container. Specifically, in the task running method provided by the embodiment of the present application, when NodeManager starts, it will automatically obtain the target mirror file; when NodeManager starts to execute the target task submitted by the user, it enables the application container engine and automatically mounts the configuration file corresponding to the target task to the previously obtained target mirror file; after the application container engine starts, it can execute the container running script to execute the Container process corresponding to the target task based on the configuration file in the target mirror file. Compared with directly running the Container process of NodeManager on the operating system, the method provided by the embodiment of the present application runs the Container process of NodeManager in the application container engine, and effectively limits the resources consumed during the runtime of the Container process through this application container engine. In addition, the method provided by the embodiment of the present application can automatically mount the configuration file corresponding to the target task to the target mirror file, without the need for the user to package the mirror file, which also greatly improves the convenience for the user to use the application container engine to execute the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic flowchart of the task running method provided by the embodiment of the present application;

[0061] Figure 2 It is a schematic implementation architecture diagram of the task running method provided by the embodiment of the present application;

[0062] Figure 3 It is a schematic structural diagram of the task running device provided by the embodiment of the present application;

[0063] Figure 4 It is a schematic structural diagram of the server provided by the embodiment of the present application;

[0064] Figure 5 The structural schematic diagram of the terminal device provided by the embodiment of the present application. Specific embodiments

[0065] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0066] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0067] In the related art, when the NodeManager in Yarn runs its Container, it usually cannot effectively isolate resources for the Container. The reason is that the Container is only a conceptual Container, and it is still running in the operating system and can still directly use the resources of the operating system. Therefore, even if the upper limit of resource usage is configured for the Container, it is impossible to effectively limit the resources consumed when the Container process runs.

[0068] In view of the problems existing in the above-mentioned related art, the embodiment of the present application provides a task running method, which runs the Container process of the NodeManager in an application container engine to effectively limit the resources consumed when the Container process runs through the application container engine.

[0069] Specifically, in the task running method provided in the embodiments of the present application, when the NodeManager starts, it can automatically obtain the target image file. When the NodeManager starts to execute the target task, it starts the application container engine and automatically mounts the configuration file corresponding to the target task into the previously obtained target image file. After the application container engine starts, it executes the container running script to execute the container process corresponding to the target task based on the configuration file in the target image file.

[0070] Compared with directly running the Container process of the NodeManager on the operating system, the method provided in the embodiments of the present application runs the Container process of the NodeManager in the application container engine, and effectively limits the resources consumed during the running of the Container process through the application container engine. In addition, the method provided in the embodiments of the present application can automatically mount the configuration file corresponding to the target task into the target image file, without the user packaging the image file, and also greatly improves the convenience for the user to use the application container engine to execute the target task.

[0071] To facilitate the understanding of the method provided in the embodiments of the present application, the basic concepts involved in the method provided in the embodiments of the present application will be introduced first below.

[0072] Yarn is a system framework for resource management and task scheduling, which mainly includes three major modules: ResourceManager, NodeManager, and ApplicationMaster. Among them, ResourceManager is responsible for the monitoring, allocation, and management of all resources, ApplicationMaster is responsible for the scheduling and coordination of each specific application program, and NodeManager is responsible for the maintenance of each node. For all application programs, ResourceManager has absolute control and the right to allocate resources. ApplicationMaster is used to negotiate resources with ResourceManager and communicate with NodeManager to execute and monitor tasks. The method provided in the embodiments of the present application is mainly proposed for the process of NodeManager executing tasks, so that when the NodeManager executes tasks through its Container, the resources consumed by the Container can be effectively limited.

[0073] The application container engine in the embodiments of the present application can be an open-source application container engine, such as Docker, etc. Developers can package an application and its dependencies into a portable container based on this application container engine, and then publish it to any Linux machine or Windows machine. Virtualization can also be achieved. Containers fully use the sandbox mechanism and there will be no interfaces between them. When performing related tasks through the application container engine, it needs to be implemented based on the image file of the application container engine.

[0074] The task running method provided by the present application will be introduced below through method embodiments.

[0075] See Figure 1 , Figure 1 which is a schematic flowchart of the task running method provided by the embodiments of the present application. As Figure 1 shown, the method includes the following steps:

[0076] Step 101: When NodeManager starts, obtain the target image file.

[0077] In practical applications, when NodeManager starts, it can automatically obtain the target image file from the container image file library. This target image file is the image file based on which NodeManager will run the Container process of the target task in the application container engine later.

[0078] It should be understood that the start of NodeManager means that there is a current need to use this NodeManager, and later it is necessary to use this NodeManager to execute the target task submitted by the user. Correspondingly, when NodeManager starts, it can automatically obtain the target image file required to run the Container process in the application container engine from the container image file library.

[0079] It should be noted that in the related art, each node needs to perform a login operation of the application container engine when starting, that is, each time a node starts, it needs to log in to the application container engine once, which is too cumbersome for users. In order to avoid performing a login operation of the application container engine every time a node starts, the technical solution provided by the embodiments of the present application, when detecting that NodeManager starts, can first detect whether the target user account is currently logged in to the application container engine. If so, the target image file can be directly obtained from the container image file library. If not, obtain the target user login information, and then log in to the target user account on the application container engine based on the target user login information. After successful login, obtain the target image file again.

[0080] Taking the application container engine Docker as an example, when NodeManager starts, it can first detect whether it is necessary to perform the docker login operation, that is, to detect whether the target user account is currently logged in to Docker. If so, the target image file can be directly obtained from the container image file library. Conversely, if not, the docker login operation needs to be performed. Exemplarily, the user can be prompted to input the login information (i.e., the target user login information) registered in advance. For example, the user can be prompted to input the user name and password reserved when registering Docker, etc. After receiving the user name and password input by the user, the user identity is verified based on the user name and password. After the verification is passed, the user is allowed to log in to Docker, and after detecting that Docker has successfully logged in to the target user account, the target image file is further obtained from the container image file library.

[0081] It should be noted that in order to ensure that the obtained target image file can better support the operation of the Container process of NodeManager, the method provided by the embodiments of the present application supports both obtaining the image file configured by the target user and obtaining the default image file preset by the operating system. This default image file can effectively support the operation of most Container processes.

[0082] Specifically, when obtaining the target image file, it can first be detected whether there is a target user-configured image file pre-stored in the container image file library; if a target user's own pre-configured image file is detected in the container image file library, then this image file is obtained as the target image file to ensure better support for the operation of the Container process of NodeManager; if a target user's own pre-configured image file is not detected in the container image file library, then the default image file preset by the operating system is obtained. Since this default image file can better support the operation of most Container processes, the stability of the application operation can also be ensured.

[0083] Step 102: When NodeManager starts to execute the target task, start the application container engine and mount the configuration file corresponding to the target task into the target image file.

[0084] When NodeManager starts to execute the target task, the application container engine can be correspondingly started. At the same time, the configuration file corresponding to the target task can also be automatically mounted into the target image file obtained in step 101. Compared with the related art where the user needs to encapsulate the image file personally, the method provided by the embodiments of the present application can automatically encapsulate the target image file when it is determined that NodeManager starts to execute the task, which is more convenient for users to use.

[0085] It should be noted that the application container engine in the embodiments of the present application can be any application container engine, such as Docker, HeroKu, NodeJitsu, etc. The present application does not make any limitation on the application container engine herein.

[0086] It should be noted that in practical applications, the configuration files mounted to the target image file include, but are not limited to, the resource configuration files of the distributed file system Hadoop, the resource configuration files of the computing engine Spark, and the target software package files (i.e., jar files). It should be understood that all kinds of files required for the Container of NodeManager to run in the application container engine can be mounted to the target image file in this step. The present application does not make any limitation on the configuration files mounted to the target image file herein.

[0087] In addition, in the related art, when a user creates and submits a task submission request, it is often necessary to set keywords related to the application container engine in the task submission request, such as the docker keyword, etc. If the keywords related to the application container engine are not set, the operating system is used to run the task by default. This method not only requires the user to configure keywords, which is cumbersome and complex to operate, but also exposes the keywords of the application container engine when transmitting the task submission request, resulting in reduced security.

[0088] In view of the problems existing in the above related art, the method provided in the embodiments of the present application sets that after receiving a task submission request, the Container process of NodeManager is defaultly run in the application container engine. That is, after receiving the task submission request, which is generated in response to a task submission operation triggered by the user, NodeManager defaultly runs the Container process corresponding to the task in the application container engine.

[0089] In this way, the user can omit the keywords of the application container engine when submitting a task job, realizing that the Container process of NodeManager runs in the application container engine without the user's awareness, and effectively improving security.

[0090] Step 103: After the application container engine is started, execute a container running script, and execute the container process corresponding to the target task based on the configuration file in the target image file.

[0091] After the application container engine is started, execute a container running script, so as to execute the Container process corresponding to the target task based on the configuration file in the target image file, that is, realize running the Container process of NodeManager in the application container engine.

[0092] Taking the application container engine Docker as an example, after Docker is started, the docker-run script is executed, so that launch_container.sh is executed based on the configuration file in the target image file.

[0093] Considering that the execution of the Container process needs to be implemented by a user or a user group, therefore, the method provided in the embodiment of the present application can also automatically detect the target user information corresponding to the submitter of the target task, and after the application container engine is started, generate a target virtual user and the permissions corresponding to the target virtual user according to the target user information. Furthermore, when executing the container run script, the target virtual user can execute the container process corresponding to the target task based on the configuration file in the target image file according to its corresponding permissions.

[0094] Still taking the application container engine Docker as an example, the user who submits the target task can be automatically detected by running a custom docker-run script, and a user or a user group and the permission information corresponding to the user or the user group can be automatically created after Docker is started. Furthermore, through the created user or user group, the Container process of NodeManager can be executed based on the configuration file in the target image file according to its corresponding permissions.

[0095] After the target task is completed in the application container engine, the entire process terminates.

[0096] The task running method provided by the embodiment of the present application runs the Container process of NodeManager in the application container engine, which can effectively limit the resources consumed when the Container runs. Specifically, in the task running method provided by the embodiment of the present application, NodeManager will automatically obtain the target image file when starting; when NodeManager starts to execute the target task submitted by the user, the application container engine is enabled, and the configuration file corresponding to the target task is automatically mounted to the target image file obtained previously; after the application container engine is started, the container run script can be executed to execute the Container process corresponding to the target task based on the configuration file in the target image file. Compared with directly running the Container process of NodeManager on the operating system, the method provided by the embodiment of the present application runs the Container process of NodeManager in the application container engine, and effectively limits the resources consumed when the Container process runs through the application container engine. In addition, the method provided by the embodiment of the present application can automatically mount the configuration file corresponding to the target task to the target image file, without the user packaging the image file, which also greatly improves the convenience of the user using the application container engine to execute the target task.

[0097] To facilitate a further understanding of the technical solution provided by the embodiments of the present application, taking the application container engine Docker as an example, the technical solution provided by the embodiments of the present application will be introduced exemplarily in combination with Figure 2 the operation process schematic diagram shown below.

[0098] As Figure 2 shown, after the NodeManager is started, it first detects whether the docker login operation needs to be executed. If so, the docker login operation is executed.

[0099] After the docker login is successful, the image can be automatically obtained. The image obtained here can be the image configured by the user himself. If there is no image configured by the user himself, the system default image can also be obtained.

[0100] When executing a task, docker is started, and the relevant configurations of Hadoop and Spark and the jar files are automatically mounted into the image.

[0101] After starting docker, the docker-run script is executed to automatically create a user or a user group, and the created user is used to execute launch_container.sh.

[0102] When the task execution is completed, it terminates.

[0103] The embodiments of the present application also provide a task running device. Refer to Figure 3 , Figure 3 which is the structural schematic diagram of the task running device provided by the embodiments of the present application. As Figure 3 shown, the task running device includes:

[0104] An image file acquisition module 301, configured to acquire a target image file when the node manager NodeManager is started;

[0105] A configuration file mounting module 302, configured to start an application container engine and mount the configuration file corresponding to the target task into the target image file when the NodeManager starts to execute the target task;

[0106] A task execution module 303, configured to execute a container running script after the application container engine is started, and execute the container process corresponding to the target task based on the configuration file in the target image file.

[0107] Optionally, the device further includes:

[0108] An information detection module, configured to detect the target user information corresponding to the submitter of the target task;

[0109] A permission generation module, which is used to generate a target virtual user and the permissions corresponding to the target virtual user according to the target user information after the application container engine is started;

[0110] Then the task execution module is specifically used for:

[0111] After the application container engine is started, execute the container running script, and based on the configuration file in the target image file, execute the container process corresponding to the target task through the target virtual user according to its corresponding permissions.

[0112] Optionally, the configuration file includes at least one of the following:

[0113] The resource configuration file of the distributed file system Hadoop, the resource configuration file of the computing engine Spark, and the target software package file.

[0114] Optionally, the device further includes:

[0115] A login detection module, which is used to detect whether a target user account is currently logged in to the application container engine when the NodeManager is started; if so, trigger the mirror file acquisition module to execute the acquisition of the target mirror file; if not, obtain the target user login information, log in the target user account on the application container engine based on the target user login information, and after detecting successful login, trigger the mirror file to execute the acquisition of the target mirror file.

[0116] Optionally, the mirror file acquisition module is specifically used for:

[0117] Detect whether there is a mirror file configured by the target user pre-stored in the container mirror file library;

[0118] If so, obtain the mirror file configured by the target user as the target mirror file;

[0119] If not, obtain the default mirror file preset by the operating system as the target mirror file.

[0120] Optionally, the device further includes:

[0121] A request receiving module, which is used to receive a task submission request; the task submission request is generated in response to a task submission operation triggered by a user;

[0122] The task execution module is specifically used for:

[0123] The NodeManager defaults to running the container process corresponding to the task in the application container engine.

[0124] Optionally, the application container engine includes any one of the following: Docker, HeroKu, and NodeJitsu.

[0125] In the task running device provided by the embodiments of the present application, running the Container process of NodeManager in the application container engine can effectively limit the resources consumed during the runtime of the Container. Specifically, in the task running device provided by the embodiments of the present application, when NodeManager starts, it will automatically obtain the target mirror file; when NodeManager starts to execute the target task submitted by the user, it enables the application container engine and automatically mounts the configuration file corresponding to the target task to the previously obtained target mirror file; after the application container engine starts, it can execute the container running script to execute the Container process corresponding to the target task based on the configuration file in the target mirror file. Compared with directly running the Container process of NodeManager on the operating system, in the device provided by the embodiments of the present application, the Container process of NodeManager is run in the application container engine, and the resources consumed during the runtime of the Container process are effectively limited through this application container engine. In addition, the device provided by the embodiments of the present application can automatically mount the configuration file corresponding to the target task to the target mirror file, without the user packaging the mirror file, which also greatly improves the convenience for the user to use the application container engine to execute the target task.

[0126] The embodiments of the present application further provide an electronic device for running tasks. Specifically, this electronic device can be a server and a terminal device. Below, the server and terminal device provided by the embodiments of the present application will be introduced from the perspective of hardware implementation.

[0127] See Figure 4 , Figure 4Schematic diagram of a structure of a server 400 provided by an embodiment of the present application. The server 400 may vary greatly due to different configurations or performances, and may include one or more central processing units (CPUs) 422 (for example, one or more processors) and a memory 432, and one or more storage media 430 (for example, one or more mass storage devices) for storing application programs 442 or data 444. Among them, the memory 432 and the storage media 430 may be transient storage or persistent storage. The programs stored in the storage media 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Further, the central processing unit 422 may be configured to communicate with the storage media 430 and execute a series of instruction operations in the storage media 430 on the server 400.

[0128] The server 400 may further include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input / output interfaces 458, and / or one or more operating systems 441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so on.

[0129] In the above embodiment, the steps executed by the server may be based on the Figure 4 shown server structure.

[0130] Among them, the CPU 422 is used to execute the following steps:

[0131] When the NodeManager starts, obtain the target mirror file;

[0132] When the NodeManager starts to execute the target task, start the application container engine and mount the configuration file corresponding to the target task into the target mirror file;

[0133] After the application container engine starts, execute the container running script and execute the container process corresponding to the target task based on the configuration file in the target mirror file.

[0134] Optionally, the CPU 422 may also be used to execute the steps of any implementation manner of the task running method provided by the embodiment of the present application.

[0135] See Figure 5 , Figure 5A schematic structural diagram of a terminal device provided by an embodiment of the present application. For ease of description, only the parts related to the embodiments of the present application are shown. For the specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal may be any terminal device including a computer, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), etc. Taking the terminal as a computer as an example:

[0136] Figure 5 What is shown is a block diagram of a part of the structure of a computer related to the terminal provided by an embodiment of the present application. Refer to Figure 5 , the computer includes: a radio frequency (English full name: Radio Frequency, English abbreviation: RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (English full name: wirelessfidelity, English abbreviation: WiFi) module 570, a processor 580, and a power supply 590 and other components. Those skilled in the art can understand that Figure 5 the computer structure shown in

[0137] does not constitute a limitation on the computer, and may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0138] The processor 580 is the control center of the computer, connects various parts of the entire computer using various interfaces and lines, and by running or executing the software programs and / or modules stored in the memory 520, and calling the data stored in the memory 520, executes various functions of the computer and processes data, thereby monitoring the computer as a whole. Optionally, the processor 580 may include one or more processing units; preferably, the processor 580 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above modem processor may not be integrated into the processor 580.

[0139] In the embodiments of the present application, the processor 580 included in the terminal further has the following functions:

[0140] When the NodeManager is started, obtain the target mirror file;

[0141] When the NodeManager starts to execute the target task, start the application container engine and mount the configuration file corresponding to the target task into the target mirror file;

[0142] After the application container engine is started, execute the container running script and execute the container process corresponding to the target task based on the configuration file in the target mirror file.

[0143] Optionally, the processor 580 is further configured to execute the steps of any implementation manner of the task running method provided in the embodiments of the present application.

[0144] The embodiments of the present application further provide a computer-readable storage medium for storing a computer program, and the computer program is used to execute any implementation manner of a task running method described in the foregoing various embodiments.

[0145] The embodiments of the present application further provide a computer program product including instructions, and when it runs on a computer, it causes the computer to execute any implementation manner of a task running method described in the foregoing various embodiments.

[0146] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be in an electrical, mechanical or other form.

[0148] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0150] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), magnetic disks or optical discs and other various media that can store computer programs.

[0151] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A task execution method, characterized in that, The method includes: When the NodeManager starts, obtain the target mirror file; When the NodeManager starts to execute the target task, start the application container engine and mount the configuration file corresponding to the target task into the target mirror file; After the application container engine starts, execute the container running script and execute the container process corresponding to the target task based on the configuration file in the target mirror file; The method further includes: Receive a task submission request; the task submission request is generated in response to a task submission operation triggered by a user; By default, the NodeManager runs the container process corresponding to the task in the application container engine.

2. The method according to claim 1, characterized in that The method further includes: Detect the target user information corresponding to the submitter of the target task; After the application container engine starts, generate a target virtual user and the permissions corresponding to the target virtual user according to the target user information; Then, after the application container engine starts, execute the container running script and execute the container process corresponding to the target task based on the configuration file in the target mirror file, including: After the application container engine starts, execute the container running script, and execute the container process corresponding to the target task based on the configuration file in the target mirror file by the target virtual user according to its corresponding permissions.

3. The method according to claim 1 or 2, characterized in that, The configuration file includes at least one of the following: Resource configuration file of the distributed file system Hadoop, resource configuration file of the computing engine Spark, target software package file.

4. The method according to claim 1, characterized in that, When the NodeManager starts, the method further includes: Detect whether a target user account is currently logged in to the application container engine; If so, execute the operation of obtaining the target mirror file; If not, obtain the target user login information, log in to the target user account on the application container engine based on the target user login information, and after detecting successful login, execute the operation of obtaining the target mirror file.

5. The method according to claim 1 or 4, characterized in that, The operation of obtaining the target mirror file includes: Detect whether there is a mirror file configured by the target user pre-stored in the container mirror file library; If so, obtain the mirror file configured by the target user as the target mirror file; If not, obtain the default mirror file preset by the operating system as the target mirror file.

6. The method according to any one of claims 1 to 5, characterized in that, The application container engine includes any one of the following: Docker, HeroKu, and NodeJitsu.

7. A task execution device, characterized in that, The device includes: A mirror file acquisition module, configured to obtain a target mirror file when the NodeManager starts; A configuration file mounting module, configured to start the application container engine when the NodeManager starts to execute the target task and mount the configuration file corresponding to the target task into the target mirror file; A task execution module, configured to execute the container running script after the application container engine starts and execute the container process corresponding to the target task based on the configuration file in the target mirror file; The device further includes: A request receiving module, configured to receive a task submission request; the task submission request is generated in response to a task submission operation triggered by a user; The task execution module is specifically configured to: The NodeManager by default runs the container process corresponding to the task in the application container engine.

8. An electronic device, characterized in that, The electronic device includes: A processor and a memory: The memory is configured to store a computer program and transmit the computer program to the processor; The processor is configured to execute the task execution method according to any one of claims 1 to 6 based on the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is configured to store a computer program, and the computer program is used to execute the task execution method according to any one of claims 1 to 6.

Images