Method, device, equipment and medium for submitting and managing artificial intelligence tasks

Abstract

The present application discloses a submission management method, apparatus, device and medium for artificial intelligence tasks. The method includes creating an artificial intelligence task according to the task type of the artificial intelligence task; receiving each instance in the artificial intelligence task; creating each container for each instance in the artificial intelligence task; and running each instance in the corresponding container.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically, to a submission management method, apparatus, device, and medium for artificial intelligence tasks. Background Art

[0002] With the rapid growth of data, the application of heterogeneous computing technology has become a hot topic in industry and academia in recent years. The greatest advantages of heterogeneous computing are its ability to deliver higher computing efficiency, better cost-performance, and lower latency. Therefore, heterogeneous computing technology has become a key way to improve the efficiency of both humans and machines in the machine learning application development process.

[0003] However, existing heterogeneous computing cannot be run on a user's personal computer. Therefore, users need to run their tasks on machines in a remote cluster. Currently, the most common approach in the industry is to describe tasks using a computing framework such as TensorFlow or Pytorch, pre-allocate resources to users, and then submit the task to a remote cluster. However, this pre-allocated resource model does not support the rapid sharing of resources with other users, resulting in resource waste. Furthermore, when multiple users have multiple tasks, the lack of isolation between tasks can lead to incompatibility and task failure. Summary of the Invention

[0004] One purpose of the embodiments of the present disclosure is to provide a new technical solution for submitting and managing artificial intelligence tasks.

[0005] According to a first aspect of the present disclosure, a method for submitting and managing artificial intelligence tasks is provided, comprising:

[0006] Create AI tasks based on their task types;

[0007] Receive instances of artificial intelligence tasks;

[0008] Create containers for each instance of the AI ​​task; and

[0009] Run the instances in the corresponding containers.

[0010] Optionally, receiving each instance of the artificial intelligence task includes:

[0011] Create a user group for the user who submitted the artificial intelligence task through a user group creation command;

[0012] Obtaining the working nodes allocated to the user group;

[0013] Obtaining resource requirements for each instance of the artificial intelligence task;

[0014] Selecting at least one working node from the allocated working nodes according to the resource requirements of each instance;

[0015] Receive each instance of the artificial intelligence task based on the at least one working node.

[0016] Optionally, running each instance in each corresponding container includes:

[0017] Based on the working nodes allocated to the instances, the instances are run in the corresponding containers.

[0018] Optionally, the task types of the artificial intelligence tasks include artificial intelligence development tasks and artificial intelligence training tasks.

[0019] Optionally, the artificial intelligence task is an artificial intelligence development task,

[0020] The creation of an artificial intelligence task includes:

[0021] Create an artificial intelligence development task through the first task creation command.

[0022] Optionally, each instance in the artificial intelligence development task is an artificial intelligence development instance.

[0023] The method further comprises:

[0024] While each artificial intelligence development instance is running in each corresponding container, the running of the artificial intelligence development instance in any of the containers is paused through a task pause command.

[0025] Optionally, each instance in the artificial intelligence development task is an artificial intelligence development instance.

[0026] The method further comprises:

[0027] After running each AI development instance in each corresponding container, obtain the AI ​​training program;

[0028] The artificial intelligence training program is saved as an image file by submitting a command.

[0029] Optionally, the method further includes:

[0030] Saving the image file to a first set folder in a predetermined storage space;

[0031] The first setting folder is a private folder created for the user who submitted the artificial intelligence development task.

[0032] Optionally, the artificial intelligence task is an artificial intelligence training task,

[0033] The creation of an artificial intelligence task includes:

[0034] Create an AI training task using the second task creation command.

[0035] Optionally, the method further includes:

[0036] Saving the training data used to train the artificial intelligence training program to a second setting folder;

[0037] The second setting folder is shared by all users.

[0038] Optionally, each instance in the artificial intelligence training task is an artificial intelligence training instance.

[0039] Running each instance in each corresponding container includes:

[0040] Acquire the image file from the first setting folder; and

[0041] Acquire the training data from the second setting folder;

[0042] Based on the working nodes allocated to the artificial intelligence training instances, the artificial intelligence training instances are run in the corresponding containers to train the image file using the training data to obtain an artificial intelligence model.

[0043] Optionally, during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance, the method further includes:

[0044] Viewing the running status information of each artificial intelligence training instance through a first task viewing command; and / or,

[0045] Viewing the operation log information of each artificial intelligence training instance through a second task viewing command; and / or,

[0046] Use the third task viewing command to view the working nodes running the artificial intelligence training instances and the resource usage information of the artificial intelligence training instances.

[0047] Optionally, during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance, the method further includes:

[0048] Based on the resource usage information of each artificial intelligence training instance, the resources required by each artificial intelligence training instance are dynamically adjusted.

[0049] According to a second aspect of the present disclosure, there is also provided an artificial intelligence task submission management device, comprising:

[0050] A first creation module is used to create an artificial intelligence task according to the task type of the artificial intelligence task;

[0051] A receiving module, used to receive instances of artificial intelligence tasks;

[0052] A second creation module is used to create containers for each instance of the artificial intelligence task; and

[0053] The running module is used to run the instances in the corresponding containers.

[0054] Optionally, the receiving module is specifically configured to:

[0055] Create a user group for the user who submitted the artificial intelligence task through a user group creation command;

[0056] Obtaining the working nodes allocated to the user group;

[0057] Obtaining resource requirements for each instance of the artificial intelligence task;

[0058] Selecting at least one working node from the allocated working nodes according to the resource requirements of each instance;

[0059] Receive each instance of the artificial intelligence task based on the at least one working node.

[0060] Optionally, the operation module is specifically configured to:

[0061] Based on the working nodes allocated to the instances, the instances are run in the corresponding containers.

[0062] Optionally, the task types of the artificial intelligence tasks include artificial intelligence development tasks and artificial intelligence training tasks.

[0063] Optionally, the artificial intelligence task is an artificial intelligence development task, and the creation module is specifically used to:

[0064] Create an artificial intelligence development task through the first task creation command.

[0065] Optionally, each instance in the artificial intelligence development task is an artificial intelligence development instance, and the device further includes a stopping module, configured to:

[0066] While each artificial intelligence development instance is running in each corresponding container, the running of the artificial intelligence development instance in any of the containers is paused through a task pause command.

[0067] Optionally, each instance in the artificial intelligence development task is an artificial intelligence development instance, and the device further includes an acquisition module for:

[0068] After running each AI development instance in each corresponding container, obtain the AI ​​training program;

[0069] The artificial intelligence training program is saved as an image file by submitting a command.

[0070] Optionally, the device further includes a storage module, configured to:

[0071] Saving the image file to a first set folder in a predetermined storage space;

[0072] The first setting folder is a private folder created for the user who submitted the artificial intelligence development task.

[0073] Optionally, the artificial intelligence task is an artificial intelligence training task, and the creation module is specifically used to:

[0074] Create an AI training task using the second task creation command.

[0075] Optionally, the storage module is further used to:

[0076] Saving the training data used to train the artificial intelligence training program to a second setting folder;

[0077] The second setting folder is shared by all users.

[0078] Optionally, each instance in the artificial intelligence training task is an artificial intelligence training instance, and the running module is specifically used to:

[0079] Acquire the image file from the first setting folder; and

[0080] Acquire the training data from the second setting folder;

[0081] Based on the working nodes allocated to the artificial intelligence training instances, the artificial intelligence training instances are run in the corresponding containers to train the image file using the training data to obtain an artificial intelligence model.

[0082] Optionally, the apparatus further includes a viewing module, wherein, during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance, the viewing module is configured to:

[0083] Viewing the running status information of each artificial intelligence training instance through a first task viewing command; and / or,

[0084] Viewing the operation log information of each artificial intelligence training instance through a second task viewing command; and / or,

[0085] Use the third task viewing command to view the working nodes running each artificial intelligence training task and the resource usage information of each artificial intelligence training task.

[0086] Optionally, the viewing module is further configured to: during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance;

[0087] Based on the resource usage information of each artificial intelligence training instance, the resources required by each artificial intelligence training instance are dynamically adjusted.

[0088] According to a third aspect of the present disclosure, there is also provided a device comprising at least one computing device and at least one storage device, wherein the at least one storage device is used to store instructions, and the instructions are used to control the at least one computing device to execute the method according to the first aspect above; or, the device implements the device according to the second aspect above through the computing device and the storage device.

[0089] According to a fourth aspect of the present disclosure, a computer-readable storage medium is further provided, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0090] One beneficial effect of the present disclosure is that the methods, apparatuses, devices, and media of the embodiments of the present disclosure create corresponding containers for each instance in an AI task, within which different computing frameworks can be mounted, thereby providing isolation between different tasks and different instances of the same task, preventing mutual interference between different tasks and different instances of the same task. Furthermore, because instances are run within containers, users can pause the container at any time to free up resources and quickly share them with other users, avoiding resource waste caused by the task occupying resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0092] Figure 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure;

[0093] Figure 2 is a flowchart of a method for submitting and managing artificial intelligence tasks according to an embodiment of the present disclosure;

[0094] Figure 3 is a schematic diagram of a task queue according to an embodiment of the present disclosure;

[0095] Figure 4 is a principle block diagram of resource allocation according to an embodiment of the present disclosure;

[0096] Figure 5 is a principle block diagram of an artificial intelligence task submission management device according to an embodiment of the present disclosure;

[0097] Figure 6 is a principle block diagram of an electronic device according to an embodiment of the present disclosure;

[0098] Figure 7 2 is a schematic diagram of the hardware structure of an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0099] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0100] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0101] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0102] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0103] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0104] <Hardware Configuration>

[0105] The method of the embodiment of the present disclosure may be implemented by at least one electronic device, that is, the apparatus 5000 for implementing the method may be disposed on the at least one electronic device. Figure 1 The hardware structure of any electronic device is shown. Figure 1 The electronic device shown may be a portable computer, a desktop computer, a workstation, a server, etc., or any other device having a computing device such as a processor and a storage device such as a memory, and is not limited here.

[0106] like Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 is configured to execute a computer program. The computer program may be written using an instruction set such as an x86, Arm, RISC, MIPS, or SSE architecture. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface or a headphone jack. The communication device 1400 may be capable of wired or wireless communication, and may specifically include Wi-Fi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, and the like. The display device 1500 may be, for example, an LCD display or a touch screen display. The input device 1600 may include, for example, a touch screen, a keyboard, or somatosensory input. The electronic device 1000 can output voice information through the speaker 1700 and can collect voice information through the microphone 1800.

[0107] Figure 1 The electronic device shown is merely illustrative and in no way limits the present invention, its application, or use. In the embodiments of the present disclosure, the memory 1200 of the electronic device 1000 is used to store instructions that control the processor 1100 to perform the method for managing the submission of artificial intelligence tasks according to the embodiments of the present disclosure. A skilled person can design instructions based on the disclosed scheme. How instructions control the processor's operation is well known in the art and will not be described in detail here.

[0108] In one embodiment, a device is provided, including at least one computing device and at least one storage device. The at least one storage device is used to store instructions for controlling the at least one computing device to execute a method according to any embodiment of the present disclosure.

[0109] The device may include at least one Figure 1 The electronic device 1000 shown provides at least one computing device such as a processor and at least one storage device such as a memory, which is not limited herein.

[0110] <Method Example>

[0111] Figure 2 1 is a flow chart of a method for submitting and managing artificial intelligence tasks according to an embodiment of the present disclosure, which is executed by an electronic device 1000. Figure 2 As shown, the method may include the following steps S2100 to S2400:

[0112] Step S2100: Create an artificial intelligence task according to the task type of the artificial intelligence task.

[0113] A job is the unit of task execution and the smallest unit submitted by a user. Each time a user submits a task, it is considered a job. After a user submits a job, the electronic device automatically generates a job ID for the job. This job ID can be used as an index to find the job. Typically, a job includes one or more instance tasks. If a job includes multiple instance tasks, these multiple instance tasks are interconnected.

[0114] AI tasks can include AI development tasks and AI training tasks. The purpose of an AI development task is to develop an AI training program, while the purpose of an AI training task is to execute the developed AI training program to obtain an AI model. For example, an AI development task might be executed to develop an anti-money laundering training program, followed by an AI training task to train the anti-money laundering training program to obtain an anti-money laundering model. Another example might be an AI development task to develop an anti-fraud training program, followed by an AI training task to train the anti-fraud training program to obtain an anti-fraud model. In other words, this embodiment distinguishes between the two processes of AI development and AI training, providing two different types of tasks to meet user needs.

[0115] In one example, when the artificial intelligence task is an artificial intelligence development task, creating the artificial intelligence task in step S2100 may further include: creating the artificial intelligence development task through a first task creation command.

[0116] The first task creation command may be an hsctl create lab command.

[0117] It is understandable that the existing technology adopts a pre-allocated resource model, that is, when submitting an artificial intelligence task, it is necessary to accurately describe the computing resource information such as the machine node IP, number of CPU cores, GPU (Graphics Processing Unit) device number, GPU video memory and memory required for the artificial intelligence task, that is, pre-allocate computing resources for the artificial intelligence task. This computing resource is a hard limit. For example, each artificial intelligence task is allocated 1 machine and 8 GPU cards. The 1 machine and 8 GPU cards are exclusively used by the artificial intelligence task, and it does not support sharing this part of the resources with other tasks. That is, the 1 machine and 8 GPU cards cannot be shared with other tasks, resulting in a waste of resources. However, looking at the entire process, the two stages of artificial intelligence development and training have different requirements for resources. The amount of resources required in the artificial intelligence training stage is often more than the amount of resources required in the artificial intelligence development stage. Moreover, the characteristic of the artificial intelligence development stage is that the development process may take a very long time, and users are not developing around the clock. There will be multiple interruptions and pauses during the period. If the resource is pre-allocated, users cannot operate frequently to release resources and retain the development site. As a result, users often directly apply for the amount of resources that meet the artificial intelligence development and training stages for use in the development process, which results in a waste of resources.

[0118] In this example, the hsctl create lab command is used to create an AI development task. This creates a lab task, and subsequent steps will show that corresponding containers will be created for each AI development instance within the task. This allows users to pause the container containing the AI ​​development instance at any time during the AI ​​development phase to free up resources. Once the resources are freed up, other users can immediately see the released resources and use them to run their own tasks. Furthermore, resources can be restored within seconds after the container corresponding to the AI ​​development instance is restarted.

[0119] In one example, when the artificial intelligence task is an artificial intelligence training task, creating the artificial intelligence task in step S2100 may further include: creating the artificial intelligence training task through a second task creation command.

[0120] The second task creation command may be the hsctl create bash command.

[0121] After creating an AI task according to the task type, enter:

[0122] Step S2200: Receive each instance of the artificial intelligence task.

[0123] In this embodiment, according to the introduction of step S2100 above, an artificial intelligence task job often includes multiple instance tasks. For example, an artificial intelligence development task job includes multiple artificial intelligence development instance tasks; for another example, an artificial intelligence training task job includes multiple artificial intelligence training instance tasks.

[0124] In this embodiment, receiving each instance of the artificial intelligence task in step S2200 may further include the following steps S2210 to S2240:

[0125] Step S2210: Create a user group for the user who submitted the artificial intelligence task through a user group creation command.

[0126] The command to create a user group can be the hsctl queue command.

[0127] In step S2210, the hsctl queue command can be used to create a user group. Here, computing resources can be allocated according to user groups. The computing resources within a user group are dynamically allocated. At the same time, since each task instance runs in a container, it can ensure that different tasks in the same user group are isolated from each other during the allocation process.

[0128] Step S2220: Obtain the working node allocated to the user group.

[0129] In step S2220, each user group can be assigned a worker node to run each task instance, allowing users to submit tasks to the worker node assigned to that user group. For example, user group 1 can be assigned worker nodes 1, 2, and 3; user group 2 can be assigned worker nodes 4, 5, and 6, and so on. Of course, the same worker node can be configured as a worker node for different user groups.

[0130] Step S2230: Obtain resource requirements for each instance in the artificial intelligence task.

[0131] In this step S2230, taking the artificial intelligence task as an artificial intelligence training task as an example, when submitting the artificial intelligence training task, it will not only specify the python file required for the artificial intelligence training task (the python file includes code information for executing the artificial intelligence training task), but also specify the required resource requirements, that is, computing resource information, such as the number of GPUs, GPU video memory and NPU number required for the artificial intelligence training task. Here, unlike the prior art in which hardware resource information such as the number of CPU cores, memory and GPU device number needs to be configured in the TensorFlow code file when submitting a task, in this embodiment, when submitting a task using a computing framework such as TensorFlow, there is no need to specify computing resource information such as the number of CPU cores, memory and GPU device number in the TensorFlow code file. Instead, the electronic device will allocate the resources automatically during the specific operation, reducing the trouble of user preset.

[0132] Step S2240: Select at least one working node from the allocated working nodes according to the resource requirements of each instance.

[0133] In this step S2240, different working nodes may be selected from the user group where the task is located according to the resource requirements of each instance in the task, so as to submit the task to the selected working nodes.

[0134] Step S2250: Receive each instance of the artificial intelligence task based on at least one working node.

[0135] According to the above steps S2210 to S2250, computing resources are allocated according to user groups, and computing resources in each user group are dynamically allocated, thereby avoiding resource waste.

[0136] In this embodiment, queue management can also be completed through the hsctl queue command. The tasks in each user group can form a task queue. In the task queue, the tasks submitted by the users are queued in sequence. Figure 3 It is the inclusion relationship of a containerized instance in the task queue, where computing frameworks such as TensorFlow are responsible for executing computing tasks in the container environment.

[0137] After receiving each instance in the artificial intelligence task, enter:

[0138] Step S2300: Create containers for each instance in the artificial intelligence task.

[0139] In this embodiment, a corresponding container will be created for each instance in the artificial intelligence task, and the computing resources required by the user will be pre-allocated during creation, such as the number of GPUs, GPU memory, and NPUs required above. By creating this container, isolation between different tasks and instances of the same task can be achieved.

[0140] After creating the containers for each instance of the artificial intelligence task, enter:

[0141] Step S2400: Run each instance in each corresponding container.

[0142] In this embodiment, running each instance in each corresponding container in step S2400 may further include: running each instance in each corresponding container based on a working node allocated to each instance.

[0143] In one example, when the artificial intelligence task is an artificial intelligence development task, each instance in the artificial intelligence development task is an artificial intelligence development instance. After running each artificial intelligence development instance in the corresponding container, an artificial intelligence training program will be obtained. Here, the artificial intelligence training program can be saved as an image file by submitting a command.

[0144] The commit command may be the hsctl commit command.

[0145] In this example, you can use the hsctl commit command to save the obtained artificial intelligence training program as an image file.

[0146] In this example, after obtaining the above image file, the image file may also be saved in a first setting folder in a predetermined storage space.

[0147] The first setting folder mentioned above is a private folder created for the user who submitted the artificial intelligence development task. The data in the private folder can only be read and written by the user.

[0148] In this example, after the AI ​​training program is saved as an image file, a large amount of training data is often needed to train the AI ​​training program to obtain an AI model in the subsequent AI training phase. In this example, the electronic device has a data storage system. In this data storage system, the user can create their own private folder path / shared / users / . The user name can be a username, and different users can be defined through the user name. Through this private folder path, the user can save the saved image file to the corresponding private folder. At the same time, each user can only read and write data in the private folder under their own username, preventing private data leakage. Here, the user can also interact with the data storage system using the hsctl file command, for example, using the hsctl file –help command to display help documentation, using the hsctl file download SOURCE DEST command to download data in the private folder, and using the hsctl file upload SOURCE DEST command to upload data in the private folder.

[0149] The data storage system also includes a public folder path, / shared / public / , accessible to all users. "public" can be the name of the public folder. Through this public folder path, users can upload commonly used datasets, such as training data for various application scenarios. These scenarios include image processing, speech recognition, natural language processing, automatic control, intelligent question-and-answer (Q&A), business decision-making, business recommendation, search, and abnormal behavior detection. All users can read and write data in this public folder, enabling data sharing. Users can also interact with the data storage system using the hsctl file command. For example, they can use the hsctl file –help command to display help documentation, the hsctl filedownload SOURCE DEST command to download data from the public folder, and the hsctl file upload SOURCE DEST command to upload data from the public folder.

[0150] In one example, when the artificial intelligence task is an artificial intelligence training task, each instance in the artificial intelligence training task is an artificial intelligence training instance. Here, running each instance in each corresponding container in step S2400 may further include the following steps S2410 to S2430:

[0151] Step S2410: Obtain an image file from a first setting folder.

[0152] The first setting folder may be the above private folder.

[0153] In this step S2410, the image file saved by the artificial intelligence training program can be obtained from the above user's private folder.

[0154] Step S2420: Acquire training data from the second setting folder.

[0155] The second setting folder can be the public folder mentioned above, and the second setting folder is shared by all users. Here, the user will save the training data used to train the artificial intelligence training program to the second setting folder in advance, so as to obtain the training data used to train the artificial intelligence training program from the second setting folder during the training phase.

[0156] In step S2420, training data can be obtained from the public folder. The more training data, the more accurate the training results. However, after a certain amount of training data is obtained, the accuracy of the training results increases more slowly until the orientation stabilizes. The amount of training data required can be determined by balancing the accuracy of the training results with the data processing cost.

[0157] Step S2430: Based on the working nodes allocated to each artificial intelligence training instance, each artificial intelligence training instance is run in each corresponding container to train the image file using the training data to obtain an artificial intelligence model.

[0158] In this step S2430, after obtaining target features by extracting and combining features from the training data according to automatic machine learning technology, training samples can be generated in combination with the real label data corresponding to the training data, and then at least one model training algorithm can be used to train the image file based on the training samples to obtain an artificial intelligence model.

[0159] According to the method of the embodiment of the present disclosure, a corresponding container is created for each instance of an AI task. Different computing frameworks can be mounted within the container, thereby providing isolation between different tasks and different instances of the same task, preventing them from interfering with each other. Moreover, because the instance is run within the container, the user can pause the container at any time to free up resources and quickly share them with other users, avoiding resource waste caused by the task occupying resources.

[0160] In one embodiment, when the artificial intelligence task is an artificial intelligence development task, each instance in the artificial intelligence development task is an artificial intelligence development instance. Here, the submission management method of the artificial intelligence task disclosed in the present disclosure may further include:

[0161] While each artificial intelligence development instance is running in the corresponding container, the operation of the artificial intelligence development instance in any container is paused through the task pause command.

[0162] The task pause command may be the hsctl stop command.

[0163] In this embodiment, since the created artificial intelligence development tasks are usually lab tasks, during the running of each artificial intelligence development instance in the corresponding container, the hsctl stop command can be used to pause the running of the artificial intelligence development instance in any container to release resources. The released resources can be used by other users in the user group, thereby realizing resource sharing and avoiding resource waste.

[0164] In one embodiment, when the artificial intelligence task is an artificial intelligence training task, during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance in step S2430 above, the artificial intelligence task submission management method of the present disclosure may further include:

[0165] The resource usage information of each artificial intelligence training instance is viewed through the third task viewing command, and the resources required for each artificial intelligence training instance are dynamically adjusted based on the resource usage information of each artificial intelligence training instance.

[0166] The third task viewing command may be the hsctl info job_usage command.

[0167] In this embodiment, the user can use the hsctl info job_usage command to view the resource usage information of each instance in the artificial intelligence training task, such as the historical amount of resources consumed, the current amount of resources consumed, etc., and dynamically adjust the resources required for each artificial intelligence training instance to ensure that each artificial intelligence training instance can operate normally.

[0168] In this embodiment, according to the above analysis of step S2230, it is known that it is only necessary to specify the required number of GPUs, GPU memory and NPU number, and there is no need to specify computing resource information such as the number of CPU cores, memory and GPU device number in the TensorFlow code file. Instead, the electronic device will allocate it by itself during operation. For example, the specific GPU device to be run on can be allocated by the electronic device, which reduces the trouble of user preset. Here, in order to allow users to not need to specify the number of CPU cores, memory and GPU device number to reduce the trouble of user preset, all hardware resources are virtualized through the virtualization module, such as Figure 4As shown, it becomes a minimum CPU core (1CPU core), a minimum GPU (1GPU), 1MB GPU memory, 1MB of memory and other resources, and is placed in a resource pool for unified perception by upper-layer applications. When the container is orchestrated, the required resources will be declared, and the resources will be allocated in the scheduling module and the required virtualized resources will be obtained. Since for containers that require CPU and memory resources, the allocation of resources is dynamically adjusted, the locking of resources is also dynamic. Therefore, when submitting tasks, users do not need to preset the number of CPU cores and the amount of memory. Instead, they are automatically adjusted during operation, which reduces the trouble of user presets and avoids the problem of task failure due to resource presets causing resource limits to be reached during operation. For containers that require GPUs, resources are currently statically locked. Since what is obtained is no longer physical resources, users need to set the number of GPUs and the amount of GPU memory, but do not need to set the GPU device number.

[0169] It is understandable that when the user submits a task and the resources are fully scheduled, the task instances will start running. Before that, the task instances are in a waiting state. Once the resources are successfully scheduled, the task instances start running automatically without the user having to operate. This strategy can be called an all-or-nothing scheduling strategy.

[0170] According to this embodiment, the user is exempted from setting resources, and instead dynamically adjusts resource usage during the execution of each task instance to avoid resource waste.

[0171] In one embodiment, when the artificial intelligence task is an artificial intelligence training task, during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance in step S2430 above, the artificial intelligence task submission management method of the present disclosure may further include:

[0172] First, the running status information of each artificial intelligence training instance is viewed through the first task viewing command.

[0173] The first task viewing command may be the hsctl list jobs command.

[0174] In this regard, the hsctl list jobs command can be used to view the running status information of each artificial intelligence training instance in the artificial intelligence training task, such as but not limited to artificial intelligence training instances in the waiting state, artificial intelligence training instances in the running state, artificial intelligence training instances in the killed state, artificial intelligence training instances in the failed state, etc.

[0175] Secondly, the running log information of each artificial intelligence training instance is viewed through the second task viewing command.

[0176] The second task viewing command may be the hsctl log job command.

[0177] In this regard, you can use the hsctl log job command to view the running log information of each AI training instance in the AI ​​training task.

[0178] Thirdly, use the third task view command to view the working nodes running each artificial intelligence training instance and the resource usage information of each artificial intelligence training instance.

[0179] In this regard, you can use the hsctl info job_usage command to view resource usage information such as which work nodes each AI training instance in the AI ​​training task runs on, the historical resource consumption during the operation, and the current resource consumption.

[0180] It is understandable that when the amount of resources is insufficient and the artificial intelligence training instance is in a waiting state, the user can adjust the queuing information of the artificial intelligence training instance. For example, the user can use the hsctl update command to put the artificial intelligence training instance at the front of the queue, so that when there are available resources, the artificial intelligence training instance can start running.

[0181] According to this embodiment, users can monitor the available resources in the resource pool and the running status of each task instance in real time, promptly feedback task scheduling information to learn the running status of the task, and can view queue information and jump the queue.

[0182] <Example>

[0183] Next, an example of an AI task submission management method is described, taking AI tasks including AI development tasks and AI training tasks as examples. In this example, the AI ​​task submission management method may include the following steps:

[0184] Step S6010: Create an artificial intelligence development task through the hsctl create lab command, create a container for each instance of the artificial intelligence development task, specify the required computing resources, and perform the entire development and debugging process to obtain an artificial intelligence training program.

[0185] Step S6020: Save the artificial intelligence training program as a mirror file through the hsctl commit command.

[0186] In step S6020, the container can be entered through the hsctl exec command to complete the debugging process, and finally a debugged image file can be saved and generated through the hsctl commit command. Here, the resources can be released through the hsctl stop command. After the resources are released, other users in the same user group can view the released resources through the hsctl status command and then occupy these resources.

[0187] Step S6030: interact with the data storage system through the hsctl command to store the saved image file in the user's private folder.

[0188] In this step S6030, the data storage system also includes a public folder that can be shared by all users. The public folder stores training data for different application scenarios. The different application scenarios can be the application scenarios mentioned in the above embodiments, which will not be described in detail in this example.

[0189] Step S6040: Create an artificial intelligence training task through the hsctl bash command, and create each container where each instance in the artificial intelligence training task is located.

[0190] In step S6040, the user can specify the required Python file, which includes the code information for executing the AI ​​training task. Here, the user can also specify the required computing resources, such as the number of GPUs, GPU memory, and the number of NPUs. Furthermore, there is no need to specify computing resource information such as the number of CPU cores, memory, and GPU device number in the TensorFlow code file. Instead, the electronic device automatically allocates computing resources at runtime. For example, the specific GPU device to run on can be assigned by the electronic device, reducing the trouble of user presets.

[0191] Step S6050: Based on the working nodes allocated to each artificial intelligence training instance, each artificial intelligence training instance is run in the corresponding container to train the image file using the training data to obtain an artificial intelligence model.

[0192] <Device Example>

[0193] In this embodiment, a submission management device 5000 for artificial intelligence tasks is also provided. Figure 5 As shown, the submission management device 5000 for artificial intelligence tasks includes a first creation module 5100, a receiving module 5200, a second creation module 5300 and an operation module 5400, which are used to implement the submission management method for artificial intelligence tasks provided in this embodiment. The various modules of the submission management device 5000 for artificial intelligence tasks can be implemented by software or by hardware, which is not limited here.

[0194] The first creation module 5100 is used to create an artificial intelligence task according to the task type of the artificial intelligence task.

[0195] The receiving module 5200 is used to receive each instance of the artificial intelligence task.

[0196] The second creation module 5300 is used to create containers for each instance in the artificial intelligence task.

[0197] The running module 5400 is used to run the instances in the corresponding containers.

[0198] In one embodiment, the receiving module 5200 is specifically used to: create a user group for the user who submitted the artificial intelligence task through a user group creation command; obtain the working node allocated to the user group; obtain the resource requirements of each instance in the artificial intelligence task; select at least one working node from the allocated working nodes based on the resource requirements of each instance; and receive each instance in the artificial intelligence task based on the at least one working node.

[0199] In one embodiment, the running module 5400 is specifically configured to run each instance in the corresponding container based on the working node allocated to each instance.

[0200] In one embodiment, the task types of the artificial intelligence tasks include artificial intelligence development tasks and artificial intelligence training tasks.

[0201] In one embodiment, the artificial intelligence task is an artificial intelligence development task, and the creation module 5300 is specifically used to: create the artificial intelligence development task through a first task creation command.

[0202] In one embodiment, each instance in the artificial intelligence development task is an artificial intelligence development instance, and the device 5000 also includes a stop module (not shown in the figure), which is used to: during the operation of each artificial intelligence development instance in the corresponding container, suspend the operation of the artificial intelligence development instance in any of the containers through a task pause command.

[0203] In one embodiment, each instance in the artificial intelligence development task is an artificial intelligence development instance, and the device 5000 also includes an acquisition module (not shown in the figure), which is used to: obtain an artificial intelligence training program after running each artificial intelligence development instance in the corresponding container; and save the artificial intelligence training program as an image file by submitting a command.

[0204] In one embodiment, the apparatus 5000 further includes a storage module configured to save the image file to a first set folder in a predetermined storage space.

[0205] The first setting folder is a private folder created for the user who submitted the artificial intelligence development task.

[0206] In one embodiment, the artificial intelligence task is an artificial intelligence training task, and the creation module 5300 is specifically used to create the artificial intelligence training task through a second task creation command.

[0207] In one embodiment, the storage module is further used to save the training data used to train the artificial intelligence training program to a second setting folder.

[0208] The second setting folder is shared by all users.

[0209] In one embodiment, each instance in the artificial intelligence training task is an artificial intelligence training instance, and the running module 5400 is specifically used to: obtain the image file from the first setting folder; and obtain the training data from the second setting folder; based on the working nodes allocated to each artificial intelligence training instance, run each artificial intelligence training instance in the corresponding container to train the image file using the training data to obtain an artificial intelligence model.

[0210] In one embodiment, the device 5000 further includes a viewing module (not shown in the figure), which is used, during the period when each artificial intelligence training instance is running in each corresponding container based on the working node allocated to each artificial intelligence training instance, to: view the running status information of each artificial intelligence training instance through a first task viewing command; and / or, view the running log information of each artificial intelligence training instance through a second task viewing command; and / or, view the working node running each artificial intelligence training task and the resource usage information of each artificial intelligence training task through a third task viewing command.

[0211] In one embodiment, the viewing module is further used to dynamically adjust the resources required for each artificial intelligence training instance based on the resource usage information of each artificial intelligence training instance during the operation of each artificial intelligence training instance in the corresponding container based on the working node allocated to each artificial intelligence training instance.

[0212] <Equipment Example>

[0213] Corresponding to the above method embodiment, in this embodiment, an electronic device is also provided, such as Figure 6 As shown, it may include an artificial intelligence task submission management device 5000 according to any embodiment of the present disclosure, which is used to implement the artificial intelligence task submission management method of any embodiment of the present disclosure.

[0214] like Figure 7As shown, the electronic device 6000 may also include a processor 6200 and a memory 6100, wherein the memory 6100 is used to store executable instructions; the processor 6200 is used to operate the electronic device according to the control of the instructions to execute the submission management method of the artificial intelligence task according to any embodiment of the present disclosure.

[0215] The various modules of the above device 5000 can be implemented by the processor 6200 running the instructions to execute the method according to any embodiment of the present disclosure.

[0216] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0217] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0218] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0219] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0220] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0221] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0222] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0223] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0224] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for submitting and managing artificial intelligence tasks, comprising: Create an artificial intelligence task according to the task type of the artificial intelligence task, wherein, when the artificial intelligence task is an artificial intelligence development task, create an artificial intelligence development task; when the artificial intelligence task is an artificial intelligence training task, create an artificial intelligence training task; Receive each instance of the artificial intelligence task, wherein, when the artificial intelligence task is an artificial intelligence development task, receive multiple instances of the artificial intelligence development task; when the artificial intelligence task is an artificial intelligence training task, receive multiple instances of the artificial intelligence training task; Creating containers for each instance in the artificial intelligence task, respectively. When the artificial intelligence task is an artificial intelligence development task, creating containers for each instance in the artificial intelligence development task; when the artificial intelligence task is a training task, receiving containers for each instance in the artificial intelligence training task; and Run each instance in the corresponding container, wherein, when the artificial intelligence task is an artificial intelligence development task, the operation of the artificial intelligence development instance in any container is suspended in response to a task pause command; when the artificial intelligence task is an artificial intelligence training task, the resource usage information of the artificial intelligence training instance is viewed in response to a view command, and the resources required for each artificial intelligence training instance are dynamically adjusted according to the resource usage information, wherein the resource usage information includes historical resource consumption and currently consumed resource consumption.

2. The method according to claim 1, wherein The receiving of each instance of the artificial intelligence task includes: Create a user group for the user who submitted the artificial intelligence task through a user group creation command; Obtaining the working nodes allocated to the user group; Obtaining resource requirements for each instance of the artificial intelligence task; Selecting at least one working node from the allocated working nodes according to the resource requirements of each instance; Receive each instance of the artificial intelligence task based on the at least one working node.

3. The method according to claim 2, wherein: Running each instance in each corresponding container includes: Based on the working nodes allocated to the instances, the instances are run in the corresponding containers.

4. The method according to claim 1, wherein The artificial intelligence task is an artificial intelligence development task, The creation of an artificial intelligence task includes: Create an artificial intelligence development task through the first task creation command.

5. The method according to claim 4, wherein Each instance in the artificial intelligence development task is an artificial intelligence development instance. The method further comprises: After running each AI development instance in each corresponding container, obtain the AI ​​training program; The artificial intelligence training program is saved as an image file by submitting a command.

6. The method according to claim 5, wherein: The method further comprises: Saving the image file to a first set folder in a predetermined storage space; The first setting folder is a private folder created for the user who submitted the artificial intelligence development task.

7. The method according to claim 6, wherein: The artificial intelligence task is an artificial intelligence training task, The creation of an artificial intelligence task includes: Create an AI training task using the second task creation command.

8. The method according to claim 7, wherein: The method further comprises: Saving the training data used to train the artificial intelligence training program to a second setting folder; The second setting folder is shared by all users.

9. The method according to claim 8, wherein Each instance in the artificial intelligence training task is an artificial intelligence training instance. Running each instance in each corresponding container includes: Acquire the image file from the first setting folder; and Acquire the training data from the second setting folder; Based on the working nodes allocated to the artificial intelligence training instances, the artificial intelligence training instances are run in the corresponding containers to train the image file using the training data to obtain an artificial intelligence model.

10. The method according to claim 9, wherein: During the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance, the method further includes: Viewing the running status information of each artificial intelligence training instance through a first task viewing command; and / or, Viewing the operation log information of each artificial intelligence training instance through a second task viewing command; and / or, Use the third task viewing command to view the working nodes running the artificial intelligence training instances and the resource usage information of the artificial intelligence training instances.

11. The method according to claim 10, wherein: During the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance, the method further includes: Based on the resource usage information of each artificial intelligence training instance, the resources required by each artificial intelligence training instance are dynamically adjusted.

12. A submission management device for artificial intelligence tasks, comprising: A first creation module is configured to create an artificial intelligence task according to the task type of the artificial intelligence task, wherein when the artificial intelligence task is an artificial intelligence development task, an artificial intelligence development task is created; when the artificial intelligence task is an artificial intelligence training task, an artificial intelligence training task is created; a receiving module, configured to receive instances of an artificial intelligence task, wherein, when the artificial intelligence task is an artificial intelligence development task, the receiving module receives multiple instances of the artificial intelligence development task; and when the artificial intelligence task is an artificial intelligence training task, the receiving module receives multiple instances of the artificial intelligence training task; A second creation module is configured to respectively create containers for each instance in the artificial intelligence task, wherein, when the artificial intelligence task is an artificial intelligence development task, the container for each instance in the artificial intelligence development task is created; when the artificial intelligence task is a training task, the container for each instance in the artificial intelligence training task is received; and, A running module is used to run the instances in the corresponding containers, wherein, when the artificial intelligence task is an artificial intelligence development task, the running of the artificial intelligence development instance in any container is suspended in response to a task pause command; when the artificial intelligence task is an artificial intelligence training task, the resource usage information of the artificial intelligence training instance is viewed in response to a view command, and the resources required for each artificial intelligence training instance are dynamically adjusted according to the resource usage information, wherein the resource usage information includes historical resource consumption and currently consumed resource consumption.

13. The device according to claim 12, wherein The receiving module is specifically configured to: Create a user group for the user who submitted the artificial intelligence task through a user group creation command; Obtaining the working nodes allocated to the user group; Obtaining resource requirements for each instance of the artificial intelligence task; Selecting at least one working node from the allocated working nodes according to the resource requirements of each instance; Receive each instance of the artificial intelligence task based on the at least one working node.

14. The device according to claim 13, wherein The operation module is specifically used to: Based on the working nodes allocated to the instances, the instances are run in the corresponding containers.

15. The device according to claim 12, wherein The artificial intelligence task is an artificial intelligence development task, and the creation module is specifically used to: Create an artificial intelligence development task through the first task creation command.

16. The device according to claim 15, wherein Each instance in the artificial intelligence development task is an artificial intelligence development instance. The device further includes an acquisition module for: After running each AI development instance in each corresponding container, obtain the AI ​​training program; The artificial intelligence training program is saved as an image file by submitting a command.

17. The device according to claim 16, wherein The device further includes a storage module, configured to: Saving the image file to a first set folder in a predetermined storage space; The first setting folder is a private folder created for the user who submitted the artificial intelligence development task.

18. The device according to claim 17, wherein The artificial intelligence task is an artificial intelligence training task, and the creation module is specifically used to: Create an AI training task using the second task creation command.

19. The device according to claim 18, wherein The storage module is further used for: Saving the training data used to train the artificial intelligence training program to a second setting folder; The second setting folder is shared by all users.

20. The device according to claim 19, wherein Each instance in the artificial intelligence training task is an artificial intelligence training instance, and the running module is specifically used to: Acquire the image file from the first setting folder; and Acquire the training data from the second setting folder; Based on the working nodes allocated to the artificial intelligence training instances, the artificial intelligence training instances are run in the corresponding containers to train the image file using the training data to obtain an artificial intelligence model.

21. The device according to claim 20, wherein The apparatus further includes a viewing module, which is configured to, during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance: View the running status information of each artificial intelligence training instance through the first task viewing command; and / or, View the running log information of each artificial intelligence training instance through the second task viewing command; and / or, Use the third task viewing command to view the working nodes running each artificial intelligence training task and the resource usage information of each artificial intelligence training task.

22. The device according to claim 21, wherein The viewing module is further configured to: during the execution of each artificial intelligence training instance in each corresponding container based on the work node allocated to each artificial intelligence training instance; Based on the resource usage information of each artificial intelligence training instance, the resources required by each artificial intelligence training instance are dynamically adjusted.

23. An apparatus comprising at least one computing device and at least one storage device, wherein: The at least one storage device is used to store instructions, and the instructions are used to control the at least one computing device to execute the method according to any one of claims 1 to 11; or, the device implements the device according to any one of claims 12 to 22 through the computing device and the storage device.

24. A computer-readable storage medium, wherein: A computer program is stored thereon, which implements the method according to any one of claims 1 to 11 when executed by a processor.

Images