Task information pushing method and device, storage medium and electronic device

Abstract

The application discloses a task information pushing method and device, a storage medium and an electronic device, and relates to the technical field of information pushing, in particular to a task information pushing method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining a plurality of to-be-trained tasks and task information of the to-be-trained tasks created by a cloud platform; obtaining running states corresponding to the plurality of to-be-trained tasks respectively under the condition that the plurality of to-be-trained tasks are successfully scheduled by a container cloud according to the task information, and pushing the task information corresponding to the plurality of to-be-trained tasks respectively into message queues corresponding to the running states of the plurality of to-be-trained tasks. The method solves the problem that, in the prior art, the task information of the to-be-trained tasks cannot be managed according to the running states of the tasks.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more specifically, to a method and apparatus for pushing task information, a storage medium, and an electronic device. Background Technology

[0002] With the continuous enhancement of computing power in hardware for machine learning (ML) and the further development of open-source machine learning frameworks, the scope of deep learning applications has expanded dramatically, and the demand has surged.

[0003] In existing technologies, algorithm training task scheduling can be achieved through methods such as comparing the task's execution time with the current time to determine whether to schedule execution; or through distributed high-concurrency scheduling systems based on decentralized job tasks, where data is sent to a designated distributed message queue for other nodes to discover and schedule tasks. However, the task information obtained from the above scheduling process does not consider that the task to be scheduled may be in different states, such as whether the training environment is ready or whether training has already ended.

[0004] In terms of related technologies, existing technologies have not yet proposed effective solutions to problems such as the inability to manage task information based on the task's running status for the task to be trained. Summary of the Invention

[0005] This invention provides a method and apparatus for pushing task information, a storage medium, and an electronic device, to at least solve the problems in the prior art, such as the inability to manage task information based on the running status of the task to be trained.

[0006] According to one aspect of the present invention, a method for pushing task information is provided, comprising: acquiring a plurality of tasks to be trained created by a cloud platform and task information of the tasks to be trained; when a container cloud successfully schedules the plurality of tasks to be trained according to the task information, acquiring the running status corresponding to each of the plurality of tasks to be trained, and pushing the task information corresponding to the plurality of tasks to message queues corresponding to the running status of the plurality of tasks to be trained.

[0007] In an exemplary embodiment, pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks includes: determining a first training task with an initialization status from the plurality of training tasks, wherein the initialization status is used to indicate that the training container for training the first training task has not been created; obtaining the task information of the first training task and pushing the task information of the first training task to the initialization task message queue.

[0008] In one exemplary embodiment, pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks includes: determining a second training task with a running status from the plurality of training tasks, wherein the running status is used to indicate that a training container for training the second training task has been created; obtaining the task information of the second training task and pushing the task information of the second training task to the running task message queue.

[0009] In one exemplary embodiment, after pushing the task information of the second training task to the running task message queue, the method further includes: obtaining the training startup script from the task information of the second training task; and training the second training task in the training container of the second training task according to the training startup script.

[0010] In one exemplary embodiment, pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks includes: determining a third training task whose running status is terminated from the plurality of training tasks; obtaining the task information of the third training task; and pushing the task information of the third training task to the terminated task message queue.

[0011] In an exemplary embodiment, after pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks, the method further includes: if the message queue includes at least one of the following: an initialization task message queue, a running task message queue, and an end-of-task message queue, determining a first training task corresponding to the task information in the initialization task message queue, a second training task corresponding to the task information in the running task message queue, and a third training task corresponding to the task information in the end-of-task message queue; and determining the running status of the plurality of training tasks based on the first training task, the second training task, and the third training task.

[0012] In an exemplary embodiment, after obtaining multiple training tasks created by the cloud platform and the task information of the training tasks, the method further includes: obtaining the target resources required for training the training tasks based on the task information; comparing the target resources with the remaining resources of the server resource group corresponding to the training tasks; and determining that the training tasks pass the first resource verification if the remaining resources are greater than the target resources.

[0013] According to another aspect of the present invention, a task information push device is also provided, comprising: an acquisition module, configured to acquire a plurality of tasks to be trained created by a cloud platform and task information of the tasks to be trained; and a push module, configured to, when the container cloud successfully schedules the plurality of tasks to be trained according to the task information, acquire the running status corresponding to each of the plurality of tasks to be trained, and push the task information corresponding to the plurality of tasks to be trained to message queues corresponding to the running status of the plurality of tasks to be trained.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes the above-described task information push method when it runs.

[0015] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, characterized in that the memory stores a computer program, and the processor runs the above-mentioned task information push method through the computer program.

[0016] In this embodiment of the invention, multiple training tasks created by a cloud platform and their task information are obtained. If the container cloud successfully schedules the multiple training tasks based on the task information, the running status of each of the multiple training tasks is obtained, and the task information corresponding to each of the multiple training tasks is pushed to a message queue corresponding to the running status of the multiple training tasks. In other words, after multiple training tasks are successfully scheduled by the container cloud, the task information of the training tasks is pushed to the message queue corresponding to the running status of the training tasks. This solves the problem in related technologies and existing technologies where the task information of training tasks cannot be managed according to the running status of the tasks. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a cloud platform for a task information push method according to an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of an optional task information push method according to an embodiment of the present invention;

[0020] Figure 3 This is a task processing flowchart of an optional task information push method according to an embodiment of the present invention;

[0021] Figure 4 This is a task status transition diagram of an optional task information push method according to an embodiment of the present invention;

[0022] Figure 5 This is a structural block diagram of a task information push device according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The method embodiments provided in this invention can run on a cloud platform. Taking running on a cloud platform as an example, Figure 1 This is a hardware structure block diagram of a cloud platform for a task information push method according to an embodiment of the present invention. Figure 1 As shown, a cloud platform may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. In one exemplary embodiment, the cloud platform may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned cloud platform. For example, the camera device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the task information push method in this embodiment of the invention. The processor 102 runs various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a secure text network via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the camera equipment's communication provider. In one example, the transmission system 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet.

[0028] This embodiment provides a method for pushing task information, including but not limited to methods applied to the aforementioned cloud platform. Figure 2 This is a flowchart of an optional task information push method according to an embodiment of the present invention, the process including the following steps:

[0029] Step S202: Obtain multiple training tasks created by the cloud platform and the task information of the training tasks;

[0030] Step S204: If the container cloud successfully schedules the multiple training tasks according to the task information, obtain the running status of each of the multiple training tasks, and push the task information corresponding to each of the multiple training tasks to the message queue corresponding to the running status of the multiple training tasks.

[0031] Through the above steps, multiple training tasks created by the cloud platform and their task information are obtained. If the container cloud successfully schedules the multiple training tasks based on the task information, the running status of each training task is obtained, and the task information corresponding to each training task is pushed to the message queue corresponding to its running status. In other words, after multiple training tasks are successfully scheduled by the container cloud, the task information of the training tasks is pushed to the message queue corresponding to their running status. This solves the problem in existing technologies where task information for training tasks cannot be managed based on the task's running status.

[0032] Optionally, pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks includes: determining a first training task with an initialization status from the plurality of training tasks, wherein the initialization status is used to indicate that the training container for training the first training task has not been created; obtaining the task information of the first training task and pushing the task information of the first training task to the initialization task message queue.

[0033] Understandably, after successfully scheduling multiple training tasks created on the cloud platform to the container cloud, the container cloud creates a unique task ID for each training task. Simultaneously, these tasks enter an initialization state, and the task information for multiple first training tasks is pushed to the initialization task message queue. Specifically: 1) In the container cloud, the initialization state is the process of creating the training environment and preparing training resources for the training tasks. This includes downloading the necessary datasets and downloading the training code from the code repository to the directory mounted on the container. These tasks are handled by the container cloud's initialization container for the same container runtime environment created for this training task. At this point, the training container used to indicate that the first training task has not yet been created. 2) The initialization task message queue receives task information for all successfully scheduled first training tasks whose training containers have not yet been created. The initialization processing node continuously iterates through the initialization task message queue.

[0034] Optionally, pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks includes: determining a second training task with a running status from the plurality of training tasks, wherein the running status is used to indicate that the training container for training the second training task has been created; obtaining the task information of the second training task and pushing the task information of the second training task to the running task message queue.

[0035] It should be noted that the "running" status indicates that the training container for training the second training task has been created. Therefore, the source of the second training task in the "running" status is as follows: the initialization processing node continuously traverses the initialization task message queue, that is, it continuously queries the container cloud for the status of the first training task based on its task ID. When the status of the first training task changes to "running," indicating that the training container has been created, the task information of the first training task is pushed to the "running task message queue." It should also be noted that if the task status of the task to be trained has not changed during the traversal of the initialization task message queue and / or the running task message queue, no task information is pushed, and the process waits for the next traversal.

[0036] Optionally, after pushing the task information of the second training task to the running task message queue, the method further includes: obtaining the training startup script from the task information of the second training task; and training the second training task in the training container of the second training task according to the training startup script.

[0037] Understandably, once the training container is created in the container cloud, training for the second training task can begin. However, it's important to note that the training startup script within the task information of the second training task needs to be determined first, and then training for the second training task can be initiated within the training container based on the training startup script and the training code.

[0038] Optionally, pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks includes: determining a third training task whose running status is terminated from the plurality of training tasks; obtaining the task information of the third training task; and pushing the task information of the third training task to the terminated task message queue.

[0039] Specifically, the second training task in the running state is the process of running training and generating a model according to the startup script of the training code. Once the second training task has finished training and the model has been generated, its task status will change to "completed." When the task status is found to be completed based on the task ID, the task information is pushed to the completed task message queue. The completed task processing node consumes the completed task message queue, and then schedules the container cloud to delete the corresponding third training task, while simultaneously recording the resource usage of the third training task and releasing its resources.

[0040] Optionally, after pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks, the method further includes: if the message queue includes at least one of the following: an initialization task message queue, a running task message queue, and a terminated task message queue, determining a first training task corresponding to the task information in the initialization task message queue, a second training task corresponding to the task information in the running task message queue, and a third training task corresponding to the task information in the terminated task message queue; and determining the running status of the plurality of training tasks based on the first training task, the second training task, and the third training task.

[0041] Understandably, by analyzing the task information in the initialization task message queue, the running task message queue, and the terminated task message queue, one can clearly determine the different task states of the training tasks corresponding to the task information in different message queues, and thus determine the running status of multiple training tasks. For example, if a training task is assigned task ID 001 after successful scheduling, and task information 001 is found in the running task message queue, then it can be known that training task 001, i.e., the second training task 001, is being trained by the training container and generating a model.

[0042] After obtaining multiple training tasks created by the cloud platform and the task information of the training tasks, it is necessary to perform resource verification on the task information, specifically including a first resource verification and a second resource verification. 1) The first resource verification includes: obtaining the target resources required for training the training task based on the task information; comparing the target resources with the remaining resources of the server resource group corresponding to the training task; and determining that the training task passes the first resource verification if the remaining resources are greater than the target resources.

[0043] Optionally, obtaining the target resources required for training the task to be trained based on the task information includes: obtaining user group quota information from the task information, wherein the user group quota information is used to indicate the resources required for training the task to be trained; and obtaining the target resources based on the user group quota information.

[0044] It's important to note that the server resource group corresponding to the training task includes the server cluster where the deployed devices are also added to the container cloud for unified management. The container cloud then divides and abstracts these resource groups based on the ML hardware. Servers within the same resource group possess the same ML hardware. For example, in the first resource verification, when training task 001 is running, after obtaining the task information for task 001, the user group quota information is read. This allows the determination of the resources required for training task 001. For instance, if the required resources account for 20% of the resource group containing task 001, then the first resource verification passes if the remaining resources in the resource group containing task 001 are greater than 20%; otherwise, it fails.

[0045] 2) Before or after the task to be trained passes the first resource verification, the method further includes: obtaining the number of servers required for training the task to be trained based on the task information; comparing the number of servers with the number of available servers in the server resource group corresponding to the task to be trained; and if the number of available servers is greater than the number of servers, determining that the task to be trained passes the second resource verification.

[0046] Optionally, obtaining the number of servers required for training the task to be trained based on the task information includes: obtaining server number information from the task information, wherein the server number information is used to indicate the number of servers required for training the task to be trained; and determining the server number information as the number of servers required for training the task to be trained.

[0047] Understandably, similar to the first resource check, taking training task 001 as an example, if training task 001 passes the first resource check, the number of servers required by training task 001 can be obtained through the task information of training task 001, for example, 3. If the number of remaining servers in the server resource group where training task 001 is located is greater than 3, then training task 001 passes the second resource check.

[0048] It should be noted that during the resource verification process, the task may not meet the requirements of the first resource verification and / or the second resource verification. In this case, the task information will be added to the waiting task dictionary and waiting reason dictionary in the in-memory database, and the task will wait for the next resource verification.

[0049] After the training task passes the resource verification, the task information is encapsulated into a target template that the container cloud of the cloud platform allows to process. This includes: if the training task passes the first resource verification and the second resource verification, determining whether there is a training startup script in the task information, wherein the training startup script is used to start a process for training the training task; if the training startup script exists, encapsulating the task information into a target template that the container cloud of the cloud platform allows to process.

[0050] Specifically, a compliance check is performed on the task information of the training tasks that pass resource verification. This involves checking whether the task information contains a training startup script used to launch the process for training the training task. If the training startup script exists, the task information is encapsulated into a target template that can be processed by the container cloud, i.e., a Kubernetes standard template. Conversely, if the compliance check fails, the push of the task information is terminated directly, or the task information can be pushed to the task termination message queue.

[0051] Optionally, obtaining the running status of the task to be trained includes: obtaining the task identifier of the task to be trained generated by the container cloud according to the target template; and querying the running status in the container cloud according to the task identifier.

[0052] Understandably, after the task identifier is sent to the container cloud as the target template, the container cloud generates a unique task ID for each task to be trained. Based on this task ID, the running status of the task can be queried, including the initialization status, the running status, and the end status.

[0053] Optionally, the different message queues include at least one of the following: priority queue, task message queue to be scheduled, task message queue to be initialized, task message queue to be running, and task message queue to be terminated.

[0054] Specifically, as shown in the attached document Figure 4 As shown, the different message queues include: priority queue, pending task message queue, initialization task message queue, running task message queue, and completed task message queue. The corresponding processing nodes are, in order: resource verification node, task scheduling processing node, task initialization processing node, running task processing node, and completed task processing node.

[0055] Among them, the resource verification processing node performs resource verification on the priority queue of each resource group (i.e., Figure 4 The resource verification processing nodes consume resources from the priority queues of each resource group. Training tasks that fail resource verification wait for the next verification, while those that pass verification enter the scheduled task message queue. The task scheduling processing nodes consume the scheduled message queue, first performing a compliance check on the task information. If the check passes, the task can be scheduled. Successful scheduling allows the task to enter the initialization task message queue; otherwise, it enters the end task message queue. The task initialization processing nodes consume the initialization task message queue, prepare the training tasks, and query the task status. Tasks in a failed or completed state enter the end task message queue, while those in a running state enter the running task message queue. The running task processing nodes consume the running task message queue, train the tasks, and query the task status. Tasks in a completed or failed state enter the end task message queue. The end task processing nodes consume the end task message queue and retrieve the failure reason from the container cloud using the task ID. If the reason exists, it is recorded in the relational database. Then, the container cloud is scheduled to delete the corresponding task. Simultaneously, the task's status and resource usage are recorded in the relational database. Finally, delete the resource group usage information corresponding to this task from the in-memory database.

[0056] It is important to note that the task initialization node and the task running node process the successfully scheduled tasks to be trained. While querying the task status from the container cloud, the task initialization node and the task running node also update the status of the tasks to be trained from the container cloud after processing them.

[0057] To better understand the implementation of the above task information push method, in an optional embodiment, a scheme is also provided to explain the above scheme.

[0058] An optional embodiment of the present invention provides a method for pushing task information. Figure 3 This is a task processing flowchart of an optional task information push method according to an embodiment of the present invention. The method includes the following steps:

[0059] Step 1: Use a cloud-based deep learning platform;

[0060] When algorithm developers create new algorithm tasks (equivalent to the training tasks in the above examples) on the cloud deep learning platform, they can select resource group clusters and training framework images, and choose the required dataset through cloud storage. Training files can be imported via cloud storage or a code repository. This task information is ultimately used by the cloud deep learning platform's training scheduling solution to select and run server nodes. Each process of the task is handled specifically by state processing nodes within the training scheduling solution, achieving pipelined task scheduling and execution.

[0061] Step 2: Submit the task;

[0062] Step 3: Task information check and resource verification;

[0063] It should be noted that: 1) Regarding resource verification:

[0064] Each resource group has a priority queue in its in-memory database, which stores the task information that algorithm engineers choose to submit training tasks (equivalent to the training tasks in the above embodiment) to that resource group. The resource verification processing node consumes the priority queues of each resource group.

[0065] Details are as attached Figure 4 As shown, the process of the resource verification processing node consuming the priority queue is as follows: It retrieves the priority queues of all resource groups in the memory database, traverses the priority queues, and obtains all uploaded algorithm training task information in each priority queue. This information undergoes an initial check, which compares the used and required resources against the maximum resources of the resource group to see if the resource requirements are met. Then, a final check is performed, comparing the number of servers available in the resource group with the available resources on the servers to determine if the number of servers required for the algorithm training task is sufficient.

[0066] During the check process, the usage status of each resource group and the usage status of each service on the resource group are saved to an in-memory database for caching. During resource verification, tasks may not meet resource quota requirements. Tasks in this situation will have their relevant information added to the waiting task dictionary and waiting reason dictionary in the in-memory database, awaiting further consumption. Once a task has undergone a series of resource checks, it can be scheduled. The task information will be pushed to the scheduling queue in the message queue, and the task information will be deleted from the priority queue, waiting task dictionary, and waiting reason dictionary in the in-memory database. In short, the resource verification processing node is a process of iterating through all priority queues and then iterating through all tasks in the priority queues.

[0067] 2) Regarding task information check:

[0068] Each resource group has a message queue for tasks to be scheduled. These messages are pushed by the resource verification processing node, which sends information about tasks that have passed the check.

[0069] like Figure 4 As shown, the scheduling processing node consumes the message queue of tasks to be scheduled. Specifically, to ensure the reasonable and orderly scheduling of a large number of algorithm tasks, the scheduling processing node performs a depth-first traversal of the message queues of tasks to be scheduled for each resource group. First, it performs a compliance check on the task information of the submitted algorithm tasks, that is, whether the necessary training startup scripts required by the platform exist. Then, it encapsulates the task information into a task template that the container cloud can process, and finally hands it over to the container cloud for actual task scheduling.

[0070] Step 4: Does it meet the requirements?

[0071] If the task information check and resource verification meet the requirements, proceed to step 5;

[0072] If the requirements are not met, there are two scenarios: 1) Scenario 1: Resource verification fails, proceed to step 3 and re-verify the resources; 2) Scenario 2: Task information check fails, proceed to step 9 and the task ends.

[0073] Step 5: Task scheduling;

[0074] It should be noted that during task scheduling, the resources used by this information will be recorded in the resource group usage dictionary of the in-memory database.

[0075] Step 6: Check if the scheduling to the container cloud (equivalent to the container cloud in the above embodiment) was successful;

[0076] If scheduling is successful, proceed to step 7. Optionally, if scheduling fails, proceed to step 9.

[0077] As shown in the appendix Figure 4 As shown. Successful scheduling will push the task information to the initialization task message queue. Tasks that fail to be scheduled during the process can be placed in the termination task message queue.

[0078] Furthermore, a task in the container cloud has three states from successful scheduling by the container cloud to final execution: initialization, running, and completed. In the initialization state, a task in the container cloud is in the process of creating the environment and preparing resources. During this stage, some preliminary work can be done for the container corresponding to the training task, including downloading the required dataset and downloading the training code from the code repository to the directory mounted on the container. This preliminary work is completed by the container cloud's initialization container, which creates the same container runtime environment for this training task. During the download and execution of these preliminary tasks, the task's state in the container cloud remains initialization. Once all the preliminary preparations are completed, the task's state changes from initialization to running. In the running state, a task in the container cloud is a training container undergoing training. When the training runs normally and finishes, its state in the container cloud changes to completed. The main difference between the initialization and running states is that in the initialization state, the corresponding training container has not yet been created and is waiting for the preliminary environment processing to be completed before it can be created and run. In the container cloud, a completed task is a training task that has finished running. The corresponding training container has finished running and stopped. It is waiting for the platform's end-of-training node to consume and record the relevant training information. After that, the end-of-training node will delete the task information from the container cloud.

[0079] Step 7: Task initialization status check;

[0080] The initialization of the processing node involves two threads to handle the successfully scheduled tasks within the processing node. For example... Figure 4 As shown, a thread first continuously consumes the initialization task message queue. The messages in the initialization task message queue are pushed by the scheduling node from successfully scheduled task information. These messages are then synchronously saved to a non-relational database and an in-memory database. Saving to the non-relational database is for persistent storage, while storing them in the in-memory database is for caching optimization during the polling iteration of another thread. This other thread polls and iterates through the information just synchronized to the in-memory database. Based on the globally unique task ID generated after each task is created by the container cloud, the thread queries the container cloud for the corresponding task's status. If the status is "running," it is placed in the running task message queue. If the status is "completed," it is placed in the completed task message queue. Tasks whose status changes are deleted from the initialization task record in the in-memory database. Simultaneously, the latest task status is synchronized back to the non-relational database. If the status is still "initialized," the initialization processing node does not perform any additional operations; its task information remains in the initialization task record in the in-memory database, awaiting consumption during the next iteration of the initialization processing node's cycle.

[0081] Step 8: Check the status of the task during operation;

[0082] The running processing node consumes the running task message queue. The process is similar to the initialization process and is handled by two threads. One thread continuously consumes the running task consumption queue. The messages in the running task consumption queue are pushed from the initialization processing node after successful initialization. Messages are synchronously saved to a non-relational database and an in-memory database. Saving to the non-relational database is for persistent storage, while storing them in the in-memory database is for caching optimization for the polling iteration of the other thread. The other thread polls and iterates through the information just synchronized to the in-memory database. Based on the globally unique task ID generated after each task is created by the container cloud, the node queries the container cloud for the relevant status of the corresponding task. If the status is "completed," it is added to the task completion consumption queue, and the corresponding record is deleted from the in-memory database. Simultaneously, the latest task status is synchronized back to the non-relational database. If the status is "running," the running processing node does not perform any additional operations; its task information remains in the running task record in the in-memory database, waiting for the next iteration of the running processing node to continue consumption.

[0083] However, it's important to note that the initial state of algorithm training primarily involves downloading the relevant dataset from cloud storage, downloading the image of the selected training framework from the image repository, and pulling the training code from the code repository. The running state of algorithm training is the process of running the training and generating the model according to the startup script of the training code.

[0084] Step 9: Task complete.

[0085] like Figure 4 As shown, the end-processing node consumes the task completion message queue and retrieves the task failure reason from the container cloud using the task ID. If the reason exists, it records it in the relational database. Next, it schedules the container cloud to delete the corresponding task. Simultaneously, it records the task's status and resource usage in the relational database. Finally, it deletes the resource group usage information corresponding to the task from the in-memory database.

[0086] To address the technical issues, 1) this embodiment utilizes cloud storage, message queues, relational / non-relational databases, and other technologies. Figure 4The system consists of multiple task status processing nodes. Algorithm engineers can uniformly store the training files they want to upload in cloud storage. Uploaded file data will be uniformly stored on the storage device, and various information about the uploaded files will be saved in a non-relational database. The uploaded files can be retrieved using the information in the database. 2) The server clusters of deployed devices are also added to the container cloud for unified management. Resource groups are defined and abstracted based on ML hardware. Servers in the same resource group have the same ML hardware. Each server interacts with cloud storage through volume mounting. Based on the container cloud, the GPU usage of resources on the server can be obtained in real time. 3) As attached... Figure 4 As shown, each task status processing node uses a single process to consume its corresponding message queue. Simultaneously, data synchronization is performed, and messages are saved to an in-memory database for caching and facilitating subsequent retry processing. A persistence strategy for the in-memory database is enabled to ensure no task data is lost. Furthermore, task status can be subdivided into resource verification, pending scheduling, initialization, running, and terminated. Each status processing node uses a message queue to implement the flow of task status, achieving pipelined task processing.

[0087] Furthermore, this embodiment has the following advantages: 1) By effectively dividing the algorithm training task into states and employing appropriate processing methods for each state, the training task can be efficiently and rationally scheduled on a large-scale computing server cluster; 2) By dividing the computing server cluster into hardware resource groups and managing the cluster using containerization, elastic scheduling and deployment services based on virtual machines or containers can be achieved, and a unified data resource pool can be provided; 3) By synchronously saving the data in the consumption queue to an in-memory database, and saving commonly used fields in the task scheduling process such as resource usage and user quotas to the in-memory database, effective scheduling processing of multiple training tasks can be achieved. Simultaneously, at the resource check processing node, tasks that fail the check are re-queued, ensuring the maximum possible scheduling of training tasks.

[0088] Furthermore, 1) This embodiment improves the overall resource utilization of the server cluster by rationally dividing the algorithm training task into states and designing a pipelined processing flow for the task. This ensures that user-submitted algorithm training tasks are queued reasonably within the cloud deep learning platform. 2) This embodiment improves the speed at which the training scheduling system processes tasks and reduces task queuing time by rationally designing the consumption queue and using it in conjunction with the database.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to run the methods of the various embodiments of the present invention.

[0090] Figure 5 This is a structural block diagram of a task information push device according to an embodiment of the present invention; as shown below. Figure 5 As shown, it includes:

[0091] The acquisition module 50 is used to acquire multiple training tasks created by the cloud platform and the task information of the training tasks;

[0092] The push module 52 is used to obtain the running status of each of the multiple tasks to be trained when the container cloud successfully schedules the multiple tasks to be trained according to the task information, and push the task information corresponding to the multiple tasks to be trained to the message queue corresponding to the running status of the multiple tasks to be trained.

[0093] The aforementioned device acquires multiple training tasks created by the cloud platform and their task information. If the container cloud successfully schedules the multiple training tasks based on the task information, it acquires the running status of each training task and pushes the task information corresponding to each training task to a message queue corresponding to its running status. In other words, after multiple training tasks are successfully scheduled by the container cloud, the task information of the training tasks is pushed to the message queue corresponding to their running status. This solves the problem in existing technologies where task information for training tasks cannot be managed based on the task's running status.

[0094] Optionally, the push module 52 is further configured to determine a first training task with an initialization state from the plurality of training tasks, wherein the initialization state is used to indicate that the training container for training the first training task has not been created; obtain the task information of the first training task; and push the task information of the first training task to the initialization task message queue.

[0095] Understandably, after successfully scheduling multiple training tasks created on the cloud platform to the container cloud, the container cloud creates a unique task ID for each training task. Simultaneously, these tasks enter an initialization state, and the task information for multiple first training tasks is pushed to the initialization task message queue. Specifically: 1) In the container cloud, the initialization state is the process of creating the training environment and preparing training resources for the training tasks. This includes downloading the necessary datasets and downloading the training code from the code repository to the directory mounted on the container. These tasks are handled by the container cloud's initialization container for the same container runtime environment created for this training task. At this point, the training container used to indicate that the first training task has not yet been created. 2) The initialization task message queue receives task information for all successfully scheduled first training tasks whose training containers have not yet been created. The initialization processing node continuously iterates through the initialization task message queue.

[0096] Optionally, the push module 52 is further configured to determine a second training task with a running status from the plurality of training tasks, wherein the running status indicates that a training container for training the second training task has been created; obtain the task information of the second training task; and push the task information of the second training task to the running task message queue.

[0097] It should be noted that the "running" status indicates that the training container for training the second training task has been created. Therefore, the source of the second training task in the "running" status is as follows: the initialization processing node continuously traverses the initialization task message queue, that is, it continuously queries the container cloud for the status of the first training task based on its task ID. When the status of the first training task changes to "running," indicating that the training container has been created, the task information of the first training task is pushed to the "running task message queue." It should also be noted that if the task status of the task to be trained has not changed during the traversal of the initialization task message queue and / or the running task message queue, no task information is pushed, and the process waits for the next traversal.

[0098] Optionally, the push module 52 is further configured to push the task information of the second training task to the running task message queue, then obtain the training startup script in the task information of the second training task; and train the second training task in the training container of the second training task according to the training startup script.

[0099] Understandably, once the training container is created in the container cloud, training for the second training task can begin. However, it's important to note that the training startup script within the task information of the second training task needs to be determined first, and then training for the second training task can be initiated within the training container based on the training startup script and the training code.

[0100] Optionally, the push module 52 is further configured to determine a third training task whose running state is in the completed state from the plurality of training tasks; obtain the task information of the third training task; and push the task information of the third training task to the completed task message queue.

[0101] Specifically, the second training task in the running state is the process of running training and generating a model according to the startup script of the training code. Once the second training task has finished training and the model has been generated, its task status will change to "completed." When the task status is found to be completed based on the task ID, the task information is pushed to the completed task message queue. The completed task processing node consumes the completed task message queue, and then schedules the container cloud to delete the corresponding third training task, while simultaneously recording the resource usage of the third training task and releasing its resources.

[0102] Optionally, the push module 52 is further configured to, after pushing the task information corresponding to the plurality of training tasks to message queues corresponding to the running status of the plurality of training tasks, determine, in the case that the message queue includes at least one of the following: an initialization task message queue, a running task message queue, and an end-of-task message queue, a first training task corresponding to the task information in the initialization task message queue, a second training task corresponding to the task information in the running task message queue, and a third training task corresponding to the task information in the end-of-task message queue; and determine the running status of the plurality of training tasks based on the first training task, the second training task, and the third training task.

[0103] Understandably, by analyzing the task information in the initialization task message queue, the running task message queue, and the terminated task message queue, one can clearly determine the different task states of the training tasks corresponding to the task information in different message queues, and thus determine the running status of multiple training tasks. For example, if a training task is assigned task ID 001 after successful scheduling, and task information 001 is found in the running task message queue, then it can be known that training task 001, i.e., the second training task 001, is being trained by the training container and generating a model.

[0104] After obtaining multiple training tasks created by the cloud platform and the task information of the training tasks, resource verification of the task information is required, specifically including a first resource verification and a second resource verification. The acquisition module 50 is also used to obtain the target resources required for training the training tasks based on the task information; compare the target resources with the remaining resources of the server resource group corresponding to the training task; and determine that the training task passes the first resource verification if the remaining resources are greater than the target resources.

[0105] Optionally, the acquisition module 50 is further configured to acquire user group quota information in the task information, wherein the user group quota information is used to indicate the resources required for training the task to be trained; and acquire the target resources according to the user group quota information.

[0106] It's important to note that the server resource group corresponding to the training task includes the server cluster where the deployed devices are also added to the container cloud for unified management. The container cloud then divides and abstracts these resource groups based on the ML hardware. Servers within the same resource group possess the same ML hardware. For example, in the first resource verification, when training task 001 is running, after obtaining the task information for task 001, the user group quota information is read. This allows the determination of the resources required for training task 001. For instance, if the required resources account for 20% of the resource group containing task 001, then the first resource verification passes if the remaining resources in the resource group containing task 001 are greater than 20%; otherwise, it fails.

[0107] Before or after the task to be trained passes the first resource verification, the acquisition module 50 is further configured to acquire the number of servers required for training the task to be trained based on the task information; compare the number of servers with the number of available servers in the server resource group corresponding to the task to be trained; and if the number of available servers is greater than the number of servers, determine that the task to be trained passes the second resource verification.

[0108] Optionally, the acquisition module 50 is further configured to acquire server number information in the task information, wherein the server number information is used to indicate the number of servers required for the training of the task to be trained; and to determine the server number information as the number of servers required for the training of the task to be trained.

[0109] Understandably, similar to the first resource check, taking training task 001 as an example, if training task 001 passes the first resource check, the number of servers required by training task 001 can be obtained through the task information of training task 001, for example, 3. If the number of remaining servers in the server resource group where training task 001 is located is greater than 3, then training task 001 passes the second resource check.

[0110] It should be noted that during the resource verification process, the task may not meet the requirements of the first resource verification and / or the second resource verification. In this case, the task information will be added to the waiting task dictionary and waiting reason dictionary in the in-memory database, and the task will wait for the next resource verification.

[0111] Optionally, the acquisition module 50 is further configured to determine whether a training startup script exists in the task information if the task to be trained passes the first resource verification and the second resource verification, wherein the training startup script is used to start a process for training the task to be trained; and if the training startup script exists, to encapsulate the task information into a target template that the container cloud of the cloud platform allows to process.

[0112] Specifically, a compliance check is performed on the task information of the training tasks that pass resource verification. This involves checking whether the task information contains a training startup script used to launch the process for training the training task. If the training startup script exists, the task information is encapsulated into a target template that can be processed by the container cloud, i.e., a Kubernetes standard template. Conversely, if the compliance check fails, the push of the task information is terminated directly, or the task information can be pushed to the task termination message queue.

[0113] Optionally, the acquisition module 50 is further configured to acquire the task identifier of the training task generated by the container cloud based on the target template; and query the running status in the container cloud based on the task identifier.

[0114] Understandably, after the task identifier is sent to the container cloud as the target template, the container cloud generates a unique task ID for each task to be trained. Based on this task ID, the running status of the task can be queried, including the initialization status, the running status, and the end status.

[0115] Optionally, the acquisition module 50 is also configured to include at least one of the following different message queues: a priority queue, a task message queue to be scheduled, an initialization task message queue, a running task message queue, and a task message queue to end.

[0116] Specifically, as shown in the attached document Figure 4As shown, the different message queues include: priority queue, pending task message queue, initialization task message queue, running task message queue, and completed task message queue. The corresponding processing nodes are, in order: resource verification node, task scheduling processing node, task initialization processing node, running task processing node, and completed task processing node.

[0117] Among them, the resource verification processing node performs resource verification on the priority queue of each resource group (i.e., Figure 4 The resource verification processing nodes consume resources from the priority queues of each resource group. Training tasks that fail resource verification wait for the next verification, while those that pass verification enter the scheduled task message queue. The task scheduling processing nodes consume the scheduled message queue, first performing a compliance check on the task information. If the check passes, the task can be scheduled. Successful scheduling allows the task to enter the initialization task message queue; otherwise, it enters the end task message queue. The task initialization processing nodes consume the initialization task message queue, prepare the training tasks, and query the task status. Tasks in a failed or completed state enter the end task message queue, while those in a running state enter the running task message queue. The running task processing nodes consume the running task message queue, train the tasks, and query the task status. Tasks in a completed or failed state enter the end task message queue. The end task processing nodes consume the end task message queue and retrieve the failure reason from the container cloud using the task ID. If the reason exists, it is recorded in the relational database. Then, the container cloud is scheduled to delete the corresponding task. Simultaneously, the task's status and resource usage are recorded in the relational database. Finally, delete the resource group usage information corresponding to this task from the in-memory database.

[0118] It is important to note that the task initialization node and the task running node process the successfully scheduled tasks to be trained. While querying the task status from the container cloud, the task initialization node and the task running node also update the status of the tasks to be trained from the container cloud after processing them.

[0119] Embodiments of the present invention also provide a storage medium comprising a stored program, wherein the program, when executed, performs any of the methods described above.

[0120] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0121] S1, Obtain multiple training tasks created by the cloud platform and the task information of the training tasks;

[0122] S2, if the container cloud successfully schedules the multiple training tasks according to the task information, obtain the running status of each of the multiple training tasks, and push the task information corresponding to each of the multiple training tasks to the message queue corresponding to the running status of the multiple training tasks.

[0123] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0124] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0125] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0126] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0127] S1, Obtain multiple training tasks created by the cloud platform and the task information of the training tasks;

[0128] S2, if the container cloud successfully schedules the multiple training tasks according to the task information, obtain the running status of each of the multiple training tasks, and push the task information corresponding to each of the multiple training tasks to the message queue corresponding to the running status of the multiple training tasks.

[0129] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0130] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0131] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be executed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for pushing task information, characterized in that, include: Obtain multiple training tasks created by the cloud platform and the task information of the training tasks; If the container cloud successfully schedules the multiple training tasks according to the task information, it obtains the running status of each of the multiple training tasks and pushes the task information corresponding to each of the multiple training tasks to the message queue corresponding to the running status of the multiple training tasks. The operating states include: initialization state, running state, and termination state; The process includes, after obtaining multiple training tasks created by the cloud platform and the task information of the training tasks, performing resource verification on the task information of the training tasks; and, if the training tasks pass the resource verification and the task information contains a training startup script, encapsulating the task information into a target template that the cloud platform's container cloud allows to process.

2. The method for pushing task information according to claim 1, characterized in that, Pushing the task information corresponding to the plurality of tasks to be trained to the message queues corresponding to the running status of the plurality of tasks to be trained, including: A first training task is determined from the plurality of training tasks and its running state is initialized, wherein the initialization state is used to indicate that the training container for training the first training task has not been created; Obtain the task information of the first training task and push the task information of the first training task to the initialization task message queue.

3. The method for pushing task information according to claim 1, characterized in that, Pushing the task information corresponding to the plurality of tasks to be trained to the message queues corresponding to the running status of the plurality of tasks to be trained, including: A second training task is determined from the plurality of training tasks to be run, wherein the running state indicates that a training container for training the second training task has been created; Obtain the task information of the second training task and push the task information of the second training task to the running task message queue.

4. The method for pushing task information according to claim 3, characterized in that, After pushing the task information of the second training task to the running task message queue, the method further includes: Obtain the training startup script from the task information of the second training task; The second training task is trained in the training container of the second training task according to the training startup script.

5. The method for pushing task information according to claim 1, characterized in that, Pushing the task information corresponding to the plurality of tasks to be trained to the message queues corresponding to the running status of the plurality of tasks to be trained, including: A third training task is determined from the plurality of training tasks whose running state is the terminated state; Obtain the task information of the third training task and push the task information of the third training task to the end task message queue.

6. The method for pushing task information according to claim 1, characterized in that, After pushing the task information corresponding to the plurality of tasks to be trained to the message queues corresponding to the running status of the plurality of tasks to be trained, the method further includes: In the case that the message queue includes at least one of the following: an initialization task message queue, a running task message queue, and an end task message queue, a first training task corresponding to the task information in the initialization task message queue, a second training task corresponding to the task information in the running task message queue, and a third training task corresponding to the task information in the end task message queue are determined. The operation status of the plurality of training tasks is determined based on the first training task, the second training task, and the third training task.

7. The method for pushing task information according to claim 1, characterized in that, After obtaining multiple training tasks created by the cloud platform and the task information of the training tasks, the method further includes: Based on the task information, obtain the target resources required for training the task to be trained; Compare the target resource with the remaining resources of the server resource group corresponding to the task to be trained; If the remaining resources are greater than the target resources, the task to be trained is determined to have passed the first resource verification.

8. A task information push device, characterized in that, include: The acquisition module is used to acquire multiple training tasks created by the cloud platform and the task information of the training tasks; The push module is used to obtain the running status of the multiple tasks to be trained respectively when the container cloud successfully schedules the multiple tasks to be trained according to the task information, and push the task information corresponding to the multiple tasks to be trained to the message queue corresponding to the running status of the multiple tasks to be trained respectively. The operating states include: initialization state, running state, and termination state; The acquisition module is further configured to perform resource verification on the task information of the task to be trained; if the task to be trained passes the resource verification and the task information contains a training startup script, the task information is encapsulated into a target template that the cloud platform's container cloud allows to process.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the method described in any one of claims 1 to 7 through the computer program.

Images