Model training task queue control method, device, equipment, medium and product

Abstract

The application provides a model training task queue control method and device, equipment, medium and product, and relates to the technical field of data processing. The method comprises the following steps: under the condition that a first preset time interval is met, reading queue task data corresponding to a training task thread pool from a queue database according to a thread pool name of the training task thread pool obtained by querying a cache, and running the queue task data; under the condition that all task queues corresponding to the training task thread pool are completed, modifying a task state of the training task thread pool to task completion, and closing the training task thread pool; wherein the queue database is stored in a storage device different from the cache.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, medium and product for controlling a model training task queue. Background Technology

[0002] With the continuous development of digital technology, model training has been more widely used. In the management of model training tasks, asynchronous queuing strategies are adopted for operations such as copying, deleting, uploading, moving, and decompressing large files. The queuing is controlled by a thread pool, but there is a risk of memory overflow. When memory overflow occurs, the platform becomes unavailable, which has a significant impact on training tasks.

[0003] Therefore, how to more effectively control the model training task queue has become an urgent problem to be solved in the industry. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for controlling a model training task queue, in order to solve the problem of how to more effectively control a model training task queue in the prior art.

[0005] This invention provides a method for controlling a model training task queue, comprising: If the first preset time interval is met, the queue task data corresponding to the training task thread pool is read from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and the queue task data is executed. When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completion, and the training task thread pool is closed. The queue database is stored on a different storage device than the cache.

[0006] According to a model training task queue control method provided by the present invention, before the step of reading queue task data corresponding to the training task thread pool from the queue database based on the thread pool name of the training task thread pool retrieved from the cache and running the queue task data, the method further includes: When creating a new training task thread pool, the queue task data corresponding to the new training task thread pool is stored in the queue database. Write the thread pool name, task status, and task type corresponding to the new training task thread pool into the cached queue data table.

[0007] According to a model training task queue control method provided by the present invention, after the step of writing the thread pool name, task status, and task type corresponding to the new training task thread pool into the queue data table in the cache, the method further includes: If there is no running queue in the cached thread pool, retrieve the target queue task data corresponding to the new training task thread pool from the queue database and run the target queue task data; wherein, the target queue task data is queue task data with the task status of being entered into the database. After the target queue task data is completed, the task status of the target queue task data is changed from the inbound status to the outbound status, and the target queue task data corresponding to the new training task thread pool continues to be read and executed. If no target queue task data corresponding to the new training task thread pool is read within the second preset time period, the task status of all queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0008] According to a model training task queue control method provided by the present invention, the queue data table includes: thread pool name, thread pool task type, thread pool task type, and queue data identity information; The thread pool task types include at least one of the following: copy task type and compression task type; The thread pool task types include at least one of the following: inbound status, outbound status, and task completion status.

[0009] According to a model training task queue control method provided by the present invention, the method reads queue task data corresponding to the training task thread pool from a queue database and runs the queue task data, including: In the queue database, the queue task data corresponding to the training task thread pool is read sequentially according to the order of the queue task data corresponding to the training task thread pool, and the queue task data is executed.

[0010] According to a model training task queue control method provided by the present invention, the step of storing the queue task data corresponding to the new training task thread pool into a queue database when creating a new training task thread pool includes: The task data corresponding to the new training task thread pool is queued. Based on the task queuing order, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0011] According to the model training task queue control method provided by the present invention, after the steps of changing the task status of the training task thread pool to task completion and closing the training task thread pool when all task queues corresponding to the training task thread pool have completed running, the method further includes: Remove the thread pool name, task status, and task type of the training task thread pool from the cached queue data table.

[0012] The present invention also provides a model training task queue control device, comprising the following modules: The query module is used to, when a first preset time interval is met, read the queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and run the queue task data. The control module is used to change the task status of the training task thread pool to "task completed" and shut down the training task thread pool when all task queues corresponding to the training task thread pool have completed running. The queue database is stored on a different storage device than the cache.

[0013] According to the model training task queue control device provided in this application, the device is further used for: When creating a new training task thread pool, the queue task data corresponding to the new training task thread pool is stored in the queue database. Write the thread pool name, task status, and task type corresponding to the new training task thread pool into the cached queue data table.

[0014] According to the model training task queue control device provided in this application, the device is further used for: If there is no running queue in the cached thread pool, retrieve the target queue task data corresponding to the new training task thread pool from the queue database and run the target queue task data; wherein, the target queue task data is queue task data with the task status of being entered into the database. After the target queue task data is completed, the task status of the target queue task data is changed from the inbound status to the outbound status, and the target queue task data corresponding to the new training task thread pool continues to be read and executed. If no target queue task data corresponding to the new training task thread pool is read within the second preset time period, the task status of all queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0015] According to the model training task queue control device provided in this application, the device is further used for: In the queue database, the queue task data corresponding to the training task thread pool is read sequentially according to the order of the queue task data corresponding to the training task thread pool, and the queue task data is executed.

[0016] According to the model training task queue control device provided in this application, the device is further used for: The task data corresponding to the new training task thread pool is queued. Based on the task queuing order, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0017] According to the model training task queue control device provided in this application, the device is further used for: Remove the thread pool name, task status, and task type of the training task thread pool from the cached queue data table.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the model training task queue control method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training task queue control method as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model training task queue control method as described above.

[0021] The model training task queue control method, apparatus, device, medium, and product provided by this invention effectively avoids the problem of queue task data occupying flash memory by storing the queue task data corresponding to the training task thread pool in a queue database instead of a cache, thus preventing flash memory overflow. Furthermore, by ensuring the sequential execution of the task queue and timely shutdown of the thread pool, system crashes due to resource exhaustion can be avoided, enhancing system stability. Intelligent management of the thread pool and task queue improves the efficiency and stability of the AI ​​training platform, while also enhancing the user experience for algorithm engineers. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the model training task queue control method provided by the present invention.

[0024] Figure 2 The overall process of operating the file operation queue provided in the embodiments of this application.

[0025] Figure 3 This is a schematic diagram illustrating the implementation process of a timed task provided in an embodiment of this application.

[0026] Figure 4 A schematic diagram of the structure of the model training task queue control device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Figure 1 This is a flowchart illustrating the model training task queue control method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: If the first preset time interval is met, read the queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and run the queue task data. In this embodiment, the system sets a preset time interval to avoid excessively frequent task queries and executions, thereby reducing system resource consumption and improving efficiency. At the end of each preset time interval, the system checks the cache to find active thread pools. The cache stores the names of the thread pools, allowing for quick location of the thread pool requiring processing.

[0029] Once the thread pool name is found in the cache, the system accesses the queue database to find the queue task data corresponding to that thread pool name. The queue database stores all pending task queues. After finding the corresponding task data, the system begins executing those tasks. These tasks may be operations such as copying, moving, deleting, or decompressing files.

[0030] Each thread pool is responsible for executing a specific type of task. In this way, multiple tasks can be processed in parallel, improving processing efficiency.

[0031] Task execution is typically asynchronous, meaning the system can begin processing the next task without waiting for the current task to complete, thus improving overall processing speed.

[0032] Step 120: When all task queues corresponding to the training task thread pool have completed running, change the task status of the training task thread pool to task completed and close the training task thread pool. The queue database is stored on a different storage device than the cache.

[0033] In this embodiment, the system continuously monitors the task execution status in each training task thread pool. The next operation will only be executed when all task queues in the thread pool have finished running.

[0034] Once all tasks are confirmed to be completed, the system updates the status of these tasks to "Task Completed". This status update is performed in the database, ensuring data consistency and traceability.

[0035] After updating the status, the system will shut down the corresponding training task thread pool. This operation is to release resources, avoid unnecessary resource consumption, and thus improve the overall efficiency of the system. During this process, the queue database is stored on a different storage device than the cache. This physical or logical separation helps improve data security and access speed, while also facilitating data management and maintenance.

[0036] In this embodiment, by storing the queued task data corresponding to the training task thread pool in a queue database instead of a cache, the problem of queued task data occupying flash memory is effectively avoided, thus preventing flash memory overflow. Furthermore, by ensuring the sequential execution of the task queue and timely shutdown of the thread pool, system crashes due to resource exhaustion can be avoided, enhancing system stability. Intelligent management of the thread pool and task queue improves the efficiency and stability of the AI ​​training platform, while also enhancing the user experience for algorithm engineers.

[0037] Optionally, prior to the step of retrieving queue task data corresponding to the training task thread pool from the queue database based on the thread pool name of the training task thread pool retrieved from the cache, and running the queue task data, provided that a first preset time interval is met, the method further includes: When creating a new training task thread pool, the queue task data corresponding to the new training task thread pool is stored in the queue database. Write the thread pool name, task status, and task type corresponding to the new training task thread pool into the cached queue data table.

[0038] In this embodiment, when a new training task is submitted to the system, the system creates a new training task thread pool. This thread pool is dedicated to this training task, ensuring that the task can be executed efficiently.

[0039] Once a thread pool is created, the system stores the queued task data associated with that thread pool in the queue database. This data includes detailed task information, such as task type and task parameters.

[0040] The system writes the name of the newly created thread pool to a cached queue data table. This name serves as an index for subsequent operations, allowing the system to quickly locate and operate on the thread pool. In addition to the thread pool name, the system also records the task's status (e.g., "pending," "in execution," "completed") and task type (e.g., "copy," "move," "delete"). This information helps the system track task execution and perform appropriate resource management and scheduling.

[0041] In this embodiment, by storing thread pool names in a cache, the system can quickly retrieve the corresponding thread pool, thereby accelerating task scheduling and execution. Storing task data in a database ensures data persistence and consistency. Even in the event of a system failure, task data will not be lost.

[0042] Optionally, after the step of writing the thread pool name, task status, and task type corresponding to the new training task thread pool into the queue data table in the cache, the method further includes: If there is no running queue in the cached thread pool, retrieve the target queue task data corresponding to the new training task thread pool from the queue database and run the target queue task data; wherein, the target queue task data is queue task data with the task status of being entered into the database. After the target queue task data is completed, the task status of the target queue task data is changed from the inbound status to the outbound status, and the target queue task data corresponding to the new training task thread pool continues to be read and executed. If no target queue task data corresponding to the new training task thread pool is read within the second preset time period, the task status of all queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0043] In this embodiment, the system checks the thread pool in the cache to confirm that there are no currently running queue tasks. Once the thread pool is confirmed to be idle, the system retrieves the queue task data corresponding to the new training task thread pool, with the task status "added to database," from the queue database. The system then begins executing these target queue task data.

[0044] After the target queue tasks have been completed, the system changes the data status of these tasks from "inbound" to "outbound". The system then continues to read and execute the target queue task data corresponding to the new training task thread pool from the queue database.

[0045] In this embodiment of the application, the system sets a second preset time period during which the system checks whether it can read the target queue task data corresponding to the new training task thread pool.

[0046] If no target queue task data is read within the second preset time period, the system will adjust the task status of all queue tasks in the new training task thread pool to "task completed". Once all task statuses are adjusted to "task completed", the system will close the new training task thread pool.

[0047] In this embodiment, by closing the thread pool after a task is completed, the system can release resources and improve resource utilization. By continuously executing tasks in the queue, the system reduces waiting time and improves task processing efficiency. By closing the thread pool when no task data is read within a second preset time period, the system avoids unnecessary resource consumption. By promptly closing idle thread pools, the system can reduce the risk of memory leaks and enhance system stability.

[0048] Optionally, the queue data table includes: thread pool name, thread pool task type, thread pool task type, and queue data identity information; The thread pool task types include at least one of the following: copy task type and compression task type; The thread pool task types include at least one of the following: inbound status, outbound status, and task completion status.

[0049] In this embodiment, the thread pool name is a unique identifier for each thread pool, used for quick indexing and operation of a specific thread pool. Thread pool task types may include: copy task types and compression task types.

[0050] In this embodiment of the application, the task status records the current state of the task, which may include the following states: Inbound status: This indicates that the task has been entered into the queue and is waiting to be executed.

[0051] Outbound status: indicates that the task has started or has been completed.

[0052] Task completion status: Indicates that the task has been fully executed.

[0053] In this embodiment of the application, the system can easily track the execution status and type of each task through the queue data table, thereby enabling effective task management and scheduling.

[0054] Optionally, reading the queue task data corresponding to the training task thread pool from the queue database and running the queue task data includes: In the queue database, the queue task data corresponding to the training task thread pool is read sequentially according to the order of the queue task data corresponding to the training task thread pool, and the queue task data is executed.

[0055] In this embodiment of the application, reading the queue task data corresponding to the training task thread pool from the queue database and running the queue task data includes: In the queue database, the queue task data corresponding to the training task thread pool is read sequentially according to the order of the queue task data, and the queue task data is executed. In order to efficiently read the task data in sequence, the task data in the database usually has an index field, such as a task ID or timestamp, to identify the order of the tasks.

[0056] The system constructs a database query to select task data based on a defined sequential index. The system executes the query and retrieves the sequentially ordered task data from the database.

[0057] The system assigns the retrieved task data to the corresponding thread pool for processing. Worker threads in the thread pool then begin executing the tasks. This may include operations such as copying, compressing, moving, and deleting files.

[0058] Optionally, when creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database includes: The task data corresponding to the new training task thread pool is queued. Based on the task queuing order, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0059] When new training tasks are submitted, the system receives them first. The system sorts the tasks according to certain rules (such as submission time, priority, etc.) to determine the execution order.

[0060] The system establishes a connection with the queue database, preparing to write task data. The system writes the task data to the database sequentially according to the order determined by the queue. Each task data entry may include information such as task type, task parameters, submission time, and priority.

[0061] After the data is written, the system will confirm that each task data has been stored correctly. The system will update the task status to "stored in database," indicating that the task is ready to be processed by the thread pool.

[0062] In this embodiment, by queuing and storing tasks sequentially, the system ensures that tasks are processed in a predetermined order. Because each thread in the thread pool, based on its characteristics, will always queue when the number of queue entries does not exceed the maximum number, effectively avoiding the problem of non-queueing.

[0063] Optionally, after changing the task status of the training task thread pool to "task completed" and closing the training task thread pool when all task queues corresponding to the training task thread pool have finished running, the method further includes: Remove the thread pool name, task status, and task type of the training task thread pool from the cached queue data table.

[0064] In this embodiment, the system modifies the task status of the training task thread pool to "task completed," indicating that all tasks have been executed. The system updates the task status in the queue database to ensure that the status of all related tasks is marked as completed.

[0065] The system shuts down the training task thread pool, releasing all resources associated with it, including memory and processor resources. The system removes the thread pool reference from the thread pool manager, ensuring that no new tasks are assigned to the closed thread pool. The system updates its cache to ensure that it no longer contains information about the closed thread pool.

[0066] In this embodiment, by clearing the cache, the system ensures the consistency and accuracy of cached data. Disabling the thread pool and clearing the cache helps release resources and improve the overall resource utilization of the system.

[0067] Figure 2The overall flow of the file operation queue provided in the embodiments of this application is as follows: Figure 2 As shown, the thread pool's task queue is not stored in memory, but in a database. This approach helps reduce memory usage and ensures the persistence of queue data.

[0068] When the system needs to execute a task, it reads task queue data from the database. This allows tasks to be loaded on demand, helping to improve the system's responsiveness and scalability.

[0069] When new tasks arrive, the system directly inserts them into the corresponding queues in the database. This operation is atomic, ensuring the consistency and integrity of task addition.

[0070] Each thread pool has a unique name, which is stored in a cache. The cache provides the ability to quickly access and retrieve thread pools, which is especially important for scheduled tasks.

[0071] The system may set up a scheduled task (e.g., running every 10 minutes) that checks the thread pool names in the cache and determines which thread pools have completed their task queues.

[0072] If all tasks in a thread pool have been completed, the scheduled task will shut down the thread pool to release resources, such as the thread resources it occupies.

[0073] When a thread pool is shut down, the system will clear the corresponding thread pool name from the cache to ensure that only active thread pool information is retained in the cache.

[0074] Figure 3 This is a schematic diagram illustrating the implementation process of the timed task provided in the embodiments of this application, such as... Figure 3 As shown, the system continuously monitors the task queue in the thread pool to determine whether all tasks have been completed.

[0075] Once the system detects that all tasks in the thread pool have been completed, it will trigger a scheduled task to shut down the thread pool and release all associated resources. Simultaneously, the scheduled task will remove all information related to the thread pool from the cache, including the thread pool name, task status, and task type.

[0076] The scheduled task executes every 10 minutes by default, but this is an adjustable setting that can be optimized based on the system's actual workload and performance requirements. System administrators can adjust the execution frequency of the scheduled task based on changes in system monitoring results or business needs.

[0077] The model training task queue control device provided by the present invention is described below. The model training task queue control device described below can be referred to in correspondence with the model training task queue control method described above.

[0078] Figure 4 This is a schematic diagram of the model training task queue control device provided in the embodiments of this application, as shown below. Figure 4 As shown, it includes: The query module 410 is used to, when a first preset time interval is met, read the queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool queried from the cache, and run the queue task data. The control module 420 is used to change the task status of the training task thread pool to "task completed" and close the training task thread pool when all task queues corresponding to the training task thread pool have completed running. The queue database is stored on a different storage device than the cache.

[0079] According to the model training task queue control device provided in this application, the device is further used for: When creating a new training task thread pool, the queue task data corresponding to the new training task thread pool is stored in the queue database. Write the thread pool name, task status, and task type corresponding to the new training task thread pool into the cached queue data table.

[0080] According to the model training task queue control device provided in this application, the device is further used for: If there is no running queue in the cached thread pool, retrieve the target queue task data corresponding to the new training task thread pool from the queue database and run the target queue task data; wherein, the target queue task data is queue task data with the task status of being entered into the database. After the target queue task data is completed, the task status of the target queue task data is changed from the inbound status to the outbound status, and the target queue task data corresponding to the new training task thread pool continues to be read and executed. If no target queue task data corresponding to the new training task thread pool is read within the second preset time period, the task status of all queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0081] According to the model training task queue control device provided in this application, the device is further used for: In the queue database, the queue task data corresponding to the training task thread pool is read sequentially according to the order of the queue task data corresponding to the training task thread pool, and the queue task data is executed.

[0082] According to the model training task queue control device provided in this application, the device is further used for: The task data corresponding to the new training task thread pool is queued. Based on the task queuing order, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0083] According to the model training task queue control device provided in this application, the device is further used for: Remove the thread pool name, task status, and task type of the training task thread pool from the cached queue data table.

[0084] This embodiment of the application effectively avoids the problem of queue task data occupying flash memory by storing the queue task data corresponding to the training task thread pool in a queue database instead of a cache, thus preventing flash memory overflow. Furthermore, by ensuring the sequential execution of the task queue and timely shutdown of the thread pool, system crashes due to resource exhaustion can be avoided, enhancing system stability. Intelligent management of the thread pool and task queue improves the efficiency and stability of the AI ​​training platform, while also enhancing the user experience for algorithm engineers.

[0085] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a model training task queue control method, which includes: when a first preset time interval is met, reading queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and running the queue task data; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completion, and the training task thread pool is closed. The queue database is stored on a different storage device than the cache.

[0086] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the model training task queue control method provided by the above methods, the method including: under the condition of satisfying a first preset time interval, reading the queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool queried from the cache, and running the queue task data; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completion, and the training task thread pool is closed. The queue database is stored on a different storage device than the cache.

[0088] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a model training task queue control method provided by the above methods, the method comprising: upon satisfying a first preset time interval, reading queue task data corresponding to the training task thread pool from a queue database according to the thread pool name of the training task thread pool queried from a cache, and running the queue task data; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completion, and the training task thread pool is closed. The queue database is stored on a different storage device than the cache.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling a model training task queue, characterized in that, include: If the first preset time interval is met, the queue task data corresponding to the training task thread pool is read from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and the queue task data is executed. When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, the training task thread pool is closed, and the thread pool name, task status, and task type of the training task thread pool are removed from the cached queue data table. The queue database is stored in a different storage device than the cache; the cache stores the thread pool name, task status and task type of the training task thread pool, and the queue database stores the queue task data corresponding to the training task thread pool.

2. The model training task queue control method according to claim 1, characterized in that, Before the step of reading the queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and running the queue task data, provided that a first preset time interval is met, the method further includes: When creating a new training task thread pool, the queue task data corresponding to the new training task thread pool is stored in the queue database. Write the thread pool name, task status, and task type corresponding to the new training task thread pool into the cached queue data table.

3. The model training task queue control method according to claim 2, characterized in that, After the step of writing the thread pool name, task status, and task type corresponding to the new training task thread pool into the queue data table in the cache, the method further includes: If there is no running queue in the cached thread pool, retrieve the target queue task data corresponding to the new training task thread pool from the queue database and run the target queue task data; wherein, the target queue task data is queue task data with the task status of being entered into the database. After the target queue task data is completed, the task status of the target queue task data is changed from the inbound status to the outbound status, and the target queue task data corresponding to the new training task thread pool continues to be read and executed. If no target queue task data corresponding to the new training task thread pool is read within the second preset time period, the task status of all queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

4. The model training task queue control method according to claim 3, characterized in that, The queue data table includes: thread pool name, thread pool task type, thread pool task status, and queue data identity information; The thread pool task types include at least one of the following: copy task type and compression task type; The thread pool task status includes at least one of the following: inbound status, outbound status, and task completion status.

5. The model training task queue control method according to claim 1, characterized in that, Read the queue task data corresponding to the training task thread pool from the queue database, and run the queue task data, including: In the queue database, the queue task data corresponding to the training task thread pool is read sequentially according to the order of the queue task data corresponding to the training task thread pool, and the queue task data is executed.

6. The model training task queue control method according to claim 3, characterized in that, The step of storing the queue task data corresponding to the new training task thread pool into the queue database when creating a new training task thread pool includes: The task data corresponding to the new training task thread pool is queued. Based on the task queuing order, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

7. A model training task queue control device, characterized in that, include: The query module is used to read the queue task data corresponding to the training task thread pool from the queue database according to the thread pool name of the training task thread pool retrieved from the cache, and run the queue task data, provided that a first preset time interval is met. The control module is used to change the task status of the training task thread pool to "task completed" and shut down the training task thread pool when all task queues corresponding to the training task thread pool have completed running; and to remove the thread pool name, task status and task type of the training task thread pool from the cached queue data table. The queue database is stored in a different storage device than the cache; the cache stores the thread pool name, task status and task type of the training task thread pool, and the queue database stores the queue task data corresponding to the training task thread pool.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model training task queue control method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training task queue control method as described in any one of claims 1 to 6.

Images