Model training task queue control method and device, equipment, medium and product

Abstract

The invention provides a model training task queue control method and device, equipment, a medium and a product, and relates to the technical field of data processing.The method comprises the steps that under the condition that a first preset time interval is met, according to a thread pool name of a training task thread pool inquired from a cache, a training task queue is obtained; reading queue task data corresponding to the training task thread pool in a queue database, and running the queue task data; under the condition that all the task queues corresponding to the training task thread pool complete operation, the task state of the training task thread pool is modified into task completion, and the training task thread pool is closed; wherein the queue database is stored in a storage device different from the cache.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a model training task queue control method, device, equipment, medium and product. Background Art

[0002] With the continuous development of digital technology, model training has also been more widely used. In the management of model training tasks in related technologies, an asynchronous queuing strategy is adopted for operations such as large file copying, deletion, uploading, moving, and decompression. The queuing adopts thread pool control, but there is a risk of memory overflow. When the memory overflows, it will cause the platform to be unavailable, which will have a great impact on the training tasks.

[0003] Therefore, how to more effectively control the model training task queue has become an urgent problem to be solved in the industry. Summary of the invention

[0004] The present invention provides a model training task queue control method, device, equipment, medium and product, which are used to solve the problem of how to more effectively control the model training task queue in the prior art.

[0005] The present invention provides a model training task queue control method, comprising: When the first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, the queue task data corresponding to the training task thread pool is read from the queue database, and the queue task data is run; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and the training task thread pool is closed; The queue database is stored in a storage device different from the cache.

[0006] According to a model training task queue control method provided by the present invention, when a first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, the queue task data corresponding to the training task thread pool is read in the queue database, and before the step of running the queue task data, the method further includes: In the case of creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database; The thread pool name, task status and task type corresponding to the new training task thread pool are written into the cached queue data table.

[0007] According to a model training task queue control method provided by the present invention, after the step of writing the thread pool name, task status and task type corresponding to the new training task thread pool into the queue data table in the cache, the method further includes: When there is no running queue in the cached thread pool, the target queue task data corresponding to the new training task thread pool is taken out from the queue database, and the target queue task data is run; wherein the target queue task data is the queue task data whose task status is the storage status; After the target queue task data is executed, the task state of the target queue task data is adjusted from the in-warehouse state to the out-warehouse state, and the target queue task data corresponding to the new training task thread pool is continued to be read and executed; Within the second preset time period, if the target queue task data corresponding to the new training task thread pool is not read, the task status of the queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0008] According to a model training task queue control method provided by the present invention, the queue data table includes: thread pool name, thread pool task type, thread pool task type, queue data identity information; The thread pool task type includes at least one of the following: a copy task type and a compression task type; The thread pool task type includes at least one of the following: storage status, outbound status, and task completion status.

[0009] According to a model training task queue control method provided by the present invention, queue task data corresponding to the training task thread pool is read in a queue database, and the queue task data is run, including: In the queue database, the queue task data corresponding to the training task thread pool are read in sequence according to the order of the queue task data corresponding to the training task thread pool, and the queue task data are run.

[0010] According to a model training task queue control method provided by the present invention, in the case of creating a new training task thread pool, storing queue task data corresponding to the new training task thread pool into a queue database, comprising: Queue the queue task data corresponding to the new training task thread pool; According to the order of task queuing, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0011] According to a model training task queue control method provided by the present invention, when all task queues corresponding to the training task thread pool have completed operation, the task state of the training task thread pool is modified to task completion, and after the step of closing the training task thread pool, the method further includes: Remove the thread pool name, task status and task type of the training task thread pool from the cached queue data table.

[0012] The present invention also provides a model training task queue control device, comprising the following modules: A query module, configured to read queue task data corresponding to the training task thread pool in the queue database according to the thread pool name of the training task thread pool queried from the cache and run the queue task data when a first preset time interval is met; A control module, used for modifying the task status of the training task thread pool to task completion and closing the training task thread pool when all task queues corresponding to the training task thread pool have completed operation; The queue database is stored in a storage device different from the cache.

[0013] According to the model training task queue control device provided by the present application, the device is also used for: In the case of creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database; The thread pool name, task status and task type corresponding to the new training task thread pool are written into the cached queue data table.

[0014] According to the model training task queue control device provided by the present application, the device is also used for: When there is no running queue in the cached thread pool, the target queue task data corresponding to the new training task thread pool is taken out from the queue database, and the target queue task data is run; wherein the target queue task data is the queue task data whose task status is the storage status; After the target queue task data is executed, the task state of the target queue task data is adjusted from the in-warehouse state to the out-warehouse state, and the target queue task data corresponding to the new training task thread pool is continued to be read and executed; Within the second preset time period, if the target queue task data corresponding to the new training task thread pool is not read, the task status of the queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0015] According to the model training task queue control device provided by the present application, the device is also used for: In the queue database, the queue task data corresponding to the training task thread pool are read in sequence according to the order of the queue task data corresponding to the training task thread pool, and the queue task data are run.

[0016] According to the model training task queue control device provided by the present application, the device is also used for: Queue the queue task data corresponding to the new training task thread pool; According to the order of task queuing, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0017] According to the model training task queue control device provided by the present application, the device is also used for: Remove the thread pool name, task status and task type of the training task thread pool from the cached queue data table.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model training task queue control method as described in any one of the above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the model training task queue control methods described above.

[0020] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned model training task queue control methods.

[0021] The model training task queue control method, device, equipment, medium and product provided by the present invention effectively avoids the problem of queue task data occupying flash memory by storing the queue task data corresponding to the training task thread pool in the queue database instead of the cache, thereby avoiding the problem of flash memory overflow. In addition, by ensuring the sequential execution of the task queue and closing the thread pool in time, system crashes caused by resource exhaustion can be avoided, thereby enhancing the stability of the system. By intelligently managing the thread pool and task queue, the efficiency and stability of the AI ​​training platform are improved, and the operating experience of algorithm personnel is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flow chart of the model training task queue control method provided by the present invention.

[0024] Figure 2 The overall process of the operation of the file operation queue provided in the embodiment of the present application.

[0025] Figure 3 A schematic diagram of the implementation process of the scheduled task provided in the embodiment of the present application.

[0026] Figure 4 A schematic diagram of the structure of a model training task queue control device provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Figure 1 is a flow chart of the model training task queue control method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 110, when the first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, read the queue task data corresponding to the training task thread pool in the queue database, and run the queue task data; In the embodiment of the present application, the system sets a preset time interval, which is to avoid querying and executing tasks too frequently, thereby reducing the consumption of system resources and improving efficiency. When each preset time interval arrives, the system checks the cache to find the active thread pool. The name of the thread pool is stored in the cache, so that the thread pool that needs to be processed can be quickly located.

[0029] Once the thread pool name is found in the cache, the system will access the queue database to find the queue task data corresponding to the thread pool name. The queue database stores all the task queues to be executed. After finding the corresponding task data, the system will start executing these tasks. These tasks may be operations such as copying, moving, deleting, or decompressing files.

[0030] Each thread pool is responsible for executing a specific type of task. In this way, multiple tasks can be processed in parallel, improving processing efficiency.

[0031] Tasks are usually executed asynchronously, which means that the system does not need to wait for the current task to complete before starting to process the next task, thereby improving the overall processing speed.

[0032] Step 120, when all task queues corresponding to the training task thread pool have completed running, modify the task status of the training task thread pool to task completion, and close the training task thread pool; The queue database is stored in a storage device different from the cache.

[0033] In the embodiment of the present application, the system will continuously monitor the task execution status in each training task thread pool. Only when all task queues in the thread pool have completed execution will the next step be executed.

[0034] Once all tasks are confirmed to be completed, the system will update the status of these tasks to "Task Completed". This status update is performed in the database, ensuring data consistency and traceability.

[0035] After updating the status, the system will close the corresponding training task thread pool. This operation is to release resources and avoid unnecessary resource occupation, thereby improving the overall efficiency of the system. In this process, the queue database is stored in a storage device different from the cache. This physical or logical separation helps to improve data security and access speed, and also facilitates data management and maintenance.

[0036] In an embodiment of the present application, by storing the queue task data corresponding to the training task thread pool in the queue database instead of the cache, the problem of queue task data occupying the flash memory is effectively avoided, thereby avoiding the problem of flash memory overflow. In addition, by ensuring the sequential execution of the task queue and closing the thread pool in time, system crashes caused by resource exhaustion can be avoided, and the stability of the system is enhanced. By intelligently managing the thread pool and task queue, the efficiency and stability of the AI ​​training platform are improved, and the operating experience of algorithm personnel is also improved.

[0037] Optionally, when the first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, the queue task data corresponding to the training task thread pool is read in the queue database, and before the step of running the queue task data, the method further includes: In the case of creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database; The thread pool name, task status and task type corresponding to the new training task thread pool are written into the cached queue data table.

[0038] In the embodiment of the present application, when a new training task is submitted to the system, the system will create a new training task thread pool. This thread pool is dedicated to serving this training task to ensure that the task can be executed efficiently.

[0039] Once the thread pool is created, the system will store the queue task data related to this thread pool in the queue database. This data includes detailed information about the task, such as task type, task parameters, etc.

[0040] The system writes the name of the newly created thread pool to the queue data table in the cache. This name is used as an index for subsequent operations, allowing the system to quickly find and operate this thread pool. In addition to the thread pool name, the system also records the task status (such as "pending", "executing", "completed") and task type (such as "copy", "move", "delete"). This information helps the system track the execution of tasks and perform appropriate resource management and scheduling.

[0041] In the embodiment of the present application, by storing the thread pool name in the cache, the system can quickly retrieve the corresponding thread pool, thereby speeding up the scheduling and execution of tasks. Storing task data in the database ensures the persistence and consistency of the data. Even in the event of a system failure, the task data will not be lost.

[0042] Optionally, after the step of writing the thread pool name, task status and task type corresponding to the new training task thread pool into the queue data table in the cache, the method further includes: When there is no running queue in the cached thread pool, the target queue task data corresponding to the new training task thread pool is taken out from the queue database, and the target queue task data is run; wherein the target queue task data is the queue task data whose task status is the storage status; After the target queue task data is executed, the task state of the target queue task data is adjusted from the in-warehouse state to the out-warehouse state, and the target queue task data corresponding to the new training task thread pool is continued to be read and executed; Within the second preset time period, if the target queue task data corresponding to the new training task thread pool is not read, the task status of the queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0043] In the embodiment of the present application, the system checks the thread pool in the cache to confirm that there is no queue task currently running. Once the thread pool is confirmed to be idle, the system will take out the queue task data corresponding to the new training task thread pool and with a task status of "in storage" from the queue database. The system starts to execute these target queue task data.

[0044] After the target queue task data is executed, the system adjusts the data status of these tasks from "inbound" to "outbound". The system continues to read and execute the target queue task data corresponding to the new training task thread pool from the queue database.

[0045] In an embodiment of the present application, the system sets a second preset time period, during which the system checks whether the target queue task data corresponding to the new training task thread pool can be read.

[0046] If no target queue task data is read within the second preset time period, the system will adjust the task status of the corresponding queue task data of the new training task thread pool to "task completed". Once all task statuses are adjusted to "task completed", the system will close the new training task thread pool.

[0047] In the embodiment of the present application, by closing the thread pool after the task is completed, the system can release resources and improve resource utilization. The system reduces waiting time and improves the efficiency of task processing by continuously executing tasks in the queue. By closing the thread pool when the task data is not read within the second preset time period, the system avoids unnecessary resource occupation. By closing the idle thread pool in time, the system can reduce the risk of memory leakage and enhance the stability of the system.

[0048] Optionally, the queue data table includes: thread pool name, thread pool task type, thread pool task type, queue data identity information; The thread pool task type includes at least one of the following: a copy task type and a compression task type; The thread pool task type includes at least one of the following: storage status, outbound status, and task completion status.

[0049] In the embodiment of the present application, the thread pool name is a unique identifier of each thread pool, which is used to quickly index and operate a specific thread pool. The thread pool task type may include: a copy task type and a compression task type.

[0050] In the embodiment of the present application, the task status records the current status of the task, which may include the following states: In-warehouse status: indicates that the task has entered the queue and is waiting to be executed.

[0051] Outbound status: indicates that the task has started or completed execution.

[0052] Task completion status: indicates that the task has been fully executed.

[0053] In the embodiment of the present application, through the queue data table, the system can easily track the execution status and type of each task, thereby performing effective task management and scheduling.

[0054] Optionally, reading queue task data corresponding to the training task thread pool in a queue database and running the queue task data includes: In the queue database, the queue task data corresponding to the training task thread pool are read in sequence according to the order of the queue task data corresponding to the training task thread pool, and the queue task data are run.

[0055] In an embodiment of the present application, the queue task data corresponding to the training task thread pool is read in the queue database, and the queue task data is executed, including: In the queue database, the queue task data corresponding to the training task thread pool is read in sequence according to the order of the queue task data corresponding to the training task thread pool, and the queue task data is run. In order to efficiently read the task data in sequence, the task data in the database usually has an index field, such as a task ID or a timestamp, for identifying the order of the tasks.

[0056] The system constructs a database query to select task data based on the defined sequential index. The system executes the query and retrieves the sequentially ordered task data from the database.

[0057] The system assigns the retrieved task data to the corresponding thread pool for processing. The worker threads in the thread pool start to execute the task. This may include operations such as copying, compressing, moving, and deleting files.

[0058] Optionally, in the case of creating a new training task thread pool, storing queue task data corresponding to the new training task thread pool in a queue database includes: Queue the queue task data corresponding to the new training task thread pool; According to the order of task queuing, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0059] When new training tasks are submitted, the system first receives these tasks. The system sorts the tasks according to certain rules (such as task submission time, priority, etc.) to determine the execution order of the tasks.

[0060] The system establishes a connection with the queue database and prepares to write the task data. The system writes the task data to the database in the order determined by the queue. Each task data may include information such as task type, task parameters, submission time, priority, etc.

[0061] After the data is written, the system will confirm that each task data has been correctly stored. The system updates the task status to "stored", indicating that the task is ready and can be processed by the thread pool.

[0062] In the embodiment of the present application, by queuing and storing in order, the system ensures that tasks are processed in a predetermined order. Because a single thread set in the thread pool, according to the characteristics of a single thread, will always queue when the number of queues does not exceed the maximum number of queues, effectively avoiding the problem of not queuing.

[0063] Optionally, when all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and after the step of closing the training task thread pool, the step further includes: Remove the thread pool name, task status and task type of the training task thread pool from the cached queue data table.

[0064] In the embodiment of the present application, the system modifies the task status of the training task thread pool to "task completed", indicating that all tasks have been executed. The system updates the task status in the queue database to ensure that the status of all related tasks is marked as completed.

[0065] The system closes the training task thread pool and releases all resources associated with the thread pool, including memory and processor resources. The system removes the reference to the thread pool from the thread pool manager to ensure that no new tasks are assigned to the closed thread pool. The system updates the cache to ensure that the cache no longer contains information about the closed thread pool.

[0066] In the embodiment of the present application, by cleaning the cache, the system ensures the consistency and accuracy of the cache data. Closing the thread pool and cleaning the cache helps to release resources and improve the overall resource utilization of the system.

[0067] Figure 2The overall process of the operation of the file operation queue provided in the embodiment of the present application is as follows: Figure 2 As shown in the figure, the task queue of the thread pool is not saved in memory, but stored in the database. This method helps to reduce memory usage and ensure the persistence of queue data.

[0068] When the system needs to execute a task, it reads the task queue data from the database. This allows tasks to be loaded on demand, helping to improve the responsiveness and scalability of the system.

[0069] When new tasks arrive, the system directly inserts them into the corresponding queues in the database. This operation is atomic, ensuring the consistency and integrity of task addition.

[0070] Each thread pool has a unique name, which is stored in a cache. The cache provides fast access and retrieval of thread pools, which is especially important for scheduled tasks.

[0071] The system might set up a scheduled task (for example, to run every 10 minutes) that checks the thread pool names in the cache and determines which thread pools have completed their task queues.

[0072] If all tasks of a thread pool are completed, the scheduled task will close the thread pool to release resources, such as releasing occupied thread resources.

[0073] When you close a thread pool, the system clears the corresponding thread pool name from the cache to ensure that only active thread pool information is retained in the cache.

[0074] Figure 3 A schematic diagram of the implementation process of the timing task provided in the embodiment of the present application is as follows: Figure 3 As shown, the system continuously monitors the task queue in the thread pool to determine whether all tasks have been completed.

[0075] Once it is detected that all the task queues in the thread pool have been executed, the system will trigger a scheduled task to close the thread pool and release all resources associated with it. When closing the thread pool, the scheduled task will remove all information related to the thread pool from the cache, including the thread pool name, task status, and task type.

[0076] The scheduled task is executed every 10 minutes by default. This is an adjustable setting that can be optimized according to the actual workload and performance requirements of the system. System administrators can adjust the execution frequency of scheduled tasks based on system monitoring results or changes in business needs.

[0077] The model training task queue control device provided by the present invention is described below. The model training task queue control device described below and the model training task queue control method described above can be referenced to each other.

[0078] Figure 4 A schematic diagram of the structure of the model training task queue control device provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, including: The query module 410 is used to read the queue task data corresponding to the training task thread pool in the queue database according to the thread pool name of the training task thread pool queried from the cache when the first preset time interval is met, and run the queue task data; The control module 420 is used to modify the task status of the training task thread pool to task completion and close the training task thread pool when all task queues corresponding to the training task thread pool have completed operation; The queue database is stored in a storage device different from the cache.

[0079] According to the model training task queue control device provided by the present application, the device is also used for: In the case of creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database; The thread pool name, task status and task type corresponding to the new training task thread pool are written into the cached queue data table.

[0080] According to the model training task queue control device provided by the present application, the device is also used for: When there is no running queue in the cached thread pool, the target queue task data corresponding to the new training task thread pool is taken out from the queue database, and the target queue task data is run; wherein the target queue task data is the queue task data whose task status is the storage status; After the target queue task data is executed, the task state of the target queue task data is adjusted from the in-warehouse state to the out-warehouse state, and the target queue task data corresponding to the new training task thread pool is continued to be read and executed; Within the second preset time period, if the target queue task data corresponding to the new training task thread pool is not read, the task status of the queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

[0081] According to the model training task queue control device provided by the present application, the device is also used for: In the queue database, the queue task data corresponding to the training task thread pool are read in sequence according to the order of the queue task data corresponding to the training task thread pool, and the queue task data are run.

[0082] According to the model training task queue control device provided by the present application, the device is also used for: Queue the queue task data corresponding to the new training task thread pool; According to the order of task queuing, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

[0083] According to the model training task queue control device provided by the present application, the device is also used for: Remove the thread pool name, task status and task type of the training task thread pool from the cached queue data table.

[0084] The embodiment of the present application effectively avoids the problem of queue task data occupying flash memory by storing the queue task data corresponding to the training task thread pool in the queue database instead of the cache, thereby avoiding the problem of flash memory overflow. In addition, by ensuring the sequential execution of the task queue and closing the thread pool in time, system crashes caused by resource exhaustion can be avoided, thereby enhancing the stability of the system. By intelligently managing the thread pool and task queue, the efficiency and stability of the AI ​​training platform are improved, and the operating experience of algorithm personnel is also improved.

[0085] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the model training task queue control method, the method comprising: when a first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, reading the queue task data corresponding to the training task thread pool in the queue database, and running the queue task data; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and the training task thread pool is closed; The queue database is stored in a storage device different from the cache.

[0086] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model training task queue control method provided by the above methods, and the method includes: when a first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, reading the queue task data corresponding to the training task thread pool in the queue database, and running the queue task data; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and the training task thread pool is closed; The queue database is stored in a storage device different from the cache.

[0088] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the model training task queue control method provided by the above methods, the method comprising: when a first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, reading the queue task data corresponding to the training task thread pool in the queue database, and running the queue task data; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and the training task thread pool is closed; The queue database is stored in a storage device different from the cache.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training task queue control method, characterized in that: include: When the first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, the queue task data corresponding to the training task thread pool is read from the queue database, and the queue task data is run; When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and the training task thread pool is closed; The queue database is stored in a storage device different from the cache.

2. The model training task queue control method according to claim 1, characterized in that: When the first preset time interval is met, according to the thread pool name of the training task thread pool queried from the cache, the queue task data corresponding to the training task thread pool is read in the queue database, and before the step of running the queue task data, the method further includes: In the case of creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database; The thread pool name, task status and task type corresponding to the new training task thread pool are written into the cached queue data table.

3. The model training task queue control method according to claim 2, characterized in that: After the step of writing the thread pool name, task status and task type corresponding to the new training task thread pool into the queue data table in the cache, the method further includes: When there is no running queue in the cached thread pool, the target queue task data corresponding to the new training task thread pool is taken out from the queue database, and the target queue task data is run; wherein the target queue task data is the queue task data whose task status is the storage status; After the target queue task data is executed, the task state of the target queue task data is adjusted from the in-warehouse state to the out-warehouse state, and the target queue task data corresponding to the new training task thread pool is continued to be read and executed; Within the second preset time period, if the target queue task data corresponding to the new training task thread pool is not read, the task status of the queue task data corresponding to the new training task thread pool is adjusted to task completion, and the new training task thread pool is closed.

4. The model training task queue control method according to claim 3, characterized in that: The queue data table includes: thread pool name, thread pool task type, thread pool task type, queue data identity information; The thread pool task type includes at least one of the following: a copy task type and a compression task type; The thread pool task type includes at least one of the following: storage status, outbound status, and task completion status.

5. The model training task queue control method according to claim 1, characterized in that: Reading queue task data corresponding to the training task thread pool in the queue database and running the queue task data includes: In the queue database, the queue task data corresponding to the training task thread pool are read in sequence according to the order of the queue task data corresponding to the training task thread pool, and the queue task data are run.

6. The model training task queue control method according to claim 3, characterized in that: In the case of creating a new training task thread pool, storing the queue task data corresponding to the new training task thread pool in the queue database includes: Queue the queue task data corresponding to the new training task thread pool; According to the order of task queuing, the queue task data corresponding to the new training task thread pool is stored in the queue database in sequence.

7. The model training task queue control method according to claim 1, characterized in that: When all task queues corresponding to the training task thread pool have completed running, the task status of the training task thread pool is changed to task completed, and after the step of closing the training task thread pool, the step further includes: Remove the thread pool name, task status and task type of the training task thread pool from the cached queue data table.

8. A model training task queue control device, characterized in that: include: A query module, configured to read queue task data corresponding to the training task thread pool in the queue database according to the thread pool name of the training task thread pool queried from the cache and run the queue task data when a first preset time interval is met; A control module, used for modifying the task status of the training task thread pool to task completion and closing the training task thread pool when all task queues corresponding to the training task thread pool have completed operation; The queue database is stored in a storage device different from the cache.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the model training task queue control method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training task queue control method as described in any one of claims 1 to 7.

Images