A data caching method and device for multiple concurrent deep learning training tasks

Abstract

The present invention discloses a data caching method and device for multiple concurrent deep learning training tasks, the method comprising: step one, performing warm-up training on each task in the multiple concurrent tasks, collecting characteristic parameters of task training batch samples, and sorting all tasks according to the characteristic parameters; step two, calculating the number of samples of each training batch sample in the cache space of each task under system pre-allocation and the number of samples of each training batch sample expected by each task in the cache space; step three, using cache dynamic allocation and management strategy to concurrently perform deep learning training of tasks; step four, when each task enters the last training cycle, no new sample data is added to the cache space of these tasks, and as the sample data in the cache space is gradually used and consumed, the occupied cache space is gradually released, and the released cache space is used by other tasks that have not yet ended. The present invention improves the utilization rate of the global cache space.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a data caching method and device for multiple concurrent deep learning training tasks. Background Art

[0002] As an important branch of machine learning, performance optimization of deep learning has been a research hotspot in recent years. Deep learning training tasks cover multiple stages such as data I / O, CPU computing, and GPU computing. As the performance of components such as CPUs and GPUs continues to improve rapidly, the I / O bottleneck problem of deep learning training has become increasingly prominent.

[0003] Cache technology is an important means to alleviate and eliminate I / O bottlenecks, but existing caches for deep learning training face the problem of "over-hitting". Specifically, during a training cycle epoch, some training batches have a large proportion of samples that hit the cache, making the data loading phase of these batches significantly shorter than the data enhancement phase or model training phase, while the situation for other batches is the opposite. This unbalanced cache usage causes the first part of the batch to waste limited cache resources. This phenomenon is called cache "over-hitting".

[0004] In addition, it is becoming more and more common for multiple deep learning training tasks to be executed concurrently. These tasks are independent of each other and may use different data sets, perform different enhancement operations, and use different model training. When these tasks are executed concurrently, a common method is to pre-allocate cache for them in a predetermined ratio based on the size of the task's data set. However, the cache utilization of this static cache allocation scheme needs to be improved: on the one hand, the cache size required for the task depends not only on the size of the data set, but also on the time overhead of the task's data enhancement phase and model training phase; on the other hand, deep learning training is cyclical, and the average interval time between two consecutive references of samples of different tasks is often different, which makes the average residence time of samples of different tasks in the cache different. Using this rule to dynamically allocate cache among multiple tasks can further improve the utilization of global cache space.

[0005] The cache design problem for concurrent deep learning training is a current research hotspot. The most representative work is Quiver, which uses the substitutability of samples to ensure that all concurrent tasks can quickly obtain samples from the cache, thereby reducing the time overhead of the task I / O stage and alleviating the task's I / O bottleneck. However, Quiver has obvious disadvantages. On the one hand, its applicable application scenarios are very narrow, that is, multiple tasks sharing the cache need to use the same data set; on the other hand, it destroys the global randomness of task access to samples during each epoch, which may have an adverse effect on the accuracy of model training. Therefore, how to dynamically allocate and manage cache for multiple concurrent deep learning training tasks has become an urgent problem to be solved. Summary of the invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a data caching method and device for multiple concurrent deep learning training tasks, dynamically allocates and manages cache for concurrent deep learning training tasks, and improves the utilization efficiency of the cache for each task by solving the "over-hit" problem, thereby alleviating and eliminating the I / O bottleneck of deep learning training tasks to the greatest extent. The specific technical solution is as follows:

[0007] A data caching method for multiple concurrent deep learning training tasks, comprising the following steps:

[0008] Step 1: For the sample set of each task in the multiple concurrent tasks, perform a warm-up training for a training cycle separately, collect the feature parameters of the training batch samples, and sort all the tasks to generate a list based on the collected feature parameters;

[0009] Step 2: Calculate the average number of samples of each training batch in the cache space of each task under the default cache allocation scheme and the expected number of samples of each training batch in the cache space of each task;

[0010] Step 3: Based on the two parameters calculated in step 2, multiple concurrent tasks use cache dynamic allocation and management strategies to concurrently execute deep learning training;

[0011] Step 4: When each task enters the last training cycle, no new sample data is added to the cache space of each task. At the same time, as the sample data in the cache space is gradually used and consumed, the occupied cache space is gradually released, and the released cache space can be used by other tasks that have not yet ended.

[0012] Furthermore, the step 1 specifically includes the following sub-steps:

[0013] Step S11, obtain the initial parameter configuration, the total number of concurrent tasks is recorded as , for the tasks The total number of samples in the dataset used is recorded as , the number of samples contained in a training batch is recorded as , the maximum number of samples that can be stored in the system pre-allocated cache space is recorded as ;

[0014] Step S12: warm-up training does not use any cache. When each task warm-up training is completed, its information is counted: task The time required to execute a training cycle alone is recorded as ; The average I / O time to load a training batch is denoted as ; The average time to load a sample is denoted by ; The average time to enhance a training batch is denoted by ; The average time to train a training batch is denoted by ;

[0015] Step S13, according to the task obtained in step S12 The time required to perform one training cycle , sort all tasks in ascending order to get an ordered task list:

[0016] ,

[0017] And each task in the list Contains a parameter As the gain coefficient of the task's request for cache space from the free cache pool, that is, whenever the task requests a sample's space from the free cache pool, the free cache pool will allocate times the cache space for this task, and at the same time, The value of are negatively correlated, and =0.

[0018] Furthermore, the step 2 specifically includes the following sub-steps:

[0019] Step S21, calculate each task The number of samples that hit the cache space for each training batch sample under the default cache allocation scheme, i.e., system pre-allocation , the expression is:

[0020] ,

[0021] For sorted tasks The dataset used contains the total number of samples, For sorted tasks The number of samples contained in a training batch of For sorted tasks The number of samples that can be stored in the system's pre-allocated buffer space;

[0022] Step S22, calculate each task The expected number of samples in the cache space for each training batch , the expression is:

[0023] .

[0024] Furthermore, the step three specifically includes the following sub-steps:

[0025] Step S31, the free cache spaces of multiple concurrent tasks are combined into a global free cache pool, the total size of which is denoted as totalMem. The cache space of each task can be logically divided into two parts, denoted as and ,in It stores the samples that entered the cache space in the previous training cycle and will be used in the current training cycle. Save the samples that enter the cache in the current training cycle and will be used in the next training cycle; where the global free cache pool totalMem is initially calculated by the following formula:

[0026]

[0027] Step S32, for the task , holds two sample access sequences in each training cycle, one of which indicates the sample access order in the current training cycle, denoted as , and the other indicates the sample access order in the next training cycle, denoted as , The task is divided into different sequence segments from the beginning to the end, each segment corresponds to a training batch, and each segment is configured with a counter to record the number of samples of the training batch that enter the cache space in the current training cycle. When a training cycle starts, all counters of the task are reset to zero, and then the process goes to step S33;

[0028] Step S33, if the task Requested Sample In its cache space Hit, then from Get the hit sample and increase the free cache pool totalMem by one, otherwise load the sample from the underlying storage system; then go to step S34;

[0029] Step S34, in task The sample access order in the next training cycle Sample retrieval request , calculate the requested sample The training batch in the next training cycle is denoted by , and then get the training batch The counter value is recorded as , and proceed to step S35;

[0030] Step S35, when and When there is no free space, go to step S36; when totalMem> 0, if , then the task Request space from the free buffer pool according to its gain coefficient (If the cache pool is empty, the request fails) and then the requested sample insert ,Then Add 1, update totalMem, and go to step S38; if ,but Do not enter the task Cache space, and go to step S38;

[0031] Step S36: If , requested sample Do not enter the task Cache space, and go to step S38; if , then go to step S37;

[0032] Step S37, if the task of Contains free space, then the sample Enter , and go to step S38; otherwise, if the task yes The first task in the list (i.e., i=0), then the requested sample Do not enter Cache space, and go to step S38; otherwise Previous task in the list of A free space is required for The specific approach is if If there is free space, move the free space of one unit directly to Otherwise, randomly select One of the samples in is eliminated, the counter of the training batch corresponding to the eliminated sample is reduced by one, and the vacated cache space is moved to , and the sample insert , the counter of the corresponding training batch is increased by one, and the process goes to step S38;

[0033] Step S38, Task Requested Sample Enter the subsequent enhancement phase and model training phase;

[0034] Step S39, when the task After the training of the current training cycle is completed, if the training of all training cycles has been completed, the task End; otherwise, go to step S32 to perform the task Training for the next training cycle.

[0035] Furthermore, the cache space of each task in the multiple concurrent tasks is isolated from each other, and can only access samples in the respective cache space.

[0036] Furthermore, for each task in its current training cycle, the samples entering the cache space are relatively evenly distributed in all training batches of the next training cycle. Tasks at the front of the list apply for free cache space from the free cache pool at a faster speed, while tasks at the back of the list can force other tasks ahead of them in the list to return part of the cache space.

[0037] A data caching device for multiple concurrent deep learning training tasks includes one or more processors for implementing a data caching method for multiple concurrent deep learning training tasks.

[0038] A computer-readable storage medium stores a program, which, when executed by a processor, implements a data caching method for multiple concurrent deep learning training tasks.

[0039] The advantages and beneficial effects of the present invention are:

[0040] The present invention designs a cache dynamic allocation and management strategy for multiple concurrent deep learning training tasks. For any training task therein, the present invention accurately selects samples to enter the cache in each training cycle, so that they are distributed as evenly as possible in all training batches of the next training cycle, thereby solving the cache "over-hit" problem of each task and improving the utilization of the cache space; on this basis, the present invention designs a real-time dynamic cache allocation strategy for multiple concurrent training tasks, so that any task can lend its cache space to other tasks at an appropriate time, and borrow cache space from other tasks when it needs cache space, thereby making full use of the cache space of all concurrent tasks, and ensuring that the cache space actually used by each task is not less than the cache space pre-allocated by the system, further improving the utilization of the global cache space. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1It is a schematic diagram of the training process of multiple concurrent tasks based on cache dynamic allocation and management strategy of the present invention;

[0042] Figure 2 It is a schematic diagram of main parameter configuration of multiple concurrent deep learning training tasks in an embodiment of the present invention;

[0043] Figure 3 It is a flowchart of a data caching method for multiple concurrent deep learning training tasks according to an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of a framework in which three concurrent tasks of an embodiment of the present invention adopt a cache dynamic allocation and management strategy;

[0045] Figure 5 It is a schematic diagram of the cache processing flow of each task in multiple concurrent tasks in an embodiment of the present invention;

[0046] Figure 6 It is a schematic diagram of the structure of a data caching device for multiple concurrent deep learning training tasks according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0048] A method for dynamic cache allocation and management for multiple concurrent deep learning training tasks aims to improve the efficiency of cache usage by deep learning training tasks, use cache to speed up the data loading phase of all tasks, and reduce or eliminate their I / O bottlenecks. Figure 1 As shown, this method collects characteristic parameters of each task through warm-up training, and then configures and initializes the cache allocation and management strategy according to these parameters. Finally, multiple tasks perform concurrent training based on real-time cache dynamic allocation and management strategy.

[0049] The method proposed in the present invention isolates the caches of different tasks from each other. For each task, in its current training cycle, it ensures that the samples received by its cache are distributed as evenly as possible in all batches of its next training cycle, thereby solving the problem of cache "over-hit". At the same time, it dynamically allocates cache resources between different tasks in real time, on the one hand solving the imbalance problem caused by the default cache static pre-allocation strategy, and on the other hand using the characteristics of the task to improve the use efficiency of the entire cache space.

[0050] The device of the present invention can be deployed on the Pytorch platform. On a single physical node, each concurrent deep learning training task has an independent GPU and CPU. The ImageNet dataset is used, and the training model is ResNet. The main parameter configuration is as follows: Figure 2As shown, in this scenario, the method of the present invention, as Figure 3 As shown, the following steps are included:

[0051] Step 1: For the sample set of each task in the multiple concurrent tasks, perform a warm-up training for a training cycle separately, and collect the feature parameters of the training batch samples. According to the collected feature parameters, sort all tasks to generate a list, which specifically includes the following sub-steps:

[0052] Step S11, obtain the initial parameter configuration, the total number of concurrent tasks is In this embodiment, the value of M is 3, for which the task , i∈ The total number of samples in the dataset used is recorded as , the number of samples contained in a training batch is recorded as , the number of samples that can be stored in the system pre-allocated cache space is recorded as ;

[0053] Step S12: When each task warm-up training is completed, its information is counted: the time required for the task to execute a training cycle alone is recorded as ; The average I / O time to load a training batch is denoted as ; The average time to load a sample is denoted by ; The average time to enhance a training batch is denoted by ; The average time to train a training batch is denoted by ;

[0054] Step S13, according to the task obtained in step S12 The time required to perform one training cycle , sort all tasks in ascending order to get an ordered task list ,

[0055] This embodiment assumes that the statistical information satisfies

[0056] ,

[0057] Then, the list is

[0058] ,

[0059] And each task in the list Contains a parameter As the gain coefficient of the task's request for cache space from the free cache pool, that is, whenever the task requests a sample's space from the free cache pool, the free cache pool will allocate times the cache space , The value of are negatively correlated, and =0, in this embodiment, the gain coefficient of each task in List can be set to <0.8, 0.4, 0>.

[0060] Step 2: Calculate the average number of samples of each training batch in the cache space of each task under the default cache allocation scheme and the expected number of samples of each training batch in the cache space of each task;

[0061] Specifically, it includes the following sub-steps:

[0062] Step S21, calculate each task The number of samples that hit the cache space for each training batch sample under the default cache allocation scheme (i.e., system pre-allocation) , the expression is:

[0063] ,

[0064] in, For sorted tasks The dataset used contains the total number of samples, For sorted tasks The number of samples contained in a training batch of For sorted tasks The number of samples that can be stored in the system's pre-allocated buffer space;

[0065] Step S22, calculate each task The expected number of samples in the cache space for each training batch , the expression is:

[0066] .

[0067] Step three, such as Figure 4 As shown, based on the two parameters calculated in step 2, multiple concurrent tasks use a cache dynamic allocation and management strategy to concurrently execute deep learning training; in multiple concurrent tasks, the cache space of each task is isolated from each other, and only samples in their own cache space can be accessed. During the current training cycle, each task makes the samples entering the cache space as evenly as possible in all training batches of the next training cycle. Tasks at the front of the list apply for free cache space from the free cache pool at a faster speed, and when the free cache pool is empty, tasks at the back can force other tasks ahead of them in the list to return part of the cache space.

[0068] The step three includes the following sub-steps:

[0069] Step S31, the free cache spaces of multiple concurrent tasks are combined into a global free cache pool, the total size of which is denoted as totalMem. The cache space of each task can be logically divided into two parts, denoted as and ,in It stores the samples that entered the cache space in the previous training cycle and will be used in the current training cycle. Save the samples that enter the cache in the current training cycle and will be used in the next training cycle; where the global free cache pool totalMem is initially calculated by the following formula:

[0070]

[0071] Step S32, as Figure 5 As shown, for the task , holds two sample access sequences in each training cycle, one of which indicates the sample access order in the current training cycle, denoted as , and the other indicates the sample access order in the next training cycle, denoted as , The task is divided into different sequence segments from the beginning to the end, each segment corresponds to a training batch, and each segment is configured with a counter to record the number of samples of the training batch that enter the cache space in the current training cycle. When a training cycle starts, all counters of the task are reset to zero, and then the process goes to step S33;

[0072] Step S33, if the task Requested Sample In its cache space Hit, then from Get the hit sample and increase the free cache pool totalMem by one, otherwise load the sample from the underlying storage system; then go to step S34;

[0073] Step S34, in task The sample access order in the next training cycle Sample retrieval request , calculate the requested sample The training batch in the next training cycle is denoted by , and then get the training batch The counter value is recorded as , and proceed to step S35;

[0074] Step S35, when and When there is no free space, go to step S36; when totalMem> 0, if , then the task Request space from the free buffer pool according to its gain coefficient (If the cache pool is empty, the request fails) and then the requested sample insert ,Then Add 1, update totalMem, and go to step S38; if ,but Do not enter the task Cache space, and go to step S38;

[0075] Step S36: If , requested sample Do not enter the task Cache space, and go to step S38; if , then go to step S37;

[0076] Step S37, if the task of Contains free space, then the sample Enter , and go to step S38; otherwise, if the task yes The first task in the list (i.e., i=0), then the requested sample Do not enter Cache space, and go to step S38; otherwise Previous task in the list of A free space is required for The specific approach is if If there is free space, move the free space of one unit directly to Otherwise, randomly select One of the samples in is eliminated, the counter of the training batch corresponding to the eliminated sample is reduced by one, and the vacated cache space is moved to , and the sample insert , the counter of the corresponding training batch is increased by one, and the process goes to step S38;

[0077] Step S38, Task Requested Sample Enter the subsequent enhancement phase and model training phase;

[0078] Step S39, when the task After the training of the current training cycle is completed, if the training of all training cycles has been completed, the task End; otherwise, go to step S32 to perform the task Training for the next training cycle.

[0079] Step 4: When each task enters the last training cycle, no new sample data is added to the cache space of each task. At the same time, as the sample data in the cache space is gradually used and consumed, the occupied cache space is gradually released, and the released cache space can be used by other tasks that have not yet ended.

[0080] Corresponding to the aforementioned embodiment of a data caching method for multiple concurrent deep learning training tasks, the present invention also provides an embodiment of a data caching device for multiple concurrent deep learning training tasks.

[0081] See also Figure 6 , an embodiment of the present invention provides a data caching device for multiple concurrent deep learning training tasks, including one or more processors, for implementing a data caching method for multiple concurrent deep learning training tasks in the above embodiment.

[0082] An embodiment of a data caching device for multiple concurrent deep learning training tasks of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, if Figure 6 As shown, it is a hardware structure diagram of a data cache device for multiple concurrent deep learning training tasks in the present invention, where any device with data processing capability is located, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0083] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0084] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0085] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a data caching method for multiple concurrent deep learning training tasks in the above embodiment is implemented.

[0086] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of a wind turbine, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0087] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data caching method for multiple concurrent deep learning training tasks, characterized in that: The following steps are involved: Step 1: For the sample set of each task in the multiple concurrent tasks, perform a warm-up training for a training cycle separately, collect the feature parameters of the training batch samples, and sort all the tasks to generate a list based on the collected feature parameters; Step 2: Calculate the average number of samples of each training batch in the cache space of each task under the default cache allocation scheme and the expected number of samples of each training batch in the cache space of each task; Step 3: Based on the two parameters calculated in step 2, multiple concurrent tasks use cache dynamic allocation and management strategies to concurrently execute deep learning training; Step 4: When each task enters the last training cycle, no new sample data is added to the cache space of each task. At the same time, as the sample data in the cache space is gradually used and consumed, the occupied cache space is gradually released, and the released cache space is used by other tasks that have not yet ended; The step 1 specifically includes the following sub-steps: Step S11, obtain the initial parameter configuration, the total number of concurrent tasks is recorded as , for the tasks The total number of samples in the dataset used is recorded as , the number of samples contained in a training batch is recorded as , the maximum number of samples that can be stored in the system pre-allocated cache space is recorded as ; Step S12: warm-up training does not use any cache. When each task warm-up training is completed, its information is counted: task The time required to execute a training cycle alone is recorded as ; The average I / O time to load a training batch is denoted as ; The average time to load a sample is denoted by ; The average time to enhance a training batch is denoted by ; The average time to train a training batch is denoted by ; Step S13, according to the task obtained in step S12 The time required to perform one training cycle , sort all tasks in ascending order to get an ordered task list: ， And each task in the list Contains parameters As the gain coefficient of the task's request for cache space from the free cache pool, that is, whenever the task requests a sample's space from the free cache pool, the free cache pool will allocate times the cache space for this task, and at the same time, The value of are negatively correlated, and =0; The step 2 specifically includes the following sub-steps: Step S21, calculate each task The number of samples that hit the cache space for each training batch sample under the default cache allocation scheme, i.e., system pre-allocation , the expression is: ， For sorted tasks The dataset used contains the total number of samples, For sorted tasks The number of samples contained in a training batch of For sorted tasks The number of samples that can be stored in the system's pre-allocated buffer space; Step S22, calculate each task The expected number of samples in the cache space for each training batch , the expression is: ； The step three specifically includes the following sub-steps: Step S31, the free cache spaces of multiple concurrent tasks are combined into a global free cache pool, the total size of which is denoted as totalMem. The cache space of each task can be logically divided into two parts, denoted as and ,in It stores the samples that entered the cache space in the previous training cycle and will be used in the current training cycle. Save the samples that enter the cache in the current training cycle and will be used in the next training cycle; where the global free cache pool totalMem is initially calculated by the following formula: ； Step S32, for the task , holds two sample access sequences in each training cycle, one of which indicates the sample access order in the current training cycle, denoted as , and the other indicates the sample access order in the next training cycle, denoted as , The task is divided into different sequence segments from the beginning to the end, each segment corresponds to a training batch, and each segment is configured with a counter to record the number of samples of the training batch that enter the cache space in the current training cycle. When a training cycle starts, all counters of the task are reset to zero, and then the process goes to step S33; Step S33, if the task Requested Sample In its cache space Hit, then from Get the hit sample and increase the free cache pool totalMem by one, otherwise load the sample from the underlying storage system; then go to step S34; Step S34, in task The sample access order in the next training cycle Sample retrieval request , calculate the requested sample The training batch in the next training cycle is denoted by , and then get the training batch The counter value is recorded as , and proceed to step S35; Step S35, when and When there is no free space, go to step S36; when totalMem > 0, if , then the task Request space from the free buffer pool according to its gain coefficient If the cache pool is empty, the request fails and the requested sample is insert ,Then Add 1, update totalMem, and go to step S38; if ,but Do not enter the task Cache space, and go to step S38; Step S36: If , requested sample Do not enter the task Cache space, and go to step S38; if , then go to step S37; Step S37, if the task of Contains free space, then the sample Enter , and proceed to step S38; On the contrary, if the task yes The first task in the list, that is, i=0, then the requested sample Do not enter Cache space, and go to step S38; otherwise Previous task in the list of A free space is required for The specific approach is if If there is free space, move the free space of one unit directly to Otherwise, randomly select One of the samples in is eliminated, the counter of the training batch corresponding to the eliminated sample is reduced by one, and the vacated cache space is moved to , and the sample insert , the counter of the corresponding training batch is increased by one, and the process goes to step S38; Step S38, Task Requested Sample Enter the subsequent enhancement phase and model training phase; Step S39, when the task After the training of the current training cycle is completed, if the training of all training cycles has been completed, the task End; otherwise, go to step S32 to perform the task Training for the next training cycle.

2. A data caching method for multiple concurrent deep learning training tasks as claimed in claim 1, characterized in that: The cache space of each task in the multiple concurrent tasks is isolated from each other, and can only access samples in the respective cache space.

3. A data caching method for multiple concurrent deep learning training tasks as claimed in claim 1, characterized in that: For each task in its current training cycle, the samples entering the cache space are relatively evenly distributed in all training batches of the next training cycle. The tasks at the front of the list apply for free cache space from the free cache pool at a faster speed. When the free cache pool is empty, the tasks at the back can force other tasks ahead of it in the list to return part of the cache space.

4. A data caching device for multiple concurrent deep learning training tasks, characterized in that: It includes one or more processors for implementing a data caching method for multiple concurrent deep learning training tasks as described in any one of claims 1-3.

5. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a data caching method for multiple concurrent deep learning training tasks as described in any one of claims 1-3 is implemented.

Images