Tensor management method, electronic device, storage medium and program product

Abstract

The present disclosure provides a tensor management method, electronic device, storage medium, and program product. The tensor management method of the present disclosure includes: during the training of a target model, obtaining the training progress and tensor unloading rules of the target model; the tensor unloading rules are generated based on the unloading benefit-to-cost ratio determined by the tensor lifecycle of the target model; obtaining the stage unloading rules of the training progress from the tensor unloading rules; obtaining the resident tensors of the target model in the video memory; determining the tensors to be unloaded from the resident tensors according to the stage unloading rules; and unloading the tensors to be unloaded to the target storage device.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a tensor management method, electronic device, storage medium, and program product. Background Art

[0002] A Deep Neural Network (DNN) model is a neural network consisting of an input layer, multiple hidden layers, and an output layer. DNN models offer advantages such as strong expressiveness (capable of modeling complex nonlinear systems), adaptability, scalability, and the ability to process large amounts of data. DNN models are the foundation of numerous AI applications, including computer vision, speech recognition, natural language processing, recommendation systems, and medical diagnostics. With the widespread adoption of DNN models, the efficiency of their training and computation has become a key issue in improving user experience.

[0003] In a DNN model, tensors are the most basic form of data representation, similar to multidimensional arrays or matrices. Whether it is the input data of a DNN model, model parameters (such as weights and biases), or intermediate calculation results (such as activation values and gradients), they all exist in the form of tensors. During the training process of a DNN model, each forward propagation and backpropagation involves the creation, reading, modification, and writing of a large number of tensors. These operations are frequent and involve large amounts of data. Especially in large models or large batch training, the number and size of tensors will increase significantly. Because graphics processing units (GPUs) can process a large number of matrix and tensor operations in parallel, the training process of DNN models is mostly performed on GPUs, thereby improving training speed.

[0004] However, as the DNN model deepens and the batch size increases, tensors such as activation values, intermediate results, and gradients need to continue to occupy a large amount of GPU memory (i.e., video memory). When there are too many tensors or the size is too large, it is easy to cause video memory overflow, making the DNN model training process unable to run normally, affecting the efficiency of DNN model training. Summary of the Invention

[0005] The present disclosure provides a tensor management method, an electronic device, a storage medium, and a program product.

[0006] According to one aspect of the present disclosure, a tensor management method is provided, including: during a target model training process, obtaining the training progress and tensor unloading rules of the target model; the tensor unloading rules are generated according to an unloading benefit-cost ratio determined based on the tensor life cycle of the target model; obtaining a stage unloading rule of the training progress from the tensor unloading rule; obtaining the resident tensors of the target model in the video memory; determining the tensors to be unloaded from the resident tensors according to the stage unloading rule; and unloading the tensors to be unloaded to a target storage device.

[0007] According to one aspect of the technical solution, unloading resident tensors in the video memory through tensor unloading rules can reduce the occupancy of the video memory, thereby alleviating the pressure on the video memory, reducing video memory overflow and improving the training efficiency of the model.

[0008] According to the tensor management method of at least one embodiment of the present disclosure, before training the target model, it also includes: obtaining the tensors of the target model and the various stages of the training process of the target model; respectively obtaining the tensor lifecycle of each tensor of the target model; respectively determining the unloading benefit-overhead ratio of the tensor at each stage according to the tensor lifecycle of each tensor; and generating tensor unloading rules according to the unloading benefit-overhead ratio of each tensor at each stage.

[0009] According to the technical solution of this embodiment, the stability of target model training can be improved and data support can be provided for video memory resource allocation.

[0010] According to the tensor management method of at least one embodiment of the present disclosure, for any tensor of the target model, the unloading benefit-overhead ratio of the tensor at each stage is determined according to the tensor life cycle of each tensor, including: obtaining the tensor size and tensor unloading overhead of the tensor; obtaining the tensor idle time of the tensor at each stage according to the tensor life cycle of the tensor; determining the tensor unloading benefit of the tensor at each stage according to the product of the tensor size and the tensor idle time of the tensor at each stage; determining the unloading benefit-overhead ratio of the tensor at each stage according to the quotient of the tensor unloading benefit of the tensor at each stage and the tensor unloading overhead; optionally, obtaining the tensor size and tensor unloading overhead of the tensor includes: obtaining the tensor size and network bandwidth of the tensor; determining the tensor unloading overhead of the tensor according to the quotient of 2 times the tensor size of the tensor and the network bandwidth.

[0011] According to the technical solution of this embodiment, when unloading tensors, tensors with relatively large unloading benefit overhead can be preferentially selected for unloading, thereby reducing the number of tensor unloading times, further reducing the unloading overhead, and improving tensor management efficiency.

[0012] According to the tensor management method of at least one embodiment of the present disclosure, after unloading the tensor to be unloaded to the target storage device, the method further includes: obtaining the tensor lifecycle and current time of the tensor to be unloaded; determining the remaining time based on the tensor lifecycle and current time of the tensor to be unloaded; judging whether the remaining time is greater than a time threshold; in response to the remaining time of the target tensor in the tensor to be unloaded being not greater than the time threshold, migrating the target tensor to the video memory; optionally, migrating the target tensor to the video memory includes: adding the target tensor to a prefetch queue; asynchronously taking out the tensor in the prefetch queue and adding it to the video memory.

[0013] According to the technical solution of this embodiment, it is possible to ensure that the target tensor arrives at the video memory in a timely manner while optimizing the utilization of the video memory without causing GPU calculation waiting. In addition, when video memory is given priority, the video memory computing performance is maximized to avoid video memory overflow or data transmission delay.

[0014] According to the tensor management method of at least one embodiment of the present disclosure, before training the target model, it also includes: in response to receiving a model training task, adding the model training task to the first queue; obtaining the real-time remaining amount of the video memory and the video memory required for each model training task in the first queue; determining the tasks to be trained and delayed training tasks from the model training tasks in the first queue based on the real-time remaining amount and the video memory required for each model training task in the first queue; migrating the delayed training tasks from the first queue to the second queue, and migrating the tasks to be trained from the first queue to the third queue; the second queue is used to manage the delayed training tasks, and migrate the delayed training tasks back to the first queue when the first migration indication is triggered; controlling the execution of the tasks to be trained in the third queue, where the tasks to be trained in the third queue include tasks for training the target model.

[0015] According to the technical solution of this embodiment, it is possible to reduce model training task conflicts, optimize load distribution, avoid resource overload, and improve the scalability and reliability of the tensor management system.

[0016] According to the tensor management method of at least one embodiment of the present disclosure, in controlling the execution of the tasks to be trained in the third queue, it also includes: monitoring whether the video memory has an over-resolution risk; in response to the existence of an over-resolution risk in the video memory, obtaining the task migration benefit ratio of each task to be trained in the third queue; determining the tasks to be migrated from the tasks to be trained in the third queue according to the task migration benefit ratio; migrating the tasks to be migrated from the third queue to the fourth queue; the fourth queue is used to manage the tasks to be migrated, and migrate the tasks to be migrated back to the third queue when the second migration indication is triggered.

[0017] According to the technical solution of this embodiment, video memory over-allocation can be avoided at a relatively low cost, thereby improving scheduling flexibility, avoiding system crashes, and maximizing overall benefits when video memory resources are tight.

[0018] According to the tensor management method of at least one embodiment of the present disclosure, the obtaining of the task migration benefit ratio of each task to be trained in the third queue includes: obtaining the growth amount of video memory demand and the historical number of migrations of each task to be trained in the third queue; adding 1 to the historical number of migrations of each task to be trained in the third queue to obtain the expected number of swap-outs for each task to be trained in the third queue; determining the task migration benefit ratio of the task to be trained based on the quotient of the growth amount of video memory demand and the expected number of swap-outs for each task to be trained in the third queue; optionally, monitoring whether there is an over-allocation risk of the video memory includes: obtaining the video memory demand of the task to be trained in the third queue and the real-time remaining amount of the video memory; obtaining the size relationship between the video memory demand and the real-time remaining amount; and determining whether there is an over-allocation risk of the video memory based on the size relationship. ; Optionally, obtaining the video memory requirement of the task to be trained in the third queue and the real-time remaining amount of the video memory includes: obtaining the total number of tensor elements, the size of tensor elements and the fixed video memory overhead of the task to be trained; determining the tensor video memory overhead according to the product of the total number of tensor elements and the size of tensor elements; determining the video memory requirement of the task to be trained according to the sum of the tensor video memory overhead and the fixed video memory overhead; Optionally, determining the task to be migrated out from the tasks to be trained in the third queue according to the task migration benefit ratio includes: sorting the tasks to be trained in the third queue according to the task migration benefit ratio of the tasks to be trained in the third queue to obtain a task sorting result; determining the number of tasks to be migrated out of the third queue according to the task migration rule; and determining the number of tasks to be migrated out from the task sorting result.

[0019] According to the technical solution of this embodiment, it is possible to give priority to migrating out tasks to be trained that have "high pressure, few swaps, and high migration benefits", thereby reducing the risk of over-allocation of video memory and avoiding problems such as frequent migration of tasks to be trained, which may result in the tasks to be trained not being executed and leading to training failure.

[0020] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the tensor management method of any embodiment of the present disclosure.

[0021] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the tensor management method of any embodiment of the present disclosure is implemented.

[0022] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the tensor management method according to any one of the embodiments of the present disclosure is implemented.

[0023] The tensor management method disclosed herein unloads resident tensors from video memory according to tensor offloading rules, reducing video memory usage and thus alleviating video memory pressure, thereby reducing video memory overflow and improving model training efficiency. This tensor management method addresses the existing problem that as DNN models deepen and batch sizes increase, tensors such as activation values, intermediate results, and gradients continue to occupy a large amount of GPU memory (i.e., video memory). When there are too many tensors or they are too large, they can easily lead to video memory overflow, preventing the DNN model training process from operating normally and affecting the efficiency of DNN model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0025] Figure 1 This is a process of a tensor management method according to an embodiment of the present disclosure. Figure 1 .

[0026] Figure 2 This is a process of a tensor management method according to an embodiment of the present disclosure. Figure 2 .

[0027] Figure 3 yes Figure 2 Flowchart of a method for calculating the offloading benefit-to-overhead ratio in the tensor management method shown.

[0028] Figure 4 yes Figure 3 The flowchart of the offloading overhead acquisition method in the offloading benefit-overhead ratio calculation method is shown.

[0029] Figure 5 This is a process of a tensor management method according to an embodiment of the present disclosure. Figure 3 .

[0030] Figure 6 yes Figure 5 Flowchart of the tensor migration method in the tensor management method shown.

[0031] Figure 7 This is a process of a tensor management method according to an embodiment of the present disclosure. Figure 4 .

[0032] Figure 8This is a process of a tensor management method according to an embodiment of the present disclosure. Figure 5 .

[0033] Figure 9 yes Figure 8 Flowchart of the method for calculating the task migration benefit ratio in the tensor management method shown.

[0034] Figure 10 yes Figure 8 Flowchart of the video memory monitoring method in the tensor management method shown.

[0035] Figure 11 yes Figure 9 The flowchart of the method for obtaining video memory requirements in the video memory monitoring method shown is shown.

[0036] Figure 12 yes Figure 8 Flowchart of the task determination method in the tensor management method shown.

[0037] Figure 13 is a schematic flowchart of a tensor management method according to an embodiment of the present disclosure.

[0038] Figure 14 yes Figure 13 Schematic diagram of task scheduling in the tensor management method shown.

[0039] Figure 15 It is a schematic block diagram of the structure of a tensor management device according to one embodiment of the present disclosure.

[0040] Figure 16 It is a schematic block diagram of the structure of an electronic device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The present disclosure is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0042] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] Suppose you are training a ResNet-152-based image classification model for autonomous vehicles to perceive and recognize traffic scenes, such as road signs, traffic lights, pedestrians, and obstacles. This image classification model has approximately 60.2 million parameters, and the training data consists of high-resolution RGB images with a resolution of 1024×1024. During training, using a batch size of 64, the input data alone occupies approximately 768MB of video memory, while the image classification model parameters and gradients occupy approximately 480MB of video memory. Intermediate activations and optimizer states occupy over 2GB of video memory. Including system overhead such as caches and temporary tensors, the total video memory usage can reach 7-8GB. Increasing the batch size to 128 to increase training speed would increase the total video memory usage to over 12GB. For GPUs with a maximum available video memory capacity of 12GB or less, this can easily trigger a video memory overflow error, causing training to abort and impacting the training efficiency of the image classification model.

[0044] To this end, the present disclosure proposes a tensor management method, electronic device, storage medium and program product. The present disclosure can be implemented by setting tensor management software on electronic devices such as servers and cloud computing platforms that have powerful parallel processing capabilities and video memory resources and can efficiently handle large-scale tensor computing tasks.

[0045] For the convenience of description and to make the technical solutions of the specific embodiments of the present disclosure easier to understand, the technical terms involved in the present disclosure are first explained as follows:

[0046] Tensors are multidimensional arrays or matrices used to store and transfer data.

[0047] The tensor lifecycle refers to the entire process from the creation to the destruction of a tensor, which usually includes the tensor active period when the tensor is frequently used, updated, and calculated, and the tensor inactive period when the tensor is no longer frequently involved in calculations.

[0048] Figure 1 FIG. 1 shows a schematic diagram of the overall process of a tensor management method M100 according to an embodiment of the present disclosure. Figure 1 The tensor management method shown includes steps S110 to S150 .

[0049] Specifically, Figure 1 The illustrated tensor management method includes the following.

[0050] Step S110: During the target model training process, the training progress and tensor unloading rules of the target model are obtained.

[0051] In some embodiments of the present disclosure, the target model in step S110 is typically a model from the fields of deep learning and machine learning, particularly a Deep Neural Network (DNN) model (such as AlexNet, VGGNet, ResNet, and other models) and their variants. The training progress of the target model can be pre-divided into multiple stages based on different tasks and optimization objectives, such as the initialization stage, the training stage, the convergence stage, and the overfitting stage. The training progress obtained in step S110 can be one of these pre-divided stages.

[0052] The tensor offloading rules in step S110 are generated based on the offloading benefit-to-overhead ratio determined based on the target model's tensor lifecycle. The offloading benefit-to-overhead ratio is the ratio of the benefit to the cost of tensor offloading, determined based on the tensor lifecycle. The benefit is the memory saved after offloading the tensor; larger tensors and longer idle times result in greater benefits; the cost is the latency of tensor offloading and prefetching.

[0053] Step S110 can obtain the training progress based on information such as the loss function or evaluation indicators. Specifically, when obtaining the training progress based on the loss function, it can be obtained based on the changes in the training loss and validation loss in the loss function; for example, when the training loss and validation loss are both high and there is no obvious downward trend, the obtained training progress is the initialization stage; when the training loss and validation loss are both decreasing, the obtained training progress is the training stage; when the training loss and validation loss are both low and tend to be stable, the obtained training progress is the convergence stage; when the training loss continues to decrease and the validation loss begins to increase or stagnates, the obtained training progress is the overfitting stage.

[0054] When obtaining training progress based on evaluation metrics, the metrics can be accuracy, recall, etc. Taking accuracy as an example, when the accuracy is low, the obtained training progress is the initialization stage; when the accuracy gradually increases, the obtained training progress is the training stage; when the accuracy tends to be stable and no longer changes significantly, the obtained training progress is the convergence stage.

[0055] Step S120 , obtaining a stage unloading rule of the training progress from the tensor unloading rule.

[0056] In some embodiments of the present disclosure, step S120 may use the training progress obtained in step S110 as a retrieval condition to obtain the stage offloading rule from the tensor offloading rule.

[0057] Step S130: Obtain the resident tensor of the target model in the video memory.

[0058] In some embodiments of the present disclosure, the resident tensors obtained in step S130 are tensors of the target model that are currently retained in the video memory and have not been released. Step S130 can use methods such as video memory usage analysis and tensor lifecycle analysis to obtain the resident tensors of the target model in the video memory.

[0059] Step S140 : determining the tensors to be unloaded from the resident tensors according to the stage unloading rule.

[0060] In some embodiments of the present disclosure, the process of determining the tensors to be unloaded through step S140 may include: first determining the unloading priority of each tensor in the resident tensors according to the stage unloading rule; and then determining a preset number or a preset proportion of tensors to be unloaded from the resident tensors according to the unloading priority from high to low.

[0061] Step S150: unload the tensor to be unloaded to the target storage device.

[0062] In some embodiments of the present disclosure, step S150 can be implemented by inserting an unload instruction during the compilation of the program for training the target model, thereby unloading the tensor to be unloaded. The target storage device in step S150 can be system memory; in particular, the target storage device can also be a solid-state drive (SSD). System memory is suitable for tasks with frequent tensor access or computationally intensive tasks, while SSDs are suitable for tasks requiring large amounts of storage space.

[0063] The tensor management method disclosed herein unloads resident tensors from video memory according to tensor offloading rules, reducing video memory usage and thus alleviating video memory pressure, thereby reducing video memory overflow and improving model training efficiency. This tensor management method addresses the existing problem that as DNN models deepen and batch sizes increase, tensors such as activation values, intermediate results, and gradients continue to occupy a large amount of GPU memory (i.e., video memory). When there are too many tensors or they are too large, they can easily lead to video memory overflow, preventing the DNN model training process from operating normally and affecting the efficiency of DNN model training.

[0064] Furthermore, the tensor management method provided by the present disclosure may further include the following steps before training the target model (i.e., before step S110): Figure 2 Steps S160 to S190 are shown.

[0065] Step S160, obtaining the tensor of the target model and each stage of the training process of the target model.

[0066] In some embodiments of the present disclosure, all tensors related to the target model can be obtained through step S160, such as input tensors for representing training data or verification data input, weight tensors for weighting and transforming input data, bias tensors for adjusting the output of each neuron, gradient tensors for updating weights and biases, etc.

[0067] The various stages of the training process of the target model obtained through step S160 include the stages corresponding to the training progress in step S110. The various stages of the training process of the target model can be pre-divided according to the training tasks and optimization goals of the target model.

[0068] Step S170 , respectively obtain the tensor lifecycle of each tensor of the target model.

[0069] In some embodiments of the present disclosure, step S170 may estimate the tensor lifecycle based on the computation graph of the target model and the execution time of the graphics processing unit (GPU) core program. Specifically, the process of obtaining the tensor lifecycle through step S170 may include: obtaining the computation graph of the target model and the execution time of the GPU core program; determining the execution order of key operations related to the tensor lifecycle based on the computation graph; obtaining time information of the key operations of each tensor of the target model based on the execution time of the GPU core program; constructing a timeline of the key operations of each tensor based on the execution order and time information; and determining the tensor lifecycle based on the timeline.

[0070] Step S180 , determining the unloading benefit-cost ratio of each tensor at each stage according to the tensor lifecycle of each tensor.

[0071] In some embodiments of the present disclosure, the offloading benefit-to-cost ratio in step S180 is an indicator of the potential benefit of offloading the tensor from the video memory to the target storage device, and can be used to evaluate whether the tensor is worth offloading.

[0072] Step S190 , generating a tensor offloading rule according to the offloading benefit-cost ratio of each tensor at each stage.

[0073] In some embodiments of the present disclosure, the tensor unloading rules in step S190 specify which tensors are unloaded at which stage during the target model training process. For any stage, step S190 may generate the tensor unloading rules by sorting the unloading benefit-to-cost ratio of each tensor in that stage from high to low, and then generating the tensor unloading rules. The tensor unloading rules may include the unloading target, the unloading quantity (or unloading ratio), etc.

[0074] In particular, in order to improve the efficiency of tensor management, steps S160 to S190 can be executed offline during the compilation phase of the target model training program, so that there is no need to make unloading decisions during the target model training process, and it is only necessary to execute according to the tensor unloading rules, thereby improving the stability of the target model training and providing data support for the allocation of video memory resources.

[0075] Steps S160 to S190 generate tensor unloading rules based on the unloading benefit-overhead ratio, so that tensors with relatively large unloading benefit-overhead are unloaded first during tensor unloading, so that the video memory can effectively support the target model training, reduce video memory overflow, and improve resource utilization and training efficiency.

[0076] For any tensor of the target model, regarding step S180, in some embodiments of the present disclosure, it may include the following: Figure 3 Steps S181 to S184 are shown.

[0077] Step S181: Obtain the tensor size and tensor unloading overhead of the tensor.

[0078] In some embodiments of the present disclosure, the tensor size obtained in step S181 is the size of the space occupied by the tensor in video memory, typically in bytes (B), KB, MB, or GB. The tensor size is typically the product of the number of elements in the tensor and the number of bytes occupied by each element. The tensor offload cost obtained in step S181 is the cost required to offload the tensor from video memory to the target storage device, typically including migration overhead and prefetch overhead.

[0079] Step S182: Obtain the tensor idle time of the tensor at each stage according to the tensor life cycle of the tensor.

[0080] In some embodiments of the present disclosure, the tensor idle time in step S182 is the time interval between the current time and the next time the tensor is used, and the tensor is usually idle at the current time; if the tensor is occupied at the current time, the tensor idle time is specifically the time interval between the current time and the next time the tensor is used.

[0081] Step S183 , determining the tensor unloading benefit of the tensor at each stage according to the product of the tensor size and the tensor idle time of the tensor at each stage.

[0082] Step S184 , determining the offloading benefit-to-overhead ratio of the tensor at each stage according to the quotient of the tensor offloading benefit and the tensor offloading overhead of the tensor at each stage.

[0083] By obtaining the tensor unloading benefit through steps S181 to S184, it is possible to give priority to unloading tensors with relatively large unloading benefit overhead when unloading tensors, thereby reducing the number of tensor unloading times, and further reducing the unloading overhead, and improving tensor management efficiency.

[0084] Regarding step S181, in some embodiments of the present disclosure, it may include the following Figure 4 Steps S1811 to S1812 are shown.

[0085] Step S1811: Obtain the tensor size and network bandwidth of the tensor.

[0086] In some embodiments of the present disclosure, the network bandwidth obtained in step S1811 is the upper limit of the amount of data that can be transmitted by the network per unit time. The network bandwidth can specifically be a theoretical network bandwidth or an actual network bandwidth.

[0087] Step S1812 : Determine the tensor offloading overhead of the tensor according to the quotient of twice the tensor size of the tensor and the network bandwidth.

[0088] Steps S1811 to S1812 determine the tensor offloading overhead based on the tensor size and network bandwidth, which can assist in offloading decisions.

[0089] Furthermore, the tensor management method provided by the present disclosure may further include the following steps after step S150: Figure 5 Steps S200 to S230 are shown.

[0090] Step S200: Obtain the tensor lifecycle and current time of the tensor to be unloaded.

[0091] Step S210 , determining the remaining time according to the tensor lifecycle of the tensor to be unloaded and the current time.

[0092] In some embodiments of the present disclosure, the remaining time in step S210 is used to determine how long the unloaded tensor will be used. The process of determining the remaining time in step S210 may include: determining the next use time of the tensor based on the tensor lifecycle; and determining the remaining time based on the difference between the next use time of the tensor and the current time.

[0093] Step S220: Determine whether the remaining time is greater than a time threshold.

[0094] In some embodiments of the present disclosure, the time threshold in step S220 may be pre-set, and the time threshold is usually related to the transmission bandwidth between the video memory and the target storage device.

[0095] When it is determined through step S220 that the remaining time of the target tensor in the tensors to be unloaded is not greater than the time threshold, step S230 is executed; otherwise, steps S200 to S220 are re-executed until the remaining time is not greater than the time threshold or the current training process of the target model is completed.

[0096] Step S230: Migrate the target tensor to the video memory.

[0097] This tensor management method dynamically migrates tensors between video memory and the target storage device through steps S110 to S150 and S200 to S230. It inserts corresponding tensor unload and prefetch instructions at the compiler intermediate expression level generated during the compilation and training of the target model program, enabling automatic management of tensors during target model training. For example, a prefetch instruction is inserted before a load or store instruction accessing a tensor, and an evict instruction is inserted after the tensor's lifecycle ends.

[0098] By controlling the time for migrating the target tensor in the tensors to be unloaded back to the video memory through steps S200 to S230, it is possible to ensure that the target tensor arrives at the video memory in a timely manner while optimizing the utilization of the video memory without causing GPU computing waits. In turn, when video memory is prioritized, the video memory computing performance is maximized to avoid video memory overflow or data transmission delays.

[0099] Regarding step S230, in some embodiments of the present disclosure, it may include the following Figure 6 Steps S231 to S232 are shown.

[0100] Step S231: Add the target tensor to the prefetch queue.

[0101] In some embodiments of the present disclosure, the prefetch queue in step S231 is a structure for asynchronous tensor migration.

[0102] Step S232 , asynchronously fetching the tensor in the pre-fetch queue and adding it to the video memory.

[0103] In some embodiments of the present disclosure, after the tensors in the prefetch queue are moved back to the video memory in step S232 , the tensor positions and the video memory mapping table of the tensors moved back to the video memory may be synchronously updated for subsequent use.

[0104] By asynchronously migrating tensors back to the video memory from step S231 to step S232, the tensors can be ready in the video memory when needed, thus avoiding the delay of computing while waiting for data to load. This can also reduce I / O bottlenecks, balance I / O loads, and thus improve throughput.

[0105] Furthermore, the tensor management method provided by the present disclosure may further include the following steps before training the target model: Figure 7 Steps S240 to S280 are shown.

[0106] Step S240: In response to receiving the model training task, add the model training task to the first queue.

[0107] In some embodiments of the present disclosure, the first queue in step S240 is used to manage all received model training tasks, and the first queue can be specifically a waiting queue.

[0108] Step S250: Obtain the real-time remaining amount of video memory and the video memory required for each model training task in the first queue.

[0109] In some embodiments of the present disclosure, the real-time remaining amount obtained by step S250 is the amount of video memory that has not been allocated or used; the video memory required for each model training task obtained by step S250 is the total amount of video memory required in the process of executing the model training task.

[0110] Step S260 , determining the tasks to be trained and delayed training tasks from the model training tasks in the first queue according to the real-time remaining amount and the video memory required by each model training task in the first queue.

[0111] In some embodiments of the present disclosure, step S260 can sort the video memory required for each model training task in the first queue from small to large, and determine whether the minimum video memory required is greater than the real-time remaining amount; if it is, all model training tasks in the current first queue are treated as delayed training tasks; if it is not, the model training task is treated as a task to be trained, and the latest real-time remaining amount is updated (the latest real-time remaining amount = the current real-time remaining amount - the minimum video memory required), and the judgment process is executed again based on the latest real-time remaining amount until all model training tasks in the first queue are determined to be completed.

[0112] Step S270: Migrate the delayed training task from the first queue to the second queue, and migrate the task to be trained from the first queue to the third queue.

[0113] In some embodiments of the present disclosure, the delayed training task indicates that the current video memory is insufficient to accommodate all the computing requirements of the task, and execution may cause video memory overflow. Therefore, it is necessary to migrate the delayed training task from the first queue to the second queue through step S270.

[0114] In step S270, the second queue is used to manage delayed training tasks and migrate the delayed training tasks back to the first queue when the first migration instruction is triggered. The second queue can be specifically a delay queue. The second queue can adopt a first-in-first-out strategy to ensure that any model training task is not delayed indefinitely.

[0115] After the delayed training task is migrated from the first queue to the second queue in step S270, the model training task in the second queue can be migrated back to the first queue according to a preset first recovery rule. The first recovery rule can be set based on one or more settings such as waiting time, video memory resources, and retry window.

[0116] In some embodiments of the present disclosure, the task to be trained indicates that the current video memory is sufficient to accommodate the computing requirements of the task, and the task to be trained can be executed smoothly. Therefore, the task to be trained can be migrated from the first queue to the third queue in step S270.

[0117] In step S270 , the third queue is used to control the execution of the task to be trained, and the third queue may be specifically an execution queue.

[0118] Step S280: Control the execution of the tasks to be trained in the third queue, where the tasks to be trained in the third queue include tasks for training the target model.

[0119] Steps S240 to S280 manage the execution of model training tasks through the first queue, the second queue, and the third queue, which can reduce model training task conflicts, optimize load distribution, avoid resource overload, and improve the scalability and reliability of the tensor management system.

[0120] Furthermore, the tensor management method provided by the present disclosure may further include the following steps in the process of controlling the execution of the tasks to be trained in the third queue through step S280: Figure 8 Steps S290 to S320 are shown.

[0121] Step S290: monitor whether the video memory has an over-allocation risk.

[0122] In some embodiments of the present disclosure, the over-allocation risk in step S290 is the risk that the total amount of memory requested by the application or system from the video memory exceeds the actual physical capacity of the video memory. Step S290 can determine whether there is an over-allocation risk for the video memory based on continuous monitoring of the video memory occupancy rate; for example, an over-allocation threshold can be pre-set. When the video memory occupancy rate is not greater than the over-allocation threshold, there is no over-allocation risk for the video memory; and when the video memory occupancy rate is greater than the over-allocation threshold, there is an over-allocation risk for the video memory.

[0123] Specifically, a time-dependent memory requirement change function can be pre-built. Based on this function and the real-time remaining memory, the risk of over-resolution can be determined. The memory requirement change function is: Total memory requirement = Total number of elements in the in-memory tensors of the model to be trained * Average size of the data types of the in-memory tensors of the model to be trained + Fixed additional training overhead of the model to be trained (such as framework overhead).

[0124] When it is monitored through step S290 that there is no risk of over-resolution of the video memory, the monitoring process continues to be executed through step S290 until the model training task in the video memory is completed; when it is monitored through step S290 that there is a risk of over-resolution of the video memory, step S300 is executed.

[0125] Step S300: Obtain the task migration benefit ratio of each task to be trained in the third queue.

[0126] In some embodiments of the present disclosure, the task migration benefit ratio in step S300 is a decision indicator for measuring whether it is worthwhile to migrate the task.

[0127] Step S310 : determining a task to be removed from the tasks to be trained in the third queue according to the task removal benefit ratio.

[0128] Step S320: Migrate the tasks to be migrated from the third queue to the fourth queue.

[0129] In some embodiments of the present disclosure, the fourth queue in step S320 is used to manage tasks to be migrated out, and migrate the tasks to be migrated back to the third queue when the second migration instruction is triggered. The fourth queue may be a pause queue.

[0130] Through steps S290 to S320, tasks with relatively high migration benefits can be continuously selected from the second queue and migrated to the fourth queue until it is predicted that video memory usage will not exceed its total amount in the future. After migrating the tasks to be migrated to the fourth queue, the memory occupied by the tasks to be migrated can be released to avoid possible memory over-allocation. When there are tasks in the fourth queue, tasks in the fourth queue can be scheduled first, rather than newly migrated tasks in the third queue.

[0131] After migrating the tasks to be migrated from the third queue to the fourth queue in step S320, the tasks to be migrated from the fourth queue can be migrated back to the third queue according to a preset second restoration rule. The second restoration rule can be set based on one or more factors, such as the availability of video memory resources, the priority of the tasks to be migrated, the time when the model training tasks were added to the fourth queue, and the task rotation strategy.

[0132] When there is a risk of over-allocation of video memory through steps S290 to S320, the tasks to be migrated out of the training tasks in the third queue are migrated to the fourth queue according to the task migration benefit ratio. This can avoid over-allocation of video memory at a relatively low cost, thereby improving scheduling flexibility, avoiding system crashes, and maximizing overall benefits when video memory resources are tight.

[0133] Steps S240 to S320 utilize the changes in video memory resource usage during the life cycle of the model training task to explore potential video memory utilization and computational parallelism. By actively selecting model training tasks and moving them backward, resource allocation conflicts can be avoided, and video memory utilization and training efficiency can be improved.

[0134] Regarding step S300, in some embodiments of the present disclosure, it may include the following: Figure 9 Steps S301 to S303 are shown.

[0135] Step S301: Obtain the memory demand growth and historical migration times of each task to be trained in the third queue.

[0136] In some embodiments of the present disclosure, the memory demand growth in step S301 is the amount of memory required for executing a task to be trained; the historical migration count in step S301 is the number of times a task to be trained was migrated to the fourth queue before the current time.

[0137] Step S302 : adding 1 to the historical number of migrations of each task to be trained in the third queue to obtain the expected number of migrations of each task to be trained in the third queue.

[0138] Step S303 : determining the task migration benefit ratio of each task to be trained in the third queue according to the quotient of the memory demand growth and the expected number of migrations.

[0139] Determining the task migration benefit ratio based on the growth in video memory demand and the expected number of swap-outs through steps S301 to S303 helps prioritize migrating out tasks to be trained that have "high pressure, few swap-outs, and high migration benefits," thereby reducing the risk of video memory over-allocation and avoiding problems such as frequent migration of a task to be trained, which results in the task not being executed and leading to training failure.

[0140] Regarding step S290, in some embodiments of the present disclosure, it may include the following Figure 10 Steps S291 to S293 are shown.

[0141] Step S291: Obtain the video memory requirements and the real-time remaining amount of video memory of the tasks to be trained in the third queue.

[0142] Step S292, obtaining the relationship between the video memory requirement and the real-time remaining amount.

[0143] Step S293: Determine whether there is a risk of over-allocation of the video memory based on the size relationship.

[0144] In some embodiments of the present disclosure, when the size relationship obtained through step S292 is that the video memory demand is greater than the real-time remaining amount, step S293 can be used to determine that there is a risk of over-allocation of the video memory; when the size relationship obtained through step S292 is that the video memory demand is not greater than the real-time remaining amount, step S293 can be used to determine that there is no risk of over-allocation of the video memory.

[0145] Determining whether the video memory is stored at over-resolution risk based on the video memory demand and real-time remaining amount through steps S291 to S293 is of great significance for ensuring the stability of DNN training and optimizing resource utilization.

[0146] Regarding step S291, in some embodiments of the present disclosure, it may include the following Figure 11 Steps S2911 to S2913 are shown.

[0147] Step S2911, obtain the total number of tensor elements, tensor element size and fixed video memory overhead of the task to be trained.

[0148] In some embodiments of the present disclosure, the total number of tensor elements of the task to be trained in step S2911 is the total number of all elements in the tensor of the task to be trained. The tensor element size in step S2911 is the number of bytes occupied by a single tensor element of the task to be trained in memory; the tensor element size can be specifically the average of the tensor element sizes of all tensors of the task to be trained. The fixed video memory overhead in step S2911 is the additional video memory required to support the training of the task to be trained, in addition to the tensor video memory overhead; the fixed video memory overhead can include stable video memory consumption such as framework overhead, memory allocator overhead, library loading overhead, and driver overhead.

[0149] Step S2912: Determine the tensor memory overhead based on the product of the total number of tensor elements and the size of the tensor elements.

[0150] Step S2913: Determine the video memory requirement of the task to be trained based on the sum of the tensor video memory overhead and the fixed video memory overhead.

[0151] Through steps S2911 to S2913, the video memory requirements can be accurately estimated based on the total number of tensor elements, the size of tensor elements and the fixed video memory overhead, thereby preventing video memory overflow in advance and improving training efficiency.

[0152] Regarding step S310, in some embodiments of the present disclosure, it may include the following Figure 12 Steps S311 to S313 are shown.

[0153] Step S311 , sorting the tasks to be trained in the third queue according to their task migration benefit ratios to obtain a task sorting result.

[0154] Step S312: Determine the number of tasks to be trained in the third queue to be migrated out according to the task migration rule.

[0155] In some embodiments of the present disclosure, the task migration rule in step S312 can be a predefined task migration ratio or task migration quantity. When the task migration rule is the task migration ratio, step S312 can specifically include: obtaining the total number of tasks to be trained in the third queue; determining the migration quantity based on the product of the total number and the task migration ratio (when the product of the total number and the task migration ratio is not an integer, the product of the total number and the task migration ratio can be rounded down or up as the migration quantity). When the task migration rule is the task migration quantity, step S312 can directly use the predefined task migration quantity as the migration quantity of the tasks to be trained in the third queue.

[0156] Step S313: Determine the number of tasks to be migrated from the task sorting result.

[0157] In some embodiments of the present disclosure, step S313 may specifically be to determine the number of tasks to be migrated in descending order of task migration efficiency ratio according to the task sorting result.

[0158] Through steps S311 to S313 , tasks with high task migration benefit ratios can be preferentially migrated, which can significantly alleviate the pressure on the video memory while minimizing performance loss.

[0159] The tensor management method provided by the present disclosure manages tensors in video memory based on the tensor unloading rules determined by the tensor lifecycle, since the access patterns of the target model's tensors are regular and predictable. This allows for the training of tensors in the video memory to complete the current calculations by retaining only a small portion of the tensors in the video memory. The remaining tensors can be unloaded to the target storage device and pre-fetched into the video memory when needed, thus avoiding impacting the execution of the model training task. This can significantly reduce video memory usage, improve video memory utilization, and mitigate the risk of video memory overflow. Furthermore, by optimizing the resource utilization of video memory and target storage devices, it can also reduce reliance on high-end graphics card hardware and lower system deployment costs.

[0160] Specifically, for most DNN models, since the forward propagation and backward propagation of the DNN model are advanced layer by layer, the current calculation of each layer of the network only involves the relevant activation tensors, weight tensors and corresponding gradient tensors. Therefore, the tensors used in the training process of the DNN model only account for a small part of all tensors (less than 10%, an average of about 1%); and since a tensor in the DNN model is usually only accessed once during the forward propagation and backward propagation respectively, the tensor access period of the DNN model is relatively long (for example, more than 60% of the tensors in the CNN model have an inactive period of more than 107 microseconds, and about 50% of the tensors in the Transformer model have an inactive period of more than 105 microseconds).

[0161] In DNN models, the size distribution of inactive tensors and the distribution of inactive period durations vary widely. For example, in the Inceptionv3-512 model, the size of inactive tensors ranges from 10KB to 2.7GB, and the duration of inactive periods ranges from 10 microseconds to 100 seconds. Based on tensor offloading rules, tensors with greater benefits can be offloaded, thereby significantly reducing the usage of video memory resources.

[0162] Figure 13 An exemplary flowchart of a tensor management method according to the present disclosure is shown.

[0163] Figure 13 In the flowchart shown, taking the simultaneous training of three tasks as an example, the tensor management method may include the following contents.

[0164] Step S410: In response to receiving three model training tasks, the three model training tasks are added to a waiting queue.

[0165] In some embodiments of the present disclosure, the first of the three model training tasks in step S410 may be a ResNet50 image classification model training task; the second model training task may be a BERT-Base text classification model training task; and the third model training task may be a UNet medical image segmentation model training task. Step S410 initially places the three model training tasks in a waiting queue.

[0166] Step S420: Obtain the real-time remaining amount of video memory and the video memory required for each model training task in the first queue.

[0167] In an embodiment of the present disclosure, the real-time remaining amount of video memory obtained through step S410 is 16GB, the video memory required for the ResNet50 image classification model training task is approximately 6GB, the video memory required for the BERT Base text classification model training task is approximately 8GB, and the video memory required for the UNet medical image segmentation model training task is approximately 10GB.

[0168] Step S430 : determining tasks to be trained and delayed training tasks from the model training tasks in the first queue according to the real-time remaining amount and the video memory required by each model training task in the first queue.

[0169] In some embodiments of the present disclosure, the processing rule for the waiting queue may be to prioritize tasks requiring less video memory and place them in the execution queue. Because the sum of the video memory required by the ResNet50 image classification model training task and the BERT Base text classification model training task (approximately 14GB) is less than the real-time remaining amount, the ResNet50 image classification model training task and the BERT Base text classification model training task can be used as pending training tasks, and the UNet medical image segmentation model training task can be used as a delayed training task.

[0170] Step S440 : Migrate the delayed training task from the waiting queue to the delay queue, and migrate the task to be trained from the waiting queue to the execution queue.

[0171] In some embodiments of the present disclosure, after migrating the delayed training task from the waiting queue to the delay queue through step S440, the delayed training task can be moved back to the waiting queue when the real-time remaining amount of video memory meets the processing requirements of the UNet medical image segmentation model training task.

[0172] Step S450: Control the execution of the model training tasks in the execution queue.

[0173] In some embodiments of the present disclosure, step S450 can control the execution of the ResNet50 image classification model training task and the BERT Base text classification model training task according to the priority of the model training tasks in the execution queue.

[0174] Step S460: monitor whether the video memory has an over-allocation risk.

[0175] In some embodiments of the present disclosure, it is assumed that a ResNet50 image classification model training task is currently being executed, a BERT Base text classification model training task is pending, and the current real-time remaining amount of video memory is 3GB. Therefore, step S460 monitors that there is a risk of over-resolution of the video memory.

[0176] Step S470: Obtain the task migration benefit ratio of each model training task in the execution queue.

[0177] Step S480 : determining tasks to be migrated from the model training tasks in the execution queue according to the task migration benefit ratio.

[0178] Step S490: Migrate the task to be migrated from the execution queue to the pause queue.

[0179] In some embodiments of the present disclosure, since there is only one unexecuted task in the execution queue in this example, steps S470 to S480 can be skipped and the BERT Base text classification model training task can be directly migrated from the execution queue to the pause queue through step S490.

[0180] The process of scheduling tasks between different queues through steps S410 to S490 can be as follows: Figure 14 shown.

[0181] Step S500: During the execution of the ResNet50 image classification model training task, the training progress and tensor unloading rules of the model are obtained.

[0182] Step S510: Obtain the stage unloading rule of the training progress from the tensor unloading rule.

[0183] Step S520: Obtain the resident tensor of the target model in the video memory.

[0184] Step S530 : Determine the tensor to be unloaded from the resident tensors according to the stage unloading rule.

[0185] Step S540: unload the tensor to be unloaded to the target storage device.

[0186] The tensor management method provided by the present disclosure can determine the execution order of model training tasks in the execution queue according to pre-set priority rules; when it is detected that there is a risk of over-resolution of the video memory, the tasks in the execution queue are added to the pause queue to free up part of the video memory space to avoid excessive use of the video memory, thereby preventing the training from being affected by over-resolution of the video memory.

[0187] The present disclosure also provides a tensor management device (corresponding to a tensor management method). Figure 15 A schematic diagram showing a hardware implementation using a processing system is shown.

[0188] like Figure 16As shown, the hardware structure of an electronic device / apparatus can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, the figure shows only one connecting line, but this does not mean that there is only one bus or one type of bus.

[0189] For ease of explanation, some steps of the above method are described as corresponding to modules. It should be understood that the corresponding modules for performing one or more steps of the above method can be one or more hardware modules specifically configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0190] like Figure 15 As shown, the tensor management device includes a progress acquisition module 1010 , a rule acquisition module 1020 , a residency acquisition module 1030 , an unloading acquisition module 1040 and a tensor unloading module 1050 .

[0191] The progress acquisition module 1010 is used to obtain the training progress and tensor unloading rules of the target model during the target model training process; the tensor unloading rules are generated based on the unloading benefit-cost ratio determined according to the tensor life cycle of the target model.

[0192] The rule acquisition module 1020 is used to obtain the stage unloading rule of the training progress from the tensor unloading rule.

[0193] The resident acquisition module 1030 is used to obtain the resident tensor of the target model in the video memory.

[0194] The unloading acquisition module 1040 is used to determine the tensors to be unloaded from the resident tensors according to the stage unloading rule.

[0195] The tensor unloading module 1050 unloads the tensor to be unloaded to the target storage device. The specific implementation of each module in the above apparatus can refer to the implementation process of the corresponding steps in the above method embodiment of the present disclosure, and will not be repeated here.

[0196] The present disclosure also provides a readable storage medium having a computer program stored therein, which is used to implement the above-mentioned method when the computer program is executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0197] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the process or function of the present disclosure is performed in whole or in part.

[0198] A computer program or instruction can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instruction can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any accessible medium or a data storage device such as a server or data center that integrates one or more accessible media. The accessible medium can be a magnetic medium such as a floppy disk, hard disk, or magnetic tape; an optical medium such as a digital video disk; or a semiconductor medium such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0199] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0203] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and the features of different embodiments / methods or examples, unless they are contradictory.

[0204] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A tensor management method, characterized in that: include: During the target model training process, the training progress and tensor unloading rules of the target model are obtained; the tensor unloading rules are generated based on the unloading benefit-cost ratio determined according to the tensor life cycle of the target model; Obtaining a stage unloading rule of the training progress from the tensor unloading rule; Get the resident tensor of the target model in the video memory; Determining a tensor to be unloaded from the resident tensors according to the stage unloading rule; as well as Unloading the tensor to be unloaded to a target storage device; Before training the target model, it also includes: Obtaining tensors of the target model and various stages of the training process of the target model; Obtain the tensor lifecycle of each tensor of the target model respectively; Determine the offloading benefit-cost ratio of each tensor at each stage based on its tensor lifecycle; and Generate tensor offloading rules based on the offloading benefit-cost ratio of each tensor at each stage; For any tensor of the target model, determining the unloading benefit-cost ratio of the tensor at each stage according to the tensor lifecycle of each tensor includes: Obtain the tensor size and tensor unloading overhead of the tensor; obtain the tensor idle time of the tensor at each stage according to the tensor life cycle of the tensor; determine the tensor unloading benefit of the tensor at each stage according to the product of the tensor size and the tensor idle time of the tensor at each stage; and determine the unloading benefit-overhead ratio of the tensor at each stage according to the quotient of the tensor unloading benefit of the tensor at each stage and the tensor unloading overhead.

2. The tensor management method according to claim 1, characterized in that: The obtaining of the tensor size and tensor unloading overhead of the tensor includes: Get the tensor size and network bandwidth of the tensor; and The tensor offloading overhead of the tensor is determined according to a quotient of twice the tensor size of the tensor and the network bandwidth.

3. The tensor management method according to claim 1, characterized in that: After unloading the to-be-unloaded tensor to the target storage device, the method further includes: Obtain the tensor lifecycle and current time of the tensor to be unloaded; Determine the remaining time according to the tensor lifecycle of the tensor to be unloaded and the current time; Determine whether the remaining time is greater than a time threshold; and In response to a remaining time of a target tensor in the to-be-unloaded tensors being not greater than the time threshold, migrating the target tensor to the video memory.

4. The tensor management method according to claim 3, characterized in that: Migrating the target tensor to the video memory includes: Adding the target tensor to a prefetch queue; and Asynchronously fetching the tensor from the prefetch queue and adding it to the video memory.

5. The tensor management method according to claim 1, characterized in that: Before training the target model, it also includes: In response to receiving the model training task, adding the model training task to the first queue; Obtaining the real-time remaining amount of the video memory and the video memory required by each model training task in the first queue; Determining tasks to be trained and delayed training tasks from the model training tasks in the first queue according to the real-time remaining amount and the video memory required by each model training task in the first queue; Migrating the delayed training task from the first queue to the second queue, and migrating the task to be trained from the first queue to the third queue; the second queue is used to manage the delayed training task and migrate the delayed training task back to the first queue when a first migration indication is triggered; and Controlling the execution of the tasks to be trained in the third queue, where the tasks to be trained in the third queue include tasks for training the target model.

6. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor performs the tensor management method according to any one of claims 1 to 5.

7. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the tensor management method according to any one of claims 1 to 5 when executed by a processor.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the tensor management method according to any one of claims 1 to 5 is implemented.

Images