Memory management system and method using memory virtualization and real-time scheduling

Abstract

The present invention relates to a memory management system using memory virtualization and real-time scheduling. According to the present invention, the memory management system using memory virtualization and real-time scheduling comprises: a processor unit which manages a mapping table for converting a virtual memory address allocated to each of multiple DNN operations into a GPU virtual address in which data required for executing a corresponding DNN operation is stored, and accesses the GPU virtual address through a virtual memory address for the DNN operation on the basis of the mapping table and scheduling information to read the data, or swaps out the data from a GPU memory to a CPU memory or swaps in the data from the CPU memory to the GPU memory; and a scheduling unit for providing the scheduling information to the processor unit by scheduling an execution order and a swap-in / out operation for multiple DNN operations over time.

Description

Memory management system and method using memory virtualization and real-time scheduling

[0001] The present invention relates to a memory system and method using memory virtualization and real-time scheduling, and more particularly, to a memory management system and method using memory virtualization and real-time scheduling that can bridge the gap between the limited capacity of GPU memory and the ever-increasing complexity of DNN models while ensuring real-time performance for multiple DNN tasks.

[0002] As applications utilizing GPUs (e.g., DNNs) increase, deeper DNNs are being utilized for higher accuracy and better services, and attempts to utilize multiple DNNs are increasing.

[0003] At this time, the size of GPU memory was smaller than that of CPU memory, and the GPU was limited to accessing only data within GPU memory, so the size of GPU memory became a new constraint.

[0004] To address these GPU memory size limitations, attempts have been made to utilize CPU memory. NVIDIA offers a technology called Oversubscription, which allows GPU hardware developed by the company to utilize more data than the actual size of the GPU memory.

[0005] In case of oversubscription, GPU memory automatically swaps out unused data to CPU memory to secure space in GPU memory, and when the data is to be used again, it is automatically swapped back in from CPU memory to GPU memory.

[0006] Another conventional approach, recent research, has attempted to address GPU memory size limitations by utilizing secondary storage devices (e.g., SSDs) rather than CPU memory. This approach involves modifying the GPU driver, which manages GPU memory, to enable GPU memory to communicate with the SSD. Furthermore, when data usage exceeds the GPU memory capacity, data can be written to and erased from the SSD.

[0007] Although these conventional technologies effectively address GPU memory size constraints, they fail to consider the impact of the delay time generated during this process on real-time tasks performed in real-time systems.

[0008] Real-time tasks operating in real-time memory systems are subject to constraints that require them to complete within a specific deadline. Under these constraints, the two previously mentioned conventional techniques fail to account for the delays caused by memory management, and thus cannot guarantee that real-time tasks will complete within their deadlines.

[0009] The technology underlying the present invention is disclosed in Korean Patent Registration No. 10-1953906 (announced on June 12, 2019).

[0010] The present invention aims to provide a memory management system and method using memory virtualization and real-time scheduling that can bridge the gap between the limited capacity of GPU memory and the ever-increasing complexity of DNN models while ensuring real-time performance for multiple DNN tasks.

[0011] The present invention relates to a memory management system using memory virtualization and real-time scheduling, comprising: a processor unit that manages a mapping table that converts a virtual memory address allocated to each of multiple DNN tasks into a GPU virtual address in which data required for execution of the corresponding DNN task is stored; and accesses the GPU virtual address through the virtual memory address for the DNN task based on the mapping table and scheduling information to read the data, or swaps the data out from GPU memory to CPU memory or swaps it in from CPU memory to GPU memory; and a scheduling unit that schedules execution orders and swap in / out operations for multiple DNN tasks according to time and provides the scheduling information to the processor unit.

[0012] In addition, if the GPU virtual address where the data is stored has changed compared to before the data is swapped out after the data is swapped in for the corresponding DNN task, the processor unit can update the GPU virtual address of the corresponding DNN task with the changed address value in the mapping table.

[0013] In addition, the processor unit can control a GPU driver connected to the GPU memory based on the mapping table and scheduling information to perform reading, execution, swap out, and swap in operations of data stored in a corresponding GPU virtual address of the GPU memory for each multiple DNN task.

[0014] In addition, the scheduling unit can schedule execution operations and swap in / out operation timings for multiple DNN tasks based on task profile information including the task duration set for each of the multiple DNN tasks, the task deadline, the task cycle, the occurrence time of each DNN task, and the real-time GPU memory remaining amount.

[0015] Additionally, the scheduling unit may not allocate swap in / out operations to ensure that data for a task whose deadline is less than a preset threshold value always remains within the GPU memory.

[0016] In addition, the scheduling unit can set a priority for executing a plurality of DNN tasks by considering at least one of the occurrence time of the task and the deadline time of the task, and schedule the tasks to be executed in order of highest priority.

[0017] In addition, the scheduling unit can set the execution of tasks in the order of earliest deadlines of the tasks, and schedule the swap-in operation for tasks to be performed dependently according to the execution order of the set tasks.

[0018] In addition, the scheduling unit sets the priorities of the multiple DNN tasks in the order of earliest deadlines of the tasks, and executes the tasks in the order of highest priority. However, when the occurrence of a second task having an earlier deadline than the first task is detected during a swap-in operation of data corresponding to the first task, the scheduling unit can schedule the execution of the second task after the execution of the first task regardless of the deadline of the second task.

[0019] In addition, the scheduling unit can schedule, after the execution of the first task is completed, to simultaneously perform swap-in of data of the third task in the next order while executing the second task.

[0020] In addition, the scheduling unit may schedule, by considering the remaining amount of GPU memory at a point in time after the execution of the first task, to skip swapping out data for the first task after the execution of the first task is completed and to perform execution of the second task and swapping in data for the third task simultaneously.

[0021] In addition, the processor unit may keep the initial GPU virtual address intact when swapping out some of the data in the GPU memory to the CPU memory, so that the GPU virtual address remains intact even when swapping in.

[0022] And, the present invention is a memory management method using memory virtualization and real-time scheduling, comprising: a step of managing a mapping table that converts a virtual memory address allocated to each of multiple DNN tasks into a GPU virtual address where data required for execution of the corresponding DNN task is stored; a step of scheduling an execution order and a swap in / out operation for multiple DNN tasks over time to generate scheduling information; and a step of accessing the GPU virtual address through the virtual memory address for the DNN task based on the mapping table and scheduling information to read the data, or swapping out the data from GPU memory to CPU memory or swapping in the data from CPU memory to GPU memory.

[0023] In addition, the present invention may further include a step of updating the GPU virtual address of the corresponding DNN task in the mapping table to the changed address value when the GPU virtual address at which the data is stored has changed compared to before the data is swapped out after the data for the corresponding DNN task has been swapped in.

[0024] In addition, the step of generating the scheduling information may schedule execution operations and swap in / out operation timings for multiple DNN tasks based on task profile information including the task duration set for each of the multiple DNN tasks, the task deadline, and the task cycle, the occurrence time of each DNN task, and the remaining real-time GPU memory.

[0025] According to the present invention, it is possible to bridge the gap between the limited capacity of GPU memory and the ever-increasing complexity of DNN models while ensuring real-time performance for multiple DNN tasks.

[0026] In addition, the present invention can provide a swap scheduling algorithm that minimizes unnecessary swap operations and maximizes overlap between calculation and data transfer, thereby alleviating the delay time of memory swap.

[0027] FIG. 1 is a diagram showing the configuration of a memory management system using memory virtualization and real-time scheduling according to an embodiment of the present invention.

[0028] FIG. 2 is a diagram exemplarily showing a mapping table update process in an embodiment of the present invention.

[0029] Figure 3 is a drawing to explain the system of Figure 1 in more detail.

[0030] FIG. 4 is a diagram for exemplarily explaining the driving mechanism of a swap scheduling algorithm for an embodiment of the present invention.

[0031] Figure 5 is a diagram comparing the real-time performance guarantee between the present invention and the prior art.

[0032] Figure 6 is a diagram comparing the response times between the present invention and the prior art.

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description have been omitted to clearly explain the present invention, and similar parts have been designated with similar reference numerals throughout the specification.

[0034] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

[0035] The present invention proposes a novel real-time memory management framework specialized for DNN (Deep Neural Network) tasks.

[0036] The memory management framework proposed in the present invention can guarantee smooth data access for DNN tasks regardless of memory swap, based on a consistent virtual memory management (virtual address mapping) mechanism.

[0037] In addition, the present invention can provide a deadline-aware swap scheduling policy that ensures timely inference by determining both the swap amount for each task and the swap in / out task schedule according to the execution schedule of the DNN task.

[0038] A memory management system according to an embodiment of the present invention may be implemented or included in a computer device or terminal including a processor, memory, a user interface input / output device and a storage device, a network input / output unit, etc., and may also be implemented as software within the device or terminal.

[0039] FIG. 1 is a diagram showing the configuration of a memory virtualization and real-time scheduling system according to an embodiment of the present invention.

[0040] As shown in FIG. 1, a memory management system (100) using memory virtualization and real-time scheduling according to an embodiment of the present invention includes a processor unit (110) and a scheduling unit (120), and may further include a GPU driver (130), GPU memory (200), and CPU memory (300).

[0041] The processor unit (110) can manage a mapping table that converts a virtual memory address allocated to each of multiple DNN tasks into a GPU virtual address where data required for executing the corresponding DNN task is stored.

[0042] The scheduling unit (120) can schedule the execution order and swap in / out operations for multiple DNN tasks according to time and provide scheduling information to the processor unit (110). The scheduling unit (120) can exchange information with the processor unit (110).

[0043] The processor unit (110) may include an address translation manager and a swap manager. Here, the address translation manager may manage a mapping table, and the swap manager may manage information on which memory used by the DNN task is in GPU memory and which is in CPU memory, and based on this, may determine which memory to swap out or swap in. The processor unit (110) may update the mapping table and scheduling information in real time.

[0044] The processor unit (110) can access the GPU virtual address through the virtual memory address for the DNN task based on the mapping table and scheduling information, read and execute the data, swap out the data from the GPU memory to the CPU memory, and swap in the data from the CPU memory back to the GPU memory.

[0045] Here, the processor unit (110) controls the GPU driver (130) connected to the GPU memory (200) based on the mapping table and scheduling information to perform reading and execution, swap out, and swap in operations of data stored in the corresponding GPU virtual address of the GPU memory (200) for each multi-DNN task.

[0046] The processor unit (110) converts a virtual memory address for a requested DNN task into a GPU virtual address based on a mapping table, accesses GPU memory through the converted GPU virtual address, and can read or execute data required for execution of the corresponding DNN task from the corresponding address location, and control a swap in or swap out operation for the data.

[0047] At this time, if the GPU virtual address where the data is stored after the swap in of the data for the corresponding DNN task has changed compared to before the swap out, the processor unit (110) can update the GPU virtual address of the corresponding DNN task with the changed address value in the mapping table.

[0048] FIG. 2 is a diagram exemplarily showing a mapping table update process in an embodiment of the present invention.

[0049] For Fig. 2, the i-th DNN task τ i The virtual memory address allocated to the i-th DNN task τ is '0x1'. i (work τ i ) is currently assumed to be '0x001'.

[0050] If the task τ is from a user program i When execution is requested for , the processor unit (110) refers to the current mapping table and executes the task τ iAfter checking the GPU virtual address '0x001' corresponding to , the GPU driver (130) accesses the '0x001' address in the GPU memory (200) and reads the data stored at that location to execute the i-th DNN task τ i can be run.

[0051] Task τ i After the execution of , the processor unit (110) stores the task τ stored in the '0x001' location in the GPU memory (200) as shown in Fig. 2 to secure data space required for the execution of another subsequent task. i Data for can be moved by swapping it out to CPU memory (300), and then the task τ i When there is an execution request for the data, the data can be moved again by swapping it in from the CPU memory (300) to the GPU memory (200).

[0052] Work τ during the movement process i It can be seen that the data for is stored at location '0x002' in GPU memory (200), and the task τ i You can see that the GPU virtual address for has changed from '0x001' to '0x002'.

[0053] At this time, the processor unit (110) performs the task τ i The existing GPU virtual address '0x001', which was mapped to the virtual memory address '0x1', can be deleted and updated to the current changed address '0x002'.

[0054] In this way, according to the present invention, even if the storage location (GPU virtual address) of data required for a DNN task within the GPU memory continuously changes due to repeated swap in / out operations of data for the DNN task, the GPU virtual address mapped to the virtual memory address of the corresponding DNN task is continuously updated within the mapping table, thereby enabling flexible response to the GPU virtual address that changes due to frequent swap operations.

[0055] Additionally, this memory management framework, with its transparent and efficient swap mechanism, allows machine learning (ML) frameworks to perform inference on DNN tasks that require memory capacity larger than the physical GPU memory without changing the DNN model or modifying the GPU device driver.

[0056] Figure 3 is a drawing to explain the system of Figure 1 in more detail.

[0057] Referring to FIG. 3, the processor unit (110) receives a DNN task τ from a user program (140). i , and the user program (140) can request multiple DNN tasks over time. The processor unit (110) can operate in conjunction with the scheduling unit (120) and the GPU driver (130).

[0058] The processor unit (110) uses an address translation manager and a swap manager to perform tasks within the GPU memory (200). i It can control access, execution, and swapping in / out of data between memory (200, 300) related to data.

[0059] In the drawing, BMW V·A represents the virtual memory address allocated for each DNN task, and GPU V·A represents the GPU virtual address where the data required to execute the corresponding DNN task is stored.

[0060] In the memory management system according to the embodiment of the present invention shown in Fig. 3, there are three major flows (Memory access, Memory allocation, Scheduling & execution).

[0061] At this time, in order to access, allocate, and operate GPU memory (200), specific APIs such as Access API, Allocation API, and Execution API must be used.

[0062] First, the memory access flow is described. This flow represents the process a DNN task goes through when attempting to access GPU memory (200).

[0063] In order for a DNN task to access GPU memory, it must know the virtual address of the GPU memory it wants to access, i.e. the GPU virtual address (GPU V·A).

[0064] If a DNN task wants to access GPU memory using the virtual memory address (BMW VA), the library (BWM library) can hook the APIs related to access and find out which virtual memory address (BMW VA) the DNN task wants to access.

[0065] Afterwards, the corresponding virtual memory address (BMW VA) can be translated into an actual GPU virtual address (GPU VA) based on the mapping table in the Address Translation Manager. After translation, access to the actual GPU virtual address (GPU VA) is performed to read data.

[0066] Next, we describe the memory allocation flow. Similar to the memory access flow, the memory allocation flow hooks into the API that performs memory allocation to determine how much GPU memory the DNN task requires. During this process, instead of providing the DNN task with a GPU virtual address (GPU VA), a virtual memory address (BMW VA) is provided, which is then reflected in the mapping table.

[0067] Next, we describe the Scheduling & Execution flow. The Scheduling & Execution flow manages the execution order of multiple DNN tasks across the system and the execution of swap operations. Based on the priorities of DNN tasks, their expected execution times, the current remaining GPU memory across the system, and the computational requests of DNN tasks, it can manage which DNN tasks should be executed at the current time and which swap operations should be performed.

[0068] In this process, if there is a need to perform a swap operation, the swap manager of the corresponding DNN task is woken up to perform a task of moving some of the data in the GPU memory (200) to the CPU memory (300) (swap out) or to perform a task of moving the data in the CPU memory (300) back to the GPU memory (200) (swap-in).

[0069] Meanwhile, when the address translation manager swaps out some of the data in the GPU memory (200) to the CPU memory (300), it copies the data and releases the virtual address (ex. 0x001), the physical memory space, and the mapping of the virtual address and the physical memory space, thereby allowing other data to occupy the space. However, when data is later swapped in from the CPU memory (300) to the GPU memory (200), the address translation manager assigns a different virtual address (ex. 0x004) rather than the initial virtual address (ex. 0x001).

[0070] Accordingly, the address translation manager according to the embodiment of the present invention can maintain the initial virtual address (e.g., 0x001) as it is without releasing it when swapping out some of the data in the GPU memory (200) to the CPU memory (300), so that the virtual address (e.g., 0x001) can be maintained as it is even if the data is swapped in later. That is, when swapping out, the address translation manager can maintain the same virtual address as the initial one by releasing the mapping between the physical memory space and the virtual address and the physical memory space without releasing the initial virtual address (e.g., 0x001).

[0071] That is, the address translation manager can keep the virtual address as it was before the swap out and when the swap in is made, even if the physical memory space changes.

[0072] Furthermore, according to an embodiment of the present invention, during the swap-out process, the amount of data to be swapped out can be known in advance, so the size of CPU memory can be allocated in advance. Therefore, according to an embodiment of the present invention, the time required to copy data can be reduced compared to the existing Pageable memory type, and the memory allocation time can be reduced compared to the case of using the existing Pinned type memory, so that the overall time required to perform the swap-out can be significantly reduced.

[0073] Meanwhile, in an embodiment of the present invention, the memory management system (100) can use a swap scheduling algorithm that can satisfy deadline constraints and guarantee real-time performance by considering the swapping delay time of multiple DNN tasks.

[0074] The proposed swap scheduling algorithm can alleviate the latency of memory swapping by minimizing unnecessary swap operations and maximizing the overlap between computation and data transfer.

[0075] The scheduling unit (120) can schedule execution operations and swap in / out operation timings for multiple DNN tasks based on task profile information including the task duration, task deadline, and task cycle set for each of the multiple DNN tasks, the release time of each DNN task, and the real-time GPU memory remaining amount.

[0076] Here, the scheduling unit (120) may not allocate swap in / out operations so that data for a task whose deadline is less than a preset threshold value always remains within the GPU memory.

[0077] In addition, the scheduling unit (120) can set the priority for executing the tasks by considering at least one of the task occurrence time (release) and the task deadline for multiple DNN tasks, and schedule the tasks to be executed in order of highest priority.

[0078] In addition, the scheduling unit (120) can set the execution of tasks in the order of earliest deadlines, and schedule the swap-in operation for tasks to be performed dependently according to the set execution order of the tasks.

[0079] A specific embodiment of the swap scheduling algorithm is described in detail through Fig. 4 below.

[0080] FIG. 4 is a diagram for exemplarily explaining the driving mechanism of a swap scheduling algorithm for an embodiment of the present invention.

[0081] Figure 4 (a) shows a case where unnecessary swap operations are performed and DNN tasks are not performed within the deadline, and Figures 4 (b) and (c) show cases where all tasks are performed within their respective deadlines using an efficient swap scheduling algorithm according to an embodiment of the present invention.

[0082] To put it simply, in case of Figure 4(a), unlike Figure 4(b), the swap-in operation and execution are separated, allowing preemption between the two. In this situation, unnecessary swap operations continue to occur, preventing the deadline from being met. In this case, we can see that task1's data is repeatedly swapped in and out meaninglessly.

[0083] In the case of (b) in Fig. 4, the swap-in operation and execution are grouped into a single non-preemptive chunk to prevent the case of the preceding (a), thereby showing that the deadline can be met.

[0084] In the case of (c) of Fig. 4, it represents the blocking delay that may occur when execution and swap operations are performed simultaneously, and τ high In this case, it means the task with the highest priority, and indicates that if any high priority task is released at that time, a delay of the indicated amount may occur.

[0085] First, the "release" marker (indicated by an upward arrow) at the bottom of each figure indicates the task's initiation point, and the "deadline" marker (indicated by a downward arrow) indicates the task's deadline. For example, "release τ1" indicates the initiation point of the first task (Task 1), and "deadline τ1" indicates the task's deadline.

[0086] Here's a more detailed explanation of the release:

[0087] Typically, when modeling a task in a real-time system, it is represented as (C, D, T). Here, C represents the task's duration, D represents the task's deadline, and T represents the task's cycle.

[0088] Therefore, (C, D, T) means that every time T, a task that takes time C occurs, and the task must be performed (completed) within D time from the time the task occurs.

[0089] For example, if the modeled information for a task is (3, 8, 10), this means that a task that takes 3 seconds occurs every 10 seconds, and the task must be completed within 8 seconds from the time it occurs. However, depending on the task sequence, the task may not be completed within 8 seconds.

[0090] The model used in the embodiments of the present invention is a representative example where the task cycle and deadline (T=D) are the same. Therefore, in the embodiments of the present invention, "release" refers to the point in time when a specific task occurs, and the release can mean that it occurs every T hours.

[0091] The deadline of a task can be modeled differently for each task, and the time at which each task is released, i.e. the time at which the task occurs, can be determined based on the request information of the user program.

[0092] Figure 4 shows the scheduling results for multiple DNN tasks including tasks 1 to 4.

[0093] The time blocks on the same line as the 'Swap' section on the left side of the graph represent the intervals during which data is swapped in / out for a specific DNN task. Specifically, SO(τ1) represents the time block during which data required for the execution of task 1 is swapped out from GPU memory (200) to CPU memory (300), and conversely, SI(τ1) represents the time block during which the corresponding data is swapped in from the CPU to the GPU.

[0094] Additionally, the time blocks displayed on the same line as the "GPU" section on the left side of the graph represent the intervals during which a specific DNN task is executed. For example, τ1 represents the time block during which Task 1 is running. The length of each time block can vary depending on factors such as the amount of data for the task and the time required for the task.

[0095] Here, for a task whose deadline is set to be shorter than a preset threshold, the task may be set not to swap in / out because performing a swap operation on the task may make it impossible to meet the task deadline.

[0096] In Fig. 4, task τ2 corresponds to this case. Accordingly, the scheduling unit (120) can unassign swap in / out operations (assigned swap volume = 0) so that data for task τ2, whose deadline is less than a preset threshold, always remains within the GPU memory (200).

[0097] In addition, the swap scheduling algorithm according to the embodiment of the present invention basically allows swap-in operations to be performed for tasks in the order of earliest occurrence time (release) of the tasks, and execution of tasks can be performed in the order of earliest deadline of the tasks.

[0098] First, among the three scenarios in Fig. 4, the operation for (a) in Fig. 4 is described. Fig. 4 (a) is a case where the swap scheduling algorithm of the present invention is not applied, and shows a case where some DNN tasks fail to meet their task deadlines due to unnecessary swap operations.

[0099] Let's explain from the earliest time. First, the task τ3 is executed, and after the task is completed, the data for the task τ3 is swapped out (SO(τ3)). Here, since the task τ1 is generated (released) during the execution of the task τ3, the swap in (SI(τ1)) for the task τ1 is performed immediately after the swap out (SO(τ3)) of the task τ3.

[0100] However, during the swap in of data of task τ1, the occurrence (release) of tasks τ2 and τ3 is detected again, and since the deadlines of tasks τ2 and τ3 are earlier than the deadline of task τ1, tasks τ2 and τ3 have priority over task τ1.

[0101] Therefore, immediately after the swap in (SI(τ1)) of task τ1, the data for task τ2 is executed. Task τ2 corresponds to a task with limited swap operations as described above.

[0102] After the execution of task τ2 is completed, a swap-out (S0(τ1)) is performed for task τ1, which has been pushed down in priority. Then, data is swapped in for task τ3 (SI(τ3)), and then task τ3 is executed. After the execution of task τ3 is completed, a swap-in (SI(τ1)) for task τ1 is performed, and then task τ1 is executed.

[0103] However, as a result, we can see that tasks τ1 and τ3 failed to meet their deadlines. This is because, as shown in the red-marked block, unnecessary swap in / out operations were added to task τ1, delaying or delaying the overall execution of the task.

[0104] To solve this problem, the scheduling unit (120) can use the following efficient swap scheduling algorithm.

[0105] The following Figure 4 (b) shows a case in which the swap scheduling algorithm proposed in the present invention is applied.

[0106] First, the execution of task τ3 is performed, and after the task is completed, the data for task τ3 is swapped out (SO(τ3)). Here, since task τ1 is generated (released) during the execution of task τ3, the swap in (SI(τ1)) for task τ1 is performed immediately after the swap out (SO(τ3)) of task τ3. Up to this point, it is the same as (a) of Fig. 4.

[0107] However, during the swap in of data of task τ1, the occurrence (release) of tasks τ2 and τ3 is detected again, and since the deadlines of tasks τ2 and τ3 are earlier than the deadline of task τ1, tasks τ2 and τ3 have priority over task τ1.

[0108] However, in an embodiment of the present invention, if a swap-in operation is already in progress for a specific task (e.g., τ1), even if the occurrence of a task with an early deadline (e.g., τ2, τ3) is detected, priority is given to the task currently being swapped in (e.g., τ1) so that the task is executed first.

[0109] That is, after performing a swap-in (SI(τ1)) for task τ1, the task τ1 for which the swap-in has been completed is first executed. Then, task τ2 is executed. Task τ2 corresponds to a task with restricted swap operations, as described above.

[0110] At this time, while executing task τ2, a swap-in (SI(τ3)) for the next task τ3 can be performed simultaneously. In the embodiment of the present invention, this is called swap-in overlapping.

[0111] In this way, by performing the function of first executing a task for which a swap-in has been performed regardless of the deadline and the swap-in overlapping function of performing swap-in and execution simultaneously, the execution time of multiple DNN tasks can be shortened compared to the case where this is not the case, and the deadlines for all DNN tasks can be met.

[0112] Afterwards, in a situation where the swap-in of task τ3 and the execution of task τ2 are performed simultaneously, after the execution of task τ2 is completed, the execution of task τ3 for which the swap-in has been completed is performed. When the execution of task τ3 is completed, a swap-out (SO(τ3)) for task τ3 is performed. Afterwards, a swap-in (SI(τ4)) for task τ4 with the latest deadline is performed, and then task τ4 is executed. In a case like this in Fig. 4 (b), it can be confirmed that all four DNN tasks are completed within their respective deadlines.

[0113] To summarize the results of (b) of FIG. 4, the scheduling unit (120) according to the embodiment of the present invention sets priorities in the order of earliest deadlines for multiple DNN tasks and executes tasks in the order of highest priority. However, if the occurrence of a second task (τ2) having an earlier deadline than the first task (τ1) is detected during a swap-in (SO) operation of data corresponding to the first task (τ1), the first task (τ1) can be executed first and then the second task (τ2) can be scheduled to be executed regardless of the deadline of the second task (τ2).

[0114] In addition, the scheduling unit (120) can schedule the swap-in of data of the third task (τ3) in the next order to be performed simultaneously while the second task (τ2) is being executed after the execution of the first task (τ1) is completed.

[0115] In addition, the scheduling unit (120) may, by considering the remaining amount of GPU memory at a point in time after the execution of the first task (τ1), schedule the execution of the second task (τ2) and the swap-in of data of the third task (τ3) to be performed simultaneously after the execution of the first task (τ1) is completed, omitting the swap-out operation of data for the first task (τ1).

[0116] Figure 4 (b) corresponds to a situation where there is room in the remaining capacity of the GPU memory (200), allowing allocation of data for the first task (τ1) to the third task (τ3). For example, if the total capacity of the GPU memory (200) is larger than the sum of the capacities of the data for the first task (τ1) to the third task (τ3), and thus there is room, the swap out operation as described above can be omitted. However, if there is no room in the remaining capacity of the GPU memory, the swap out for the first task (τ1) can be performed immediately after the execution of the first task (τ1) is completed to secure memory space, and then the swap in operation for the third task (τ3) can be performed.

[0117] As described above, (c) of Fig. 4 shows an arbitrary high priority task (τ high ) is released at that point, it shows that the indicated amount of delay may occur.

[0118] Figure 5 is a diagram comparing real-time performance between the present invention and the prior art, and Figure 6 is a diagram comparing response times between the present invention and the prior art.

[0119] In the case of Fig. 5, as the amount of memory required by DNN tasks becomes larger than the GPU memory size, this is an experiment comparing the real-time performance between the memory management technique proposed in the present invention (BMW), NVIDIA's Oversubscription (Baseline), and the heuristic method (Partial-X). In the case of the present invention, it can find a combination of DNN tasks that can guarantee real-time performance even in a situation where the memory is overflowing by up to 45% or 15.9 GB, whereas in the case of the Baseline, it can only find a combination in a situation where the memory is overflowing by about 10% or 3.5 GB.

[0120] Figure 6 shows the response speeds of DNN tasks measured under the BMW technique proposed by the present invention and NVIDIA's Oversubscription (Baseline). In the case of NVIDIA's Oversubscription, memory is managed according to the Least-Recently Used (LRU) policy, resulting in significant variability in the DNN's response time, and task_2 fails to complete within the specified deadline.

[0121] On the other hand, in the case of DNN tasks operating under the present invention, the variability in the response time of DNNs is small by moving only a fixed amount of memory, and it can be confirmed that all DNN tasks are completed before the deadline with policies that take the deadline into account.

[0122] According to the present invention, unlike conventional technologies, there is an advantage in that there are no restrictions such as having to use a specific GPU manufacturer or modifying a GPU driver.

[0123] Additionally, since the present invention transfers only a fixed amount of data between GPU memory and CPU memory a fixed number of times, rather than a non-deterministic and unpredictable swap mechanism, the delay time experienced by each DNN task becomes deterministic.

[0124] In addition, by using a real-time guarantee analysis technique that can determine whether each DNN task can be completed within the deadline based on the set delay time, real-time guarantee of all DNN tasks can be guaranteed.

[0125] Furthermore, while existing memory management techniques manage memory in units of pages, the present invention manages memory at the memory allocation level (memory-object), thereby enabling more efficient memory management.

[0126] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. In a memory management system using memory virtualization and real-time scheduling, A processor unit that manages a mapping table that converts a virtual memory address allocated to each of multiple DNN tasks into a GPU virtual address where data required for execution of the corresponding DNN task is stored, and accesses the GPU virtual address through the virtual memory address for the DNN task based on the mapping table and scheduling information to read the data, or swaps the data out from GPU memory to CPU memory or swaps the data in from CPU memory to GPU memory; and A memory management system including a scheduling unit that schedules execution order and swap in / out operations for multiple DNN tasks according to time and provides the scheduling information to the processor unit.

2. In claim 1, The above processor unit, A memory management system that updates the GPU virtual address of the DNN task with the changed address value in the mapping table when the GPU virtual address where the data is stored has changed compared to before the data is swapped out after the data for the DNN task has been swapped in.

3. In claim 1, The above processor unit, A memory management system that controls a GPU driver connected to the GPU memory based on the above mapping table and scheduling information to read, execute, swap out, and swap in data stored in the corresponding GPU virtual address of the GPU memory for each multi-DNN task.

4. In claim 1, The above scheduling section, A memory management system that schedules execution operations and swap in / out operation timing for multiple DNN tasks based on task profile information including task duration, task deadline, and task cycle set for each DNN task, the occurrence time of each DNN task, and the remaining real-time GPU memory.

5. In claim 4, The above scheduling section, A memory management system that does not allocate swap in / out operations for tasks whose deadlines are below a preset threshold so that data for the task always remains within GPU memory.

6. In claim 4, The above scheduling section, A memory management system that schedules tasks to be executed in order of highest priority by setting the priority of execution of tasks by considering at least one of the occurrence time of the task and the deadline time of the task for multiple DNN tasks.

7. In claim 6, The above scheduling section, A memory management system that sets the execution of tasks in the order of earliest deadlines for the tasks above, and schedules swap-in operations for tasks to be performed dependently according to the execution order of the set tasks.

8. In claim 6, The above scheduling section, For multiple DNN tasks, the tasks are executed in order of priority by setting the priority in the order of the earliest deadline of the tasks. A memory management system that, when the occurrence of a second task having a deadline earlier than that of the first task is detected during a swap-in operation of data corresponding to the first task, schedules the second task to be executed after the first task is executed regardless of the deadline of the second task.

9. In claim 8, The above scheduling section, A memory management system that schedules, after the execution of the first task is completed, to concurrently perform swap-in of data of the third task in the next order while executing the second task.

10. In claim 9, The above scheduling section, A memory management system that schedules, after the execution of the first task is completed, the swap out of data for the first task to be skipped and the execution of the second task and the swap in of data for the third task to be performed simultaneously, taking into account the remaining amount of GPU memory at a point in time after the execution of the first task.

11. In claim 2, The above processor unit, A memory management system that keeps the original GPU virtual address intact when swapping out some data in GPU memory to CPU memory, so that the GPU virtual address remains intact even when swapped in.

12. In a memory management method using memory virtualization and real-time scheduling, A step for managing a mapping table that converts the virtual memory address allocated to each of the multiple DNN tasks into a GPU virtual address where the data required for executing the corresponding DNN task is stored; A step of generating scheduling information by scheduling execution order and swap in / out operations for multiple DNN tasks in time; and A memory management method comprising a step of accessing the GPU virtual address through the virtual memory address for the DNN task based on the mapping table and scheduling information to read the data, or swapping the data out from GPU memory to CPU memory or swapping it in from CPU memory to GPU memory.

13. In claim 12, A memory management method further comprising a step of updating the GPU virtual address of the DNN task in the mapping table with the changed address value when the GPU virtual address where the data is stored has changed compared to before the data is swapped out after the data for the DNN task has been swapped in.

14. In claim 12, The step of generating the above scheduling information is: A memory management method for scheduling execution operations and swap in / out operation timing for multiple DNN tasks based on task profile information including task duration, task deadline, and task cycle set for each DNN task, the occurrence time of each DNN task, and real-time GPU memory remaining amount.

Images