Memory virtualization method, deep learning system and task switching method thereof

Abstract

The invention discloses a memory virtualization method, which comprises the following steps of: firstly, configuring an address conversion table, and then performing conversion between a virtual address and a physical address of a memory through the address conversion table. Through memory virtualization, memory resources of the deep learning system can be stored in a delayed manner during task switching, so that the memory moving amount stored on site is reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a memory virtualization method, a deep learning system and a task switching method thereof. Background Art

[0002] The application of deep learning is becoming more and more widespread, and AI / DL tasks often require the use of GPU / NPU computing power. In more complex AI task scenarios, such as autonomous driving and other high-real-time scenarios, it is necessary to have strong scheduling capabilities for computing resources in order to reasonably plan the execution of computing kernels and efficiently complete complex computing tasks.

[0003] The main directions of existing task / kernel scheduling algorithms for DL ​​models include NPU non-preemptive scheduling and GPU preemptive scheduling. GPU / NPU needs to undertake computing tasks with high computing density, and the computing kernel is very large, so its task scheduling / task switching overhead is often large, far greater than the computing core designed for concurrent scenarios such as CPU.

[0004] NPU processors usually contain multiple NPU computing cores, and different computing cores are independent of each other. The execution order of computing tasks is planned according to the execution priority and resource usage of computing tasks. When the NPU computing core is idle, the computing task is issued according to the allocation scheduling algorithm. Therefore, the next computing task must wait for the completion of the currently executed task, and high-priority tasks cannot preempt existing computing resources, resulting in poor real-time performance. And when multiple NPU computing tasks are carried out in parallel, multiple independent NPU computing cores are required, and some hardware resources are difficult to isolate, which will increase the burden on the hardware and have poor flexibility. In addition, the NPU cannot implement a time-slice round-robin task scheduling method, and cannot virtualize computing power from the time dimension, which reduces resource utilization.

[0005] GPU can support time slice rotation of computing tasks. After the current time slice is completed, the running computing task will be saved and switched to the next computing task that needs to be rotated. The next rotating task needs to restore the previously saved scene to continue execution. In GPU, this scheduling method involves scene saving and recovery. GPU scheduling is based on SIMT architecture, and there are many SM units. Therefore, when saving the scene, the registers of each SM unit need to be saved and cached to the outer storage unit. When restoring the scene, the scene information needs to be read from the outer storage unit, and the overhead of scene switching is heavy. Taking A100 as an example, the number of SM units is 108. Each SM unit has 64KB of regs and 384KB of cache, a total of 448KB. Based on the highest bandwidth of 1.5TB / s, it still takes 30us to complete the scene saving. Summary of the invention

[0006] In view of some or all of the problems in the prior art, the present invention provides a memory virtualization method in a first aspect, comprising:

[0007] configuring an address translation table; and

[0008] The address conversion table is used to realize the conversion between the virtual address and the physical address of the memory.

[0009] Furthermore, configuring the address translation table includes:

[0010] Setting at least one address mapping group includes setting a starting virtual address, a physical address corresponding to the starting virtual address, and an address length included in the address mapping group.

[0011] Furthermore, in each address mapping group, the mapping relationship between the virtual address and the physical address is unique.

[0012] Further, the range of addresses in each address mapping group overlaps with the range of addresses in the remaining N address mapping groups, where N is an integer greater than or equal to zero.

[0013] Furthermore, configuring the address translation table also includes:

[0014] Configure the priority of each address mapping group. The priority of each address mapping group is different.

[0015] Furthermore, configuring the address translation table also includes:

[0016] Set the enable status of each address mapping group.

[0017] Furthermore, implementing the conversion between the virtual address and the physical address of the memory through the address conversion table includes:

[0018] Determine the address mapping group corresponding to the input address; and

[0019] Address translation is performed according to the information of the corresponding address mapping group.

[0020] Further, determining the address translation group corresponding to the input virtual address includes:

[0021] The input address is compared with the address range of each address mapping group one by one according to the priority, and the first address mapping group that successfully matches is used as the corresponding address mapping group, wherein successful matching means that the input address is within the address range of the address mapping group.

[0022] Based on the virtualization method as described above, the second aspect of the present invention provides a deep learning system, comprising:

[0023] A task management unit, which is used to schedule and manage computing tasks;

[0024] a computing unit, which is communicatively connected to the task management unit, adopts a DSA architecture, and includes at least one computing core, a first address translation module, and a shared cache module, wherein each computing core includes an independent on-chip cache area, and the first address translation module is used to translate a virtual address into a physical address according to the virtualization method as described above, thereby making the physical addresses of multiple available memory segments of the shared cache module be mapped into continuous virtual addresses of the memory space; and

[0025] An external storage unit is communicatively connected to the computing unit via a second address conversion module, wherein the second address conversion module is used to convert the address of the virtual space into a physical address according to the virtualization method as described above, thereby allowing the program code of the computing task to be mapped to the specified virtual address space.

[0026] Based on the deep learning system as described above, the third aspect of the present invention provides a task switching method of the deep learning system, comprising:

[0027] When the task management unit receives a computing task with a higher priority, it notifies the computing unit to perform on-site switching;

[0028] After the current calculation of each core is completed, the computing unit saves the calculation result of each core to the shared cache module, and sends the memory occupancy of the shared cache module to the task management unit;

[0029] The task management unit determines whether the remaining memory of the shared cache module meets the memory requirement of the computing task:

[0030] If so, configuring the first address translation module to map the physical address of the remaining memory to a continuous virtualized address; and

[0031] If not, first move at least part of the data in the shared cache module to the external storage module so that the remaining memory of the shared cache module is not less than the memory requirement of the computing task, and then configure the first address translation module to map the physical address of the remaining memory to a continuous virtualized address;

[0032] The computing unit saves the scene of each core and sends a saving completion signal to the task management unit;

[0033] The task management unit configures the second address translation module to map the computing task to a specified virtual address space;

[0034] The task management unit notifies the computing unit to execute a computing task; and

[0035] After the computing task is completed, the computing unit and the task management unit perform on-site recovery.

[0036] The present invention provides a memory virtualization method, a deep learning system and a task switching method. Through memory virtualization, memory resources can be saved with a delay when switching tasks, thereby reducing the amount of memory moved on-site. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more specific description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof. In the accompanying drawings, for clarity, identical or corresponding parts will be represented by identical or similar reference numerals.

[0038] Figure 1 A schematic diagram showing the structure of an address conversion table according to an embodiment of the present invention;

[0039] Figure 2 A schematic diagram showing the structure of a deep learning system according to an embodiment of the present invention; and

[0040] Figure 3 A flowchart of a task switching method for a deep learning system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0041] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that various embodiments can be implemented without one or more specific details or with other replacement and / or additional methods, materials or components. In other cases, well-known structures, materials or operations are not shown or described in detail to avoid blurring the inventive point of the present invention. Similarly, for the purpose of explanation, specific quantities, materials and configurations are set forth to provide a comprehensive understanding of embodiments of the present invention. However, the present invention is not limited to these specific details. In addition, it should be understood that the various embodiments shown in the drawings are illustrative representations and are not necessarily drawn in correct proportions.

[0042] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in this specification does not necessarily all refer to the same embodiment.

[0043] It should be noted that the embodiments of the present invention describe the process steps in a specific order, but this is only for the purpose of illustrating the specific embodiment, rather than limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to the adjustment of the process.

[0044] During the calculation process of multiple computing cores of NPU, when the task needs to preempt the current computing task, the scene needs to be saved. NPU usually adopts a specific domain (DSA, Domain-Specific Architecture) architecture, which is characterized by a small control flow register scene, which is the same order of magnitude as the CPU processor, but the cache scene is usually large, which is the main overhead of scene preservation. Scene recovery is the reverse process of scene preservation, and the core overhead is also the movement of the cache scene. Based on this, if you need to reduce the scene switching overhead of NPU, you can consider reducing the movement of the cache scene. The inventor found through research that when users write computing tasks, the address space they think is a fixed continuous address space. The memory space can be regarded as a virtual address memory. The user does not actually need to perceive the actual physical memory space. Therefore, it can be considered to map the discontinuous actual memory space to a continuous virtual address space through the memory virtualization method, and then when saving on the scene, the existing data in the cache can be moved without moving, thereby reducing the cache movement overhead. Taking the computing task A[0:1024]=B[0:1024]+C[0:1024] as an example, memory virtualization is briefly described, where A, B, and C are all 1024B arrays in units of int8. From the user's perspective, the address space is continuously available, so the address A space can be allocated to 0-1024B, the address B space can be allocated to 1024-2048B, and the address C space can be allocated to 2048-3072B. However, the actual available memory space is 1024-4096B, so a memory virtualization method can be provided, such as establishing a mapping relationship of VA[0:3072]->PA[1024:4096] through an address translation unit (ATU), so that the user only sees the virtual address. By virtualizing the memory scene and performing the task scene saving and recovery process based on this, delayed memory saving and switching can be achieved, reducing the scene switching overhead of the NPU.

[0045] The scheme of the present invention is further described below in conjunction with the accompanying drawings of the embodiments.

[0046] Figure 1 FIG. 2 is a schematic diagram showing the structure of an address conversion table according to an embodiment of the present invention. Figure 1As shown, an address translation table ATU includes at least one group of address mapping group Entry, wherein each Entry includes a starting virtual address VitrualAddr, a physical address Physical Addr corresponding to the starting virtual address, and an address length Size included in the Entry, wherein the mapping relationship between the virtual address and the physical address is unique. In one embodiment of the present invention, the virtual address ranges in different Entries may overlap. Based on this, in one embodiment of the present invention, each Entry has a different priority. When the address mapping range overlaps, the unique mapping relationship is determined by the priority of the Entry. In another embodiment of the present invention, each Entry can be set to an enabled or disabled state. Specifically, each Entry includes an enable flag bit. If the enable flag bit is low, the Entry is invalid and does not participate in the ATU address translation query.

[0047] Based on this, in one embodiment of the present invention, a method for virtualizing a memory includes: configuring an address translation table, and realizing the conversion between a virtual address and a physical address of the memory through the address translation table. Configuring the address translation table refers to setting at least one group of address mapping groups, wherein the setting of each group of address mapping groups includes setting a starting virtual address, a physical address corresponding to the starting virtual address, and an address length included in the address mapping group, wherein the range of addresses in any group of address mapping groups may overlap with the range of addresses in the remaining N groups of address mapping groups, wherein N is an integer greater than or equal to zero. In one embodiment of the present invention, configuring the address translation table also includes configuring the priority of each group of address mapping groups, wherein the priority of each group of address mapping groups is different. In one embodiment of the present invention, configuring the address translation table also includes setting the enable status of each address mapping group, etc.

[0048] In one embodiment of the present invention, the conversion between the virtual address and the physical address of the memory is realized by the address conversion table. First, the address mapping group corresponding to the input address needs to be determined, and then the address conversion is performed according to the information of the corresponding address mapping group. Specifically, in one embodiment of the present invention, the address conversion process is as follows:

[0049] First, query each Entry in order of priority to determine the Entry corresponding to the input address. If the enable flag of an Entry is low, the Entry is invalid and does not participate in the address conversion query. If no enabled Entry is found for the input address, the address is not converted by default. In one embodiment of the present invention, the address matching rules of the Entry are as follows:

[0050] If the input address is in the interval [VA, VA+Size), that is, the virtual address VA (Virtual Addr) <= input address (Input Addr) < (virtual address VA (Virtual Addr) + memory size (Size)), then the input address is considered to match the Entry;

[0051] Next, address conversion is performed based on the Entry. When the Entry matches, the output address (Output Addr) = VA - Input Addr + physical address PA (Physical Addr).

[0052] Based on the virtualization method described above, Figure 2 A schematic diagram of the structure of a deep learning system according to an embodiment of the present invention is shown. Figure 2 As shown, a deep learning system includes a task management unit 201, a computing unit 202 and an external storage unit 203.

[0053] The task management unit 201 is used to schedule and manage computing tasks, and may be, for example, an MCU.

[0054] The computing unit 202 is responsible for computing tasks, and is communicatively connected to the task management unit, following the common DSA architecture, without a large amount of control logic, and a single instruction can correspond to a large number of computing tasks. As shown in the figure, the computing unit includes at least one computing core Core 221, a first address conversion module 222, and a shared cache module L2 share memory 223. Each computing core 221 includes an independent on-chip cache area L1 memory, which can perform calculations. The first address conversion module 222 is used to map the physical addresses of multiple available memory segments of the shared cache module 223 to continuous memory space virtual addresses according to the virtualization method described above, so that when switching on site, the delayed overflow of the system computing cache can be realized, the amount of data movement can be reduced, and the speed of on-site switching can be accelerated.

[0055] The external storage unit 203, the shared cache module L2 share memory 223, and the on-chip cache area L1 memory constitute a three-level hierarchical memory of the deep learning system. At the same time, as shown in the figure, the external storage unit 203 is provided with a corresponding second address conversion module 231, and the second address conversion module 231 is used to map the program code of the computing task to the specified virtual address space according to the virtualization method as described above, so that each computing task uses the same virtual address space, so that the computing task does not need to be moved to a unified physical space, reducing the overhead of address relocation and speeding up the loading of computing tasks. In one embodiment of the present invention, the external storage unit 203 may include storage modules such as DDR (HBM), for example.

[0056] Based on the virtualization method as described above, the deep learning system can virtualize the memory scene, realize delayed storage and switching of memory, and reduce the overhead of scene switching. In one embodiment of the present invention, the computing unit is notified that the scene switching is required through an external unit, such as the task management unit 201 as described above. After receiving the switching request, the computing unit will wait for the current instruction to be completed, and perform scene saving after completion. Then the computing unit notifies the external unit of the memory occupancy information at the moment of the current task switching, including the address / length of the memory occupancy, etc. After the external unit receives the memory occupancy information, it determines whether the remaining memory size meets the memory requirements of the task to be issued. If the remaining memory meets the requirements of the task to be issued, there is no need to save the current scene memory back to the external storage, but through the virtualization method as described above, the current multiple available memory segments are remapped into continuous memory space for use by the task to be issued. If the remaining memory does not meet the requirements of the task to be issued, the excess memory needs to be overflowed and saved to the external storage, and then the virtualization operation is performed, that is, the address mapping of the ATU unit required by the task to be issued is calculated. At the same time, the computing unit actively saves the required registers and other resources to achieve parallel ATU switching and register saving, and notifies the external unit after the register saving is completed. After the external unit processes the ATU address mapping, the memory overflows. After the computing unit completes register saving, it can send a new computing task to the computing unit. After the computing unit processes the current task, it notifies the external unit that the task is completed. The external unit then determines whether to restore the scene. When the scene needs to be restored, the external unit notifies the computing unit to restore the scene from the specified location. The computing unit restores the register and other configurations and waits for the ATU / memory scene recovery message. If there is a memory overflow, the external unit needs to copy the scene memory back to the corresponding location. The external unit restores the ATU address mapping and notifies the computing unit that the ATU memory scene has been restored. After receiving the recovery message, the computing unit jumps the PC pointer and continues to execute the switched computing task.

[0057] In order to better describe the task switching method, Figure 3 A flowchart of a task switching method for a deep learning system according to an embodiment of the present invention is shown. Figure 3 As shown, a task switching method for a deep learning system includes:

[0058] First, in step 301, it is confirmed to perform a field switch. When the computing unit is executing a computing task, if the task management unit receives a computing task with a higher priority, the task management unit needs to interrupt the existing computing task and align the task with the higher priority, that is, the task management unit notifies the computing unit that a field switch is required;

[0059] Next, in step 302, wait for the switching point. When the computing unit receives the field switch, it needs to wait for the calculation of the current multiple computing cores to be completed. At this time, the calculation results of the computing cores are all in the L2 level memory, that is, the shared cache module is reserved, and the L1 memory on the chip of the computing core is free. At this time, it can be considered that a switching point has been reached;

[0060] Next, in step 303, the memory occupancy is confirmed. After reaching the switching point, the computing unit notifies the task management unit of the memory occupancy of the shared cache area. The shared cache module usually occupies multiple discontinuous memory segments, so the task management unit needs to be notified of the address and length of each memory segment;

[0061] Next, in step 304, it is determined whether the memory requirement is met. After receiving the memory usage information, the task management unit determines whether the total remaining memory is greater than the memory requirement of the task to be issued according to the resource situation of the task to be issued. If it is satisfied, it directly proceeds to step 306 for memory virtualization. If it is not satisfied, it first proceeds to step 305 for memory overflow.

[0062] In step 305, memory overflows. When the remaining memory is less than the memory requirement of the task to be issued, the task management unit needs to move the redundant shared cache module memory at the current site. At this time, there is no need to move all the current shared cache module on-site memory, only the redundant part needs to be moved to meet the memory requirement of the task to be issued. The task management unit can move the memory of the shared cache module to the external storage unit by configuring modules such as DMA, and then enter step 306, memory virtualization;

[0063] In step 306, memory virtualization: When the remaining memory can meet the memory requirement of the task to be issued, the task management unit configures the first address translation module on the shared cache module side to remap multiple memory segments with discontinuous physical addresses of the current shared cache module to a memory segment with continuous virtual addresses, so that the task to be issued will have a whole segment of continuous memory that meets the memory requirement of the shared cache module.

[0064] At the same time, in step 307, the register scene is saved. While the memory is virtualized, the computing unit saves the scene in the computing core in parallel. Since the on-chip cache area in the computing core is idle at this time, it does not need to be saved, and the register scene of the instruction stream needs to be saved. The register scene is usually various general registers such as PC, which are used to control the instruction stream. The size of the control register scene is usually below 1KB, which is very lightweight compared to the GPU. It is saved by the computing unit to the external storage unit, and after saving the register scene, it notifies the task management unit that the scene in the computing core is saved;

[0065] Next, in step 308, the computing task is executed. When the task management unit completes memory virtualization and receives a message from the computing unit notifying that the kernel scene has been saved, it indicates that the current task scene has been saved. At this time, the second address translation module is configured to map the computing task program code to be issued to a unified address space, and the computing unit is notified to execute the next computing task; and

[0066] Finally, in step 309, the scene is restored. When the computing task of the computing unit is executed, the task management unit is notified that the current task is finished. If there are computing tasks to be restored, the task management unit will restore them in sequence according to the nested order that has been preempted. In one embodiment of the present invention, the scene recovery mainly includes: the task management unit notifies the computing unit of the storage address of the current scene in the external storage unit, and the computing unit restores the scene from the specified address. At this time, the computing unit and the task management unit work in parallel, the computing unit is responsible for restoring the lightweight scene in the computing core, and the task management unit is responsible for restoring other scenes. That is, the computing unit restores the scene information such as registers, and waits for the notification of the task management unit after restoration. If there is memory overflow to the external storage unit side before, the task management unit needs to move the memory on the external storage unit side to the shared cache module, and reconfigure the first and second address conversion modules to restore the virtual address mapping of the external storage unit and the shared cache module of the previous scene. After completion, the computing unit is notified that the external scene has been restored. After receiving the message, the computing unit jumps to the corresponding PC pointer and continues to execute the previous computing task.

[0067] In one embodiment of the present invention, when the computing unit is executing any computing task, as long as it receives a higher priority task, it can switch tasks according to the method described above. It should be noted that in the scenario where task switching is required, the mapping relationship between virtual addresses and physical addresses between different tasks is different, so it is necessary to establish the address mapping relationship in real time and dynamically, that is, the physical memory space is changing, while the virtual space seen by the user is continuous and fixed.

[0068] Although various embodiments of the present invention are described above, it should be understood that they are presented as examples only and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should only be defined according to the attached claims and their equivalents.

Claims

1. A memory virtualization method, characterized in that: Includes steps: configuring an address translation table; and The address translation table is used to translate the virtual address into the physical address of the memory.

2. The virtualization method according to claim 1, characterized in that: Configuring the address translation table includes the following steps: At least one address mapping group is set, wherein the setting of each address mapping group includes: setting a starting virtual address, a physical address corresponding to the starting virtual address, and an address length included in the address mapping group.

3. The virtualization method according to claim 2, characterized in that: In each address mapping group, the mapping relationship between the virtual address and the physical address is unique.

4. The virtualization method according to claim 2, characterized in that: The range of addresses in each address mapping group overlaps with the range of addresses in the remaining N address mapping groups, where N is an integer greater than or equal to zero.

5. The virtualization method according to claim 2, wherein: Configuring the address translation table also includes the following steps: Configure the priority of each address mapping group. The priority of each address mapping group is different.

6. The virtualization method according to claim 2, wherein: Configuring the address translation table also includes the following steps: Set the enable status of each address mapping group.

7. The virtualization method according to claim 2, characterized in that: The conversion between the virtual address and the physical address of the memory is realized by the address conversion table, comprising the steps of: Determine the address mapping group corresponding to the input address; and Address translation is performed according to the information of the corresponding address mapping group.

8. The virtualization method according to claim 7, characterized in that: Determining the address translation group corresponding to the input virtual address comprises the steps of: The input address is compared with the address range of each address mapping group one by one according to the priority, and the first address mapping group that successfully matches is used as the corresponding address mapping group, wherein successful matching means that the input address is within the address range of the address mapping group.

9. A deep learning system, characterized in that: include: A task management unit, which is configured to schedule and manage computing tasks; A computing unit, which is communicatively connected to the task management unit, adopts a DSA architecture, and includes at least one computing core, a first address translation module, and a shared cache module, wherein each computing core includes an independent on-chip cache area, and the first address translation module is configured to map the physical addresses of multiple available memory segments of the shared cache module into continuous memory space virtual addresses according to the virtualization method according to any one of claims 1 to 8; and An external storage unit is communicatively connected to the computing unit via a second address translation module, wherein the second address translation module is configured to map the program code of the computing task to a specified virtual address space according to the virtualization method as described in any one of claims 1 to 8.

10. A task switching method for a deep learning system as claimed in claim 9, characterized in that: Includes steps: After receiving a computing task with a higher priority, the task management unit notifies the computing unit to perform on-site switching; After the current calculation of each core is completed, the calculation unit saves the calculation result of each core to the shared cache module, and sends the memory occupancy of the shared cache module to the task management unit; The task management unit determines whether the remaining memory of the shared cache module meets the memory requirement of the computing task: If the conditions are met, configuring the first address translation module to map the physical address of the remaining memory to a continuous virtualized address; as well as If not, first move at least part of the data in the shared cache module to the external storage module so that the remaining memory of the shared cache module is not less than the memory requirement of the computing task, and then configure the first address translation module to map the physical address of the remaining memory to a continuous virtualized address; The computing unit saves the scene of each core and sends a saving completion signal to the task management unit; The task management unit configures the second address translation module to map the computing task to a specified virtual address space; The task management unit notifies the computing unit to execute a computing task; as well as After the computing task is completed, the computing unit and the task management unit perform on-site recovery.

Images