A DMA method and system for a system-on-chip in a virtualized environment

Abstract

The present invention provides a DMA method and system for a system-on-chip in a virtualized environment, including DMA hardware in two directions: reading host memory - writing FPGA memory and reading FPGA memory - writing host memory. Taking reading host memory - writing FPGA memory as an example, a virtual machine running on the host initiates an instruction to access FPGA memory. The DMA module receives this instruction, inputs the FPGA memory access address and the size of the transferred bytes in the form of a data stream to the command word generation module to convert the data stream into a command word, and the stream data conversion module accesses the FPGA memory according to this command word. The present invention realizes using only one DMA descriptor in each of the above two directions by designing DMA software and hardware, so as to reduce the overhead of host hardware address translation competition. In an SoC FPGA, the CPU and FPGA interconnected by the on-chip bus are tightly coupled, and the above two-direction DMA is directly passed through in a virtual machine running on its CPU host.

Description

Technical Field

[0001] The present invention relates to the fields of computer system architecture, system virtualization, and heterogeneous processing systems, and particularly relates to a direct memory access (DMA) system for a system on chip in a virtualized environment. Background Art

[0002] As Moore's Law slows down, and at the same time, the demand for computing power from new applications such as artificial intelligence in the cloud has increased explosively, it has become increasingly difficult for general-purpose CPUs to meet the requirements of these applications. To address the computing power bottleneck and reduce the burden on the CPU, more and more accelerators have begun to be deployed in the cloud. Different types of hardware accelerators such as GPUs, field programmable gate arrays (FPGAs), and tensor processing units (TPUs) have emerged continuously in commercial clouds.

[0003] Due to its hardware programmability and high computing energy efficiency ratio, FPGAs have been widely deployed in the cloud for application acceleration. Cloud providers such as Amazon, Microsoft, and Alibaba have deployed FPGAs in the cloud to accelerate applications such as machine learning, graph computing, video image processing, blockchain, and gene analysis.

[0004] The deployment of FPGA accelerators relies on dynamic partial reconfiguration technology to reconfigure FPGAs. The dynamic partial reconfiguration technology divides the FPGA into a static area (SHELL) and a dynamic area (ROLE) where FPGA accelerators are deployed, ensuring that the static area and the dynamic areas of other accelerators remain unchanged and unaffected when reconfiguring the dynamic area of the user accelerator.

[0005] The use of cloud FPGAs relies on data copying between the virtual machines running on the host and the FPGA memory, and the data copying efficiency between the virtual machine and the accelerator FPGA memory affects the acceleration efficiency of the FPGA accelerator to a certain extent. However, for an SoC (System On Chip) FPGA embedded with a CPU core, since the interface between the host and the FPGA is not a PCIe interface, there is a lack of support for PCIe DMA, resulting in imperfect software and hardware support for DMA data copying between different address spaces of the virtual machine and the FPGA memory, especially in the case of multiple accelerators sharing the FPGA memory in the FPGA dynamic partial reconfiguration scenario.

[0006] Currently, for SoC FPGA, whether it is from the host to the FPGA memory or from the FPGA memory to the host, the existing DMA hardware needs to access the host memory to obtain the DMA descriptor, and the hardware address translation competition overhead is relatively large in the virtualization environment. In addition, for SoC FPGA, when multiple dynamic regions are partitioned on the FPGA to deploy multiple accelerators, there is currently no mature solution for performing DMA data copying of different accelerators in the virtual machines running on the CPU host. Summary of the Invention

[0007] In view of the lack of a mature solution for performing DMA copying in the CPU virtual machine on the current SoC FPGA platform, a DMA system for a system-on-chip in a virtualization environment is proposed, including DMA hardware and driver software. The present invention can be used to perform data copying between a virtual machine and the FPGA accelerator memory in a virtual machine running on the host of the SoC FPGA, and only one DMA descriptor needs to be obtained in each direction.

[0008] Specifically, the present invention proposes a DMA method for a system-on-chip in a virtualization environment, which includes:

[0009] The system-on-chip includes a tightly coupled CPU host and FPGA interconnected by an on-chip bus. When the virtual machine running on the host transfers data to the FPGA memory, step 1 is executed; when transferring data from the FPGA memory to the virtual machine, step 2 is executed;

[0010] Step 1: The virtual machine running on the host initiates an instruction to transfer data to the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction to access the FPGA memory, and inputs the FPGA memory access address and the transfer byte size of the instruction to access the FPGA memory in the form of a data stream to the command word generation module. The command word generation module converts the data stream into a command word. After the stream data conversion module accepts the command word, the module accesses the FPGA memory according to the FPGA memory access address and the transfer byte size.

[0011] Step 2: The virtual machine running on the host initiates an instruction to read data from the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction, and inputs the virtual machine address and the transfer byte size of the instruction in the form of a data stream to the command word generation module. The command word generation module converts the data stream into a command word. After the stream data conversion module accepts the command word, the module returns data to the virtual machine according to the virtual machine address and the transfer byte size.

[0012] For the DMA method for a system-on-chip in a virtualization environment, the DMA hardware is directly connected to each virtual machine on the host. The DMA hardware includes the DMA module, the command word generation module, and the stream data conversion module.

[0013] The described DMA method for the system - on - chip in a virtualized environment, where the FPGA memory is divided according to the number of dynamically - reconfigurable regions (simply referred to as dynamic regions), and different FPGA dynamic regions have the same FPGA memory view;

[0014] According to the current FPGA dynamic region resource usage, after the system allocates dynamic region resources, it records the FPGA accelerator numbers of each dynamic region. When executing step 1 or step 2, it obtains the FPGA memory base addresses of different dynamic regions according to the FPGA accelerator numbers, and by configuring different memory base - address offsets for each dynamic region, it enables the accelerators deployed in different dynamic regions to only access the FPGA memory of the corresponding address space.

[0015] The described DMA method for the system - on - chip in a virtualized environment, where the DMA descriptor includes user - field data, and the command - word generation module constructs the command word according to the user - field data and sends it to the stream - data conversion module.

[0016] The present invention also proposes a DMA system for the system - on - chip in a virtualized environment, which includes:

[0017] The system - on - chip includes a CPU host and an FPGA interconnected on - chip. When the virtual machine running on the host transfers data to the FPGA memory, it calls the first transfer module, and when transferring data from the FPGA memory to the virtual machine, it calls the second transfer module;

[0018] The first transfer module is used to enable the virtual machine running on the host to initiate an instruction to transfer data to the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction to access the FPGA memory, and inputs the FPGA memory access address and the transfer byte size of the instruction to access the FPGA memory in the form of a data stream to the command - word generation module. The command - word generation module converts the data stream into a command word. After the stream - data conversion module accepts the command word, the module accesses the FPGA memory according to the FPGA memory access address and the transfer byte size.

[0019] The second transfer module is used to enable the virtual machine running on the host to initiate an instruction to read data from the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction, and inputs the virtual - machine address and the transfer byte size of the instruction in the form of a data stream to the command - word generation module. The command - word generation module converts the data stream into a command word. After the stream - data conversion module accepts the command word, the module returns data to the virtual machine according to the virtual - machine address and the transfer byte size.

[0020] The DMA hardware is directly passed through in the virtual machine, and the DMA hardware includes the DMA module, the command - word generation module, and the stream - data conversion module.

[0021] The described DMA system for a system-on-chip in a virtualized environment, where the FPGA memory is divided according to the number of dynamically reconfigurable regions (simply referred to as dynamic regions), and different FPGA dynamic regions have the same FPGA memory view;

[0022] According to the current FPGA dynamic region resource usage, the system records the FPGA accelerator numbers of each dynamic region after allocating dynamic region resources. When performing this step 1 or this step 2, the FPGA memory base address is obtained according to the FPGA accelerator number, and by configuring different memory base address offsets for each dynamic region, it is realized that accelerators in different dynamic regions can only access the specified range of FPGA memory.

[0023] The described DMA system for a system-on-chip in a virtualized environment, where the DMA descriptor includes user field data, and the command word generation module constructs the command word according to the user field data and sends it to the stream data conversion module.

[0024] The present invention also proposes a storage medium for storing a program for executing any one of the DMA methods for a system-on-chip in a virtualized environment. The present invention also proposes a client for any one of the DMA systems for a system-on-chip in a virtualized environment.

[0025] As can be seen from the above solutions, the advantages of the present invention are:

[0026] The method of the present invention, for the SoC FPGA platform, supports implementing DMA data copying between the virtual machine and the accelerator FPGA memory in the virtual machine running on its host, and only needs to access the DMA descriptor once in each direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the DMA hardware structure diagram of the present invention;

[0028] Figure 2 It is the virtual machine DMA software and hardware framework diagram;

[0029] Figure 3 It is the data flow diagram for the virtual machine to transfer data using the DMA method. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] Currently, there is no mature solution for executing DMA copy in the CPU virtual machine of SoC FPGA. Moreover, when the existing DMA hardware performs data copy between the host and FPGA memory, it needs to read descriptors from the host memory twice in each direction. Therefore, the competition overhead of hardware address translation is relatively large. By customizing the design of the DMA hardware, the present invention uses only one DMA descriptor in each direction. Therefore, the DMA accesses the descriptor only once in the host memory in each direction, thus reducing the competition overhead of hardware address translation. Secondly, the present invention directly passes through the DMA in both directions in each virtual machine running on the SoC FPGA host, and by using different physical address offsets of the FPGA memory in the FPGA memory direction, it is possible to achieve the same FPGA memory address space for different partially reconfigurable regions, and perform secure DMA copy in the virtual machine.

[0031] In order to achieve the above technical effects, the present invention mainly includes the following key technical points:

[0032] Key point 1: By customizing the metadata for accessing the FPGA memory in the user field of the DMA descriptor, it is realized that only one DMA descriptor is used in each direction when performing DMA copy between the host and the FPGA memory;

[0033] Key point 2: By directly passing through two DMA hardwares in each virtual machine running on the FPGA host, DMA copy in both the H2C and C2H directions is realized;

[0034] Key point 3: By using different FPGA memory base addresses, different FPGA memories with a unified address space are provided for the accelerators deployed on different FPGA partially reconfigurable regions, and DMA data copy between different virtual machines and accelerators is supported through memory access offsets in the DMA driver.

[0035] In order to make the above features and effects of the present invention more clearly and understandably described, specific embodiments are hereinafter given and described in detail in conjunction with the accompanying drawings of the specification as follows.

[0036] Taking the SoC FPGA with the ARM architecture of the host CPU as an example, the DMA system framework for the on-chip system in the virtualization environment proposed by the present invention includes two parts: hardware and software, which are introduced separately below.

[0037] The DMA hardware system framework of the present invention is as Figure 1As shown in the figure. In both the H2C (Host to Card) and C2H (Card to Host) directions, DMA data copying is achieved through a customized command word generation module, in cooperation with a DMA (including but not limited to multi-channel DMA) that only contains an MM2S (Memory-Mapped to Stream) channel and a stream data conversion module (including but not limited to DataMover).

[0038] Taking reading the host memory and writing to the FPGA memory as an example, that is, the specific implementation principle of the H2C direction DMA is as follows:

[0039] In the DMA driver, customize the user APP fields of the DMA descriptor. These fields will be output from the CNTRL interface of the DMA in the form of multiple stream data. On the other hand, the stream data conversion module converts the input stream data into AXI4 data and outputs it to the downstream according to the command word input from the cmd interface. The command word contains the memory access address and the transfer byte size. Therefore, in the present invention, the APP field of the BD descriptor is defined in the DMA driver to include information such as the accelerator FPGA memory access address and the transfer size. Then these key information will be output in the form of a Stream stream through the DMA CNTRL interface. Then Figure 1 the command word generation module in the figure can piece together the command word required by the downstream stream data conversion module using the user field data in the obtained multiple DMA descriptors in hardware. Finally, the stream data conversion accesses the FPGA memory according to the specified FPGA memory physical address and size. The hardware implementation principle of the other direction C2H is similar to that of H2C. Therefore, whether it is the H2C or C2H direction, the DMA data copying between the host and the FPGA memory only requires one DMA descriptor of the DMA module.

[0040] The system framework for customizing the DMA working principle in the virtual machine is as Figure 2, in the FPGA SHELL, a pair of custom DMAs (in two directions of H2C and C2H) are allocated for each ROLE to perform DMA data copying between the host and the FPGA memory. At the same time, the accelerators deployed in the dynamic area perform accelerated computing by accessing the FPGA memory. In software implementation, the DMA driver and the custom DMA agent driver are deployed in the kernel of each virtual machine operating system. Therefore, the virtual machine tenant can perform DMA transmission through the user-mode h2c (corresponding to the H2C DMA hardware) and c2h (corresponding to the C2H DMA hardware) character devices exposed by the driver. Specifically, the operation of the custom DMA hardware in the virtual machine depends on the IOMMU (Input / Output Memory ManagemnetUnit, I / O memory management unit) of the ARM host to implement hardware address translation, and performs device passthrough through the VFIO (Virtual Function I / O) framework. In QEMU (Quick Emulator), the device passthrough model (vfio-axi-mcdma) of DMA is implemented by encapsulating the vfio-platform abstract device model. When the tenant virtual machine running on the host starts, DMA in two directions of H2C and C2H is passed through for bidirectional data copying. The QEMU startup parameters for enabling DMA passthrough are configured as follows:

[0041] -device vfio-axi-mcdma,host=81008000.axi_mcdma

[0042] -device vfio-axi-mcdma,host=81007000.axi_mcdma

[0043] The virtual machine DMA copy should access the legal accelerator FPGA memory address. In the present invention, the FPGA memory is divided according to the number of dynamically partially reconfigurable regions. For example, if the FPGA memory is 16GB and the FPGA logic is divided into 4 dynamic areas, then the memory from 0 to 4GB is used for dynamic area 1, 4 to 8GB is used for dynamic area 2, and so on. Moreover, different accelerators have the same memory view. For example, for a 16GB FPGA memory, the memory view of each accelerator is 0 to 4GB, which is achieved by configuring different base address offsets for the FPGA memory access addresses of each accelerator.

[0044] The implementation of the virtual machine DMA proxy driver ensures that accelerators deployed in different dynamic regions can only access the FPGA memory in the corresponding address space. The reason is that: in the present invention, the usage of dynamic region resources is recorded in real time in the SHELL, and after allocating the dynamic region resources, QEMU will record the accelerator number. Therefore, when the DMA proxy driver performs DMA transmission, it obtains the base address of the FPGA memory according to the accelerator number of the allocated dynamic region, so as to access the FPGA memory in the corresponding address space.

[0045] On the other hand, the virtual machine DMA proxy driver needs to cooperate with the customized DMA hardware. Since the customized DMA hardware initiates a memory access command to the stream data conversion module through the CNTRL stream data output by DMA, the present invention includes a custom user field in the DMA descriptor of the DMA driver that is consistent with the input stream data format of the command word generation module in Figure 1 the command word generation module.

[0046] Such as Figure 3 , the method for transmitting data through DMA in a virtual machine includes: Step 1, Step 2, Step 3, Step 4, Step 5, Step 6, and Step 7; wherein, Step 1 is to implement customized DMA in the H2C and C2H directions in the SHELL of the FPGA, Step 2 is to enable the host IOMMU software and hardware and the VFIO driver, Step 3 is to implement the reset function of DMA device passthrough in the host operating system kernel, Step 4 is to implement the device model of DMA passthrough in the virtual machine monitor, Step 5 is to pass through the H2C and C2H DMA devices when the tenant virtual machine starts, Step 6 is to implement and load the DMA-related driver in the virtual machine operating system kernel, and Step 7 is for the tenant in the virtual machine to perform DMA data transmission through the character device exposed by the DMA driver.

[0047] In addition, in order to implement Step 1 to implement customized DMA in the H2C and C2H directions in the SHELL of the FPGA, as Figure 1 , Step 1 specifically includes: Step 1.1 and Step 1.2; wherein, Step 1.1 is to implement the command word generation module, which converts the data stream (AXI Stream) containing information such as the FPGA memory access address and size output by DMA into the command word of the downstream stream data conversion module, and Step 1.2 is to implement the customized DMA data transmission hardware in the two directions of H2C and C2H between the host and the FPGA memory by integrating DMA, the command word generation module, and the stream data conversion module.

[0048] In addition, to enable the host IOMMU software and hardware and the VFIO driver in step 2, step 2 specifically includes: step 2.1, step 2.2, and step 2.3; where step 2.1 is to enable the host IOMMU hardware, step 2.2 is to enable and load the IOMMU driver in the host operating system kernel, and step 2.3 is to enable and load the VFIO driver in the host kernel.

[0049] In addition, to implement and load the DMA-related driver in the virtual machine operating system kernel in step 6, step 6 specifically includes: step 6.1, step 6.2, and step 6.3; where step 6.1 is to enable the DMA driver in the virtual machine, and step 6.2 is to implement and load the DMA proxy driver in the virtual machine, which cooperates with the customized DMA hardware and ensures that the virtual machine accesses the FPGA memory of the corresponding address space.

[0050] The following is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in cooperation with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment, and in order to reduce repetition, they will not be elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0051] The present invention also proposes a DMA system for a system-on-chip in a virtualized environment, which includes:

[0052] The system-on-chip includes a tightly coupled CPU host and an FPGA. When a virtual machine running on the host transfers data to the FPGA memory, it calls the first transfer module, and when transferring data from the FPGA memory to the virtual machine, it calls the second transfer module;

[0053] The first transfer module is used to enable a virtual machine running on the host to initiate an instruction to transfer data to the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction to access the FPGA memory, and inputs the FPGA memory access address and the transfer byte size of the instruction to access the FPGA memory in the form of a data stream to the command word generation module. The command word generation module converts the data stream into a command word. After the stream data conversion module accepts the command word, the module accesses the FPGA memory according to the FPGA memory access address and the transfer byte size.

[0054] The second transfer module is used to enable a virtual machine running on the host to initiate an instruction to read data from the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction, and inputs the virtual machine address and the transfer byte size of the instruction in the form of a data stream to the command word generation module. The command word generation module converts the data stream into a command word. After the stream data conversion module accepts the command word, the module returns data to the virtual machine according to the virtual machine address and the transfer byte size.

[0055] Direct the DMA hardware in a virtual machine, where the DMA hardware includes the DMA module, the command word generation module, and the stream data conversion module.

[0056] The DMA system for a system-on-chip in a virtualization environment, where the FPGA memory is divided according to the number of dynamically partial reconfigurable regions (simply referred to as dynamic regions), and the accelerators deployed in different FPGA dynamic regions have the same FPGA memory view;

[0057] According to the current FPGA dynamic region resource usage, after the system allocates dynamic region resources, it records the FPGA accelerator numbers of each dynamic region. When performing step 1 or step 2, it obtains the FPGA memory base address according to the FPGA accelerator number, and by configuring different memory base address offsets for each dynamic region, it realizes that the accelerators in different dynamic regions access the FPGA memory of the corresponding address space.

[0058] The DMA system for a system-on-chip in a virtualization environment, where the DMA descriptor includes user field data, and the command word generation module constructs the command word according to the user field data and sends it to the stream data conversion module.

[0059] The present invention also proposes a storage medium for storing a program for executing any one of the DMA methods for a system-on-chip in a virtualization environment. The present invention also proposes a client for any one of the DMA systems for a system-on-chip in a virtualization environment.

Claims

1. A DMA method for a system-on-chip in a virtualized environment, characterized in that, Comprising: The system - on - chip includes a tightly - coupled CPU host and an FPGA interconnected via an on - chip bus. When a virtual machine running on the host transfers data to the FPGA memory, step 1 is executed. When transferring data from the FPGA memory to the virtual machine, step 2 is executed; Step 1: The virtual machine running on the host initiates an instruction to transfer data to the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction to access the FPGA memory and inputs the FPGA memory access address and the transfer byte size of the instruction to access the FPGA memory in the form of a data stream to the command - word generation module. The command - word generation module converts the data stream into a command word. After the stream - data conversion module accepts the command word, the module accesses the FPGA memory according to the FPGA memory access address and the transfer byte size; Step 2: The virtual machine running on the host initiates an instruction to read data from the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction and inputs the virtual machine address and the transfer byte size of the instruction in the form of a data stream to the command - word generation module. The command - word generation module converts the data stream into a command word. After the stream - data conversion module accepts the command word, the module returns data to the virtual machine according to the virtual machine address and the transfer byte size; 2. The DMA method for the system-on-chip in the virtualization environment according to claim 1, wherein For each virtual machine on the host, a direct - pass DMA hardware is provided. The DMA hardware includes the DMA module, the command - word generation module, and the stream - data conversion module; 3. The DMA method for a system-on-chip in a virtualized environment according to claim 1 or 2, characterized in that, The FPGA memory is divided according to the number of dynamically - reconfigurable regions (simply referred to as dynamic regions), and different FPGA dynamic regions have the same FPGA memory view; According to the current resource usage of the FPGA dynamic regions, after the system allocates dynamic - region resources, it records the FPGA accelerator numbers of each dynamic region. When executing step 1 or step 2, it obtains the FPGA memory base address according to the FPGA accelerator number, and by configuring different memory base - address offsets for each dynamic region, it realizes that the accelerators deployed in different dynamic regions have the same FPGA memory address space; 4. The DMA method for a system-on-chip in a virtualized environment according to claim 1 or 2, characterized in that, The DMA descriptor includes user - field data, and the command - word generation module constructs the command word according to the user - field data and sends it to the stream - data conversion module; 5. A DMA system for a system-on-chip in a virtualized environment, characterized in that, Comprising: The system - on - chip includes a tightly - coupled CPU host and an FPGA interconnected via an on - chip bus. When a virtual machine running on the host transfers data to the FPGA memory, it calls the first transfer module. When transferring data from the FPGA memory to the virtual machine, it calls the second transfer module; The first transfer module is used to enable the virtual machine running on the host to initiate an instruction to transfer data to the FPGA memory according to the DMA descriptor. The DMA module accepts the instruction to access the FPGA memory and inputs the FPGA memory access address and the transfer byte size of the instruction to access the FPGA memory in the form of a data stream to the command - word generation module. The command - word generation module converts the data stream into a command word. After the stream - data conversion module accepts the command word, the module accesses the FPGA memory according to the FPGA memory access address and the transfer byte size; A second transmission module is configured to enable a virtual machine running on a host to initiate an instruction to read data from the FPGA memory according to a DMA descriptor. The DMA module receives the instruction and inputs the virtual machine address and the transmission byte size of the instruction to a command word generation module in the form of a data stream. The command word generation module converts the data stream into a command word. After receiving the command word, a stream data conversion module returns data to the virtual machine according to the virtual machine address and the transmission byte size.

6. The DMA system for a system-on-chip in a virtualized environment according to claim 5, wherein, Direct the DMA hardware for each virtual machine on the host, where the DMA hardware includes the DMA module, the command word generation module, and the stream data conversion module.

7. The DMA system for a system-on-chip in a virtualized environment according to claim 5 or 6, characterized in that, The FPGA memory is divided according to the number of dynamically partial reconfigurable regions (simply referred to as dynamic regions), and different FPGA dynamic regions have the same FPGA memory view. According to the current resource usage of the FPGA dynamic regions, the system records the FPGA accelerator numbers of each dynamic region after allocating dynamic region resources. When performing step 1 or step 2, the FPGA memory base address is obtained according to the FPGA accelerator number, and by configuring different memory base address offsets for each dynamic region, it is realized that the accelerators deployed in different dynamic regions have the same FPGA memory address space.

8. The DMA system for the system-on-chip in a virtualized environment according to claim 5 or 6, characterized in that, The DMA descriptor includes user field data, and the command word generation module constructs the command word according to the user field data and sends it to the stream data conversion module.

9. A storage medium is configured to store a program for executing any one of the DMA methods for a system on a chip in a virtualization environment as claimed in claims 1 to 4.

10. A client is configured to be used for any one of the DMA systems for a system on a chip in a virtualization environment as claimed in claims 5 to 8.

Images