Method, system, terminal and medium for accessing host space across bus domains

Abstract

The present invention discloses a method, system, terminal and medium for accessing the host space across bus domains, the method comprising: obtaining the request access information of the virtual device in the device lender to access the device borrower, and determining the request address of the request access information; obtaining the real address of the virtual device accessing the device borrower based on the BAR space of the connection bridge, and replacing the request address with the real address; forwarding the real address to the bus domain of the device borrower based on the connection bridge. The present invention can realize access to the device borrower from the virtual device, support DMA, MSI / MSI-X operations of the virtual device, and improve the performance of the read operation by reasonably caching the address of the Memory read operation sent by the virtual device.

Description

Technical Field

[0001] The present invention relates to the field of device virtualization technology, and in particular to a method, system, terminal and medium for accessing a host machine space across bus domains. Background Art

[0002] PCIe device virtualization is usually the process of emulating and managing physical PCIe devices in a virtual machine environment under only one PCIe bus domain. Virtualizing PCIe devices allows virtual machines to access and use these devices as if they were directly connected to the virtual machine.

[0003] Currently, PCIe devices across PCIe bus domains (two or more host systems) can expose access interfaces to each other through PCIe NTB (Non-Transparent Bridge), such as interrupts, BARs, and DMA execution. Current PCIe device virtualization cannot achieve PCIe device virtualization across PCIe bus domains (i.e., two or more host systems). In addition, the operating performance of the prior art is not high when accessing the device borrower from the virtualized device.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method, system, terminal and medium for accessing the host space across bus domains in view of the above-mentioned defects of the prior art. The technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for accessing a host space across bus domains, wherein the method comprises:

[0007] Obtaining access request information of the device lender for the virtual device to access the device borrower, and determining a request address of the access request information;

[0008] Acquire the real address accessed by the virtual device to the device borrower based on the BAR space of the connection bridge, and replace the requested address with the real address;

[0009] The real address is forwarded to the bus domain of the device borrower based on the connection bridge.

[0010] In one implementation, the BAR space of the connection bridge includes: MS I / MS IX interrupt space, Memory write access space, and Memory read access space;

[0011] Wherein, the MS I / MS IX interrupt space provides an address for each MS I / MS IX interrupt of each virtualized device;

[0012] The memory write access space provides an address space for each virtualized device;

[0013] The memory read access space provides an address space for each virtualized device.

[0014] In one implementation, the obtaining of the access request information of the virtual device in the device lender to access the device borrower includes:

[0015] If the access type of the requested access information is a Memory write operation, then when the control bridge hijacks the Memory write operation, the Memory write operation is split into a first write operation and a second write operation;

[0016] Based on the first write operation, write the real address corresponding to the memory write operation of the virtual device into the write address of the memory write access space of the connection bridge;

[0017] Based on the second write operation, the request for the virtual device Memory write operation is sent to the write request address space of the Memory write access space of the connection bridge.

[0018] In one implementation, the bridge-based BAR space obtains the real address accessed by the virtual device to the device borrower, and replaces the request address with the real address, including:

[0019] If the access type of the requested access information is a Memory write operation, a real address in the Memory write access space of the connection bridge is obtained, and the real address replaces the requested address of the Memory write operation of the virtual device.

[0020] In one implementation, the obtaining of the access request information of the virtual device in the device lender to access the device borrower includes:

[0021] If the access type of the requested access information is a memory read operation, when the control bridge hijacks the memory read operation, the memory read operation is split into a write operation and a read operation;

[0022] Based on the write operation, write the real address corresponding to the memory read operation of the virtual device into the write area of ​​the memory read access space of the connection bridge;

[0023] Based on the read operation, the request for the Memory read operation of the virtual device is sent to the read request address space of the Memory read access space of the connection bridge.

[0024] In one implementation, the bridge-based BAR space obtains the real address accessed by the virtual device to the device borrower, and replaces the request address with the real address, including:

[0025] If the access type of the requested access information is a Memory read operation, a real address in the Memory read access space is obtained, and the real address replaces the requested address of the Memory read operation of the virtual device.

[0026] In one implementation, the method further includes:

[0027] When the access type of the requested access information is an MS I / MS IX interrupt request, the MS I / MSI-X interrupt request of the virtualized device is redirected by the bridge driver or the control bridge to the MS I / MS IX interrupt space of the connection bridge;

[0028] After the MSI / MS IX interruption space of the connection bridge receives the MSI / MS IX interruption request, determining the interruption number according to the request address of the MSI / MS IX interruption request;

[0029] The corresponding interrupt is sent to the bus domain of the device borrower based on the connection bridge.

[0030] In a second aspect, an embodiment of the present invention provides a system for accessing a host space across bus domains, wherein the system is applied to a method for accessing a host space across bus domains as described in any one of the above schemes, and the system includes: a device borrower, a device lender, and a connection bridge, wherein the connection bridge is used to connect buses corresponding to the device borrower and the device lender respectively;

[0031] The device lender includes the control bridge and the virtualized device, and the control bridge is used to connect the bus of the virtualized device and the device lender;

[0032] The connection bridge is used to receive access request information and obtain the real address accessed by the virtual device to the device borrower, replace the request address with the real address, and forward the real address to the bus domain of the device borrower.

[0033] In the third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a cross-bus domain host space access program stored in the memory and executable on the processor, and when the processor executes the cross-bus domain host space access program, the steps of the cross-bus domain host space access method of any one of the above-mentioned schemes are implemented.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a program for accessing the host machine space across bus domains is stored on the computer-readable storage medium, and when the program for accessing the host machine space across bus domains is executed by a processor, the steps of the method for accessing the host machine space across bus domains described in any one of the above-mentioned schemes are implemented.

[0035] Beneficial effect: Compared with the prior art, the present invention provides a method for accessing the host space across bus domains. The present invention first obtains the request access information of the virtual device in the device lender to access the device borrower, and determines the request address of the request access information. Then, based on the BAR space of the connection bridge, the real address of the virtual device accessing the device borrower is obtained, and the real address replaces the request address. Finally, the real address is forwarded to the bus domain of the device borrower based on the connection bridge. The present invention can realize access to the device borrower from the virtual device, support DMA, MS I / MS IX operations of the virtual device, and improve the performance of the read operation by reasonably caching the address of the Memory read operation sent by the virtual device. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a preferred embodiment of a method for accessing a host space across bus domains provided in an embodiment of the present invention.

[0037] Figure 2 A structural diagram of a system for accessing host space across bus domains provided in an embodiment of the present invention.

[0038] Figure 3 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0041] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0042] This embodiment provides a method for accessing the host space across bus domains. The method based on this embodiment can realize access to the device borrower from the virtualized device, support DMA, MS I / MS IX operations of the virtualized device, and improve the performance of the read operation by reasonably caching the address of the memory read operation sent by the virtualized device.

[0043] like Figure 1 As shown in , the cross-bus domain method for accessing the host space of this embodiment can be applied to a terminal or a server. The cross-bus domain method for accessing the host space includes the following steps:

[0044] Step S100, obtaining access information requested by the virtual device in the device lender to access the device borrower, and determining a request address for the access information;

[0045] Step S200: Acquire the real address accessed by the virtual device to the device borrower based on the BAR space of the connection bridge, and replace the requested address with the real address;

[0046] Step S300: forwarding the real address to the bus domain of the device borrower based on the connection bridge.

[0047] For example, host machine H1 is the device lender, and the bus of H1 is mounted with device A through the control bridge. Host machine H2 is the device borrower. In order to borrow device A, H2 generates a virtual device a corresponding to device A through virtualization technology, and device A is the virtualized device. This embodiment is to achieve the access of the virtualized device to the host machine H2. Since H1 and H2 correspond to different bus domains, in order to achieve cross-bus domain device access, a memory space (i.e., BAR space) EP0 is set in H2, and a memory space EP1 is set in H1. EP0 and EP1 form the two ends of the connection bridge, and the connection bridge forwards the access request information between H1 and H2 across the bus domain. When the virtualized device (i.e., device A) accesses the host machine H2 (device borrower), the request address of the request access information is determined, the BAR space of the connection bridge obtains the real address of device A accessing the host machine H2, and replaces the request address with the real address, and then the connection bridge forwards the real address to the bus domain of the host machine H2.

[0048] Specifically, this embodiment provides a system for accessing the host space across bus domains, combining Figure 2 As shown, the system includes: a device borrower, a device lender, and a connection bridge, wherein the connection bridge is used to connect the buses corresponding to the device borrower and the device lender respectively; the device lender includes the control bridge and the virtualized device, wherein the control bridge is used to connect the buses of the virtualized device and the device lender. The connection bridge is used to receive access request information and obtain the real address of the virtualized device accessing the device borrower, replace the request address with the real address, and forward the real address to the bus domain of the device borrower.

[0049] In this embodiment, the connection bridge is a PCIe non-transparent bridge (Non-transparent Bridge), which is represented as a PCIe EP (Endpoint) in the PCIe bus of the device borrower and the device lender. In this embodiment, the EP of the device borrower is called EP0, and the EP of the device lender is called EP1. Both EP0 and EP1 of the connection bridge have specific address spaces. When the device borrower accesses the specific space of EP0, the connection bridge will convert the access through the address space and send it on the PCIe bus of the device lender through EP1, so as to realize the access from the device borrower to the device lender; conversely, when the device lender accesses the specific space of EP1, the connection bridge will convert the access through EP0 on the PCIe bus of the device borrower after the address conversion operation, so as to realize the access from the device lender to the device borrower.

[0050] One end of the control bridge of this embodiment is connected to the PCIe bus of the device lender, and the other end is connected to the virtualized device, and it has the functions of a PCIe transparent bridge and a non-transparent bridge. As a transparent bridge, the PCIe bus topology of the device lender is not changed, ensuring the normal use of the virtualized device at the device lender. As a non-transparent bridge, when the device borrower uses the virtualized device, the PCIe data entering and leaving the virtualized device can be further processed and controlled, including: realizing the access of the connection bridge to the configuration space of the virtualized device; realizing the redirection of the DMA request and MS I / MS IX request of the virtualized device.

[0051] The BAR space of the connection bridge of this embodiment is divided into three areas (which can be implemented on one BAR space or using multiple BAR spaces), including: MS I / MS IX interrupt space, Memory write access space and Memory read access space, as shown in the connection bridge EP1 in 2. The MS I / MS IX interrupt space provides an address for each MS I / MS IX interrupt of each virtualized device; the Memory write access space provides an address space for each virtualized device, including: a real address write address and a write request address space; the size of the write request address space is not less than 4kB to support the feature of the PC I e bus writing a maximum of 4kB of data at a time. The Memory read access space provides an address space for each virtualized device, including: a real address write area and a read request address space. The size of the real address write area is not less than 256 to support the feature that a PC I e device has a maximum of 256 uncompleted non-posted requests; the size of the read request address space is not less than 4kB to support the feature that the PC I e reads a maximum of 4kB of data at a time.

[0052] When accessing, if the access type of the requested access information is a Memory write operation, then when the control bridge hijacks the Memory write operation, the Memory write operation is split into a first write operation (i.e., write operation 1) and a second write operation (i.e., write operation 2) in order. Then, based on the first write operation, this embodiment writes the real address corresponding to the Memory write operation of the virtual device into the write address of the Memory write access space of the connection bridge. Then, based on the second write operation, the request for the Memory write operation of the virtual device is sent to the write request address space of the Memory write access space of the connection bridge. When the connection bridge EP1 receives "write operation 1", it records the data of the first write operation, i.e., the real address corresponding to the Memory write operation of the virtual device. When it receives "write operation 2", it obtains the real address in the Memory write access space of the connection bridge, and replaces the request address of the Memory write operation of the virtual device with the real address. The request after the address replacement is sent to the PCI e bus domain of the device borrower through EP0.

[0053] If the access type of the requested access information is a Memory read operation, when the control bridge hijacks the Memory read operation, the Memory read operation is split into a write operation and a read operation in order. Then, based on the write operation, the real address corresponding to the Memory read operation of the virtual device is written into the write area of ​​the Memory read access space of the connection bridge; based on the read operation, the request for the Memory read operation of the virtual device is sent to the read request address space of the Memory read access space of the connection bridge. When the connection bridge EP1 receives the write operation, it caches the data of the write operation, that is, the real address corresponding to the Memory read operation of the virtual device. After receiving the read operation, it obtains the real address in the Memory read access space and replaces the request address of the Memory read operation of the virtual device with the real address. The request after the address is replaced is sent to the PCI e bus domain of the device borrower through EP0. When the connection bridge EP1 forwards the access of the virtual device, the address of the Memory read operation is cached, and the cache size for each virtual device is not less than 256 (a PCI e device has a maximum of 256 unfinished non-posted requests) to improve the performance of the read operation.

[0054] In addition, when the access type of the requested access information is an MSI / MS IX interrupt request, the MSI / MS IX interrupt request of the virtualized device is redirected by the bridge driver or the control bridge to the MSI / MS IX interrupt space of the connection bridge. Specifically, the bridge driver directly modifies the interrupt address in the interrupt vector table of the virtualized device during the configuration phase, or the control bridge intercepts the MSI / MS IX interrupt request and reissues it after modifying the interrupt address. After the MSI / MS IX interrupt space of the connection bridge receives the MSI / MS IX interrupt request, the interrupt number corresponding to the connection bridge EP0 is determined according to the request address of the MSI / MS IX interrupt request; and the corresponding interrupt is sent to the bus domain of the device borrower based on the connection bridge EP0. When the device borrower loads the driver of the virtualized device itself (such as the driver provided by NVID IA for its GPU), the interrupt registration of the virtual device is hijacked, and the interrupt handling function is associated with the corresponding interrupt response of the connection bridge EP0, and the corresponding interrupt is generated on the device borrower and the correct interrupt handling function is called.

[0055] Based on the above embodiments, the present invention also provides a system for accessing host space across bus domains, such as Figure 2 As shown in , the system of this embodiment is applied to the method for accessing the host space across bus domains described in the above embodiments, and the system includes: a device borrower, a device lender, and a connection bridge, the connection bridge is used to connect the buses corresponding to the device borrower and the device lender respectively; the device lender includes the control bridge and the virtualized device, the control bridge is used to connect the bus of the virtualized device and the device lender; the connection bridge is used to receive request access information and obtain the real address of the virtualized device to access the device borrower, replace the request address with the real address, and forward the real address to the bus domain of the device borrower.

[0056] The functions of each module in the cross-bus domain access to the host space system of this embodiment are the same as those in each step of the above method embodiment, and will not be repeated here.

[0057] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 3 The terminal may include one or more processors 100 ( Figure 3Only one is shown), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a program for accessing a host space across bus domains. When one or more processors 100 execute the computer program 102, the various steps in the embodiment of the method for accessing a host space across bus domains can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the embodiment of the method for accessing a host space across bus domains can be implemented, which is not limited here.

[0058] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0059] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0060] Those skilled in the art will understand that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0061] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accessing a host space across bus domains, characterized in that: The method comprises: Obtaining access request information of the device lender for the virtual device to access the device borrower, and determining a request address of the access request information; Acquire the real address accessed by the virtual device to the device borrower based on the BAR space of the connection bridge, and replace the requested address with the real address; Forwarding the real address to the bus domain of the device borrower based on the connection bridge; The obtaining of the access request information of the virtual device in the device lender to access the device borrower includes: If the access type of the requested access information is a Memory write operation, then when the control bridge hijacks the Memory write operation, the Memory write operation is split into a first write operation and a second write operation; Based on the first write operation, write the real address corresponding to the memory write operation of the virtual device into the write address of the memory write access space of the connection bridge; Based on the second write operation, the request for the virtual device Memory write operation is sent to the write request address space of the Memory write access space of the connection bridge.

2. The method for accessing the host space across bus domains according to claim 1, characterized in that: The BAR space of the connection bridge includes: MSI / MSI-X interrupt space, Memory write access space and Memory read access space; Wherein, the MSI / MSI-X interrupt space provides an address for each MSI / MSI-X interrupt of each virtualized device; The memory write access space provides an address space for each virtualized device; The Memory read access space provides an address space for each virtualized device.

3. The method for accessing the host space across bus domains according to claim 1, characterized in that: The bridge-based BAR space obtains the real address accessed by the virtual device to the device borrower, and replaces the request address with the real address, including: If the access type of the requested access information is a Memory write operation, a real address in the Memory write access space of the connection bridge is obtained, and the real address replaces the requested address of the Memory write operation of the virtual device.

4. The method for accessing the host space across bus domains according to claim 2, characterized in that: The obtaining of the access request information of the virtual device in the device lender to access the device borrower includes: If the access type of the requested access information is a memory read operation, when the control bridge hijacks the memory read operation, the memory read operation is split into a write operation and a read operation; Based on the write operation, write the real address corresponding to the memory read operation of the virtual device into the write area of ​​the memory read access space of the connection bridge; Based on the read operation, the request for the Memory read operation of the virtual device is sent to the read request address space of the Memory read access space of the connection bridge.

5. The method for accessing the host space across bus domains according to claim 2, characterized in that: The bridge-based BAR space obtains the real address accessed by the virtual device to the device borrower, and replaces the request address with the real address, including: If the access type of the requested access information is a Memory read operation, a real address in the Memory read access space is obtained, and the real address replaces the requested address of the Memory read operation of the virtual device.

6. The method for accessing the host space across bus domains according to claim 2, characterized in that: The method further comprises: When the access type of the requested access information is an MSI / MSI-X interrupt request, the MSI / MSI-X interrupt request of the virtualized device is redirected to the MSI / MSI-X interrupt space of the connection bridge by the bridge driver or the control bridge; After receiving the MSI / MSI-X interrupt request in the MSI / MSI-X interrupt space of the connection bridge, determining the interrupt number according to the request address of the MSI / MSI-X interrupt request; The corresponding interrupt is sent to the bus domain of the device borrower based on the connection bridge.

7. A system for accessing host space across bus domains, characterized in that: The system is applied to the method for accessing the host space across bus domains as described in any one of claims 1 to 6 above, and the system comprises: a device borrower, a device lender, and a connection bridge, wherein the connection bridge is used to connect the buses corresponding to the device borrower and the device lender respectively; The device lender includes a control bridge and a virtualized device, wherein the control bridge is used to connect the virtualized device and a bus of the device lender; The connection bridge is used to receive access request information and obtain the real address accessed by the virtual device to the device borrower, replace the request address with the real address, and forward the real address to the bus domain of the device borrower.

8. A terminal, characterized in that: The terminal includes a memory, a processor, and a cross-bus domain host space access program stored in the memory and executable on the processor. When the processor executes the cross-bus domain host space access program, the steps of the cross-bus domain host space access method as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for accessing the host space across bus domains. When the program for accessing the host space across bus domains is executed by the processor, the steps of the method for accessing the host space across bus domains as described in any one of claims 1-6 are implemented.

Images