System and method for high-speed data communication architecture for cloud gaming data storage and retrieval

Abstract

A cloud gaming system includes a cloud storage system and a cloud computing system connected together by a PCIe switch and a PCIe architecture. The PCIe switch is configured to map resources within the cloud storage system to a memory map of the cloud computing system. The PCIe switch is configured to map resources within the cloud computing system to a memory map of the cloud storage system. The PCIe architecture enables data input / output command buffers generated by the cloud computing system to be transferred to the cloud storage system by direct writes or direct reads between the cloud computing system and the cloud storage system. The PCIe architecture enables data to be transferred between the cloud storage system and the cloud computing system by direct writes or direct reads between the cloud computing system and the cloud storage system, as indicated by the command buffers.

Description

Background Art

[0001] The video game industry has seen many changes over the years. As computing power has expanded, video game developers have likewise created gaming software that takes advantage of the increased computing power. To do this, video game developers have been coding games that incorporate complex calculations and mathematics to produce a very realistic gaming experience.

[0002] In addition to adapting to the ever-expanding computing power, game developers are also developing online games that increase the interaction between users and gaming systems. Interactive online games provide users with the ability to interactively compete or play with other users on the Internet and capture such interactions. In addition, with the growing trend of cloud-based computing, game developers are developing new ways to play existing traditional games and develop new games using increased computing power and increased user interactivity. A cloud-based computing system may include a cloud-based processing server that executes a game application and communicates with a remote client system that is present at the location of the game player, wherein the remote client system may be configured to receive input from the game player and present video on a display for the game player to watch. In this case, the present invention arises as the times require. Summary of the invention

[0003] In an exemplary embodiment, a cloud gaming system is disclosed. The cloud gaming system includes a cloud storage system having a peripheral component interconnect express (PCIe) interface. The cloud storage system also includes a computer memory. The cloud gaming system includes a PCIe switch connected to the PCIe interface of the cloud storage system. The cloud gaming system includes a cloud computing system having a PCIe interface connected to the PCIe switch. The cloud computing system has a client memory mapping, and the client memory mapping stores memory addresses assigned to components of the cloud computing system. The PCIe switch is configured to convert the memory address stored in the client memory mapping to a memory address in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch to access a given memory address in the computer memory of the cloud storage system instead.

[0004] In an exemplary embodiment, a cloud gaming system is disclosed. The cloud gaming system includes a cloud storage system having a PCIe interface. The cloud storage system includes a computer memory. The cloud gaming system includes a first PCIe switch connected to the PCIe interface of the cloud storage system. The cloud gaming system includes a second PCIe switch connected to the first PCIe switch. The cloud gaming system includes a third PCIe switch connected to the first PCIe switch. The cloud gaming system includes a first cloud computing system having a PCIe interface connected to the second PCIe switch. The first cloud computing system has a first client memory mapping, which stores memory addresses assigned to components of the first cloud computing system. One or both of the first PCIe switch and the second PCIe switch are configured to convert the memory addresses stored in the first client memory mapping to memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the first client memory mapping is converted to access a given memory address in the computer memory of the cloud storage system instead. The cloud gaming system includes a second cloud computing system having a PCIe interface connected to the third PCIe switch. The second cloud computing system has a second client memory map that stores memory addresses assigned to components of the second cloud computing system. One or both of the first PCIe switch and the third PCIe switch are configured to translate the memory addresses stored in the second client memory map into memory addresses in the computer memory of the cloud storage system, such that access to a given memory address in the second client memory map is translated to access a given memory address in the computer memory of the cloud storage system instead.

[0005] In an exemplary embodiment, a method for requesting data in a cloud gaming system is disclosed. The method includes having a cloud gaming system, the cloud gaming system including a cloud storage system having a PCIe interface. The cloud storage system includes a computer memory. The cloud gaming system also includes a PCIe switch, the PCIe switch connected to the PCIe interface of the cloud storage system. The cloud gaming system also includes a cloud computing system, the cloud computing system having a PCIe interface connected to the PCIe switch. The cloud computing system includes a computer memory. The PCIe switch exposes a doorbell register monitored by the cloud storage system. The method also includes operating the cloud computing system to write to the doorbell register, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud storage system. The method also includes operating the cloud storage system to handle the interrupt. The interrupt instructs the cloud storage system to read a message from a specified location in the computer memory of the cloud storage system. The message instructs the cloud storage system to read the requested data from a storage device accessible to the cloud storage system.

[0006] In an exemplary embodiment, a method for requesting and transmitting data in a cloud gaming system is disclosed. The method includes having a cloud gaming system, the cloud gaming system including a cloud storage system and a cloud computing system connected to each other through a PCIe fabric. The cloud storage system includes a computer memory. The cloud computing system includes a computer memory. The method also includes operating the cloud computing system to generate a command buffer for a data read operation. The method also includes operating the cloud computing system to write the command buffer to the computer memory of the cloud storage system through the PCIe fabric. The method also includes operating the cloud computing system to notify the cloud storage system through the PCIe fabric that the command buffer has been written to the computer memory of the cloud storage system. The method also includes operating the cloud storage system to read the command buffer from the computer memory of the cloud storage system. The method also includes operating the cloud storage system to process the command buffer to read the requested data from a storage device accessible to the cloud storage system and write the requested data to the computer memory of the cloud storage system. The method also includes operating the cloud storage system to notify the cloud computing system through the PCIe fabric that the requested data is in the computer memory of the cloud storage system. The method also includes operating the cloud computing system to read the requested data from the computer memory of the cloud storage system over the PCIe architecture.

[0007] In an exemplary embodiment, a method for requesting and transmitting data in a cloud gaming system is disclosed. The method includes having a cloud gaming system, the cloud gaming system including a cloud storage system and a cloud computing system connected to each other through a PCIe fabric. The cloud storage system includes a computer memory. The cloud computing system includes a computer memory. The method also includes operating the cloud computing system to generate a command buffer for a data read operation. The method also includes operating the cloud computing system to write the command buffer to the computer memory of the cloud storage system through the PCIe fabric. The method also includes operating the cloud computing system to notify the cloud storage system through the PCIe fabric that the command buffer has been written to the computer memory of the cloud storage system. The method also includes operating the cloud storage system to read the command buffer from the computer memory of the cloud storage system. The method also includes operating the cloud storage system to process the command buffer to read the requested data from a storage device accessible to the cloud storage system. The method also includes operating the cloud storage system to write the requested data to the computer memory of the cloud computing system through the PCIe fabric. The method also includes operating the cloud storage system to notify the cloud computing system via the PCIe architecture that the requested data is in the computer memory of the cloud computing system.

[0008] In an exemplary embodiment, a method for requesting and transmitting data in a cloud gaming system is disclosed. The method includes having a cloud gaming system, the cloud gaming system including a cloud storage system and a cloud computing system connected to each other through a PCIe fabric. The cloud storage system includes a computer memory. The cloud computing system includes a computer memory. The method also includes operating the cloud computing system to generate a command buffer for a data read operation. The method also includes operating the cloud computing system to write the command buffer to the computer memory of the cloud computing system. The method also includes operating the cloud computing system to notify the cloud storage system through the PCIe fabric that the command buffer has been written to the computer memory of the cloud computing system. The method also includes operating the cloud storage system to read the command buffer from the computer memory of the cloud computing system through the PCIe fabric. The method also includes operating the cloud storage system to process the command buffer to read the requested data from a storage device accessible to the cloud storage system. The method also includes operating the cloud storage system to write the requested data to the computer memory of the cloud computing system through the PCIe fabric. The method also includes operating the cloud storage system to notify the cloud computing system via the PCIe architecture that the requested data is in the computer memory of the cloud computing system.

[0009] In an exemplary embodiment, a method for requesting and transmitting data in a cloud gaming system is disclosed. The method includes having a cloud gaming system, the cloud gaming system including a cloud storage system and a cloud computing system connected to each other through a PCIe fabric. The cloud storage system includes a computer memory. The cloud computing system includes a computer memory. The method also includes operating the cloud computing system to generate a command buffer for a data read operation. The method also includes operating the cloud computing system to write the command buffer to the computer memory of the cloud computing system. The method also includes operating the cloud computing system to notify the cloud storage system through the PCIe fabric that the command buffer has been written to the computer memory of the cloud computing system. The method also includes operating the cloud storage system to read the command buffer from the computer memory of the cloud computing system through the PCIe fabric. The method also includes operating the cloud storage system to process the command buffer to read the requested data from a storage device accessible to the cloud storage system and write the requested data to the computer memory of the cloud storage system. The method also includes operating the cloud storage system to notify the cloud computing system through the PCIe fabric that the requested data is in the computer memory of the cloud storage system. The method also includes operating the cloud computing system to read the requested data from the computer memory of the cloud storage system over the PCIe architecture.

[0010] In an exemplary embodiment, a method for data transfer in a cloud gaming system is disclosed. The method includes having a cloud gaming system, the cloud gaming system including a cloud storage system and a cloud computing system connected to each other via a PCIe fabric. The cloud storage system includes a computer memory. The cloud computing system includes a computer memory. The method also includes operating the cloud storage system to generate a file identification table, the file identification table specifying memory addresses storing different data files in the computer memory of the cloud storage system. The method also includes operating the cloud computing system to use the file identification table to determine the memory address of the storage required data file in the computer memory of the cloud storage system. The method also includes operating the cloud computing system to read the required data file from the computer memory of the cloud storage system via the PCIe fabric.

[0011] Other aspects of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrates the present invention by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1AAn overview of the PCIe architecture is shown.

[0013] Figure 1B An exemplary configuration of a PCIe switch is shown.

[0014] Figure 1C A diagram showing a PCIe link comprising one PCIe lane is shown.

[0015] Figure 2A A diagram is shown regarding how the PCIe architecture can be used to connect two different host computing devices according to some embodiments of the present invention.

[0016] Figure 2B The invention shows a method according to some embodiments of the present invention. Figure 2A An exemplary diagram of a memory mapping function performed by a PCIe switch in a host device to host device connection arrangement.

[0017] Figure 3A A diagram is shown regarding how the PCIe architecture can be used to connect a cloud storage system with a cloud computing system according to some embodiments of the present invention.

[0018] Figure 3B An exemplary diagram of some internal components of a cloud storage system and a cloud computing system according to some embodiments is shown.

[0019] Figure 4 An exemplary diagram of memory mapping functions performed by an NTB in a PCIe switch and an IOMMU in a cloud storage system according to some embodiments of the present invention is shown.

[0020] Figure 5 A cloud gaming configuration according to some embodiments is shown, in which a cloud storage system is connected to a certain number (D) of cloud computing systems via a PCIe switch, where D is greater than 1.

[0021] Figure 6 A cloud gaming configuration according to some embodiments is shown, wherein a cloud storage system is connected to a certain number (E) of PCIe switches via corresponding PCIe links, where E is greater than 1.

[0022] Figure 7 A cloud gaming configuration according to some embodiments is shown, in which a cloud storage system is connected to multiple cloud computing systems via a fan-out configuration of PCIe switches.

[0023] Figure 8 An exemplary compute sled is shown according to some embodiments.

[0024] Fig. 9An exemplary diagram of a rack that can be implemented within a cloud gaming center according to some embodiments is shown.

[0025] Fig.10 An example of a mapping table that maps file names to file identifiers according to some embodiments is shown.

[0026] Fig.11 A method for requesting and transmitting data in a cloud gaming system including a cloud storage system connected to a cloud computing system via a PCIe architecture is shown according to some embodiments.

[0027] Fig.12 A method for requesting and transmitting data in a cloud gaming system including a cloud storage system connected to a cloud computing system via a PCIe architecture is shown according to some embodiments.

[0028] Fig.13 A method for requesting and transmitting data in a cloud gaming system including a cloud storage system connected to a cloud computing system via a PCIe architecture is shown according to some embodiments.

[0029] Fig.14 A method for requesting and transmitting data in a cloud gaming system including a cloud storage system connected to a cloud computing system via a PCIe architecture is shown according to some embodiments.

[0030] Fig.15 A method for requesting and transmitting data between a cloud storage system and a cloud computing system in a cloud gaming system using doorbell registers exposed through a PCIe architecture is shown according to some embodiments.

[0031] Fig.16 A method for data transmission in a cloud gaming system according to some embodiments is shown. DETAILED DESCRIPTION

[0032] In the following description, many specific details are proposed in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without some or all of these specific details. In other cases, well-known process operations are not described in detail to avoid unnecessary confusion of the present invention.

[0033] The following detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings show diagrams according to exemplary embodiments. These exemplary embodiments (also referred to as "examples" in this article) are described in sufficient detail to enable those skilled in the art to practice the proposed subject matter. Without departing from the scope of protection claimed, the embodiments can be combined, other embodiments can be used, or structural, logical and electrical changes can be made. Therefore, the following detailed description should not be viewed in a restrictive sense, and the scope is limited by the attached claims and their equivalents. In this document, as is common in patent documents, the terms "one" and "a kind of" are used to include one or more. In this document, unless otherwise indicated, the term "or" is used to refer to a non-exclusive "or", so that "A or B" includes "A but not B", "B but not A" and "A and B".

[0034] The rack design for a cloud gaming system can include a cloud storage system and many cloud computing systems, such as a game console, where the cloud storage system is shared by the cloud computing system. The cloud storage system can include fast storage devices, such as solid-state drives (SSDs) and hard disk drives (HDDs). The rack design of the cloud gaming system assumes that the data storage and retrieval capabilities of the cloud storage system are faster than those of the cloud computing system (e.g., the SSD or HDD of the local game console in the rack). However, the data storage and retrieval capabilities (e.g., data write speed and / or data read speed) of the cloud computing system (game console) have been greatly improved. For example, the data storage and retrieval speed of the cloud computing system has become faster than the speed that can be achieved by the currently available fast non-volatile memory (NVMe) SSD. Therefore, the current challenge of the rack-based cloud gaming system is to ensure that data can be exchanged between the cloud computing system and the cloud storage system in the rack in a sufficiently efficient manner, thereby avoiding operational delays in the cloud computing system.

[0035] Furthermore, it is not currently possible to address the above challenges simply by replacing existing data storage devices within a cloud storage system with faster data storage devices, because the fastest data storage devices currently available are not fast enough to keep up with the data demands of a cloud computing system using conventional data exchange infrastructure (e.g., networking) that connects the cloud computing system with the cloud storage system. Additionally, the current data exchange infrastructure that connects the cloud computing system with the cloud storage system in a cloud gaming rack design presents performance challenges in delivering the required data across multiple cloud computing systems simultaneously. Furthermore, the increasing cost of the current data exchange infrastructure that connects the cloud computing system with the cloud storage system in a cloud gaming rack design is a concern.

[0036] Disclosed herein are systems and methods, wherein multiple cloud computing systems are connected to a cloud storage system using a fast peripheral component interconnect (PCIe) architecture within a rack of a cloud gaming system. Multiple cloud computing systems are connected to a cloud storage system using a PCIe architecture within a rack of a cloud gaming system so that game content can be directly stored in a computer memory on a cloud storage system for use by multiple cloud computing systems. In some embodiments, the computer memory on the cloud storage system is a random access memory (RAM). In some embodiments, the computer memory on the cloud storage system is a storage class memory that behaves like RAM, but is persistent because when the power supplied to the computer memory is turned off, the data stored in the storage class memory will not be lost. In some embodiments, the storage class memory provides a higher data storage capacity than RAM. In some embodiments, the storage class memory is configured as a hybrid memory module, which uses a large amount of flash memory or equivalent technology paired with RAM. Since the storage class memory behaves like RAM, the PCIe switch can address the storage class memory in the same manner as the manner used to address RAM. Furthermore, multiple cloud computing systems are connected to a cloud storage system using a PCIe architecture within a rack of the cloud gaming system to provide for initial storage of video game content on an NVMe data storage device accessible to the cloud storage system, and filling of the video game content from the NVMe data storage device to a computer memory of the cloud storage system upon a cache miss event, which occurs when any of the cloud computing systems attempts to read specific video game content from the computer memory of the cloud storage system. NVMe is an open logic device interface specification for accessing non-volatile storage media (e.g., NAND flash memory, etc.) attached via a PCIe link.

[0037] In the system and method disclosed herein, multiple cloud computing systems are connected to one or more shared cloud storage systems using PCIe switching capabilities. In various embodiments, multiple cloud computing systems and one or more shared cloud storage systems are installed in one or more racks in a computing center. Do not confuse the cloud computing system installed in the rack in the computing center with the remote client system of the user (e.g., the player of the video game) local. More specifically, it should be understood that in some embodiments, the cloud computing system and the cloud storage system in the rack in the computing center work together to generate the streaming video of the video game and transmit the streaming video to the remote client system of the user who is playing the video game. Using PCIe switching capabilities to connect multiple cloud computing systems to one or more shared cloud storage systems provides a technical basis for solving cloud game performance challenges (such as ensuring that data can be exchanged between the cloud computing system and the cloud storage system in the rack in a sufficiently efficient manner to avoid operational delays of the cloud computing system, and such as simultaneously transmitting required data across many cloud computing systems).

[0038] PCIe is a PCI-based interconnect protocol developed by the Peripheral Component Interconnect Special Interest Group (PCI-SIG). A high-performance, general-purpose input / output interconnect described in the Base Specification Revision 5.0 Version 1.0, PCI-SIG is the electronics industry consortium responsible for specifying the Peripheral Component Interconnect (PCI) computer bus, the Peripheral Component Interconnect-Extended (PCI-X) computer bus, and the PCIe computer bus. PCIe is a bidirectional, point-to-point data bus originally designed for personal computers (PCs) and servers to provide communication between host devices and other devices such as graphics processing units (GPUs), network cards, sound cards, storage controllers, etc. PCI The Base Specification Revision 5.0 Version 1.0 is incorporated herein by reference in its entirety for all purposes. It should be understood that any PCIe feature referenced herein refers to the PCIe 1.0 specification published by the PCI-SIG. The corresponding PCIe features described in any revision and / or version of the base specification. Therefore, it should be understood that any PCIe feature mentioned herein is not limited to only the PCIe The description of the corresponding PCIe feature in Revision 5.0 Version 1.0 of the Base Specification and may correspond to the PCI Description of the corresponding PCIe features in any revisions and / or versions of the Base Specification.

[0039] Figure 1A An overview of the PCIe architecture is shown. The PCIe architecture includes point-to-point PCIe links that interconnect various components. The components and PCIe links in the PCIe architecture define a hierarchy. Figure 1A An exemplary PCIe architecture of shows a central processing unit (CPU) 101 and computer memory 103 of a host device connected to a root complex 105 of the host device. In various embodiments, computer memory 103 can be RAM or storage class memory, or another type of computer memory substantially equivalent to RAM and / or storage class memory. Root complex 105 represents the root of an input / output hierarchy that connects CPU 101 and computer memory 103 to the input / output ports of the host device. Root complex 105 can have any number of PCIe ports. Each PCIe port of root complex 105 is an interface that defines a separate hierarchical domain, where each hierarchical domain corresponding to a port of root complex 105 may include any number of endpoints or sub-hierarchical domains including one or more PCIe switches and associated endpoints. For example, Figure 1ASome ports of the root complex 105 are shown connected to a plurality of PCIe endpoints 107 through corresponding PCIe links. Each PCIe endpoint 107 can be any type of system element that utilizes the PCIe bus architecture and protocol, such as a processor, a controller, a module, etc. In general, an endpoint in a PCIe architecture refers to a class of functions that can act as a requester or a completer of a PCIe transaction. An endpoint in a PCIe architecture can be a traditional endpoint, a PCIe endpoint, or a root complex integrated endpoint.

[0040] and, Figure 1A The ports of the root complex 105 are shown connected via PCIe links to a PCIe to PCI / PCI-X bridge 109. The PCIe to PCI / PCI-X bridge 109 is used to virtually or physically connect a PCI / PCI-X hierarchy 111 to a PCIe fabric, such as to a PCIe port of the root complex 105. Figure 1A Also shown is a PCIe port of the root complex 105 connected to a PCIe switch 113 via a PCIe link. The PCIe switch 113 is used to connect two or more PCIe ports to allow packets to be routed from one port to another. Figure 1A In the example of FIG. 1 , the PCIe switch 113 includes an upstream PCIe port connected to the root complex 105. The PCIe switch 113 also includes a plurality of downstream PCIe ports connected to a plurality of legacy endpoints 115 through corresponding PCIe links. The PCIe switch 113 also includes a plurality of downstream PCIe ports connected to a plurality of PCIe endpoints 117 through corresponding PCIe links. The PCIe switch 113 appears to the configuration software as a collection of two or more virtual (logical) PCI to PCI bridges.

[0041] Figure 1BAn exemplary configuration of a PCIe switch 113 is shown. The PCIe switch 113 includes an upstream PCIe port 119 and one or more downstream PCIe ports 125(1)-125(n). Each of the upstream PCIe port 119 and the one or more downstream PCIe ports 125(1)-125(n) is connected to a corresponding PCIe link. The PCIe switch 113 includes a virtual PCI-PCI bridge 121 connected to the upstream PCIe port 119. The PCIe switch 113 also includes one or more virtual PCI-PCI bridges 123(1)-123(n) connected to the one or more downstream PCIe ports 125(1)-125(n), respectively. The collection of virtual PCI-PCI bridges 121 and 123(1)-123(n) are connected together within the PCIe switch 113, as shown by line 127. The PCIe switch 113 is used to forward transactions using a PCI bridge mechanism such as address-based routing. The PCIe switch 113 is also used to forward all types of transaction layer packets between any group of PCIe ports. Each PCIe port of the PCIe switch 113 is configured to comply with the PCIe transaction flow control specification.

[0042] A PCIe link represents a dual simplex data communication channel between two components. A PCIe link supports at least one PCIe lane. Figure 1C A diagram of a PCIe link including one PCIe lane 129 is shown. Figure 1C A PCIe channel 129 is shown to include a low voltage differential driven transmit conductor pair 131 and a low voltage differential driven receive conductor pair 133, where component A is considered a transmitter component and component B is considered a receiver component. The transmit conductor pair 131 forms a unidirectional differential signaling mechanism. The receive conductor pair 133 forms a unidirectional differential signaling mechanism. Therefore, a single PCIe channel 129 includes four electrical signal conductors or traces.

[0043] Data is transmitted across PCIe lanes at one bit per cycle. To increase bandwidth and obtain higher data throughput, PCIe links can aggregate multiple PCIe lanes. The size of a PCIe link is represented by xN, where N is the number of PCIe lanes in the PCIe link. A x1 PCIe link has one PCIe lane (four wires) and carries one bit per cycle in each of the transmit and receive directions. A x2 PCIe link has two PCIe lanes (8 wires) and carries two bits per cycle in each of the transmit and receive directions. A x4 PCIe link has four PCIe lanes (16 wires) and carries four bits per cycle in each of the transmit and receive directions. The PCIe specification describes the operation of PCIe link sizes of x1, x2, x4, x8, x12, x16, and x32.

[0044] PCIe connections are serial connections that operate like a network rather than a bus. In the PCIe architecture, each PCIe link is mapped through a virtual PCI-to-PCI bridge and has an associated logical PCI bus. Figure 1A In the example of FIG. 1 , the virtual PCI-to-PCI bridge for a given PCIe link may be part of a PCIe port of the root complex 105, or part of an upstream PCIe port 119 of the PCIe switch 113, or part of a downstream PCIe port 125(1)-125(n) of the PCIe switch 113. The PCIe port of the root complex 105 is referred to as a root port, and is a virtual PCI-to-PCI bridge that originates a PCIe hierarchy domain from the root complex 105. Devices within the PCIe fabric are mapped into a configuration space such that each device will respond to a specific device number.

[0045] The PCIe architecture was originally designed to provide data communication between a host computing device and a client computing device. However, the PCIe architecture can also be used to provide data communication between different host computing devices, but such applications and corresponding technologies are not included in the PCIe specification. In order to implement host device to host device data communication through the PCIe architecture, additional hardware in the form of a PCIe switch is required. Figure 2A A diagram showing how the PCIe architecture can be used to connect two different host computing devices according to some embodiments of the present invention. Specifically, Figure 2AA first host computing device 201 is shown connected to a second host computing device 203 via a PCIe switch 205. The root complex of the first host computing device 201 has a PCIe port connected to the PCIe port of the PCIe switch 205 via a PCIe link 207. Also, the root complex of the second host computing device 203 has a PCIe port connected to the PCIe port of the PCIe switch 205 via a PCIe link 209. The PCIe switch 205 includes programmable logic to bridge the memory address spaces of the first host computing device 201 and the second host computing device 203. For example, the logic to bridge the memory address spaces of the first host computing device 201 and the second host computing device 203 may include a non-transparent bridge (NTB) or a special direct memory access (DMA) engine that is capable of communicating with both the first host computing device 201 and the second host computing device 203 and handling any memory address translations required.

[0046] In order to understand why it is not trivial to connect different host computing devices to each other to have direct data communication capabilities, it is helpful to consider the boot process of the host computing device. During the boot process of the host computing device, the host computing device creates a memory map through its basic input / output system (BIOS) or through its unified extensible firmware interface (UEFI) or through its operating system (OS) according to the configuration of the host computing device. The memory map of the host computing device is a component device that assigns memory addresses to the computer memory of the host computing device and the memory map of the host computing device. It should be understood that from a software development perspective, modern component devices that can be connected to the host computing device look like memory. Some examples of component devices include graphics processing units (GPUs), storage CPUs, decryption / encryption engines, decompression / compression engines, direct memory access (DMA) controllers, fast non-volatile memory (NVMe) solid-state drives (SSDs), serial advanced technology attachments (SATA) controllers, hard drives, etc. Many component devices can be connected to host computing devices. For a host computing device with a PCIe root complex, an enumeration process is performed at system power-up, during which component devices connected to the host computing device are discovered and each component device is assigned a unique memory address range in the host computing device's memory map.

[0047] Although multiple host computing devices can be electrically connected together in a direct manner via a PCIe link, such direct connection is problematic because each host computing device independently creates its own memory map when powered on, which can result in memory address conflicts between the multiple memory maps created by the multiple host computing devices. Figure 2AAs shown, the connection of PCIe switches between different host computing systems solves the problem of memory address conflicts between different host computing systems because the PCIe switch appears as another component device in the memory map created independently by the host computing device. Logic (such as NTB logic) within the PCIe switch is used to "map" the memory in one host computing device to a location within the memory map of another host computing device, thereby allowing "shared memory" between host computing devices. For example, with respect to Figure 2A , logic within the PCIe switch 205 is used to map a memory address range in the first host computing device 201 to a memory address range in the second host computing device 203, and vice versa.

[0048] Figure 2B The invention shows a method according to some embodiments of the present invention. Figure 2A 2 is an exemplary diagram of a memory mapping function performed by a PCIe switch 205 in a host device to host device connection arrangement. Figure 2B The left part of 201 shows the memory map of the first host computing device 201, which is independently created by the first host computing device when it is powered on. Figure 2B The right side of shows the memory map of the second host computing device 203, which is independently created by the second host computing device at boot time. The memory map of the first host computing device 201 includes a memory address range of 0xC0000000 to 0x0FFFFFFF (as an example, a size of 2 gigabytes (GB)) for the PCIe switch 205. Figure 2BIn the example of , accesses by the first host computing device 201 to the memory address range 0xC0000000 to 0x0FFFFFFF of the PCIe switch 205 are remapped by the remapping logic in the PCIe switch 205 to the corresponding memory address range in the memory map of the second host computing device 203, such as the first 2GB in the memory map of the second host computing device 203 in this example. It should be understood that a given memory address range in the memory map of the first host computing device 201 can be remapped by the PCIe switch 205 to any memory address range of corresponding size in the memory map of the second host computing device 203. Moreover, it should be understood that a given memory address range in the memory map of the first host computing device 201 can be remapped by the PCIe switch 205 to any multiple memory address ranges of cumulative corresponding size in the memory map of the second host computing device 203. In general, it should be understood that memory mapping between different host computing devices performed by the PCIe switch effectively provides shared memory between different host computing devices connected by the PCIe switch.

[0049] Figure 3A A diagram showing how the PCIe architecture can be used to connect a cloud storage system 301 with a cloud computing system 303 according to some embodiments of the present invention. Specifically, Figure 3A A cloud storage system 301 is shown connected to a cloud computing system 303 via a PCIe switch 305. The PCIe switch 305 connects the cloud storage system 301 and the cloud computing system 303 and enables storage sharing between the cloud storage system 301 and the cloud computing system 303. The cloud storage system 301 is connected to the PCIe switch 305 via a PCIe link 307. The cloud computing system 303 is connected to the PCIe switch 305 via a PCIe link 309. In some embodiments, the PCIe switch 305 is disposed within the client computing system 303. However, it should be understood that in other embodiments, the PCIe switch 305 can be located external to the cloud computing system 303.

[0050] Figure 3BAn exemplary diagram of some internal components of a cloud storage system 301 and a cloud computing system 303 according to some embodiments is shown. A root complex 311 of the cloud storage system 301 has a PCIe port 313 connected to a PCIe port 315 of a PCIe switch 305 via a PCIe link 307. Also, a root complex 317 of the cloud computing system 303 has a PCIe port 316 connected to a PCIe port 319 of the PCIe switch 305 via a PCIe link 309. The PCIe switch 305 includes logic to bridge the memory address space of the cloud storage system 301 and the cloud computing system 303. For example, the logic to bridge the memory address space of the cloud storage system 301 and the cloud computing system 303 may include a non-transparent bridge (NTB) or a special direct memory access (DMA) engine that is capable of communicating with both the cloud storage system 301 and the cloud computing system 303 and handling any memory address translations required. In some embodiments, the DMA engine may be implemented within the PCIe switch 305. It should be understood that both the root complex 311 of the cloud computing system 303 and the root complex 317 of the cloud storage system 301 include a PCIe interface or an equivalent interface, such as a Gen-Z interface defined according to the Gen-Z communication / interconnect specification developed, for example, by the Gen-Z Consortium, or other future developed interfaces that are substantially equivalent to the PCIe interface or the Gen-Z interface.

[0051] For example, in some embodiments, the PCIe switch 305 optionally includes an NTB 321. If the PCIe switch 305 includes an NTB 321 and is in NTB mode, the NTB 321 is used to convert physical memory addresses in memory access commands written by the cloud computing system 303 to data I / O (input / output) command buffers into memory addresses within the computer memory 323 of the cloud storage system 301 that can be reached by the cloud storage system 301. The NTB 321 functionally operates to modify memory addresses at the PCIe transaction layer, in which transaction layer packets (TLPs) are assembled and disassembled. In some embodiments, the NTB 321 also operates to modify the requestor identifier. If the cloud computing system 303 generates a command buffer, such as by the storage CPU 331, the command buffer is ultimately wrapped in a TLP by the root complex 311. The storage CPU 331 operates to ensure the correctness of any memory addresses within the command buffer generated by the storage CPU 331. Moreover, in the event that the cloud storage system 301 needs to write data back to the cloud computing system 303, the storage CPU 331 operates to ensure that any memory addresses within the command buffer are translated addresses. When the cloud computing system 303 writes the command buffer to a memory address in the cloud storage system 301, the NTB 321 comes into play. In this case, the memory address to which the command buffer is written is translated by the NTB 321 and / or by the output memory management unit (IOMMU) 381 in the computer memory 323 of the cloud storage system 301. The IOMMU 381 maps the virtual memory addresses visible to the device to the physical memory addresses. Moreover, in this case, the NTB 321 or the IOMMU 381 will not modify the contents of the command buffer itself.

[0052] The NTB mode enables multiple root complexes to communicate with each other, i.e., to exchange data with each other. In some embodiments, if the PCIe switch 305 is connected to a secondary PCIe switch to which only NTB devices are connected, the secondary PCIe switch does not need to use the NTB because it will only forward transaction layer packets (TLPs). In various embodiments, the computer memory 323 may be a RAM or a storage class memory, or another type of computer memory that is substantially equivalent to the RAM and / or storage class memory. The NTB 321 handles the conversion of memory addresses in the data I / O command buffer and provides the converted memory addresses to the cloud storage system 301. If the PCIe switch 305 does not use the NTB 321, the memory access commands written to the data I / O command buffer may be generated to include other metadata, such as a host identifier or other types of metadata, which may be used to determine the memory address in the computer memory 323 of the cloud storage system to which the memory address in the data I / O command buffer should be mapped. In some embodiments where the PCIe switch 305 does not use the NTB 321, a DMA engine may be used to send packets across the PCIe switch 305 to the correct destination. Furthermore, in some embodiments, if the PCIe switch 305 does not use the NTB 321 and is connected to a secondary PCIe switch, the secondary PCIe switch operates to forward the packet to the correct destination.

[0053] The cloud computing system 303 includes at least one CPU 325 connected to an internal data bus 327. In various embodiments, the cloud computing system 303 may also include one or more GPUs 329 connected to the internal data bus 327, and / or one or more storage CPUs 331 connected to the internal data bus 327, and / or one or more decryption / encryption engines 333 connected to the internal data bus 327, and / or one or more decompression / compression engines 335 connected to the internal data bus 327. The internal data bus 327 is connected to the root complex 311 of the cloud computing system 303. In various embodiments, the GPU 329 is a processor configured to perform parallel operations for rendering images, animations, and videos for display on an electronic display screen. Moreover, in some embodiments, the parallel processing capabilities of the GPU 329 can be used for vector processing in non-graphics applications that require repeated calculations.

[0054] In various embodiments, the storage CPU 331 is configured to handle requests for storage commands and to dispatch storage on a specific type of storage medium. In various embodiments, the storage CPU 331 and the main CPU 325 may have the same or different architectures. For example, in some embodiments, the storage CPU 331 may be an advanced RISC (reduced instruction set computer) machine (ARM) processor or a MIPS processor, and the main CPU 325 is an x86 processor. It should be understood that in various embodiments, the storage CPU 331 may be a suitable computer processor of substantially any type, and the main CPU 325 may be a suitable computer processor of substantially any type. The specific type of storage medium to which the storage CPU 331 dispatches data for storage may vary according to the performance capabilities of the storage medium (e.g., data read speed and / or data write speed) and / or the form in which data is stored on the storage medium (e.g., file-based data storage, folder-based data storage, or byte-based data storage). In some embodiments, the storage CPU 331 operates to divide the data into a plurality of data blocks to be placed in corresponding locations in the computer memory 337 of the cloud computing system 303, wherein each storage location in the computer memory 337 is tracked and updated based on a storage table. In various embodiments, the computer memory 337 may be a RAM or a storage class memory, or another type of computer memory substantially equivalent to the RAM and / or storage class memory. The storage CPU 331 may also be used to optimize storage based on the number of input / output operations per second (IOPS). In some embodiments, storage optimization based on IOPS may be related to the access speed of available storage media, which may include rotating storage media, solid-state storage media, and / or hybrid storage media. In various embodiments, the decryption / encryption engine 333 is a processor configured and dedicated to applying a cryptographic algorithm to decrypt encrypted data and encrypt unencrypted data. In various embodiments, the decompression / compression engine 335 is configured and dedicated to applying an algorithm to decompress compressed data and compress uncompressed data.

[0055] In some embodiments, a computer memory 337 of the cloud computing system 303 is connected to the root complex 311. In some embodiments, the cloud computing system 303 also includes a DMA controller 339 connected to the root complex 311. In some embodiments, the computer memory 337 and / or the DMA controller 339 can be connected to the internal data bus 327, as shown by lines 361 and 363, respectively. Also, in some embodiments, the cloud computing system 303 may include one or more of the NVMe SSDs 341 connected to the PCIe port 345 of the root complex 311 via a PCIe link 343. Also, in some embodiments, the cloud computing system 303 may include one or more of the SATA controllers 347 connected to the PCIe port 351 of the root complex 311 via a PCIe link 349. The SATA controller 347 may serve as a bridge between one or more of the HDDs 353 and the PCIe fabric. Furthermore, in various embodiments, one or more other PCIe-enabled component devices 355 can be connected to one or more corresponding PCIe ports 357 of the root complex 311 via corresponding PCIe links 359. In various embodiments, the one or more other PCIe-enabled component devices 355 may include one or more GPUs, one or more field programmable gate arrays (FPGAs), one or more network adapters, one or more SSDs, one or more SATA / HHDs, and other devices. It should be understood that the above are provided by way of example. Figure 3B The architecture of the cloud computing system 303 is shown in FIG. In various embodiments, the cloud computing system 303 may include Figure 3B More or fewer components may be used than those shown in the examples.

[0056] The cloud storage system 301 includes at least one CPU 365 connected to an internal data bus 367. In various embodiments, the cloud storage system 301 may also include one or more of the GPUs 369 connected to the internal data bus 367 and / or one or more of the network interface cards (NICs) 373 connected to the internal data bus 367. The internal data bus 367 is connected to a root complex 317 of the cloud storage system 301. The cloud storage system 301 may also include one or more of the NVMe SSDs 371 connected to the root complex 317. Moreover, in some embodiments, the GPU 369 and / or the NIC 373 and / or other devices may be configured to use PCIe and be able to connect directly to the root complex 317 instead of the internal data bus 367.

[0057] In some embodiments, the cloud storage system 301 is configured and operated to distribute data storage among multiple physical storage media and utilize deduplication methods to optimize the use of multiple physical storage media. Multiple physical storage media accessible / controllable by the cloud storage system 301 may be located in multiple different storage boxes interconnected by high-speed interconnects and high-speed switches. In some embodiments, multiple different storage boxes including multiple physical storage media accessible / controllable by the cloud storage system 301 may be installed in a separate location within a data center, or in multiple data centers that are geographically dispersed. In some embodiments, a cache system may be used to manage data storage between multiple data centers so that data utilized or required by a specific requesting entity (e.g., a specific remote client device used by a user to play a cloud gaming application) may be moved to a specific data center that is closest to the requesting entity or provides acceptable quality of service to the requesting entity. In some embodiments, the cache system may utilize a migration algorithm defined as migrating user data and / or game data to a specific data center, which may include a specific edge computing server or node, such as a cloud computing system 303. The cloud computing system 303 generates and transmits memory access commands to the data I / O command buffer of the cloud storage system 301 to ensure that the requesting entity will have the necessary data / information for presenting application output (such as graphical output of a game) with an allowed latency.

[0058] Data storage for cloud gaming applications can be managed by storing in a cloud system memory, such as in a computer memory 337 of a cloud computing system 303 and / or in a plurality of data buffers in a computer memory 323 of a cloud storage system 301. In some embodiments, these data buffers are optimized for presenting video frames that will be compressed and transmitted to a requesting entity via a network. Data buffers can be used to store data for compressing video frames (e.g., I frames and P frames). In a typical cloud gaming application, a scene change in the game will require a new I frame to be generated and transmitted to the requesting entity. When this happens, the storage CPU 331 may receive information from the decompression / compression engine 335 to clear the P frame from the data buffer in the computer memory 337, because a new scene (e.g., a cut scene) will require a new I frame. In some embodiments, the storage CPU 331 may not track which data buffer stores data for video frames (such as I frames and P frames). In these embodiments, the information received by the storage CPU 331 from the decompression / compression engine 335 may specify which data buffers to clear in order to clear the necessary P frames. Therefore, it should be understood that the data storage operation is a coordinated operation that requires handling of data buffers and storage devices in the cloud computing system 303 and the cloud storage system 301.

[0059] In some embodiments, data storage handling and buffering in the cloud computing system 303 and / or the cloud storage system 301 can utilize artificial intelligence (AI) to load data expected to be used by the cloud gaming application to the computer memory 337 and / or the computer memory 323, so that more efficient loading and processing of graphics data can be performed in anticipation of new scenes that may be required to execute the cloud computing application. It should be understood that AI operations can be performed at various levels of the cloud computing system 303 and / or the cloud storage system 301 to optimize the retrieval of data in anticipation of the data required for the video frames for rendering the cloud gaming application. In some embodiments, AI operations can be optimized over time based on machine learning and continuous feedback from one or more requesting entities (e.g., from one or more users playing the cloud gaming application). For example, in some embodiments, machine learning can be used to identify specific types of scenes in the game, which will require faster retrieval of data from the cloud storage system 301 for timely processing of graphics. In turn, specific types of game scenes identified as requiring faster data retrieval can trigger data retrieval operations in advance in time to reduce the waiting time associated with streaming video frames to the requesting entity.

[0060] The systems and methods disclosed herein utilize the PCIe architecture connecting the cloud computing system 303 and the cloud storage system 301 to allow the CPU 325 and / or storage CPU 331 and / or DMA controller 339 (and / or other devices) on the cloud computing system 303 to direct message sending and data copying between the computer memory 323 of the cloud storage system 301 and the computer memory 337 of the cloud computing system 303, and vice versa. The result of connecting the cloud computing system 303 and the cloud storage system 301 using the PCIe architecture and after configuring the PCIe switch 305 is a "shared memory" mechanism across the cloud computing system 303 and the cloud storage system 301. The cloud computing system 303 can directly access the designated portion 377 of the computer memory 323 within the cloud storage system 301 with very little overhead (e.g., as shown by arrow 379 while bypassing the CPU 365 of the cloud storage system 301). Direct access by the cloud computing system 303 to the computer memory 323 within the cloud storage system 301 may be directed by any device within the cloud computing system 303 that is capable of managing data storage requests and is connected to the PCIe switch 305 through the root complex 311, such as the CPU 325 and / or the storage CPU 331 and / or the DMA controller 339. However, it should be appreciated that managing data storage requests may not be the most efficient use of the CPU 325 of the cloud computing system 303. Therefore, the storage CPU 331 and / or the DMA controller 339 may be primarily used to manage the data storage requests of the cloud computing system 303 by generating memory access commands that are written to the data I / O command buffer in the computer memory 323 of the cloud storage system 301.

[0061] In addition to providing a "shared memory" mechanism, connecting the cloud computing system 303 to the cloud storage system 301 through the PCIe switch 305 also provides for sending "messages" from the cloud computing system 303 to the cloud storage system 301, and vice versa, by writing the messages directly into the computer memory 337 of the cloud computing system 303 or into the computer memory 323 of the cloud storage system 301 or into a memory within the PCIe switch 305. The message sending between the cloud computing system 303 and the cloud storage system 301 can be used to facilitate the execution of memory access commands written to the data I / O command buffer. For example, if the cloud computing system 303 expects the cloud storage system 301 to execute a certain memory access command that results in retrieving a certain requested data from the data storage device and storing it somewhere in the shared memory present in the cloud computing system 303 and the cloud storage system 301, then the cloud storage system 301 can write a message to a memory location monitored by the cloud computing system 303 to notify the cloud computing system 303 when and where the requested data is available for use by the cloud computing system 303.

[0062] In some embodiments, a message may be sent by the cloud computing system 303 or the cloud storage system 301 to one or more doorbell registers 375 exposed through the PCIe switch 305. In some embodiments, writing to the doorbell register 375 by the cloud computing system 303 will cause an interrupt to be triggered on the cloud storage system 301, which in turn causes the cloud storage system 301 to handle the interrupt. In some embodiments, handling of the interrupt by the cloud storage system 301 causes the cloud storage system 301 to read the message (certain data) from a specified location in a shared memory present in the cloud computing system 303 and the cloud storage system 301. Similarly, writing to the doorbell register 375 by the cloud server system 301 will cause an interrupt to be triggered on the cloud computing system 303, which in turn causes the cloud computing system 303 to handle the interrupt. In some embodiments, handling of the interrupt by the cloud computing system 303 causes the cloud computing system 303 to read the message (certain data) from a specified location in a shared memory present in the cloud computing system 303 and the cloud storage system 301.

[0063] The content of the message depends on the use case. In some use cases, the message may convey a request to read a certain amount of data from a storage device and store the read data in a shared memory present in the cloud computing system 303 and the cloud storage system 301. Then, after the request in the message is completed, another message may be sent to notify the completion. For example, if the cloud computing system 303 sends a first message requesting the cloud storage system 301 to read a certain specific data from the data storage area, the cloud storage system 301 will execute the request in the first message and then send a second message, which notifies the cloud computing system 303 that the first message request is completed and where the specific requested data is stored in the shared memory present in the cloud computing system 303 and the cloud storage system 301.

[0064] The cloud computing system 303 and the cloud storage system 301 are independent systems, each with its own memory map. The memory map stores the memory addresses of the computer memory and other devices in the system. The PCIe switch 305 occupies a user-configurable region of the memory address space in each memory map of the cloud computing system 303 and the cloud storage system 301. In various embodiments, depending on the configuration of the PCIe switch 305, a physical memory address in the computer memory 337 of the cloud computing system 303 can be mapped to the same or different physical memory addresses in the computer memory 323 of the cloud storage system 301.

[0065] In some embodiments, for security and / or other reasons, the cloud storage system 301 may be configured to hide its actual physical memory address, such as by implementing an IOMMU 381 that maps device-visible virtual memory addresses to physical memory addresses. The IOMMU 381 is configured to map device memory addresses to physical memory addresses, while a conventional memory management unit (MMU) is configured to map virtual memory addresses to physical memory addresses. For a conventional MMU, the virtual memory addresses are continuous, so the application can see a memory block (e.g., a 64MB memory block) and operate on the memory block, but in fact the MMU maps continuous virtual memory addresses to multiple separate physical memory addresses and hides the memory mapping from the application. Similar to a conventional MMU, the IOMMU 381 maps virtual memory addresses visible to the cloud computing system 303 to multiple separate physical memory addresses in the computer memory 323, and hides the memory mapping from the cloud computing system 303. In this manner, use of the IOMMU 381 allows the cloud computing system 303 to see the contiguous blocks of virtual memory addresses present on the cloud storage system 301, while hiding the complexity of how the contiguous blocks of virtual memory addresses are actually mapped to different physical memory addresses across the computer memory 323 of the cloud storage system 301. In some embodiments, the IOMMU 381 is included in the root complex 317 of the cloud storage system 301. In some embodiments, the root complex 311 of the cloud computing system 303 may also include the IOMMU 383.

[0066] exist Figure 3BIn the configuration of FIG. 3 , the IOMMU 381 may allow the cloud computing system 303 to view a specific memory address (device address) as having a one-to-one correspondence with a specific physical memory address in the computer memory 323 of the cloud storage system 301, but in fact the IOMMU 381 is capable of remapping the specific memory address viewed by the cloud computing system 303 to a different physical memory address in the computer memory 323 of the cloud storage system 301. Moreover, in some embodiments, the IOMMU 381 may be used to prevent access to one or more designated physical memory areas in the computer memory 323 of the cloud storage system 301 that are not authorized to be accessed by the cloud computing system 303. Moreover, because the IOMMU 381 effectively hides memory fragmentation in the computer memory 323 of the cloud storage system 301 from the cloud computing system 303, the use of the IOMMU 381 can simplify DMA transfer operations in the cloud computing system 303. For example, if IOMMU 381 is not used to present the fragmented physical memory addresses in computer memory 323 to cloud computing system 303 as a continuous block of virtual memory addresses, cloud computing system 303 would need to perform multiple DMA transfer operations or scatter / gather transactions to read data across the fragmented physical memory addresses in computer memory 323. However, by using IOMMU 381 to present the fragmented physical memory addresses in computer memory 323 to cloud computing system 303 as a continuous block of virtual memory addresses, cloud computing system 303 is able to perform a single DMA transfer operation to read data across the fragmented physical memory addresses in computer memory 323.

[0067] Both the NTB 321 in the PCIe switch 305 and the IOMMU 381 in the cloud storage system 301 perform memory address translation. Figure 4 An exemplary diagram of memory mapping functions performed by NTB 321 in PCIe switch 305 and IOMMU 381 in cloud storage system 301 is shown according to some embodiments of the present invention. Figure 4 An exemplary physical memory address range 0x30000000-0x400000000 in the computer memory 337 of the cloud computing system 303 is shown mapped to the physical memory address range 0x70000000-0x800000000 in the computer memory 323 of the cloud storage system 301 through the NTB 321 in the PCIe switch 305 and the IOMMU 381 in the cloud storage system 301. The cloud computing system 303 is unaware of the actual physical memory address range in the computer memory 323 of the cloud storage system 301 to which the visible memory address range 0x30000000-0x40000000 is mapped. Figure 4Also shown is an exemplary physical memory address range 0x10000000-0x200000000 in the computer memory 337 of the cloud computing system 303, which is mapped in a fragmented manner to two physical memory address ranges 0x30000000-0x3fffffff and 0x40000000-0x4ffffffff in the computer memory 323 of the cloud storage system 301 through the NTB 321 in the PCIe switch 305 and the IOMMU 381 in the cloud storage system 301. The cloud computing system 303 does not know how the visible memory address range 0x30000000-0x40000000 is mapped to the actual physical memory address range in the computer memory 323 of the cloud storage system 301. The cloud computing system 303 also does not know that the visible memory address range 0x30000000-0x40000000 is actually fragmented in the computer memory 323 of the cloud storage system 301. Furthermore, the cloud computing system 303 does not need to be concerned about the visible memory address range 0x30000000-0x40000000 being fragmented in the computer memory 323 of the cloud storage system 301 .

[0068] although Figure 3A and Figure 3B A cloud computing system 303 is shown connected to a cloud storage system 301 via a PCIe switch 305, but it should be understood that in other embodiments, the cloud storage system 301 can be connected to multiple cloud computing systems via a PCIe architecture. Figure 5A cloud gaming configuration according to some embodiments is shown, wherein a cloud storage system 301 is connected to a certain number (D) of cloud computing systems 303(1)-303(D) via a PCIe switch 305, where D is greater than one. The PCIe switch 305 includes the number (D) of PCIe ports 315-315(D), and the cloud computing systems 303(1)-303(D) are connected to the number of PCIe ports 315-315(D) via corresponding PCIe links 309(1)-309(D), respectively. In some embodiments, each of the PCIe links 309(1)-309(D) has a certain number (M) of PCIe lanes, and the PCIe link 307 connecting the cloud storage system 301 to the PCIe switch 305 has a certain number (N) of PCIe lanes. In some embodiments, cloud storage system 301 is configured to provide the same data communication interface bandwidth to each of cloud computing systems 303(1)-303(D). In these embodiments, the number of PCIe lanes (N) in PCIe link 307 is equal to the number of cloud computing systems 303(1)-303(D) (D) multiplied by the number of PCIe lanes (M) in each PCIe link 309(1)-309(D). In some embodiments, it is not necessary that each cloud computing system 303(1)-303(D) requires the maximum data communication interface bandwidth at the same time. For example, the data access patterns in different cloud computing systems 303(1)-303(D) may indicate that each cloud computing system 303(1)-303(D) does not require the maximum data communication interface bandwidth at the same time. In these embodiments, the number of PCIe lanes (N) in PCIe link 307 can be less than the number of cloud computing systems 303(1)-303(D) (D) multiplied by the number of PCIe lanes (M) in each PCIe link 309(1)-309(D). The reduction in the number of PCIe lanes (N) in PCIe link 307 can provide cost and energy savings.

[0069] In some embodiments, the root complex 317 of the cloud storage system 301 can be connected to multiple PCIe switches. For example, Figure 6A cloud gaming configuration according to some embodiments is shown, wherein a cloud storage system 301 is connected to a number (E) of PCIe switches 305 (1) -305 (E) via corresponding PCIe links 307 (1) -307 (E), where E is greater than one. The cloud storage system 301 includes a number (E) of PCIe ports 316 (1) -316 (E), to which corresponding PCIe links in the number (E) of PCIe links 307 (1) -307 (E) are connected. Each of the PCIe switches 305 (1) -305 (E) has a corresponding PCIe port 319 (1) -319 (E), to which corresponding PCIe links in the number (E) of PCIe links 307 (1) -307 (E) are connected. Each PCIe switch 305(1)-305(E) has a certain number (F) of PCIe ports 315(1)-315(F), and the corresponding cloud computing systems 303(1)-303(F) are connected to the number of PCIe ports 315(1)-315(F) through corresponding PCIe ports 313(1)-313(F) and corresponding PCIe links 309(1)-309(F). The number (F) of PCIe ports 315(1)-315(F) may vary between different PCIe switches 305(1)-305(E).

[0070] In some embodiments, PCIe switches 305(1)-305(E) are disposed within cloud storage system 301. In some embodiments, PCIe switches 305(1)-305(E) are disposed within the PCIe architecture at a location between cloud storage system 301 and various cloud computing systems 303(1)-303(F). When a single PCIe switch 305(1)-305(E) does not provide sufficient data communication interface bandwidth (PCIe lanes) to support the aggregation of downstream cloud computing systems 303(1)-303(E*F), Figure 6 Configurations may be particularly useful. Also, Figure 6 The configuration provides a certain degree of redundancy in the PCIe architecture. For example, the use of multiple PCIe switches 305(1)-305(E) reduces the failure domain so that if one (or even more) PCIe switches 305(1)-305(E) fails within a given cloud storage system 301, only a small amount of the data communication interface bandwidth capacity of the cloud storage system 301 is lost.

[0071] In various scenarios, one or more of the cloud computing systems 303(1)-303(E*F) may fail, and / or one or more of their corresponding PCIe links 309(1)-309(E*F) may fail, and / or one or more of the PCIe switches 305(1)-305(E) may fail. These various failure modes may be detected and handled. For example, the PCIe switches 305(1)-305(E) and / or the cloud computing systems 303(1)-303(E*F) and / or the cloud storage system 301 may be configured to detect when a given PCIe link in the PCIe links 309(1)-309(E*F) fails, such as due to an I / O request timeout. When such a failure is detected, an attempt may be made to reestablish the failed given PCIe link in the PCIe links 309(1)-309(E*F). If a given failed PCIe link in the PCIe links 309(1)-309(E*F) cannot be reestablished, the corresponding active cloud gaming session may need to be terminated. In this case, an error is reported to the appropriate software component to terminate the cloud gaming session. Then, at an appropriate time, such as when the system is restarted, the given failed PCIe link in the PCIe links 309(1)-309(E*F) can be checked and verified. Moreover, when a given failed PCIe link in the PCIe links 309(1)-309(E*F) cannot be reestablished, the corresponding cloud computing system in the cloud computing systems 303(1)-303(E*F) can be reported as "offline" so that users will not be assigned to it.

[0072] In some embodiments, the PCIe fabric that connects one or more cloud computing systems 303 to the cloud storage system 301 may include a fan-out configuration of multiple PCIe switches 305. For example, Figure 7A cloud gaming configuration according to some embodiments is shown, in which a cloud storage system 301 is connected to multiple cloud computing systems 303 via a fan-out configuration of PCIe switches 305A, 305B, 305C. PCIe port 316 of cloud storage system 301 is connected to PCIe port 319A of PCIe switch 305A via PCIe link 307. PCIe port 315A of PCIe switch 305A is connected to PCIe port 319B of PCIe switch 305B via PCIe link 701. PCIe port 315B of PCIe switch 305A is connected to PCIe port 319C of PCIe switch 305C via PCIe link 703. The PCIe switch 305B has a certain number (F) of PCIe ports 315(1)-315(F), and the corresponding cloud computing systems 303(1)-303(F) are connected to the said number of PCIe ports 315(1)-315(F) through the corresponding PCIe ports 313(1)-313(F) and the corresponding PCIe links 309(1)-309(F). The PCIe switch 305C has a certain number (F) of PCIe ports 315(1)-315(F), and the corresponding cloud computing systems 303(1)-303(F) are connected to the said number of PCIe ports 315(1)-315(F) through the corresponding PCIe ports 313(1)-313(F) and the corresponding PCIe links 309(1)-309(F). The number (F) of PCIe ports 315(1)-315(F) may vary for different PCIe switches 305B and 305C. Figure 7 The example shows three PCIe switches 305A, 305B, 305C in a fan-out configuration, but it should be understood that other embodiments may include more than three PCIe switches 305 in a fan-out configuration and may include more than two PCIe switches 305 connected to the downstream PCIe ports 315 of a given PCIe switch 305.

[0073] When a single PCIe switch 305 does not provide enough data communication interface bandwidth (PCIe channel) to support the aggregation of downstream cloud computing systems 303, the fan-out configuration of multiple PCIe switches 305 may be particularly useful. Moreover, when multiple cloud computing systems 303 need the maximum data communication interface bandwidth (PCIe channel) at the same time and / or at all times, the fan-out configuration of multiple PCIe switches 305 may be particularly useful. Moreover, the fan-out configuration of multiple PCIe switches 305 provides improved signal integrity through the PCIe architecture. For example, the PCIe specification has a maximum length for electrical traces on a circuit board or in a cable. Moreover, the maximum length of the electrical trace depends on the generation of the PCIe technology utilized. The PCIe specification defines a certain amount of signal loss, which is about 26dB for PCIe4.0. This signal loss budget is throughout the entire signal path, such as throughout the I / O pins, connectors, traces / cables, etc. of the chip that constitutes the entire signal path. In an actual system that implements PCIe 4.0, the maximum length of the entire signal path is about one meter. However, if care is taken and very high quality materials are used, the maximum length of the entire signal path can be up to 4 meters. Since each PCIe switch 305 regenerates the signal traveling through the PCIe fabric, a fan-out configuration using multiple PCIe switches 305 can provide a method to increase the overall electrical trace distance that a signal can travel through the PCIe fabric. In some embodiments, increasing the overall electrical trace distance that a signal can travel through the PCIe fabric can be achieved using special logic to regenerate and / or boost signals at various points within the PCIe fabric between the cloud storage system 301 and any one or more of the cloud computing systems 303.

[0074] although Figure 3A , Figure 3B , Figure 5 , Figure 6 and Figure 7The example of shows a PCIe architecture including a cloud storage system 301, multiple PCIe switches 305, multiple client computing systems 303, and associated PCIe links, but it should be understood that the PCIe architecture in various embodiments may also include other PCIe compatible devices, such as GPUs, network adapters, NVMe storage devices, and / or substantially any other PCIe compatible devices, each of which is connected to an assigned PCIe switch via an appropriately sized PCIe link. At system power-up, any given PCIe compatible device within the PCIe architecture can be assigned to a given cloud computing system 303 through the configuration of the PCIe architecture. Typically, the system hardware configuration is fixed at system power-up. However, in some cases, PCIe hot-plugging can be used, such as to hot-plug, for example, NVMe SSD U.2, a device with a form factor of 2.5 inches. In this way, PCIe hot-plugging is similar to cabled PCI Express. In some embodiments, the root complex is configured to recognize the hot-plug connection of a new PCIe device and notify the operating system, which in turn allocates memory address space for use by the new PCIe device. Moreover, in various embodiments, one or more PCIe compatible devices having a PCIe architecture can be shared between multiple cloud computing systems 303 within the PCIe architecture. Such sharing of PCIe compatible devices by multiple cloud computing systems 303 within the PCIe architecture may be possible if the PCIe compatible devices support standards such as single root input / output virtualization (SR-IOV), which is a specification that provides sharing of a single PCIe device within a PCIe architecture by virtually slicing PCIe compatible devices into virtual functions. For example, using SR-IOV to implement sharing of a single PCIe device within a PCIe architecture is particularly useful for network adapters, NVMe devices, and even some GPUs.

[0075] Computations for cloud gaming can be organized in compute drawers, where each compute drawer includes one or more independent cloud computing systems 303, a power delivery system, a rack management interface, a PCIe interface, and optional network connectivity. Figure 8An exemplary computing drawer 801 according to some embodiments is shown. The computing drawer 801 includes a certain number (Y) of cloud computing systems 303 (1)-303 (Y), where (Y) is one or more. The computing drawer also includes a PCIe interface 805 and the number (Y) of PCIe links 807 (1)-807 (Y) that connect the cloud computing systems 303 (1)-303 (Y) to the PCIe interface 805. The PCIe interface 805 is configured to connect the computing drawer 801 to an external PCIe link 815. The computing drawer 801 also includes a power delivery system 803 connected to deliver power to all electrical components on the computing drawer 801. The power delivery system 803 is configured to receive power from a source external to the computing drawer 801. For example, in some embodiments, the power delivery system 803 is configured to be connected to a power rail. In some embodiments, the power delivery system 803 includes a power supply. The compute drawer 801 also includes a rack management interface 813, such as a universal serial bus (USB), a universal asynchronous receiver-transmitter (UART) controller, an inter-integrated circuit (I2C) system, etc. The compute drawer 801 may also include one or more additional components 811, any of which may have a PCIe interface connected to the PCIe interface 805 of the compute drawer 801 through a corresponding PCIe link 809. Moreover, in some embodiments, the compute drawer 801 may optionally include network connectivity, such as to the compute drawer 801 and / or to an Ethernet connection and / or to the cloud computing system 303(1)-303(Y).

[0076] In some embodiments, each cloud computing system 303(1)-303(Y) on the computing drawer 801 includes, for example, Figure 3B In some embodiments, each cloud computing system 303(1)-303(Y) includes a PCIe interface connected to the PCIe interface 805 of the computing drawer 801. In some embodiments, the PCIe interface on each cloud computing system 303(1)-303(Y) is a PCIe slot and / or an M2 interface and / or an OCuLink connector and / or another type of interface capable of carrying PCIe signals and interfacing with a PCIe architecture (or an equivalent architecture).

[0077] In some embodiments, the PCIe interface 805 in the computing drawer 801 is a board to which each cloud computing system 303(1)-303(Y) is connected. The board of the PCIe interface 805 is configured to collect PCIe signals from each cloud computing system 303(1)-303(Y) and expose a connector on the outside (e.g., the back) of the computing drawer 801. The connector exposed on the outside of the computing drawer 801 can be connected to the backplane of the rack, where the backplane connects multiple computing drawers 801 to the cloud storage system 301. In some embodiments, the connector exposed on the outside of the computing drawer 801 can provide a cable (wired) connection between the computing drawer 801 and the cloud storage system 301. In some embodiments, the PCIe interface 805 of the computing drawer 801 may include electronic components, such as components for enhancing signal integrity and / or for implementing as described above. Figure 7 The PCIe switches and PCIe links of the fan-out configuration of the multiple PCIe switches described. In some embodiments, the PCIe interface 805 includes auxiliary PCIe devices, such as UART controllers, temperature sensors, general purpose input / output devices (GPIO) components and / or other components for implementing the management of the computing drawer 801. Moreover, the various devices / components within the PCIe interface 805 can be used by the cloud computing system 303 (1)-303 (Y) and / or another management server for the operational purposes of the computing drawer 801, such as for powering on a specific cloud computing system 303 (1)-303 (Y). Such a management server can be configured to manage one or more computing drawers 801, an array of computing drawers 801, or an entire rack of computing drawers 801. In some embodiments, the cloud storage system 301 can be used as a management server. In some embodiments, the management server is different from the cloud storage system 301.

[0078] Fig. 9An exemplary diagram of a rack 901 that can be implemented in a cloud gaming center according to some embodiments is shown. Rack 901 includes the top of a rack switch 905 so that components within rack 901 can be connected to an external network, such as to the Internet. In some embodiments, rack 901 includes multiple computing arrays 903A, 903B, 903C. Each computing array 903A, 903B, 903C includes multiple computing drawers 801. Rack 901 also includes a cloud storage system 301. Each computing array 903A, 903B, 903C is connected to the cloud storage system 301 via a PCIe architecture. In some embodiments, a given computing array 903A, 903B, 903C includes a backplane 907, and the PCIe interfaces 805 of various computing drawers 801 are connected to the backplane 907. In some embodiments, the backplane 907 of a given computing array 903A, 903B, 903C may include a PCIe switch or one or more other electronic components to enhance signal integrity. In some embodiments, the backplane 907 of a given computing array 903A, 903B, 903C can be connected to the cloud server system 301 via a cable such as an OCuLink cable or other type of connection device compatible with the PCIe architecture. In some embodiments, the backplane 907 of a given computing array 903A, 903B, 903C can be configured to provide the maximum amount of PCIe bandwidth required for the aggregation of computing drawers 801 within a given computing array 903A, 903B, 903C. However, in some embodiments, the backplane 907 of a given computing array 903A, 903B, 903C can be configured to reduce the PCIe bandwidth available to individual computing drawers 801 within a given computing array 903A, 903B, 903C, such as by implementing a fan-out configuration of a PCIe switch.

[0079] It should be understood that for the purpose of description, the Fig. 9901. In various embodiments, a rack 901 may include substantially any number of computing arrays 903A-903C, wherein each computing array 903A-903C includes substantially any number of computing drawers 801. Moreover, in various embodiments, a rack 901 may include more than one cloud storage system 301. Moreover, in some embodiments, a computing drawer 801 in a given rack 901 is connected to communicate only with one or more cloud storage systems 301 within a given rack 901. However, in some embodiments, a computing drawer 801 in a given rack 901 may be connected to communicate with a cloud storage system 301 in one or more additional racks 901. Moreover, in some embodiments, a cloud storage system 301 in a given rack 901 may be connected to communicate with another cloud storage system 301 in another rack 901. Moreover, in some embodiments, various components of a cloud storage system 301 may be distributed among multiple racks 901.

[0080] Therefore, it should be understood that in some embodiments, the cloud storage system 301 and the cloud computing system 303 may be located in the same rack 901. However, in some embodiments, the cloud storage system 301 and the cloud computing system 303 may be located in different racks 901. Moreover, in some embodiments, a single rack 901 will include multiple cloud storage systems 301. Moreover, in some embodiments, each of the multiple cloud storage systems 301 may be configured to host different content, for example, different cloud gaming content, which may be different games in some embodiments. In some embodiments, depending on the gaming content required by a given cloud computing system 303, a given cloud computing system 303 will interact with different cloud storage systems in the multiple cloud storage systems 301. In some embodiments, before the cloud computing system 303 is installed, details about which cloud storage system 301 will be communicated with are provided to the cloud computing system 303. In some embodiments, the cloud computing system 303 is configured to maintain a database / list that provides information about which cloud storage systems 301 are available for data access and about the content (e.g., gaming content) stored on each cloud storage system 301.

[0081] In advanced cloud gaming systems, local data storage solutions for cloud computing systems 303 may have very high data throughput and very low latency. In cloud gaming, it is beneficial to virtualize data storage for cloud computing systems 303, because it is usually impossible to store all provided game content in a data storage area local to the cloud computing system 303, and because it will be impossible to manage such a large amount of provided game content stored in a data storage area local to an individual cloud computing system 303. For at least these reasons, multiple cloud computing systems 303 share data storage in one or more cloud storage systems 301, and one or more cloud storage systems may provide a large amount of data storage capacity. Cloud storage system 301 may store data that may be shared between multiple cloud computing systems 303, such as read-only game content. Cloud storage system 301 may also store other data that is specific to a given cloud computing system 303 and may not be shared between multiple cloud computing systems 303. Cloud computing system 303 is configured to operate based on the data access speed of its local storage system (e.g., the data access speed of NVMe SSD, which may be very fast). Therefore, the cloud storage system 301 needs to match the local data access speed of the cloud computing system 303 in order to avoid adversely affecting the operational performance of the cloud computing system 303. However, the data storage devices currently available for use in the cloud storage system 301 combined with the network-based data access latency of the cloud storage system 301 do not provide a fast enough data access solution to the cloud computing system 303 to achieve a match of the local data access speed of the cloud computing system 303. In addition, it is currently impossible to obtain individual server-class storage devices that can match the data access performance requirements of the cloud computing system 303 using conventional cloud gaming rack architectures.

[0082] The system and method disclosed herein for connecting multiple cloud computing systems 303 to a cloud storage system 301 using a PCIe architecture within a rack 901 of a cloud gaming system enables each cloud computing system 303 to directly access game content stored in a computer memory 323 of the cloud storage system 301, with access speed and bandwidth as if the game content were stored in a local computer memory 337 of the cloud computing system 303. In the system and method disclosed herein, PCIe switching technology is utilized to provide high-speed access to the computer memory 323 of the cloud storage system 301 to the cloud computing system 303, wherein such high-speed access exceeds the speed that can be achieved using a network card and a network switch. Moreover, the storage CPU 331 of the cloud computing system 303 is configured to efficiently process storage commands, but may not be powerful enough to run an operating system and manage the high-speed networking operations that would be required to arrange data communications between the cloud storage system 301 and the cloud computing system 303 using a network card and a network switch. However, the use of PCIe switching technology in the systems and methods disclosed herein allows the storage CPU 331 to handle data communications between the cloud storage system 301 and the cloud computing system 303, which helps to offload the main CPU 325 of the cloud computing system 303. In this way, the use of the PCIe architecture to connect the cloud computing system 303 and the cloud storage system 301 provides a relatively natural extension of how the cloud computing system 303 is configured to operate.

[0083] As previously described, in some embodiments, the cloud computing system 303 includes its own local data storage device, such as SATA-HDD 353, NVMe SSD 341 and / or other data storage devices, such as SATA-SSD, etc. One or more local data storage devices of the cloud computing system 303 can store a variety of different types of data, such as operating system data (e.g., system libraries, system programs, services, configuration files, fonts, images, audio files, and other types of data), user-specific data (saved game data, user character data, download data, browser bookmarks, pictures, music, videos, and other types of data), and / or game data (data for a given game, including executable files, libraries, textures, audio files, videos, game geometry, and other types of data). Each type of data stored locally on the cloud computing system 303 may have different performance requirements and access modes. For example, except for configuration files, operating system data is mostly read-only. Moreover, user-specific data may be frequently modified. Moreover, game data is usually read-only. In addition, any write operation of the game will obtain user-specific data, such as saved game data files for storing game progress, etc. In some embodiments, operating system data, user-specific data, and game data are stored on a single storage device. Furthermore, in some embodiments, because operating system data, user-specific data, and game data have different performance requirements, operating system data, user-specific data, and game data are stored on different partitions or namespaces within a single storage device. In some embodiments, each partition within a storage device may use a different file system, such as FAT32, exFAT, NTFS, UFS, EXT4, and the like.

[0084] When the cloud computing system 303 is turned on, the operating system of the cloud computing system 303 enumerates the storage devices visible to the cloud computing system 303 and uses the file system required by the partition to install any required partitions. Most of the data available to the cloud computing system 303 is installed when the cloud computing system 303 is turned on. However, other installation operations may occur during the operation of the cloud computing system 303. For example, a user may insert additional data storage media, such as a DVD, a Blu-ray disc, a USB key, and / or a portable hard drive, etc. Moreover, in some embodiments, the data stored on the local data storage device of the cloud computing system 303 itself may need to be installed before use. For example, game content can be organized in a game package format including game data and game executable files. In this example, before starting the game, it may be necessary to install the game package so that any required digital rights management (DRM) can be applied as needed to decrypt any data.

[0085] In the context of cloud computing system 303, the performance (access and processing) of game data is the most critical because it greatly affects game loading time and user experience. In some embodiments, special file systems and file APIs (application programming interfaces) are used for game data, while other data types (operating system data and / or user-specific data) use traditional file systems and traditional file APIs, such as POSIX (Portable Operating System Interface) or Win32 (WINDOWS 95 and higher 32-bit API) specified file systems and file APIs. Specifically, when using high-speed NVMe devices, the overhead of traditional POSIX and Win32 APIs is large, because the APIs are very complex and require a lot of CPU-side processing, which is not optimal for game data performance and latency reasons. In some embodiments, an example of a dedicated game-specific API for reading and writing files may look like this:

[0086] readFile(uint32_t id,void*buffer,size_t size,size_t offset)

[0087] writeFile(uint32_t id,void*buffer,size_t size,size_t offset)

[0088] In this case, "id" is an identifier for the file. In traditional operating systems, file identifiers are obtained through matching "open" API calls that map a "file path" to an identifier, where the details of how this works depend on the operating system and the file system. This can also be done in different ways. For example, in a cloud computing system 303, data can be organized in game packages (such as ISO files, ISO images, or other formats), which can include a mapping table of file names to file identifiers. For example, Fig.10 An example of a mapping table that maps file names to file identifiers according to some embodiments is shown. In some embodiments, as an example, an application can use an API call such as Fig.10 The mapping table shown performs file identifier mapping.

[0089] A buffer is a block of memory that holds data to be read or written. In the example of a dedicated game-specific API for reading and writing files shown above, "size" is the amount of data to be read or written. A buffer needs to have data storage capacity that can accommodate at least the "size" (i.e., amount) of data to be read or written. In the example of a dedicated game-specific API for reading and writing files shown above, "offset" is the position in memory relative to the beginning of the file at which the data to be read is located / starts. The offset is typically zero, but non-zero values ​​can be used to read data from a location in the file that is a certain distance from the beginning of the file.

[0090] In some embodiments, game-specific APIs for reading and writing files (readFile / writeFile) are mapped to command buffers. A command buffer is a data structure that contains at least the parameters passed to the readFile / writeFile API call, in addition to other internal data structures. The "void*buffer" provided by the application is usually a virtual memory address local to the application. In some embodiments, the command buffer requires a physical memory address, which in turn requires a conversion between the virtual memory address and the physical memory address. In some embodiments, this conversion between the virtual memory address and the physical memory address is performed by the operating system kernel. This conversion capability allows other processes or coprocessors (such as storage CPU 331, etc.) in the cloud computing system 303 to reach the memory buffer.

[0091] In some embodiments, the command buffer is placed on a command queue for processing by an operating system running on the main CPU 325 of the cloud computing system 303. In some embodiments, the command buffer is sent to the storage CPU 331, such as through a computer memory 337 shared by both the main CPU 325 and the storage CPU 331. In some embodiments, the storage CPU 331 is used to manage storage input / output requests, which provides offloading of storage input / output requests from the main CPU 325. Because high-performance storage request processing is very demanding in terms of interrupts, the use of the storage CPU 331 can reduce interrupts handled by the main CPU 325 and correspondingly improve the processing performance of the main CPU 325. Moreover, since the storage CPU 331 is completely isolated from the main CPU 325, the handling of storage input / output requests by the storage CPU 331 can improve data security to a certain extent, understanding that the storage CPU 331 can access the computer memory 337 and may send messages between systems, such as by using doorbell registers.

[0092] In some embodiments, the systems and methods disclosed herein (where PCIe architecture is used to connect cloud computing system 303 to cloud storage system 301) include using storage CPU 331 and its command buffer processing to enable direct data access between cloud computing system 303 and cloud storage system 301. In cloud computing system 303, file identifiers are used as file system tables (e.g., Fig.10 ) to find the metadata of the file. In some embodiments, the metadata of the file includes the location of the file in the memory, such as on an HDD or SSD. The location of the file in the memory as specified in the metadata may also include an offset to read the data, such as a logical block addressing (LBA) sector on an HDD or SSD, or an offset relative to the game package. The metadata of the file may also include the size of the file and other optional metadata, such as metadata for compression or encryption purposes.

[0093] The metadata of the file is used by the cloud computing system 303 to perform DMA operations on the storage device to retrieve / store the data identified by the command buffer. After the DMA operation is completed, as indicated by an interrupt from the DMA controller 363, the storage CPU 331 notifies the main CPU 325 and transmits the results back to the application. In some embodiments, other operations after retrieving the data from the storage device may include decompression and decryption of the data, as managed by the storage CPU 331 through other coprocessors such as the decompression / compression engine 335.

[0094] Although the game-specific APIs for reading files and writing files (readFile / writeFile) discussed above are used for dedicated game APIs, conventional POSIX and / or Win32 APIs can be used by the cloud computing system 303 for other data types, and these conventional APIs can be mapped to different types of command buffers. In general, the entire data path is expected to be different with more processing and handling by the operating system. However, the final data access request can still be passed through the storage CPU 331.

[0095] In some cloud gaming implementations, according to the embodiments disclosed herein, the cloud computing system 303 may not have its own local data storage area, such as an HDD and / or an SSD. Alternatively, the cloud computing system 303 may have a minimal local data storage device for storing a portion of the operating system. It should be understood that in these embodiments, the PCIe architecture connecting the cloud computing system 303 and the cloud storage system 301 can be used to expand the handling of command buffers by the storage CPU 331 of the cloud computing system 303. In some cases, a command buffer indicating access to data stored at a certain location within the cloud storage system 301 may be processed by the cloud storage system 301. Optionally, in some cases, a command buffer indicating access to data stored at a certain location within the cloud storage system 301 may be managed by the cloud computing system 301 while bypassing the CPU 365 in the cloud storage system 301.

[0096] Various embodiments are described below, wherein a command buffer indicating access to data stored at a location within the cloud storage system 301 is processed by the cloud storage system 301. In these embodiments, the CPU 365 of the cloud storage system 301 receives a command buffer from the cloud computing system 303, wherein the command buffer includes information about which data to read or write. The command buffer is initially generated by a readFile or writeFile operation. In some embodiments, the command buffer is passed to the storage CPU 331 of the cloud computing system 303 and is handled by the storage CPU 331. The command buffer is a data structure including information specifying which data to read or write. In these embodiments, instead of the storage CPU 331 accessing the local storage area on the cloud computing system 303, the storage CPU 331 interacts with the cloud storage system 301 through the PCIe architecture.

[0097] Several exemplary embodiments for data read operations are described below, wherein a command buffer indicating the reading of data stored at a certain location within the cloud storage system 301 is processed by the cloud storage system 301. The exemplary embodiments vary with respect to the manner in which the command buffer is transmitted to the cloud storage system 301 and the manner in which the requested data is transmitted to the cloud computing system 303. The process of exchanging command buffers between the cloud computing system 303 and the cloud storage server 301 is called a control path. The process of exchanging data between the cloud storage system 301 and the cloud computing system 303 is called a data path. Each of the exemplary embodiments described below represents a different combination of a control path and a data path. In some cases, both the control path and the data path may be unloaded to dedicated hardware. However, this may be difficult in some cases. Therefore, in some cases, a CPU such as a storage CPU 331 handles a control path that is usually a limited workload, and the data path is unloaded from the CPU for handling by dedicated hardware (such as a DMA controller 339 or other dedicated hardware).

[0098] For ease of discussion, the following exemplary embodiments are described about the handling of a single command buffer. However, it should be understood that, in fact, the task of the cloud storage server 301 is to handle multiple command buffers at a time, wherein these multiple command buffers can come from the same cloud computing system 303 or from different cloud computing systems 303. Therefore, in some embodiments, the cloud storage system 301 implements a queue mechanism for handling incoming command buffers. Moreover, in some embodiments, the queue mechanism implemented by the cloud storage system 301 includes multiple command buffer queues for each cloud computing system 303. In these embodiments, different processing priority levels can be assigned to different command buffer queues so that the command buffers with higher priority enter the command buffer queues with higher priority. Moreover, the exemplary embodiments discussed below mainly relate to game content read-only data access command buffers, such as command buffers generated by game-specific APIs for reading files and writing files (readFile / writeFile) as discussed above. However, it should be understood that the principles of the exemplary embodiments discussed below can also be used to read and write other data types, such as operating system data and / or user-specific data, which can use more conventional POSIX and / or Win32 APIs and correspondingly adjusted command buffers. For example, a read or write operation performed using a regular POSIX API would be similar to a dedicated game read operation, which would require some adjustments to transfer write command buffer data from the cloud computing system 303 to the cloud storage system 301.

[0099] In the first exemplary embodiment, the command buffer is transmitted to the cloud storage system 301 by the cloud computing system 303 operating to write the command buffer to the computer memory 323 of the cloud storage system 301 via the PCIe architecture. The cloud storage server 301 executes the command buffer to read the requested data from a data storage device accessible to the cloud storage server 301. The requested data is transmitted to the cloud computing system 303 by the cloud computing system 303 operating to read the requested data from the computer memory 323 of the cloud storage system 301 via the PCIe architecture.

[0100] In a first exemplary embodiment, the cloud computing system 303 generates a command buffer to read data from the cloud storage system 301. The data to be read is referred to as the requested data. The cloud computing system 303 writes the command buffer to the computer memory 323 on the cloud storage system 301. The cloud computing system 303 notifies the cloud storage system 301 that the command buffer has arrived in the computer memory 323 of the cloud storage system 301. In some embodiments, this notification of the arrival of the command buffer in the computer memory 323 of the cloud storage system 301 is accomplished by the cloud computing system 303 writing to a doorbell register or sending a message to the cloud storage system 301. The doorbell register may be in any memory location monitored by the cloud storage system 301. For example, the doorbell register may be in the computer memory 323 of the cloud storage system 301 or in the PCIe switch 305, or even in the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The notification of the arrival of the command buffer in the computer memory 323 of the cloud storage system 301 causes an interrupt to be triggered on the cloud storage system 301.

[0101] In response to the interrupt, the cloud storage system 301 reads the command buffer from the computer memory 323 of the cloud storage system 301 and processes the command buffer to obtain the requested data from a data storage device accessible to the cloud storage system 301. The cloud storage system 301 writes the requested data to the computer memory 323 of the cloud storage system 301. The cloud storage system 301 then notifies the cloud computing system 303 that the requested data is available in the computer memory 323 of the cloud storage system 301 and where the requested data is located in the computer memory 323 of the cloud storage system 301. In some embodiments, this notification that the requested data is available in the computer memory 323 of the cloud storage system 301 is accomplished by the cloud storage system 301 writing to a doorbell register or sending a message to the cloud computing system 303. The doorbell register may be in any memory location monitored by the cloud computing system 303. For example, the doorbell register may be in the computer memory 337 of the cloud computing system 303 or in the PCIe switch 305, or even in the computer memory 323 of the cloud storage system 301 through the PCIe architecture. Moreover, in some embodiments, the doorbell register is an actual register in hardware dedicated for this purpose, such as a hardware register in an SRAM / flip-flop.

[0102] In response to the notification that the requested data is available in the computer memory 323 of the cloud storage system 301, the cloud computing system 303 reads the requested data from the computer memory 323 on the cloud storage system 301. In some embodiments, the cloud computing system 303 uses the memory address returned by the cloud storage system 301 to read the requested data from the computer memory 323 on the cloud storage system 301. In some embodiments, the pre-assigned fixed memory address in the computer memory 323 on the cloud storage system 301 is dedicated to the cloud computing system 303 and is known by the cloud computing system 303 as a memory address that can be used to read the requested data. In some embodiments, the storage CPU 331 on the cloud computing system 303 reads the requested data directly from the computer memory 323 on the cloud storage system 301 through the PCIe architecture. In some embodiments, the storage CPU 331 on the cloud computing system 303 instructs the DMA engine to read the requested data from the computer memory 323 on the cloud storage system 301 through the PCIe architecture. In various embodiments, the DMA engine may be implemented within the cloud computing system 303 (e.g., DMA controller 339), or within the PCIe switch 305, or within another component accessible through the PCIe architecture of the cloud computing system 303. The DMA engine may be used to read requested data from the computer memory 323 on the cloud storage system 301 to help offload the storage CPU 331 of the cloud computing system 303.

[0103] Fig.11 A method for requesting and transmitting data in a cloud gaming system according to some embodiments is shown, the cloud gaming system including a cloud storage system 301 connected to a cloud computing system 303 via a PCIe architecture. The method includes an operation 1101 for having a cloud gaming system including a cloud storage system 301 and a cloud computing system 303 connected to each other via a PCIe architecture. The cloud storage system 301 includes a computer memory 323, and the cloud computing system 303 includes a computer memory 337. The PCIe architecture includes a PCIe switch 305 connected to a PCIe interface of the cloud storage system 301 and to a PCIe interface of the cloud computing system 303. The method includes an operation 1103, wherein the cloud computing system 303 operates to generate a command buffer for a data read operation. The method includes an operation 1105, wherein the cloud computing system 303 operates to write the command buffer to the computer memory 323 of the cloud storage system 301 via the PCIe architecture. The method includes operation 1107, where the cloud computing system 303 operates to notify the cloud storage system 301 through the PCIe architecture that the command buffer has been written to the computer memory 323 of the cloud storage system 301. In some embodiments, operation 1107 includes operating the cloud computing system 303 to write to a doorbell register exposed through the PCIe switch 305, where the doorbell register is monitored by the cloud storage system 301.

[0104] The method includes an operation 1109, wherein the cloud storage system 301 operates to read a command buffer from a computer memory 323 of the cloud storage system 301. The method includes an operation 1111, wherein the cloud storage system 301 operates to process the command buffer to read the requested data from a storage device accessible to the cloud storage system 301 and write the requested data in the computer memory 323 of the cloud storage system 301. The method includes an operation 1113, wherein the cloud storage system 301 operates to notify the cloud computing system 303 through the PCIe architecture that the requested data is in the computer memory 323 of the cloud storage system 301. In some embodiments, operation 1113 includes operating the cloud storage system 301 to write to a doorbell register exposed through the PCIe switch 305, wherein the doorbell register is monitored by the cloud computing system 303. The method includes an operation 1115, wherein the cloud computing system 303 operates to read the requested data from the computer memory 323 of the cloud storage system 301 through the PCIe architecture.

[0105] exist Fig.11 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321. Fig.11 In some embodiments of the method, the cloud storage system 301 includes an IOMMU 381, and the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. Fig.11 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the cloud storage system 301 includes an IOMMU 381, wherein the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321, and wherein the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381.

[0106] In a second exemplary embodiment, the command buffer is transmitted to the cloud storage system 301 by the cloud computing system 303 operating to write the command buffer to the computer memory 323 of the cloud storage system 301 via the PCIe architecture. The cloud storage server 301 executes the command buffer to read the requested data from a data storage device accessible to the cloud storage server 301. The requested data is transmitted to the cloud computing system 303 by the cloud storage system 301 operating to write the requested data to the computer memory 337 of the cloud computing system 303 via the PCIe architecture.

[0107] In a second exemplary embodiment, the cloud computing system 303 generates a command buffer to read data from the cloud storage system 301. The data to be read is referred to as requested data. The command buffer specifies a physical memory address in the computer memory 337 of the cloud computing system 303 at which the cloud storage system 301 can write the requested data via the PCIe architecture. If the PCIe switch 305 includes an NTB 321, the physical memory address in the computer memory 337 of the cloud computing system 303 that is specified in the command buffer as indicating that the cloud storage system 301 can write the requested data therein is a converted memory address generated by the NTB 321. In this case, the storage CPU 331 of the cloud computing system 303 operates to modify the original command buffer to include the converted memory address generated by the NTB 321.

[0108] The cloud computing system 303 writes the command buffer to the computer memory 323 on the cloud storage system 301 through the PCIe architecture. The cloud computing system 303 notifies the cloud storage server 301 that the command buffer has arrived in the computer memory 323 of the cloud storage system 301. In some embodiments, this notification of the command buffer's arrival in the computer memory 323 of the cloud storage system 301 is accomplished by the cloud computing system 303 writing to a doorbell register or sending a message to the cloud storage system 301. The doorbell register may be in any memory location monitored by the cloud storage system 301. For example, the doorbell register may be in the computer memory 323 of the cloud storage system 301 or in the PCIe switch 305, or even in the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The notification of the command buffer's arrival in the computer memory 323 of the cloud storage system 301 causes an interrupt to be triggered on the cloud storage system 301.

[0109] In response to the interrupt, the cloud storage system 301 reads the command buffer from the computer memory 323 of the cloud storage system 301 and processes the command buffer to obtain the requested data from the data storage device accessible to the cloud storage system 301. The cloud storage system 301 may optionally write the requested data to the computer memory 323 of the cloud storage system 301. Then, the cloud storage server 301 writes the requested data to the memory address specified in the command buffer on the cloud computing system 303 through the PCIe architecture. In some embodiments, the cloud storage system 301 writes the requested data directly to the computer memory 337 on the cloud computing system 303 through the PCIe architecture. In some embodiments, the cloud storage system 301 instructs the DMA engine to write the requested data to the computer memory 337 on the cloud computing system 303 through the PCIe architecture. In various embodiments, the DMA engine may be implemented within the cloud storage system 301, or within the PCIe switch 305, or within another component of the cloud storage system 301 that is accessible through the PCIe architecture. A DMA engine may be used to write the requested data to computer memory 337 on cloud computing system 303 to help offload CPU 365 of cloud storage system 301 .

[0110] The cloud storage system 301 then notifies the cloud computing system 303 that the requested data is available in the computer memory 337 of the cloud computing system 303. In some embodiments, this notification that the requested data is available in the computer memory 337 of the cloud computing system 303 is accomplished by the cloud storage system 301 writing to a doorbell register or sending a message to the cloud computing system 303. The doorbell register may be in any memory location monitored by the cloud computing system 303. For example, the doorbell register may be in the computer memory 337 of the cloud computing system 303 or in the PCIe switch 305, or even in the computer memory 323 of the cloud storage system 301 through the PCIe architecture.

[0111] Fig.12 A method for requesting and transmitting data in a cloud gaming system according to some embodiments is shown, the cloud gaming system including a cloud storage system 301 connected to a cloud computing system 303 via a PCIe architecture. The method includes an operation 1201 for having a cloud gaming system, the cloud gaming system including a cloud storage system 301 and a cloud computing system 303 connected to each other via a PCIe architecture. The cloud storage system 301 includes a computer memory 323, and the cloud computing system 303 includes a computer memory 337. The PCIe architecture includes a PCIe switch 305 connected to a PCIe interface of the cloud storage system 301 and to a PCIe interface of the cloud computing system 303. The method includes an operation 1203, wherein the cloud computing system 303 operates to generate a command buffer for a data read operation. The method also includes an operation 1205, wherein the cloud computing system 303 operates to write the command buffer to the computer memory 323 of the cloud storage system 301 via the PCIe architecture. The method also includes operation 1207, where the cloud computing system 303 operates to notify the cloud storage system 301 through the PCIe architecture that the command buffer has been written to the computer memory 323 of the cloud storage system 301. In some embodiments, operation 1207 includes operating the cloud computing system 303 to write to a doorbell register exposed through the PCIe switch 305, where the doorbell register is monitored by the cloud storage system 301.

[0112] The method also includes an operation 1209, wherein the cloud storage system 301 operates to read the command buffer from the computer memory 323 of the cloud storage system 301. The method also includes an operation 1211, wherein the cloud storage system 301 operates to process the command buffer to read the requested data from a storage device accessible to the cloud storage system 301. The method also includes an operation 1213, wherein the cloud storage system 301 operates to write the requested data to the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The method also includes an operation 1215, wherein the cloud storage system 301 operates to notify the cloud computing system 303 through the PCIe architecture that the requested data is in the computer memory 337 of the cloud computing system 303. In some embodiments, operation 1215 includes operating the cloud storage system 301 to write to a doorbell register exposed through the PCIe switch 305, wherein the doorbell register is monitored by the cloud computing system 303.

[0113] exist Fig.12 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321. Fig.12 In some embodiments of the method, the cloud storage system 301 includes an IOMMU 381, and the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. Fig.12 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the cloud storage system 301 includes an IOMMU 381, wherein the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321, and wherein the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. If the PCIe switch 305 includes the NTB 321, the physical memory address in the computer memory 337 of the cloud computing system 303 that is designated in the command buffer as indicating where the cloud storage system 301 can write the requested data is a converted memory address generated by the NTB 321. In this case, the storage CPU 331 of the cloud computing system 303 operates to modify the original command buffer to include the converted memory address generated by the NTB 321.

[0114] In a third exemplary embodiment, the command buffer is transmitted to the cloud storage system 301 by the cloud storage server 301 operating to read the command buffer from the computer memory 337 of the cloud computing system 303 via the PCIe architecture. The cloud storage server 301 executes the command buffer to read the requested data from a data storage device accessible to the cloud storage server 301. The requested data is transmitted to the cloud computing system 303 by the cloud storage system 301 operating to write the requested data to the computer memory 337 of the cloud computing system 303 via the PCIe architecture.

[0115] In a third exemplary embodiment, the cloud computing system 303 generates a command buffer to read data from the cloud storage system 301. The data to be read is referred to as requested data. The command buffer specifies a physical memory address in the computer memory 337 of the cloud computing system 303 at which the cloud storage system 301 can write the requested data via the PCIe architecture. If the PCIe switch 305 includes an NTB 321, the physical memory address in the computer memory 337 of the cloud computing system 303 that is specified in the command buffer as indicating that the cloud storage system 301 can write the requested data therein is a converted memory address generated by the NTB 321. In this case, the storage CPU 331 of the cloud computing system 303 operates to modify the original command buffer to include the converted memory address generated by the NTB 321.

[0116] The cloud computing system 303 writes the command buffer to computer memory 337 on the cloud computing system 303. The cloud computing system 303 notifies the cloud storage server 301 that the command buffer has arrived in the computer memory 337 of the cloud computing system 303. In some embodiments, this notification of the command buffer's arrival in the computer memory 337 of the cloud computing system 303 is accomplished by the cloud computing system 303 writing to a doorbell register or sending a message to the cloud storage system 301. The doorbell register may be in any memory location monitored by the cloud storage system 301. For example, the doorbell register may be in the computer memory 323 of the cloud storage system 301 or in the PCIe switch 305, or even in the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The notification of the command buffer's arrival in the computer memory 337 of the cloud computing system 303 causes an interrupt to be triggered on the cloud storage system 301.

[0117] In response to the interrupt, the cloud storage system 301 reads the command buffer from the computer memory 337 of the cloud computing system 303 through the PCIe architecture, and processes the command buffer to obtain the requested data from the data storage device accessible to the cloud storage system 301. The cloud storage system 301 may optionally write the requested data to the computer memory 323 of the cloud storage system 301. Then, the cloud storage system 301 writes the requested data to the memory address specified in the command buffer on the cloud computing system 303 through the PCIe architecture. In some embodiments, the cloud storage system 301 writes the requested data directly to the computer memory 337 on the cloud computing system 303 through the PCIe architecture. In some embodiments, the cloud storage system 301 instructs the DMA engine to write the requested data to the computer memory 337 on the cloud computing system 303 through the PCIe architecture. In various embodiments, the DMA engine may be implemented within the cloud storage system 301, or within the PCIe switch 305, or within another component accessible to the cloud storage system 301 through the PCIe architecture. A DMA engine may be used to write the requested data to computer memory 337 on cloud computing system 303 to help offload CPU 365 of cloud storage system 301 .

[0118] The cloud storage system 301 then notifies the cloud computing system 303 that the requested data is available in the computer memory 337 of the cloud computing system 303. In some embodiments, this notification that the requested data is available in the computer memory 337 of the cloud computing system 303 is accomplished by the cloud storage system 301 writing to a doorbell register or sending a message to the cloud computing system 303. The doorbell register may be in any memory location monitored by the cloud computing system 303. For example, the doorbell register may be in the computer memory 337 of the cloud computing system 303 or in the PCIe switch 305, or even in the computer memory 323 of the cloud storage system 301 through the PCIe architecture.

[0119] Fig.13A method for requesting and transmitting data in a cloud gaming system according to some embodiments is shown, the cloud gaming system including a cloud storage system 301 connected to a cloud computing system 303 via a PCIe architecture. The method includes an operation 1301 for having a cloud gaming system, the cloud gaming system including a cloud storage system 301 and a cloud computing system 303 connected to each other via a PCIe architecture. The cloud storage system 301 includes a computer memory 323, and the cloud computing system 303 includes a computer memory 337. The PCIe architecture includes a PCIe switch 305 connected to a PCIe interface of the cloud storage system 301 and to a PCIe interface of the cloud computing system 303. The method includes an operation 1303, wherein the cloud computing system 303 operates to generate a command buffer for a data read operation. The method also includes an operation 1305, wherein the cloud computing system 303 operates to write the command buffer to the computer memory 337 of the cloud computing system 303. The method also includes operation 1307, where the cloud computing system 303 operates to notify the cloud storage system 301 through the PCIe architecture that the command buffer has been written to the computer memory 337 of the cloud computing system 303. In some embodiments, operation 1307 includes operating the cloud computing system 303 to write to a doorbell register exposed through the PCIe switch 305, where the doorbell register is monitored by the cloud storage system 301.

[0120] The method also includes operation 1309, wherein the cloud storage system 301 operates to read the command buffer from the computer memory 323 of the cloud computing system 301 through the PCIe architecture. The method also includes operation 1311, wherein the cloud storage system 301 operates to process the command buffer to read the requested data from a storage device accessible to the cloud storage system 301. The method also includes operation 1313, wherein the cloud storage system 301 operates to write the requested data to the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The method also includes operation 1315, wherein the cloud storage system 301 operates to notify the cloud computing system 303 through the PCIe architecture that the requested data is in the computer memory 337 of the cloud computing system 303. In some embodiments, operation 1315 includes operating the cloud storage system 301 to write to a doorbell register exposed through the PCIe switch 305, wherein the doorbell register is monitored by the cloud computing system 303.

[0121] exist Fig.13 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321. Fig.13In some embodiments of the method, the cloud storage system 301 includes an IOMMU 381, and the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. Fig.13 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the cloud storage system 301 includes an IOMMU 381, wherein the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321, and wherein the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. If the PCIe switch 305 includes the NTB 321, the physical memory address in the computer memory 337 of the cloud computing system 303 that is designated in the command buffer as indicating where the cloud storage system 301 can write the requested data is a converted memory address generated by the NTB 321. In this case, the storage CPU 331 of the cloud computing system 303 operates to modify the original command buffer to include the converted memory address generated by the NTB 321.

[0122] In a fourth exemplary embodiment, the command buffer is transmitted to the cloud storage system 301 by the cloud storage server 301 operating to read the command buffer from the computer memory 337 of the cloud computing system 303 via the PCIe architecture. The cloud storage server 301 executes the command buffer to read the requested data from a data storage device accessible to the cloud storage server 301. The requested data is transmitted to the cloud computing system 303 by the cloud computing system 303 operating to read the requested data from the computer memory 323 of the cloud storage system 301 via the PCIe architecture.

[0123] In a fourth exemplary embodiment, the cloud computing system 303 generates a command buffer to read data from the cloud storage system 301. The data to be read is referred to as the requested data. The cloud computing system 303 writes the command buffer to a computer memory 337 on the cloud computing system 303. The cloud computing system 303 notifies the cloud storage server 301 that the command buffer has arrived in the computer memory 337 of the cloud computing system 303. In some embodiments, this notification of the arrival of the command buffer in the computer memory 337 of the cloud computing system 303 is accomplished by the cloud computing system 303 writing to a doorbell register or sending a message to the cloud storage system 301. The doorbell register may be in any memory location monitored by the cloud storage system 301. For example, the doorbell register may be in the computer memory 323 of the cloud storage system 301 or in the PCIe switch 305, or even in the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The notification of the arrival of the command buffer in the computer memory 337 of the cloud computing system 303 causes an interrupt to be triggered on the cloud storage system 301.

[0124] In response to the interrupt, the cloud storage system 301 reads the command buffer from the computer memory 337 of the cloud computing system 303 through the PCIe architecture and processes the command buffer to obtain the requested data from the data storage device accessible to the cloud storage system 301. The cloud storage system 301 writes the requested data to the computer memory 323 of the cloud storage system 301. The cloud storage system 301 then notifies the cloud computing system 303 that the requested data is available in the computer memory 323 of the cloud storage system 301 and where the requested data is located in the computer memory 323 of the cloud storage system 301. In some embodiments, this notification that the requested data is available in the computer memory 323 of the cloud storage system 301 is accomplished by the cloud storage system 301 writing to a doorbell register or sending a message to the cloud computing system 303. The doorbell register can be in any memory location monitored by the cloud computing system 303. For example, the doorbell register may be in the computer memory 337 of the cloud computing system 303 or in the PCIe switch 305, or even in the computer memory 323 of the cloud storage system 301 via the PCIe architecture.

[0125] In response to the notification that the requested data is available in the computer memory 323 of the cloud storage system 301, the cloud computing system 303 reads the requested data from the computer memory 323 on the cloud storage system 301. In some embodiments, the cloud computing system 303 uses the memory address returned by the cloud storage system 301 to read the requested data from the computer memory 323 on the cloud storage system 301. In some embodiments, the pre-assigned fixed memory address in the computer memory 323 on the cloud storage system 301 is dedicated to the cloud computing system 303 and is known by the cloud computing system 303 as a memory address that can be used to read the requested data. In some embodiments, the storage CPU 331 on the cloud computing system 303 reads the requested data directly from the computer memory 323 on the cloud storage system 301 through the PCIe architecture. In some embodiments, the storage CPU 331 on the cloud computing system 303 instructs the DMA engine to read the requested data from the computer memory 323 on the cloud storage system 301 through the PCIe architecture. In various embodiments, the DMA engine may be implemented within the cloud computing system 303 (e.g., DMA controller 339), or within the PCIe switch 305, or within another component accessible through the PCIe architecture of the cloud computing system 303. The DMA engine may be used to read requested data from the computer memory 323 on the cloud storage system 301 to help offload the storage CPU 331 of the cloud computing system 303.

[0126] Fig.14A method for requesting and transmitting data in a cloud gaming system according to some embodiments is shown, the cloud gaming system including a cloud storage system 301 connected to a cloud computing system 303 via a PCIe architecture. The method includes operation 1401 for having a cloud gaming system, the cloud gaming system including a cloud storage system 301 and a cloud computing system 303 connected to each other via a PCIe architecture. The cloud storage system 301 includes a computer memory 323, and the cloud computing system 303 includes a computer memory 337. The PCIe architecture includes a PCIe switch 305 connected to a PCIe interface of the cloud storage system 301 and to a PCIe interface of the cloud computing system 303. The method includes operation 1403, wherein the cloud computing system 303 operates to generate a command buffer for a data read operation. The method also includes operation 1405, wherein the cloud computing system 303 operates to write the command buffer to the computer memory 337 of the cloud computing system 303. The method also includes operation 1407, where the cloud computing system 303 operates to notify the cloud storage system 301 through the PCIe architecture that the command buffer has been written to the computer memory 337 of the cloud computing system 303. In some embodiments, operation 1407 includes operating the cloud computing system 303 to write to a doorbell register exposed through the PCIe switch 305, where the doorbell register is monitored by the cloud storage system 301.

[0127] The method further includes operation 1409, in which the cloud storage system 301 operates to read the command buffer from the computer memory 337 of the cloud computing system 303 through the PCIe architecture. The method includes operation 1411, in which the cloud storage system 301 operates to process the command buffer to read the requested data from the storage device accessible to the cloud storage system 301 and write the requested data in the computer memory 323 of the cloud storage system 301.

[0128] The method includes operation 1413, in which the cloud storage system 301 operates to notify the cloud computing system 303 through the PCIe fabric that the requested data is in the computer memory 323 of the cloud storage system 301. In some embodiments, operation 1413 includes operating the cloud storage system 301 to write to a doorbell register exposed through the PCIe switch 305, wherein the doorbell register is monitored by the cloud computing system 303. The method includes operation 1415, in which the cloud computing system 303 operates to read the requested data from the computer memory 323 of the cloud storage system 301 through the PCIe fabric.

[0129] exist Fig.14In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321. Fig.14 In some embodiments of the method, the cloud storage system 301 includes an IOMMU 381, and the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. Fig.14 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the cloud storage system 301 includes an IOMMU 381, wherein the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321, and wherein the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381.

[0130] Fig.15 A method for requesting and transmitting data between a cloud storage system 301 and a cloud computing system 303 in a cloud gaming system using doorbell registers exposed through a PCIe architecture is shown according to some embodiments. The method includes an operation 1501 for having a cloud gaming system, the cloud gaming system including a cloud storage system 301 having a PCIe interface, wherein the cloud storage system 301 includes a computer memory 323. The cloud gaming system includes a PCIe switch 305 connected to the PCIe interface of the cloud storage system 301. The cloud gaming system also includes a cloud computing system 303 having a PCIe interface connected to the PCIe switch 305. In some embodiments, the PCIe switch 305 exposes one or more doorbell registers. The method includes an operation 1503 for operating the cloud computing system 303 to write to the doorbell registers exposed through the PCIe switch 305. In response to the cloud computing system 303 writing to the doorbell registers, the method continues with an operation 1505 for triggering an interrupt on the cloud storage system 301. The method includes operation 1507, wherein the cloud storage system 301 is operated to handle the interruption. The interruption instructs the cloud storage system 301 to read a message from a specified location in the computer memory 323 of the cloud storage system 301, wherein the message instructs the cloud storage system 301 to read the requested data from a storage device accessible to the cloud storage system 301.

[0131] In some embodiments, Fig.15The method also includes operating the cloud storage system 301 to write the requested data to the computer memory 323 of the cloud storage system 301. In these embodiments, the method also includes notifying the cloud computing system 303 that the requested data is stored in the computer memory 323 of the cloud storage system 301. In some embodiments, notifying the cloud computing system 303 of the memory address includes writing to a doorbell register. In some embodiments, the method also includes operating the cloud computing system 303 to read the requested data from the computer memory 323 of the cloud storage system 301.

[0132] In some embodiments, Fig.15 The method includes operating the cloud storage system 301 to write the requested data to the computer memory 337 of the cloud computing system 303. In some embodiments, the cloud computing system 301 has a client memory map that stores memory addresses assigned to components of the cloud computing system 301. In these embodiments, the PCIe switch 305 is configured to convert the memory addresses stored in the client memory map to memory addresses in the computer memory 323 of the cloud storage system 301, so that access to a given memory address in the client memory map is converted by the PCIe switch 305 to access the given memory address in the computer memory 323 of the cloud storage system 301 instead.

[0133] The cloud computing system 303 includes firmware to start its hardware. This firmware may have code to start from a local storage device, via a network, or via a PCIe architecture. In the case where the cloud computing system 303 is started via a network, the firmware of the cloud computing system 303 may issue a dynamic host configuration protocol (DHCP) request to request an Internet Protocol (IP) address and information about a storage server (such as the cloud storage system 301). The information about the cloud storage system 301 may include an IP address and a file system path for locating data, but may also include PCIe information. The PCIe information may include a memory address, PCIe bus / device / function information, or other information that allows the cloud computing system 303 to locate the cloud storage system 301 on which data is stored. In some embodiments, information that supports the startup of the cloud computing system 303 is preloaded into the firmware of the cloud computing system 303, such as at the factory or through another mechanism (e.g., a UART connection), etc. In the case where the cloud computing system 303 uses the PCIe architecture for data access to the startup information, the data access may occur through the storage CPU 331, and the corresponding data access code for the storage CPU 331 is part of the firmware of the cloud computing system 303. Furthermore, the cloud storage system 301 may preload the boot data required by the cloud computing system 303 into a memory accessible through the PCIe architecture, or may load the boot data into the memory accessible through the PCIe architecture as needed.

[0134] When the user opens the cloud game application, as part of the game opening process, the cloud game system may perform DRM-related checks to verify that the user's game license is currently valid, and obtain an optional decryption key as needed to decrypt the game data. After verifying the user's game license, the cloud game system operates to install the game data, especially when the game data is stored in a special game package format. In the cloud game use case, installation may trigger operations on the cloud storage system 301. For example, if the cloud storage system 301 does not have a copy of the game on one or more of its local storage devices (e.g., computer memory 323, NVMe SSD 371 or other local storage areas, such as HDD), the cloud storage system 301 may need to download the game to one or more of its local storage devices. Once the cloud storage system 301 has identified the game content, the cloud storage system 301 can transfer all or part of the game content to a fast storage area, for example, transferring the game content from the HDD to the computer memory, or transferring the game content from the HDD to the SSD and then to the computer memory. Because many users may play the same game, when determining that game content already exists in the appropriate location on cloud storage system 301, cloud storage server 301 can skip the download of game and transmit game content to fast storage area. For example, in some embodiments, cloud storage system 301 can use database to track the list of games currently loaded and the position in the memory into which corresponding game data is loaded. In some embodiments, the cloud service module outside cloud storage server 301 may have ensured that a given user is assigned to cloud storage server 301, wherein cloud storage server 301 has loaded the game content requested by a given user. In order to promote this embodiment, cloud storage server 301 can provide indicators and / or other metadata to cloud service module. It should be understood and understood that, from the perspective of user experience, loading game content from cloud storage system 301 that has stored game content locally may be the best, because the game loading time of the user is reduced.

[0135] When the cloud computing system 303 sends the read command buffer to the cloud storage system 301, the cloud storage system 301 has options on how to obtain the requested data. In some cases, the cloud storage system 301 may have preloaded all the requested data into the computer memory 323 during the installation operation. If this is the case, the cloud storage system 301 only needs to convert the offset and length of the data requested by the command buffer to the offset and length of the computer memory 323 in the cloud storage system 301. In some cases, the offset in the command buffer may be relative to the storage location of the game package. If so, the cloud storage system 301 needs to add the offset to the storage location at the beginning of the game package. In some cases, the game package can be stored across multiple memory areas, such as when there is not enough continuous memory space for a complete package. In these cases, the cloud storage system 301 needs to add the offset to the address of the portion of the memory containing the requested file. If multiple memory areas are involved, the cloud storage system 301 can return multiple memory areas. When the IOMMU 381 is implemented in the cloud storage system 301 , assuming that the cloud computing system 303 is responsible for reading the requested data from the cloud storage system 301 , the details of the multiple memory areas will be hidden from the cloud computing system 303 .

[0136] In some cases, the cloud storage system 301 may not load all game data into the local storage area during the installation of the game. For example, the cloud storage system 301 may only load the file system table and other data structures into the computer memory 323 of the cloud storage system 301. When the cloud storage system 301 receives a command buffer for data that is not currently in the computer memory 323, the cloud storage system 301 operates to retrieve the requested data based on the command buffer. For example, in a block storage setting, one method is that the cloud storage system 301 retrieves the requested data based on an offset parameter and a length parameter, and then caches the retrieved data in the computer memory 323 for future access. Caching data in the computer memory 323 bypasses the need to retrieve data from a slower storage device such as an HDD or SSD, and improves the performance of the cloud storage system 301. In some cases, it may be difficult to predict which other data the game application will need to access. In this case, cache misses may occur frequently. Thus, in this case, even though the gaming application may not use the predictively cached data, data stored after the memory block just accessed is also predictively cached in the computer memory 323 .

[0137] In some embodiments, the gaming application has a Fig.10A dedicated game file system for the file system table discussed. In this case, the file system is quite "flat" and can be viewed as a key-value store, where each file is identified by a file identifier. In this case, the cloud storage system 301 has more information about the input / output request than the location of the data. More specifically, in this case, the cloud storage system 301 knows what data belongs to a given file. Therefore, instead of the cloud storage system 301 operating to perform caching based on memory blocks, in this case, the cloud storage system 301 can perform a smarter cache of data based on the file level. For example, the cloud storage system 301 can pre-cache a complete file from beginning to end based on a data request for that file. Moreover, the cloud storage system 301 can pre-cache portions of files when appropriate. In the key-value approach, the cache layer implemented by the cloud storage system 301 can operate to maintain a set of cache metadata, which includes one or more counters, timestamps, and other information about the data cached in the computer memory 323, such as how frequently or recently each data file is accessed. In some embodiments, a cache layer implemented by the cloud storage system 301 may use cache metadata to evict data from the computer memory 323 that has not been accessed within a specified amount of time or has not been accessed frequently enough.

[0138] In some cases, data access may be for user-specific data, which may be located on a HDD that is accessible locally by the cloud storage system 301 or through a network (the Internet) accessible to the cloud storage system 301. The POSIX API may be mapped via a command buffer that triggers data access to such user-specific data. In order to improve cloud gaming performance, the cloud storage system 301 may be operated to pre-fetch the user-specific data from a specific data storage device and save any modified versions of the user-specific data back to the specific data storage device. The pre-fetching of user-specific data may require deep interaction with the hardware, software, and possibly multiple servers of the cloud computing system 303.

[0139] Above about Figures 11 to 15 The embodiments and methods discussed involve a client / server relationship in which command buffers specify input / output data requests communicated from cloud computing system 303 (client) to cloud storage system 301 (server). In these embodiments, CPU 365 on cloud storage system 301 participates in some capacity in the processing of command buffers and may participate in the exchange of requested data between cloud storage system 301 and cloud computing system 303.

[0140] In some embodiments, the processing of the command buffer and the exchange of the requested data may be performed by the cloud storage system 301 while bypassing the CPU 365 of the cloud storage system 301. For example, in some embodiments, the CPU 365 of the cloud storage system 301 may include one or more CPU cores, one or more memory controllers, one or more PCIe root complexes, and other components. The unloading of the CPU 365 of the cloud storage system 301 refers to bypassing one or more CPU cores of the CPU 365 to perform any software-level processing. When the CPU 365 is unloaded, other non-CPU core components within the CPU 365 (such as a memory controller, a DMA engine, or other components) may still be utilized to implement the processing of the command buffer and the exchange of the requested data. One benefit of bypassing the CPU 365 when processing the command buffer and exchanging the requested data is to avoid interrupting the CPU core within the CPU 365, and thus improve the performance of the cloud storage system 301. It should be understood and appreciated that the processing performed by one or more CPU cores within the CPU 365 adds additional overhead and causes more waiting time. The CPU core or cores within CPU 365 are also expensive and consume power, so there are cost savings and energy savings associated with offloading CPU 365.

[0141] In some embodiments, the CPU 365 of the cloud storage system 301 can be offloaded by obtaining the cloud computing system 303 DMA data directly from the cloud storage system 301. In these embodiments, the cloud computing system 303 uses a file identifier table to locate data on the local storage area of ​​the cloud storage system 301. In some embodiments, during the installation operation of the game application, the cloud storage system 301 generates a custom file identifier table that includes the memory addresses of the data listed in the game package for the installed game application in the computer memory 323 of the cloud storage system 301. The cloud storage server 301 makes the file identifier table of the game application available to the cloud computing system 303. The cloud computing system 303 can then use the file identifier table to determine the memory address within the computer memory 323 of the cloud storage system 301 where the specific required data is located, and perform a corresponding DMA operation to read the required data directly from the computer memory 323 of the cloud storage system 301 without having to interrupt one or more CPU cores of the CPU 365 of the cloud storage system 301. Additionally, in this method, the cloud computing system 303 does not need to transfer the input / output command buffer to the cloud storage system 301. This method of using the file identifier table and DMA operations on the cloud computing system 303 is particularly suitable for accessing read-only data (such as game packages, and other types of data) stored in the cloud storage system 301. Moreover, this method of using the file identifier table and DMA operations on the cloud computing system 303 is likely to achieve a high level of data access performance because it does not require the cloud storage server 301 to process each command buffer.

[0142] Fig.16 A method for data transfer in a cloud gaming system according to some embodiments is shown. The method includes an operation 1601 for having a cloud gaming system, the cloud gaming system including a cloud storage system 301 and a cloud computing system 303 connected to each other via a PCIe architecture. The cloud gaming system includes a PCIe switch 305 connected to a PCIe interface of the cloud storage system 301. The cloud gaming system also includes a PCIe switch 305 connected to a PCIe interface of the cloud computing system 303. The cloud storage system 301 includes a computer memory 323. Moreover, the cloud computing system 303 includes a computer memory 337. The method includes an operation 1603, wherein the cloud storage system 301 operates to generate a file identification table, the file identification table specifies the memory addresses of the computer memory 323 of the cloud storage system 301 storing different data files. In some embodiments, operation 1603 is completed during the installation of the game package, wherein the different data files are data files of the game package.

[0143] The method also includes an operation 1605 for storing the file identification table in the computer memory 337 of the cloud computing system 303. In some embodiments, the operation 1605 includes operating the cloud storage system 301 to notify the cloud computing system 303 through the PCIe fabric that the file identification table is available in the computer memory 323 of the cloud storage system 301 and where the file identification table is stored in the computer memory 323 of the cloud storage system 301. Moreover, the operation 1605 may include operating the cloud computing system 303 to read the file identification table from the computer memory 323 of the cloud storage system 301 through the PCIe fabric. In some embodiments, operating the cloud storage system 301 to notify the cloud computing system 303 through the PCIe fabric that the file identification table is in the computer memory 323 of the cloud storage system 301 includes operating the cloud storage system 301 to write to a doorbell register exposed through a PCIe switch 305 to which both the cloud storage system 301 and the cloud computing system 303 are connected. The method further includes operation 1607, wherein the cloud computing system 303 operates to use the file identification table to determine a memory address storing the desired data file in the computer memory 323 of the cloud storage system 301. The method further includes operation 1609, wherein the cloud computing system 303 operates to read the desired data file from the computer memory 323 of the cloud storage system 301 through the PCIe architecture.

[0144] exist Fig.16 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321. Fig.16 In some embodiments of the method, the cloud storage system 301 includes an IOMMU 381, and the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381. Fig.16 In some embodiments of the method, the PCIe switch 305 includes an NTB 321, and the cloud storage system 301 includes an IOMMU 381, wherein the PCIe interface of the cloud computing system 303 is connected to the PCIe switch 305 through the NTB 321, and wherein the PCIe interface of the cloud storage system 301 is connected to the IOMMU 381.

[0145] Although some method operations are described herein in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted so that they occur at slightly different times, or operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing. Embodiments of the present invention may be practiced using various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronic devices, minicomputers, mainframe computers, etc. The present invention may also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked by wired or wireless networks.

[0146] In view of the above embodiments, it should be understood that the present invention may employ various computer-implemented operations involving data stored in a computer system. These operations are operations that require physical manipulation of physical quantities. Any operation forming a part of the present invention described herein is a useful machine operation. The present invention also relates to a device or apparatus for performing these operations. The apparatus may be specially constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used together with computer programs written according to the teachings of this article, or it may be more convenient to construct more specialized equipment to perform the desired operations.

[0147] The various components used with the present invention may be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device capable of storing data that can thereafter be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. Computer readable media may include computer readable tangible media distributed on a network coupled computer system so that computer readable code is stored and executed in a distributed manner.

[0148] Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments are to be considered illustrative rather than restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalent range of the described embodiments.

Claims

1. A cloud gaming system, comprising: A cloud storage system having a peripheral component interconnect express (PCIe) interface and a first root complex, the cloud storage system comprising a computer memory; A PCIe switch, the PCIe switch connected to the PCIe interface of the cloud storage system; as well as A cloud computing system having a PCIe interface and a second root complex connected to the PCIe switch, the cloud computing system having a client memory map storing memory addresses assigned to components of the cloud computing system, the PCIe switch being configured to convert the memory addresses stored in the client memory map to memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory map is converted by the PCIe switch to access the given memory address in the computer memory of the cloud storage system instead.

2. The cloud gaming system of claim 1, wherein the cloud storage system and the cloud computing system are positioned in a rack.

3. A cloud gaming system as described in claim 2, wherein the PCIe interface of the cloud computing system is connected to the backplane of the rack, and wherein the PCIe switch is arranged in the backplane of the rack, and wherein the backplane of the rack is connected to the cloud storage system via a cable.

4. The cloud gaming system of claim 1, wherein the cloud computing system is a first cloud computing system among a plurality of cloud computing systems, each of the plurality of cloud computing systems having a corresponding PCIe interface connected to the PCIe switch.

5. The cloud gaming system of claim 4, wherein each of the plurality of cloud computing systems is a corresponding cloud computing system having a corresponding client memory mapping, the corresponding client memory mapping storing memory addresses assigned to components of the corresponding cloud computing system, and the PCIe switch being configured to convert the memory addresses stored in the corresponding client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the corresponding client memory mapping is converted by the PCIe switch into accessing a given memory address in the computer memory of the cloud storage system instead.

6. A cloud gaming system as described in claim 5, wherein the PCIe interface of the cloud storage system is connected to the PCIe switch through a certain number N of channels, and wherein the corresponding PCIe interface of each of the multiple cloud computing systems is connected to the PCIe switch through a certain number M of channels, wherein each channel has a first pair of electrical signal conductors for receiving data through differential signaling and a second pair of electrical signal conductors for transmitting data through differential signaling.

7. A cloud gaming system as described in claim 6, wherein the number of channels N is equal to the number of channels M multiplied by the number of the multiple cloud computing systems.

8. The cloud gaming system of claim 6, wherein the number N of channels is less than the number M of channels multiplied by the number of the plurality of cloud computing systems.

9. A cloud gaming system as described in claim 4, wherein the cloud computing system is the first cloud computing system among multiple cloud computing systems implemented on a computing drawer, the computing drawer includes a PCIe interface, and each of the multiple cloud computing systems has a corresponding PCIe interface connected to the PCIe interface of the computing drawer.

10. A cloud gaming system as described in claim 9, wherein the computing drawer and the cloud storage system are installed in a rack, wherein the PCIe interface of the computing drawer is connected to the backplane of the rack, and wherein the PCIe switch is arranged in the backplane of the rack, and wherein the backplane of the rack is connected to the cloud storage system via a cable.

11. A cloud gaming system as described in claim 10, wherein the computing drawer is the first computing drawer among a plurality of computing drawers installed in the rack, wherein each of the plurality of computing drawers has a corresponding PCIe interface, wherein each PCIe interface of the plurality of computing drawers is connected to the backplane of the rack.

12. The cloud gaming system of claim 1, wherein the PCIe switch is a first PCIe switch among a plurality of PCIe switches, each of the plurality of PCIe switches being connected to the PCIe interface of the cloud storage system.

13. The cloud gaming system of claim 12, wherein each of the PCIe switches is a corresponding PCIe switch, the corresponding PCIe switch having at least one cloud computing system connected to the corresponding PCIe switch, each of the at least one cloud computing system being a corresponding cloud computing system having a corresponding client memory mapping, the corresponding client memory mapping storing memory addresses assigned to components of the corresponding cloud computing system, the corresponding PCIe switch being configured to convert the memory addresses stored in the corresponding client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the corresponding client memory mapping is converted by the corresponding PCIe switch into accessing a given memory address in the computer memory of the cloud storage system instead.

14. The cloud gaming system of claim 1, wherein the PCIe switch comprises a PCIe non-transparent bridge (NTB) and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, or wherein the cloud storage system comprises an input / output memory management unit (IOMMU) and the PCIe interface of the cloud storage system is connected to the IOMMU, or The PCIe switch includes the PCIe NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, and the cloud storage system includes the IOMMU and the PCIe interface of the cloud storage system is connected to the IOMMU.

15. The cloud gaming system according to claim 1, further comprising: At least one additional component, the at least one additional component having a PCIe interface connected to the PCIe switch, the at least one additional component being one or more of a graphics processing unit GPU, a network adapter, a fast non-volatile memory NVMe device, a universal asynchronous receiver-transmitter UART controller, a sensor device and / or a general purpose input / output GPIO device, the PCIe switch being configured to assign the at least one additional component to a memory address range in the computer memory of the cloud storage system.

16. A cloud gaming system as described in claim 1, wherein the PCIe switch exposes a doorbell register, wherein writing to the doorbell register by the cloud computing system causes an interrupt to be triggered on the cloud storage system, which in turn causes the cloud storage system to handle the interrupt, the interrupt instructing the cloud storage system to read a message from a specified location in the computer memory of the cloud storage system, the message instructing the cloud storage system to perform a specified action.

17. A cloud gaming system as described in claim 16, wherein the specified action is to read a specific amount of data from a storage device accessible to the cloud storage system and write the specific amount of data to the computer memory of the cloud storage system, and to notify the cloud computing system of the memory address where the specific amount of data is stored in the computer memory of the cloud storage system.

18. A cloud gaming system, comprising: A cloud storage system having a peripheral component interconnect express (PCIe) interface and a first root complex, the cloud storage system comprising a computer memory; a first PCIe switch, the first PCIe switch being connected to the PCIe interface of the cloud storage system; a second PCIe switch, wherein the second PCIe switch is connected to the first PCIe switch; a third PCIe switch, the third PCIe switch being connected to the first PCIe switch; a first cloud computing system having a PCIe interface connected to the second PCIe switch and a second complex, the first cloud computing system having a first client memory map storing memory addresses assigned to components of the first cloud computing system, one or both of the first PCIe switch and the second PCIe switch being configured to translate the memory addresses stored in the first client memory map into memory addresses in the computer memory of the cloud storage system such that access to a given memory address in the first client memory map is translated to access a given memory address in the computer memory of the cloud storage system instead; as well as a second cloud computing system having a PCIe interface and a third root complex connected to the third PCIe switch, the second cloud computing system having a second client memory map storing memory addresses assigned to components of the second cloud computing system, one or both of the first PCIe switch and the third PCIe switch being configured to convert the memory addresses stored in the second client memory map to memory addresses in the computer memory of the cloud storage system, such that access to a given memory address in the second client memory map is converted to access a given memory address in the computer memory of the cloud storage system instead.

19. The cloud gaming system of claim 18, wherein the first cloud computing system is one of a first plurality of cloud computing systems, each of the first plurality of cloud computing systems having a corresponding PCIe interface connected to the second PCIe switch, and The second cloud computing system is one of a second plurality of cloud computing systems, each of the second plurality of cloud computing systems having a corresponding PCIe interface connected to the third PCIe switch.

20. The cloud gaming system of claim 18, wherein the first cloud computing system is one of a first plurality of cloud computing systems implemented on a first computing drawer, the first computing drawer comprising a PCIe interface, each of the first plurality of cloud computing systems having a corresponding PCIe interface connected to the PCIe interface of the first computing drawer, and The second cloud computing system is one of a second plurality of cloud computing systems implemented on a second computing drawer, the second computing drawer includes a PCIe interface, and each of the second plurality of cloud computing systems has a corresponding PCIe interface connected to the PCIe interface of the second computing drawer.

21. A cloud gaming system as described in claim 20, wherein both the first computing drawer and the second computing drawer are installed in the same rack, or wherein the first computing drawer is installed in a first rack and the second computing drawer is installed in a second rack.

22. A method for requesting data in a cloud gaming system, the method comprising: There is a cloud gaming system, the cloud gaming system comprising a cloud storage system having a peripheral component interconnect express (PCIe) interface, the cloud storage system comprising a computer memory and a first root complex, the cloud gaming system comprising a PCIe switch connected to the PCIe interface of the cloud storage system, the cloud gaming system comprising a cloud computing system having a PCIe interface connected to the PCIe switch, the cloud computing system comprising a computer memory and a second root complex, the PCIe switch exposing a doorbell register monitored by the cloud storage system; operating the cloud computing system to write to the doorbell register, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud storage system; as well as operating the cloud storage system to handle the interrupt, the interrupt instructing the cloud storage system to read a message from a specified computer memory location, the message instructing the cloud storage system to read the requested data from a storage device accessible to the cloud storage system, The cloud computing system has a client memory mapping that stores memory addresses assigned to components of the cloud computing system, and the PCIe switch is configured to convert the memory addresses stored in the client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch into access to a given memory address in the computer memory of the cloud storage system.

23. The method of claim 22, further comprising: operating the cloud storage system to write the requested data to the computer memory of the cloud storage system; as well as The cloud computing system is informed of a memory address where the requested data is stored in the computer memory of the cloud storage system.

24. The method of claim 23, wherein notifying the cloud computing system of the memory address comprises writing to a doorbell register monitored by the cloud computing system.

25. The method of claim 23, further comprising: The cloud computing system is operated to read the requested data from the computer memory of the cloud storage system.

26. The method of claim 22, further comprising: operating the cloud storage system to write the requested data into a computer memory of the cloud computing system; as well as The cloud computing system is informed of a memory address where the requested data is stored in the computer memory of the cloud computing system.

27. The method of claim 26, wherein notifying the cloud computing system of the memory address comprises writing to a doorbell register monitored by the cloud computing system.

28. The method of claim 26, further comprising: The cloud computing system is operated to read the requested data from the computer memory of the cloud computing system.

29. The method of claim 22, wherein the designated computer memory location is in the computer memory of the cloud storage system.

30. The method of claim 22, wherein the designated computer memory location is in the computer memory of the cloud computing system.

31. A method for transmitting data in a cloud gaming system, the method comprising: A cloud gaming system is provided, the cloud gaming system comprising a cloud storage system and a cloud computing system connected to each other via a peripheral component interconnect express (PCIe) architecture, the cloud storage system comprising a computer memory and a first root complex, the cloud computing system comprising a computer memory and a second root complex; operating the cloud storage system to generate a file identification table, the file identification table specifying memory addresses in the computer memory of the cloud storage system storing different data files; operating the cloud computing system to determine a memory address in the computer memory of the cloud storage system storing a desired data file using the file identification table; as well as operating the cloud computing system to read the desired data file from the computer memory of the cloud storage system via the PCIe architecture, Wherein, the PCIe architecture includes a PCIe switch connected to a PCIe interface of the cloud storage system and to a PCIe interface of the cloud computing system, the cloud computing system has a client memory mapping that stores memory addresses assigned to components of the cloud computing system, and the PCIe switch is configured to convert the memory addresses stored in the client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch into access to a given memory address in the computer memory of the cloud storage system.

32. The method of claim 31, wherein operating the cloud storage system to generate the file identification table is performed during installation of a game package, wherein the different data files are data files of the game package.

33. The method of claim 31, further comprising: The cloud storage system is operated to store the file identification table in the computer memory of the cloud storage system.

34. The method of claim 33, further comprising: The cloud storage system is operated to notify the cloud computing system of the location of the file identification table in the computer memory of the cloud storage system through the PCIe architecture, and the cloud computing system is operated to read the file identification table from the computer memory of the cloud storage system through the PCIe architecture.

35. The method of claim 34, wherein operating the cloud storage system to notify the cloud computing system via the PCIe architecture of the location of the file identification table in the computer memory of the cloud storage system comprises operating the cloud storage system to write to a doorbell register exposed via a PCIe switch, to which both the cloud storage system and the cloud computing system are connected.

36. The method of claim 31, further comprising: The cloud storage system is operated to store the file identification table in the computer memory of the cloud computing system via the PCIe architecture.

37. The method of claim 36, further comprising: The cloud storage system is operated to notify the cloud computing system of the location of the file identification table in the computer memory of the cloud computing system through the PCIe architecture, and the cloud computing system is operated to read the file identification table from the computer memory of the cloud computing system.

38. The method of claim 37, wherein operating the cloud storage system to notify the cloud computing system via the PCIe architecture of the location of the file identification table in the computer memory of the cloud computing system comprises operating the cloud storage system to write to a doorbell register exposed via a PCIe switch, to which both the cloud storage system and the cloud computing system are connected.

39. The method of claim 31, wherein the PCIe switch comprises a PCIe non-transparent bridge (NTB) and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, or wherein the cloud storage system comprises an input / output memory management unit (IOMMU) and the PCIe interface of the cloud storage system is connected to the IOMMU, or The PCIe switch includes the PCIe NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, and the cloud storage system includes the IOMMU and the PCIe interface of the cloud storage system is connected to the IOMMU.

40. A method for requesting and transmitting data in a cloud gaming system, the method comprising: A cloud gaming system is provided, the cloud gaming system comprising a cloud storage system and a cloud computing system connected to each other via a peripheral component interconnect express (PCIe) architecture, the cloud storage system comprising a computer memory and a first root complex, the cloud computing system comprising a computer memory and a second root complex; operating the cloud computing system to generate a command buffer for a data read operation; operating the cloud computing system to write the command buffer to the computer memory of the cloud storage system via the PCIe architecture; operating the cloud computing system to notify the cloud storage system via the PCIe architecture that the command buffer has been written to the computer memory of the cloud storage system; operating the cloud storage system to read the command buffer from the computer memory of the cloud storage system; operating the cloud storage system to process the command buffer to read requested data from a storage device accessible to the cloud storage system and to write the requested data to the computer memory of the cloud storage system; operating the cloud storage system to notify the cloud computing system via the PCIe architecture that the requested data is in the computer memory of the cloud storage system; as well as operating the cloud computing system to read the requested data from the computer memory of the cloud storage system via the PCIe architecture, Wherein, the PCIe architecture includes a PCIe switch connected to a PCIe interface of the cloud storage system and to a PCIe interface of the cloud computing system, the cloud computing system has a client memory mapping that stores memory addresses assigned to components of the cloud computing system, and the PCIe switch is configured to convert the memory addresses stored in the client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch into access to a given memory address in the computer memory of the cloud storage system.

41. The method of claim 40, wherein operating the cloud computing system to notify the cloud storage system that the command buffer has been written to the computer memory of the cloud storage system comprises operating the cloud computing system to write to a doorbell register exposed through the PCIe switch.

42. A method as claimed in claim 41, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud storage system, which in turn causes the cloud storage system to handle the interrupt, the interrupt instructing the cloud storage system to read a message from a specified computer memory location, the message instructing the cloud storage system to read the command buffer from the computer memory of the cloud storage system.

43. The method of claim 42, wherein the designated computer memory location is in the computer memory of the cloud storage system.

44. The method of claim 42, wherein the designated computer memory location is in the computer memory of the cloud computing system.

45. The method of claim 40, wherein operating the cloud storage system to notify the cloud computing system that the requested data is in the computer memory of the cloud storage system comprises operating the cloud storage system to write to a doorbell register exposed through the PCIe switch.

46. ​​A method as claimed in claim 45, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud computing system, which in turn causes the cloud computing system to handle the interrupt, the interrupt instructing the cloud computing system to read a message from a specified computer memory location, the message notifying the cloud computing system where the requested data is stored in the computer memory of the cloud storage system.

47. The method of claim 46, wherein the designated computer memory location is in the computer memory of the cloud storage system.

48. The method of claim 46, wherein the designated computer memory location is in the computer memory of the cloud computing system.

49. The method of claim 40, wherein the PCIe switch comprises a PCIe non-transparent bridge NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, or wherein the cloud storage system comprises an input / output memory management unit (IOMMU) and the PCIe interface of the cloud storage system is connected to the IOMMU, or The PCIe switch includes the PCIe NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, and the cloud storage system includes the IOMMU and the PCIe interface of the cloud storage system is connected to the IOMMU.

50. A method for requesting and transmitting data in a cloud gaming system, the method comprising: A cloud gaming system is provided, the cloud gaming system comprising a cloud storage system and a cloud computing system connected to each other via a peripheral component interconnect express (PCIe) architecture, the cloud storage system comprising a computer memory and a first root complex, the cloud computing system comprising a computer memory and a second root complex; operating the cloud computing system to generate a command buffer for a data read operation; operating the cloud computing system to write the command buffer to the computer memory of the cloud storage system via the PCIe architecture; operating the cloud computing system to notify the cloud storage system via the PCIe architecture that the command buffer has been written to the computer memory of the cloud storage system; operating the cloud storage system to read the command buffer from the computer memory of the cloud storage system; operating the cloud storage system to process the command buffer to read requested data from a storage device accessible to the cloud storage system; operating the cloud storage system to write the requested data to the computer memory of the cloud computing system via the PCIe architecture; operating the cloud storage system to notify the cloud computing system via the PCIe architecture that the requested data is in the computer memory of the cloud computing system; as well as operating the cloud computing system to read the requested data from the computer memory of the cloud computing system, Wherein, the PCIe architecture includes a PCIe switch connected to a PCIe interface of the cloud storage system and to a PCIe interface of the cloud computing system, the cloud computing system has a client memory mapping that stores memory addresses assigned to components of the cloud computing system, and the PCIe switch is configured to convert the memory addresses stored in the client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch into access to a given memory address in the computer memory of the cloud storage system.

51. The method of claim 50, wherein operating the cloud computing system to notify the cloud storage system that the command buffer has been written to the computer memory of the cloud storage system comprises operating the cloud computing system to write to a doorbell register exposed through the PCIe switch.

52. A method as claimed in claim 51, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud storage system, which in turn causes the cloud storage system to handle the interrupt, the interrupt instructing the cloud storage system to read a message from a specified computer memory location, the message instructing the cloud storage system to read the command buffer from the computer memory of the cloud storage system.

53. The method of claim 52, wherein the designated computer memory location is in the computer memory of the cloud storage system.

54. The method of claim 52, wherein the designated computer memory location is in the computer memory of the cloud computing system.

55. The method of claim 50, wherein operating the cloud storage system to notify the cloud computing system that the requested data is in the computer memory of the cloud computing system comprises operating the cloud storage system to write to a doorbell register exposed through the PCIe switch.

56. A method as claimed in claim 55, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud computing system, which in turn causes the cloud computing system to handle the interrupt, the interrupt instructing the cloud computing system to read a message from a specified computer memory location, the message notifying the cloud computing system where the requested data is stored in the computer memory of the cloud computing system.

57. The method of claim 56, wherein the designated computer memory location is in the computer memory of the cloud storage system.

58. The method of claim 56, wherein the designated computer memory location is in the computer memory of the cloud computing system.

59. The method of claim 50, wherein the PCIe switch comprises a PCIe non-transparent bridge NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, or wherein the cloud storage system comprises an input / output memory management unit (IOMMU) and the PCIe interface of the cloud storage system is connected to the IOMMU, or The PCIe switch includes the PCIe NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, and the cloud storage system includes the IOMMU and the PCIe interface of the cloud storage system is connected to the IOMMU.

60. A method for requesting and transmitting data in a cloud gaming system, the method comprising: A cloud gaming system is provided, the cloud gaming system comprising a cloud storage system and a cloud computing system connected to each other via a peripheral component interconnect express (PCIe) architecture, the cloud storage system comprising a computer memory and a first root complex, the cloud computing system comprising a computer memory and a second root complex; operating the cloud computing system to generate a command buffer for a data read operation; operating the cloud computing system to write the command buffer to the computer memory of the cloud computing system; operating the cloud computing system to notify the cloud storage system via the PCIe architecture that the command buffer has been written to the computer memory of the cloud computing system; operating the cloud storage system to read the command buffer from the computer memory of the cloud computing system via the PCIe architecture; operating the cloud storage system to process the command buffer to read requested data from a storage device accessible to the cloud storage system; operating the cloud storage system to write the requested data to the computer memory of the cloud computing system via the PCIe architecture; operating the cloud storage system to notify the cloud computing system via the PCIe architecture that the requested data is in the computer memory of the cloud computing system; as well as operating the cloud computing system to read the requested data from the computer memory of the cloud computing system, Wherein, the PCIe architecture includes a PCIe switch connected to a PCIe interface of the cloud storage system and to a PCIe interface of the cloud computing system, the cloud computing system has a client memory mapping that stores memory addresses assigned to components of the cloud computing system, and the PCIe switch is configured to convert the memory addresses stored in the client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch into access to a given memory address in the computer memory of the cloud storage system.

61. The method of claim 60, wherein operating the cloud computing system to notify the cloud storage system that the command buffer has been written to the computer memory of the cloud computing system comprises operating the cloud computing system to write to a doorbell register exposed through the PCIe switch.

62. A method as claimed in claim 61, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud storage system, which in turn causes the cloud storage system to handle the interrupt, the interrupt instructing the cloud storage system to read a message from a specified computer memory location, the message instructing the cloud storage system to read the command buffer from the computer memory of the cloud computing system.

63. The method of claim 62, wherein the designated computer memory location is in the computer memory of the cloud storage system.

64. The method of claim 62, wherein the designated computer memory location is in the computer memory of the cloud computing system.

65. The method of claim 60, wherein operating the cloud storage system to notify the cloud computing system that the requested data is in the computer memory of the cloud computing system comprises operating the cloud storage system to write to a doorbell register exposed through the PCIe switch.

66. A method as claimed in claim 65, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud computing system, which in turn causes the cloud computing system to handle the interrupt, the interrupt instructing the cloud computing system to read a message from a specified computer memory location, the message notifying the cloud computing system where the requested data is stored in the computer memory of the cloud computing system.

67. The method of claim 66, wherein the designated computer memory location is in the computer memory of the cloud storage system.

68. The method of claim 66, wherein the designated computer memory location is in the computer memory of the cloud computing system.

69. The method of claim 60, wherein the PCIe switch comprises a PCIe non-transparent bridge NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, or wherein the cloud storage system comprises an input / output memory management unit (IOMMU) and the PCIe interface of the cloud storage system is connected to the IOMMU, or The PCIe switch includes the PCIe NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, and the cloud storage system includes the IOMMU and the PCIe interface of the cloud storage system is connected to the IOMMU.

70. A method for requesting and transmitting data in a cloud gaming system, the method comprising: A cloud gaming system is provided, the cloud gaming system comprising a cloud storage system and a cloud computing system connected to each other via a peripheral component interconnect express (PCIe) architecture, the cloud storage system comprising a computer memory and a first root complex, the cloud computing system comprising a computer memory and a second root complex; operating the cloud computing system to generate a command buffer for a data read operation; operating the cloud computing system to write the command buffer to the computer memory of the cloud computing system; operating the cloud computing system to notify the cloud storage system via the PCIe architecture that the command buffer has been written to the computer memory of the cloud computing system; operating the cloud storage system to read the command buffer from the computer memory of the cloud computing system via the PCIe architecture; operating the cloud storage system to process the command buffer to read requested data from a storage device accessible to the cloud storage system and to write the requested data to the computer memory of the cloud storage system; operating the cloud storage system to notify the cloud computing system via the PCIe architecture that the requested data is in the computer memory of the cloud storage system; as well as operating the cloud computing system to read the requested data from the computer memory of the cloud storage system via the PCIe architecture, Wherein, the PCIe architecture includes a PCIe switch connected to a PCIe interface of the cloud storage system and to a PCIe interface of the cloud computing system, the cloud computing system has a client memory mapping that stores memory addresses assigned to components of the cloud computing system, and the PCIe switch is configured to convert the memory addresses stored in the client memory mapping into memory addresses in the computer memory of the cloud storage system, so that access to a given memory address in the client memory mapping is converted by the PCIe switch into access to a given memory address in the computer memory of the cloud storage system.

71. A method as claimed in claim 70, wherein operating the cloud computing system to notify the cloud storage system that the command buffer has been written to the computer memory of the cloud computing system includes operating the cloud computing system to write to a doorbell register exposed through the PCIe switch.

72. A method as claimed in claim 71, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud storage system, which in turn causes the cloud storage system to handle the interrupt, the interrupt instructing the cloud storage system to read a message from a specified computer memory location, the message instructing the cloud storage system to read the command buffer from the computer memory of the cloud computing system.

73. The method of claim 72, wherein the designated computer memory location is in the computer memory of the cloud storage system.

74. The method of claim 72, wherein the designated computer memory location is in the computer memory of the cloud computing system.

75. The method of claim 70, wherein operating the cloud storage system to notify the cloud computing system that the requested data is in the computer memory of the cloud storage system comprises operating the cloud storage system to write to a doorbell register exposed through the PCIe switch.

76. A method as claimed in claim 75, wherein writing to the doorbell register causes an interrupt to be triggered on the cloud computing system, which in turn causes the cloud computing system to handle the interrupt, the interrupt instructing the cloud computing system to read a message from a specified computer memory location, the message notifying the cloud computing system where the requested data is stored in the computer memory of the cloud storage system.

77. The method of claim 76, wherein the designated computer memory location is in the computer memory of the cloud storage system.

78. The method of claim 76, wherein the designated computer memory location is in the computer memory of the cloud computing system.

79. The method of claim 70, wherein the PCIe switch comprises a PCIe non-transparent bridge NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, or wherein the cloud storage system comprises an input / output memory management unit (IOMMU) and the PCIe interface of the cloud storage system is connected to the IOMMU, or The PCIe switch includes the PCIe NTB and the PCIe interface of the cloud computing system is connected to the PCIe switch through the PCIe NTB, and the cloud storage system includes the IOMMU and the PCIe interface of the cloud storage system is connected to the IOMMU.

Images