Computer system

Abstract

Provided is a computer system comprising: a CPU (10a); an OCS (2) that selectively connects PCIe devices (3-1 to 3-4) to PCIe lanes of the CPU (10a); an OCS controller (4); and a management control system (5) that, when the effective bandwidth of the PCIe lanes used by the PCIe devices (3-1 to 3-3) which are connected to the CPU (10a) is less than or equal to a threshold value, instructs the CPU (10a) to reduce the number of PCIe lanes used by said devices and instructs the OCS controller (4) to newly connect, to the PCIe lanes that have been made available, the PCIe devices (3-4) that are not connected to the CPU (10a).

Description

Computer Systems

[0001] The present invention relates to computer systems.

[0002] In recent years, the use of graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and other devices that can perform calculations more efficiently than central processing units (CPUs) has increased, leading to the use of many peripheral component interconnect express (PCIe) devices. In conventional server-centric configurations, PCIe devices had to be installed on a board that housed the CPU, and only 10 or fewer devices could be used.

[0003] For this reason, research into disaggregated computing has progressed in recent years, allowing CPU hosts to be connected to PCIe device pools and used as if PCIe devices were inside the server. Furthermore, while there is a limit to the distance that cables can travel when transmitting and receiving electrical signals, the introduction of optical communications has extended the connectable distance, increasing the possibility of connecting to more devices.

[0004] However, there is a limit to the number of PCIe lanes that a CPU or PCIe switch can have, and there is also a limit to the number of ports on an optical path switch, so it has been difficult to actually increase the number of PCIe device connections. Also, while servers generally use 16 PCIe lanes, the bandwidth of these 16 lanes is rarely used to its full potential, resulting in ineffective use of the PCIe bandwidth.

[0005] A lane is a data transfer path, and more specifically, a bundle of data transfer paths for transmission and reception. In the following description, the number of lanes is represented by "x number." For example, x16 indicates that the number of lanes is 16. In PCIe, the number of lanes can be x1, x2, x4, x8, x12, x16, etc.

[0006] 17 is a block diagram showing a schematic configuration of disaggregated computing. A CPU board 1 is equipped with a CPU 10, a memory 11, a photoelectric conversion device 12, and a NIC (Network Interface Card) 13. The CPU 10 has three channels of PCIe buses with 16 lanes each, and is connected to three PCIe devices 3-1 to 3-3 via the photoelectric conversion device 12 and an OCS (Optical Circuit Switch) 2. The photoelectric conversion device 12 converts electrical signals from the CPU 10 to the PCIe devices 3-1 to 3-3 into optical signals and sends them to the OCS 2, and converts optical signals received from the PCIe devices 3-1 to 3-3 via the OCS 2 into electrical signals and sends them to the CPU 10.

[0007] The PCIe devices 3-1 to 3-4 are equipped with GPUs 30-1 to 30-4 and photoelectric conversion devices 31-1 to 31-4, respectively. The photoelectric conversion devices 31-1 to 31-4 convert electrical signals from the GPUs 30-1 to 30-4 to the CPU 10 into optical signals and send them to the OCS 2, and convert optical signals received from the CPU 10 via the OCS 2 into electrical signals and send them to the GPUs 30-1 to 30-4. The CPU 10 manages the OCS controller 4 via the NIC 13 and controls the connection between itself and the PCIe devices 3-1 to 3-4.

[0008] 17, the CPU 10 is connected to three PCIe devices 3-1 to 3-3, all of which are connected via ×16 lanes, so even if the GPUs 30-1 to 30-3 are not using the bandwidth of ×16 lanes, another PCIe device 3-4 cannot be connected. For example, even if it is desired to increase the load during the execution of an application and there is spare PCIe bandwidth, the PCIe device 3-4 cannot be added.

[0009] A technology for changing the number of lanes of a PCIe bus is disclosed in Patent Document 1. In the technology disclosed in Patent Document 1, when connecting an information processing device to a device using an adapter inserted into a slot of the information processing device, the number of lanes of the slot of the information processing device is changed depending on the type of adapter. However, with the technology disclosed in Patent Document 1, the power supply to the remaining lanes not assigned to a device out of the ×16 lanes of the slot is cut off, making it impossible to connect other devices to the remaining lanes, and therefore the PCIe bandwidth cannot be effectively utilized.

[0010] Japanese Patent Application Laid-Open No. 2018-022267

[0011] The present invention has been made to solve the above-mentioned problems, and aims to provide a computer system that can utilize a new PCIe device when there is spare PCIe bandwidth, and can effectively utilize the PCIe bandwidth.

[0012] The computer system of the present invention includes a plurality of PCIe devices, a CPU configured to execute processing according to a program, a switch configured to selectively connect the plurality of PCIe devices to PCIe lanes of the CPU, a first control unit configured to be able to change the number of PCIe lanes on the CPU side, a second control unit configured to control the switch, and a management control system configured to instruct the first control unit to reduce the number of PCIe lanes used by a PCIe device connected to the CPU when the effective bandwidth of the PCIe lane used by the device is equal to or less than a first threshold, and to instruct the second control unit to newly connect a PCIe device not connected to the CPU to the PCIe lane made available by the reduction, wherein the CPU assigns processing to the newly connected PCIe device.

[0013] According to the present invention, the effective bandwidth of PCIe lanes used by PCIe devices connected to a CPU is monitored by a management control system, and when the effective bandwidth is equal to or less than a first threshold value, the number of PCIe lanes used by the PCIe devices is reduced, and a PCIe device not connected to the CPU is newly connected to the PCIe lane that becomes free by the reduction. This makes it possible to efficiently add new PCIe devices without increasing the number of switch ports, thereby increasing device utilization efficiency and improving power consumption efficiency and overall computing performance.

[0014] FIG. 1 is a block diagram showing the configuration of a computer system according to a first embodiment of the present invention. FIG. 2 is a flowchart illustrating the operation of the computer system according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating an example in which the number of lanes on the CPU side is changed in the first embodiment of the present invention. FIG. 4 is a block diagram showing the configuration of a computer system according to a second embodiment of the present invention. FIG. 5 is a block diagram showing the configuration of a computer system according to a third embodiment of the present invention. FIG. 6 is a block diagram showing another configuration of a computer system according to the third embodiment of the present invention. FIG. 7 is a flowchart illustrating the operation of a computer system according to a fourth embodiment of the present invention. FIG. 8 is a block diagram showing the configuration of a computer system according to a fifth embodiment of the present invention. FIG. 9 is a diagram illustrating the relationship between the wavelength multiplicity, the number of PCIe devices connected to an OCS, the number of connection ports between each PCIe device and the OCS, and the number of connection ports between each wavelength selective switch and the OCS. FIG. 10 is a diagram illustrating lane number switching control using a wavelength selective switch. FIG. 11 is a diagram illustrating lane number switching control using a wavelength selective switch. FIG. 12 is a diagram illustrating an example of how a CPU and each PCIe device are connected when the wavelength multiplicity is 4. Fig. 13 is a diagram showing an example of how a CPU and each PCIe device are connected when the wavelength multiplexing degree is 4. Fig. 14 is a block diagram showing the configuration of a computer system according to a sixth embodiment of the present invention. Fig. 15 is a block diagram showing another configuration of a computer system according to the sixth embodiment of the present invention. Fig. 16 is a block diagram showing the configuration of a computer system according to a seventh embodiment of the present invention. Fig. 17 is a block diagram showing a schematic configuration of conventional disaggregated computing.

[0015] [First embodiment] An embodiment of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing the configuration of a computer system according to a first embodiment of the present invention, and the same components as those in Figure 17 are assigned the same reference numerals. The computer system is composed of a CPU board 1a, an OCS 2, PCIe devices 3-1 to 3-4, an OCS controller 4, and a management control system 5.

[0016] The CPU 10a mounted on the CPU board 1a includes a PCIe controller 100 (first control unit) for connecting to the PCIe devices 3-1 to 3-4 via a PCIe bus. The OCS 2 is an optical switch capable of switching the connection between the CPU board 1a and the PCIe devices 3-1 to 3-4 in response to a control signal from the OCS controller 4.

[0017] The OCS controller 4 (second control unit) includes a CPU 40, a memory 41, and a NIC 42. The management control system 5 includes a CPU 50, a memory 51, and a NIC 52. The CPU 10a can communicate with the CPU 50 of the management control system 5 via the NIC 13 and the NIC 52 of the management control system 5. The CPU 50 of the management control system 5 can communicate with the CPU 40 of the OCS controller 4 via the NIC 52 and the NIC 42 of the OCS controller 4.

[0018] The CPUs 10a, 40, and 50 execute the following processes according to the programs stored in the memories 11, 41, and 51, respectively. In the example of Fig. 1, the management control system 5 is provided separately from the CPU board 1a, but the CPU board 1a and the management control system 5 may be integrated, and the functions of the management control system 5 may be realized by the CPU 10a.

[0019] 2 is a flowchart illustrating the operation of the computer system of this embodiment. Here, a case where an additional device is used during execution of an application will be described. When adding processing by a PCIe device (YES in step S100 in FIG. 2), the CPU 10a requests the management control system 5 to add processing by the PCIe device. In response to this request, the CPU 50 of the management control system 5 determines whether or not there is an unused PCIe device among the PCIe devices connected to the CPU 10a via the PCIe bus (step S101 in FIG. 2).

[0020] If there is an unused PCIe device among the PCIe devices connected to the CPU 10a (YES in step S101), the CPU 50 instructs the CPU 10a to cause the unused PCIe device to execute the process to be added. In response to this instruction, the CPU 10a causes the unused PCIe device to execute the process (step S102 in FIG. 2). Steps S100 to S102 are repeatedly executed until all processes are completed (YES in step S103 in FIG. 2).

[0021] Next, if there is no unused PCIe device among the PCIe devices connected to CPU 10a via the PCIe bus (NO in step S101), CPU 50 determines whether there is an extra PCIe device not connected to CPU 10a among the PCIe devices connected to OCS2 (step S104 in FIG. 2).

[0022] If there are no spare PCIe devices (NO in step S104), the CPU 50 waits until a spare PCIe device becomes available among the PCIe devices in use (step S105 in FIG. 2). Next, if there are spare PCIe devices (YES in step S104), the CPU 50 determines whether there are spare PCIe lanes on the CPU 10a side (step S106 in FIG. 2).

[0023] If there are spare PCIe lanes on the CPU 10a side (YES in step S106), the CPU 50 instructs the OCS controller 4 to connect the spare PCIe devices confirmed in step S104 to the spare PCIe lanes on the CPU 10a side. In response to this instruction, the CPU 40 of the OCS controller 4 outputs a control signal to the OCS 2 to connect the PCIe devices instructed by the CPU 50 to the spare PCIe lanes on the CPU 10a side instructed by the CPU 50. In this way, the PCIe devices that were not connected to the CPU 10a are connected to the CPU 10a via the OCS 2 and the spare PCIe lanes on the CPU 10a side (step S107 in FIG. 2).

[0024] If there are no unused PCIe lanes on the CPU 10a side (NO in step S106), the CPU 50 detects the effective bandwidth of the PCIe device in use (step S108 in FIG. 2). The process of step S108 can be realized by inquiring about the effective bandwidth (sending a command) to the CPU 10a.

[0025] If there is a PCIe device whose effective bandwidth is equal to or less than a predetermined threshold value TH1 among the PCIe devices in use, the CPU 50 determines that there is sufficient bandwidth (YES in step S109 in FIG. 2 ) and instructs the CPU 10a to reduce the number of PCIe lanes used by the PCIe device whose effective bandwidth is equal to or less than the threshold value TH1. In response to this instruction, the CPU 10a changes the number of PCIe lanes being used by the PCIe device instructed by the CPU 50 from ×16 to, for example, ×8 (step S110 in FIG. 2 ). This process of reducing the number of PCIe lanes can be implemented by the PCIe controller 100.

[0026] The threshold value TH1 may be set to a value smaller than the maximum bandwidth of the PCIe lane, for example, half the maximum bandwidth of the PCIe lane. However, in the present invention, the setting of the threshold value TH1 is not limited to this example.

[0027] Also, although an example has been described in which the number of PCIe lanes in use is changed from ×16 to ×8, it may be changed to any of ×1, ×2, ×4, ×8, and ×12. To determine what number of lanes to change to, for example, a change rule may be set in advance in the memory 51 of the management control system 5, and the CPU 50 may instruct the new number of lanes according to this change rule.

[0028] After the processing of step S110 creates free lanes on the CPU 10a side, the CPU 50 commands the OCS controller 4 to connect the remaining PCIe devices confirmed in step S104 to the free PCIe lanes. In response to this command, the CPU 40 of the OCS controller 4 outputs a control signal to the OCS 2 to connect the PCIe devices instructed by the CPU 50 to the PCIe lanes on the CPU 10a side instructed by the CPU 50. In this way, the PCIe devices that were not connected to the CPU 10a are connected to the CPU 10a via the OCS 2 and the PCIe lanes on the CPU 10a side (step S107).

[0029] An example in which the number of lanes on the CPU 10a side has been changed is shown in FIG. 3. In the example of FIG. 1, the CPU 10a is connected to PCIe devices 3-1 to 3-3. Here, if the effective bandwidth used by the PCIe device 3-3 is equal to or less than the threshold value TH1, the number of PCIe lanes on the CPU 10a side connected to the PCIe device 3-3 is changed from ×16 to ×8. This creates free lanes worth ×8. By connecting the PCIe device 3-4 to this free lane, the CPU 10a can cause the PCIe device 3-4 to execute a new process.

[0030] If there is no PCIe device whose effective bandwidth is equal to or less than the threshold value TH1 among the PCIe devices in use (NO in step S109), the CPU 50 waits until the processing of the PCIe device in use is completed (step S111 in FIG. 2).

[0031] As described above, in this embodiment, the management control system 5 monitors the bandwidth of PCIe lanes, and reduces the number of lanes for lanes with redundant bandwidth, thereby enabling new PCIe devices to be added efficiently without increasing the number of ports on the OCS2, thereby increasing device utilization efficiency and improving power consumption efficiency and overall computing performance.

[0032] Second Embodiment In the first embodiment, the number of PCIe lanes is changed by the CPU 10a, but a PCIe bridge chip that can change the number of lanes in the same way may be used.

[0033] 4 is a block diagram showing the configuration of a computer system according to this embodiment, in which the same components as those in Fig. 1 and Fig. 17 are assigned the same reference numerals. In this embodiment, a PCIe bridge chip 14 (first control unit) is provided between a CPU 10b and a photoelectric conversion device 12 in a CPU board 1b.

[0034] Since the processing flow of the computer system is the same as that of the first embodiment, the operation of this embodiment will be described with reference to Fig. 2. When the CPU 10b receives a command from the CPU 50 of the management control system 5 in step S110, the CPU 10b commands the PCIe bridge chip 14 to reduce the number of PCIe lanes used by PCIe devices whose effective bandwidth is equal to or less than the threshold value TH1. In response to this command, the PCIe bridge chip 14 changes the number of PCIe lanes used by the PCIe devices instructed by the CPU 10b (CPU 50) from x16 to, for example, x8.

[0035] Other processing is the same as in Example 1. Thus, in this example, the same effects as in Example 1 can be obtained.

[0036] Third Embodiment In the first and second embodiments, an OCS, which is an optical switch, is used to connect the CPU board and the PCIe device, but an electric switch may be used instead of the OCS.

[0037] 5 is a block diagram showing the configuration of a computer system according to this embodiment, with the same components as those in FIGS. 1, 4, and 17 being assigned the same reference numerals. In this embodiment, an electric switch 6 is used instead of an OCS, eliminating the need for the photoelectric conversion device 12 in the CPU board 1c. Furthermore, PCIe controllers 32-1 to 32-4 are provided in the PCIe devices 3c-1 to 3c-4 instead of the photoelectric conversion device.

[0038] Furthermore, instead of the OCS controller 4, a controller 7 (second control unit) is provided that controls the electric switch 6. The controller 7 includes a CPU 70, a memory 71, and a NIC 72. The CPU 70 executes the following processes in accordance with a program stored in the memory 71.

[0039] Since the processing flow of the computer system is the same as that of the first embodiment, the operation of this embodiment will be described with reference to Fig. 2. If there is no unused PCIe device among the PCIe devices connected to the CPU 10a via the PCIe bus (NO in step S101 in Fig. 2), the CPU 50 of the management control system 5 determines whether there is an extra PCIe device not connected to the CPU 10a among the PCIe devices connected to the electric switch 6 (step S104 in Fig. 2).

[0040] If there are spare PCIe devices (YES in step S104), the CPU 50 determines whether there are spare PCIe lanes on the CPU 10a side (step S106 in FIG. 2 ). If there are spare PCIe lanes on the CPU 10a side (YES in step S106), the CPU 50 instructs the controller 7 to connect the spare PCIe devices confirmed in step S104 to the spare PCIe lanes on the CPU 10a side. In response to this instruction, the CPU 70 of the controller 7 outputs a control signal to the electric switch 6 to connect the PCIe devices instructed by the CPU 50 to the spare PCIe lanes on the CPU 10a side instructed by the CPU 50. In this way, the PCIe devices that were not connected to the CPU 10a are connected to the CPU 10a via the electric switch 6 and the spare PCIe lanes on the CPU 10a side (step S107 in FIG. 2 ).

[0041] After the processing of step S110 creates free lanes on the CPU 10a side, the CPU 50 commands the controller 7 to connect the remaining PCIe devices confirmed in step S104 to the free PCIe lanes. In response to this command, the CPU 70 of the controller 7 outputs a control signal to the electric switch 6 to connect the PCIe devices instructed by the CPU 50 to the PCIe lanes on the CPU 10a side instructed by the CPU 50. In this way, the PCIe devices that were not connected to the CPU 10a are connected to the CPU 10a via the electric switch 6 and the PCIe lanes on the CPU 10a side (step S107).

[0042] Other processing is the same as in the first embodiment. Thus, in this embodiment, when an electric switch is used instead of the OCS, the same effects as in the first embodiment can be obtained. In FIG. 5, an electric switch is applied to the first embodiment, but it may also be applied to the second embodiment. The configuration in this case is shown in FIG. 6.

[0043] [Fourth embodiment] In the first to third embodiments, the effective bandwidth of the PCIe lane in use is monitored, but the utilization rate of the PCIe device may also be monitored in addition to the effective bandwidth. In this embodiment, the configuration of the computer system is the same as in the first to fourth embodiments, and therefore will be described using the reference numerals in Figures 1 and 4 to 6.

[0044] 7 is a flowchart illustrating the operation of the computer system of this embodiment. The CPU 50 of the management control system 5 detects the usage rate of PCIe devices in use (step S200 in FIG. 7). If the usage rate of the PCIe devices is equal to or greater than a predetermined threshold value TH2 (YES in step S201 in FIG. 7), the CPU 50 determines whether there is an unused PCIe device among the PCIe devices connected to the CPUs 10a and 10b via the PCIe bus (step S202 in FIG. 7).

[0045] The threshold value TH2 may be set to, for example, a utilization rate of 80%. This threshold value TH2 may be determined taking into consideration, for example, the increase in power consumption of the PCIe device that accompanies an increase in utilization rate.

[0046] If there is an unused PCIe device among the PCIe devices connected to the CPUs 10a and 10b (YES in step S202), the CPU 50 instructs the CPUs 10a and 10b to cause the unused PCIe device to execute part of the processing of the PCIe device whose usage rate is equal to or greater than the threshold value TH2. In response to this instruction, the CPUs 10a and 10b cause the unused PCIe device to execute part of the processing of the instructed PCIe device (step S203 in FIG. 7).

[0047] The percentage of processing to be allocated to unused PCIe devices can be determined, for example, by setting an allocation rule in advance in the memory 51 of the management control system 5, and the CPU 50 can instruct the percentage of unused PCIe devices to execute processing according to this allocation rule.

[0048] Next, if there is no unused PCIe device among the PCIe devices connected to the CPUs 10a and 10b via the PCIe bus (NO in step S202), the CPU 50 determines whether there is an extra PCIe device that is not connected to the CPUs 10a and 10b among the PCIe devices connected to the OCS 2 or the electrical switch 6 (step S205 in FIG. 7).

[0049] If there are no spare PCIe devices (NO in step S205), the CPU 50 waits until a spare PCIe device becomes available among the PCIe devices in use (step S206 in FIG. 7). Next, if there are spare PCIe devices (YES in step S205), the CPU 50 determines whether there are spare PCIe lanes on the CPU 10a, 10b side (step S207 in FIG. 7).

[0050] If there are spare PCIe lanes on the CPU 10a, 10b side (YES in step S207), the CPU 50 instructs the OCS controller 4 or the controller 7 to connect the spare PCIe devices confirmed in step S205 to the spare PCIe lanes on the CPU 10a, 10b side. In response to this instruction, the CPU 40 of the OCS controller 4 or the CPU 70 of the controller 7 outputs a control signal to the OCS 2 or the electric switch 6 to connect the PCIe devices instructed by the CPU 50 to the spare PCIe lanes on the CPU 10a, 10b side instructed by the CPU 50. In this way, the PCIe devices that were not connected to the CPUs 10a, 10b are connected to the CPUs 10a, 10b via the OCS 2 or the electric switch 6 (step S208 in FIG. 7 ).

[0051] If there are no unused PCIe lanes on the CPU 10a, 10b side (NO in step S207), the CPU 50 detects the effective bandwidth of the PCIe devices in use (step S209 in FIG. 7). If there is a PCIe device whose effective bandwidth is equal to or less than the threshold value TH1 among the PCIe devices in use (YES in step S210 in FIG. 7), the CPU 50 instructs the CPUs 10a, 10b to reduce the number of PCIe lanes used by the PCIe devices whose effective bandwidth is equal to or less than the threshold value TH1. In response to this instruction, the CPUs 10a, 10b change the number of PCIe lanes in use by the PCIe devices instructed by the CPU 50 from ×16 to, for example, ×8 (step S211 in FIG. 7). This process of reducing the number of PCIe lanes can be implemented by the PCIe controller 100 or the PCIe bridge chip 14.

[0052] After the processing of step S211 creates free lanes on the CPU 10a, 10b side, the CPU 50 commands the OCS controller 4 or the controller 7 to connect the remaining PCIe devices confirmed in step S205 to the free PCIe lanes. In response to this command, the CPU 40 of the OCS controller 4 or the CPU 70 of the controller 7 outputs a control signal to the OCS 2 or the electric switch 6 to connect the PCIe devices instructed by the CPU 50 to the PCIe lanes on the CPU 10a, 10b side instructed by the CPU 50. In this way, the PCIe devices that were not connected to the CPUs 10a, 10b are connected to the CPUs 10a, 10b via the OCS 2 or the electric switch 6 (step S208).

[0053] By the above processing, in this embodiment, it is possible to monitor the utilization rate of PCIe devices. When the utilization rate of PCIe devices is high, there are cases where power consumption is lower when multiple PCIe devices are used. This embodiment is effective when the utilization rate of PCIe devices is high and power consumption is increasing.

[0054] In the first to fourth embodiments, a management control system 5 is provided separately from the CPU board, but the functions of the management control system 5 may be realized by the CPUs 10a, 10b, and 10b. Also, in the first to fourth embodiments, a single CPU board is used, but multiple CPU boards may be provided. In the case where multiple CPU boards are used, the processing described in the first to fourth embodiments may be executed for each CPU on each CPU board.

[0055] [Fifth Embodiment] Figure 8 is a block diagram showing the configuration of a computer system according to a fifth embodiment of the present invention, with the same components as in Figure 4 being assigned the same reference numerals. This embodiment is similar to the second embodiment in that a plurality of wavelength selective switches (WSSs) 8 are inserted between the photoelectric conversion device 12 in the CPU board 1b and the OCS 2. In Figure 8, the number of connection ports between each of the PCIe devices 3-1 to 3-N (N is an integer of 2 or greater) and the OCS 2 is represented as A, and the number of connection ports between each of the WSSs 8 and the OCS 2 is represented as B.

[0056] The advantage of using OCS2 is that by applying wavelength multiplexing, a feature of optical technology, to a switch with a limited number of ports, it is possible to connect more PCIe devices 3-1 to 3-N than with an electrical switch with the same number of ports. Also, WSS8 is used to enable efficient access by the CPU 10b to the multiple PCIe devices 3-1 to 3-N.

[0057] 2 and 7. The CPU 40 of the OCS controller 4 can control the WSS 8 via the NIC 42, and can control the WSS 8 so that a PCIe device instructed by the CPU 50 of the management control system 5 is connected to the CPU 10b. Fig. 9 shows the relationship between the wavelength multiplexing degree P, the number N of PCIe devices 3-1 to 3-N connected to the OCS 2, the number A of connection ports between each of the PCIe devices 3-1 to 3-N and the OCS 2, and the number B of connection ports between each WSS 8 and the OCS 2.

[0058] In this embodiment, it is assumed that the number of PCIe devices N increases in proportion to the wavelength multiplexing degree P, and the number of PCIe devices N = 5 × P. That is, in this case, in the case of the electrical switch 6 or the OCS 2 to which wavelength multiplexing is not applied, it is assumed that five PCIe devices can be connected per switch. In FIG. 8, this is related to the fact that the number of PCIe × 16 lanes on the photoelectric conversion device 12 side of the PCIe bridge chip 14 is 5.

[0059] Next, specific numerical values ​​of the variables P, N, A, and B are shown. For example, when the wavelength multiplicity P=4 and the number of PCIe devices N=20, assuming 16 PCIe lanes, the wavelength multiplicity P is 4, so the signal input port of one PCIe device is connected to the OCS2 via four optical fibers. Similarly, the signal output port of one PCIe device is connected to the OCS2 via four optical fibers. Therefore, when the wavelength multiplicity P=4, the number of connection ports between each of the PCIe devices 3-1 to 3-N and the OCS2 is A=4. In this embodiment, A is given by A=16 / P.

[0060] As described above, the number of PCIe x16 lanes on the photoelectric conversion device 12 side of the PCIe bridge chip 14 is fixed at five ports, and the number of lanes cannot be increased as the number of PCIe devices increases. In order for the CPU 10b to access the PCIe devices 3-1 to 3-N connected to the OCS2, the relationship "Number of PCIe devices N" ≥ "Total number of ports on the CPU side of the OCS2" must be satisfied. In this embodiment, since the number of PCIe x16 lanes on the photoelectric conversion device 12 side of the PCIe bridge chip 14 is fixed at five ports, N = 5 x B. To satisfy this relationship, a WSS 8 is inserted, enabling the CPU 10b to access each of the PCIe devices 3-1 to 3-N.

[0061] The lane number switching control using the WSS8 will be explained using Figures 10(A) to 10(C) and 11(A) to 11(E). Note that when the wavelength multiplicity P=4, A=B, so the WSS8 can be bypassed and the WSS8 is not required, so the case of wavelength multiplicity P=4 is not shown.

[0062] When the wavelength multiplexing degree P=8, the photoelectric conversion device 12 is connected to one WSS 8 via two optical fibers. The photoelectric conversion device 12 converts the PCIe×16 lane electrical signal from the PCIe bridge chip 14 into two 8-wave multiplexed optical signals and sends them to the two optical fibers. The photoelectric conversion device 12 also converts each of the 8-wave multiplexed optical signals from the two optical fibers into a PCIe×8 lane electrical signal and sends them to the PCIe bridge chip 14.

[0063] 10A shows a case where the wavelength multiplexing degree P is 8, and the WSS 8 is switched and controlled so that an 8-wavelength multiplexed optical signal from one port on the photoelectric conversion device 12 side of the WSS 8 is demultiplexed into two wavelengths and output to four ports on the OCS2 side of the WSS 8, and an 8-wavelength multiplexed optical signal from the other port on the photoelectric conversion device 12 side is demultiplexed into two wavelengths and output to four other ports on the OCS2 side of the WSS 8. Each output port on the OCS2 side is two-wavelength multiplexed. The example in FIG. 10A corresponds to ×2.

[0064] FIG. 10B illustrates a case where the wavelength multiplexing level P is 8, and the WSS 8 is switched so that an 8-wavelength multiplexed optical signal from one port on the optoelectronic conversion device 12 side of the WSS 8 is demultiplexed into four wavelengths and output to two ports on the OCS 2 side of the WSS 8, and an 8-wavelength multiplexed optical signal from the other port on the optoelectronic conversion device 12 side is demultiplexed into four wavelengths and output to two other ports on the OCS 2 side of the WSS 8. Each output port on the OCS 2 side is four-wavelength multiplexed. The example of FIG. 10B corresponds to ×4. In the example of FIG. 10B, four ports on the OCS 2 side are unused, but the selection of the four output ports on the OCS 2 side may be an example other than that of FIG. 10B.

[0065] FIG. 10C shows a case where the wavelength multiplexing degree P is 8, and the WSS 8 is switched so that an 8-wavelength multiplexed optical signal from one port on the optoelectronic conversion device 12 side of the WSS 8 is output to one port on the OCS 2 side of the WSS 8, and an 8-wavelength multiplexed optical signal from the other port on the optoelectronic conversion device 12 side is output to another port on the OCS 2 side. Each output port on the OCS 2 side is 8-wave multiplexed. The example of FIG. 10C corresponds to ×8. In the example of FIG. 10C, six ports on the OCS 2 side are unused, but the selection of two output ports on the OCS 2 side may be an example other than that of FIG. 10C.

[0066] When the wavelength multiplexing degree P=16, the photoelectric conversion device 12 is connected to one WSS 8 via one optical fiber. The photoelectric conversion device 12 converts a PCIe×16 lane electrical signal from the PCIe bridge chip 14 into a 16-wave multiplexed optical signal and sends it out to one optical fiber. The photoelectric conversion device 12 also converts a 16-wave multiplexed optical signal from one optical fiber into a PCIe×16 lane electrical signal and sends it out to the PCIe bridge chip 14.

[0067] 11A shows a case where the wavelength multiplexing degree P is 16, and the WSS 8 is switched and controlled so that a 16-wavelength multiplexed optical signal from a port on the optoelectronic conversion device 12 side of the WSS 8 is demultiplexed into individual wavelengths and output to 16 ports on the OCS 2 side of the WSS 8. No wavelength multiplexing occurs at each output port on the OCS 2 side. The example in FIG. 11A corresponds to ×1.

[0068] FIG. 11B shows a case where the wavelength multiplexing degree P is 16, and the WSS 8 is switched and controlled so that a 16-wavelength multiplexed optical signal from a port on the optoelectronic conversion device 12 side of the WSS 8 is demultiplexed into two wavelengths and output to eight ports on the OCS 2 side of the WSS 8. Each output port on the OCS 2 side is two-wavelength multiplexed. The example of FIG. 11B corresponds to ×2. In the example of FIG. 11B, the eight ports on the OCS 2 side are unused, but the selection of the eight output ports on the OCS 2 side may be an example other than that of FIG. 11B.

[0069] FIG. 11C shows a case where the wavelength multiplexing degree P is 16, and the WSS 8 is switched and controlled so that a 16-wavelength multiplexed optical signal from a port on the optoelectronic conversion device 12 side of the WSS 8 is demultiplexed into four wavelengths and output to four ports on the OCS 2 side of the WSS 8. Each output port on the OCS 2 side is four-wavelength multiplexed. The example of FIG. 11C corresponds to ×4. In the example of FIG. 11C, 12 ports on the OCS 2 side are unused, but the selection of the four output ports on the OCS 2 side may be an example other than that of FIG. 11C.

[0070] Figure 11(D) shows a case where the wavelength multiplexing degree P is 16, and the WSS 8 is switched and controlled so that a 16-wavelength multiplexed optical signal from a port on the optoelectronic conversion device 12 side of the WSS 8 is demultiplexed into eight wavelengths and output to two ports on the OCS 2 side of the WSS 8. Each output port on the OCS 2 side is eight-wave multiplexed. The example of Figure 11(D) corresponds to x8. In the example of Figure 11(D), 14 ports on the OCS 2 side are unused, but the selection of the two output ports on the OCS 2 side may be an example other than that of Figure 11(D).

[0071] Figure 11(E) shows a case where the wavelength multiplexing degree P is 16, and the WSS 8 is switched and controlled so that a 16-wave multiplexed optical signal from a port on the optoelectronic conversion device 12 side of the WSS 8 is output to one port on the OCS 2 side of the WSS 8. The output port on the OCS 2 side is 16-wave multiplexed. The example of Figure 11(E) corresponds to x16. In the example of Figure 11(E), 15 ports on the OCS 2 side are unused, but the selection of one output port on the OCS 2 side may be an example other than that of Figure 11(E).

[0072] Note that Figures 10(A) to 10(C) and Figures 11(A) to 11(E) show the signal flow from the photoelectric conversion device 12 to the OCS2, but it is clear that by changing the direction of each arrow in Figures 10(A) to 10(C) and Figures 11(A) to 11(E), the signal flow will become from the OCS2 to the photoelectric conversion device 12.

[0073] In the example of Figure 10(A), in the case of a signal flow from the OCS2 toward the photoelectric conversion device 12, the WSS8 is switched so that two-wavelength multiplexed optical signals from four ports on the OCS2 side of the WSS8 are multiplexed and output to one port on the photoelectric conversion device 12 side of the WSS8. In the example of Figure 10(B), in the case of a signal flow from the OCS2 toward the photoelectric conversion device 12, the WSS8 is switched so that four-wavelength multiplexed optical signals from two ports on the OCS2 side of the WSS8 are multiplexed and output to one port on the photoelectric conversion device 12 side of the WSS8. In the example of Figure 10(C), in the case of a signal flow from the OCS2 toward the photoelectric conversion device 12, the WSS8 is switched so that eight-wavelength multiplexed optical signals from one port on the OCS2 side of the WSS8 are multiplexed and output to one port on the photoelectric conversion device 12 side of the WSS8.

[0074] 11A, in the case of a signal flow from the OCS2 toward the photoelectric conversion device 12, the WSS8 is switched so that optical signals from 16 ports on the OCS2 side of the WSS8 are multiplexed and output to a single port on the photoelectric conversion device 12 side of the WSS8. In FIG. 11B, in the case of a signal flow from the OCS2 toward the photoelectric conversion device 12, the WSS8 is switched so that two-wavelength multiplexed optical signals from eight ports on the OCS2 side of the WSS8 are multiplexed and output to a single port on the photoelectric conversion device 12 side of the WSS8. In FIG. 11C, in the case of a signal flow from the OCS2 toward the photoelectric conversion device 12, the WSS8 is switched so that four-wavelength multiplexed optical signals from four ports on the OCS2 side of the WSS8 are multiplexed and output to a single port on the photoelectric conversion device 12 side of the WSS8.

[0075] 11(D), in the case of a signal flow from OCS2 toward the photoelectric conversion device 12, WSS8 is switched so that 8-wave multiplexed optical signals from two ports on the OCS2 side of WSS8 are multiplexed and output to one port on the photoelectric conversion device 12 side of WSS8. In Fig. 11(E), in the case of a signal flow from OCS2 toward the photoelectric conversion device 12, WSS8 is switched so that a 16-wave multiplexed optical signal from one port on the OCS2 side of WSS8 is output to one port on the photoelectric conversion device 12 side of WSS8.

[0076] 12 and 13 show an example of how the CPU 10b is connected to each of the PCIe devices 3-1 to 3-N when the wavelength multiplicity P=4. In the example of FIG. 12, the CPU 10b is connected to the minimum number of five PCIe devices connected to the OCS2 via PCIe ×16 lanes. The lane number switching control is performed while maintaining ×16. Because the wavelength multiplicity P=4, one port of the photoelectric conversion device 12 is connected to the OCS2 via four optical fibers 20. Similarly, one port of each of the photoelectric conversion devices 31-1 to 31-N is connected to the OCS2 via four optical fibers 21. The photoelectric conversion devices 12, 31-1 to 31-N assign four wavelengths to one optical fiber (corresponding to ×4 lanes).

[0077] When the wavelength multiplexing degree P=4, the photoelectric conversion device 12 converts the PCIe×16 lane electrical signal from the PCIe bridge chip 14 into four 4-wave multiplexed optical signals and sends them out to the four optical fibers. The photoelectric conversion device 12 also converts the 4-wave multiplexed optical signals from the four optical fibers into PCIe×4 lane electrical signals, respectively, and sends them out to the PCIe bridge chip 14. The photoelectric conversion devices 31-1 to 31-N of the PCIe devices 3-1 to 3-N convert each of the four 4-wave multiplexed optical signals transmitted by the four optical fibers into a PCIe×4 lane electrical signal. The photoelectric conversion devices 31-1 to 31-N also convert the PCIe×16 lane electrical signals from the GPUs 30-1 to 30-N into four 4-wave multiplexed optical signals and send them out to the four optical fibers.

[0078] 13 shows a case where the CPU 10b can access all PCIe devices 3-1 to 3-N connected to the OCS 2. The OCS 2 is controlled so that one PCIe device is connected to the CPU 10b via one of the four optical fibers 21. As a result, the CPU 10b is connected to all 20 PCIe devices via PCIe×4 lanes.

[0079] This embodiment utilizes wavelength multiplexing, a feature of optical technology, to achieve scalability by increasing the number of PCIe devices connected to the CPU 10b, while also achieving flexibility in terms of CPU 10b's access to each PCIe device through lane number switching control using the WSS 8. According to this embodiment, the management and control system 5 monitors the effective bandwidth of the PCIe lanes used by the PCIe devices connected to the CPU 10b, and when the effective bandwidth is equal to or less than a first threshold, the state is changed, for example, from that shown in FIG. 12 to that shown in FIG. 13 . This reduces the number of PCIe lanes used by the PCIe devices, and allows new PCIe devices not connected to the CPU 10b to be connected to the freed-up PCIe lanes. This improves device utilization efficiency, power consumption efficiency, and overall computing performance. While FIGS. 12 and 13 illustrate the case where the WSS 8 is not used, the number of PCIe lanes can be similarly reduced even when the WSS 8 is switched.

[0080] In addition to the effective bandwidth, the utilization rate of the PCIe devices 3-1 to 3-N associated with the second threshold may be monitored, which also increases the utilization efficiency of the devices, thereby improving power consumption efficiency and overall computing performance.

[0081] [Sixth Embodiment] Figure 14 is a block diagram showing the configuration of a computer system according to a sixth embodiment of the present invention, with the same components as those in Figure 8 being assigned the same reference numerals. This embodiment is configured by connecting two blocks 22-1 and 22-2 through an unused port of the OCS2 in the configuration of the fifth embodiment. Blocks 22-1 and 22-2 each include a CPU board 1b, an OCS2, PCIe devices 3-1 to 3-N, and a WSS8. Figure 14 omits the illustration of components other than the OCS2, NIC 13, and WSS8 in block 22-2.

[0082] The CPU 10b of each block 22-1, 22-2 is capable of communicating with the CPU 50 of the management control system 5 via the NIC 13 and the NIC 52 of the management control system 5. The CPU 40 of the OCS controller 4 is capable of controlling the OCS2 and WSS8 of each block 22-1, 22-2 via the NIC 42. The number of ports connecting the OCS2 of block 22-1 and the OCS2 of block 22-2 is 2×B×k (k is a positive integer other than 0). As described above, B is the number of connection ports with the OCS2 of each WSS8.

[0083] In this embodiment, the number of PCIe devices in one block may be less than or equal to the number of devices N in the fifth embodiment, and the number of PCIe x16 lanes on the photoelectric conversion device 12 side of the PCIe bridge chip 14 in one block may be five or more.

[0084] The two blocks 22-1 and 22-2 are managed and controlled by a common management control system 5 and OCS controller 4. The operation of this embodiment is similar to the operation described with reference to FIGS. 2 and 7. The CPU 10b of the block 22-1 can access the PCIe devices of the block 22-2 via the OCS2 of the block 22-1 and the OCS2 of the block 22-2. Similarly, the CPU 10b of the block 22-2 can access the PCIe devices of the block 22-1 via the OCS2 of the block 22-2 and the OCS2 of the block 22-1. In other words, the number of PCIe devices that can be accessed by one CPU 10b can be considered to be twice that of the fifth embodiment.

[0085] 14, two blocks 22-1 and 22-2 are connected through a free port of OCS2, but it is possible to connect more blocks depending on the number of free ports. For example, a configuration in which three blocks 22-1 to 22-3 are connected is shown in FIG. 15. In this case, the number of PCIe devices that one CPU 10b can access can be considered three times that in the fifth embodiment. The actual overall system configuration is determined by taking into account the amount of calculation of the target workload and changes in calculation over time, and by determining the number of blocks and the value of k, which relates to the number of ports connecting the blocks.

[0086] In this embodiment, it is considered common practice to set the number of PCIe devices in one block to be less than the number of devices N in the fifth embodiment, or to set the number of PCIe × 16 lanes on the photoelectric conversion device 12 side of the PCIe bridge chip 14 in one block to 5 or more. The reason for this is that, as shown in Fig. 13, if one CPU 10b can access all of the PCIe devices 3-1 to 3-N connected to the OCS 2 connected via the PCIe bridge chip 14, it is considered that in a configuration in which multiple blocks are connected, the CPU 10b in one block has very few opportunities to access another block.

[0087] For example, if a PCIe device in a block fails, it is expected that a PCIe device in another block will be accessed to replace the failed PCIe device, so connecting multiple blocks is effective in terms of operation. However, since PCIe device failure is considered to be a rare case, in many cases, when realizing the configurations of Figures 14 and 15, it is considered that the number of PCIe devices in one block will be less than the number of devices N in the fifth embodiment, or the number of PCIe x16 lanes on the photoelectric conversion device 12 side of the PCIe bridge chip 14 in one block will be five or more.

[0088] 16 is a block diagram showing the configuration of a computer system according to a seventh embodiment of the present invention. This embodiment is similar to the second embodiment in that a plurality of WSSs 8 are inserted between the OCS 2 and the PCIe devices 3-1 to 3-N, and is configured such that the insertion positions of the WSSs 8 are moved closer to the PCIe devices 3-1 to 3-N than in the fifth embodiment.

[0089] The relationship between the wavelength multiplexing degree P, the number N of PCIe devices 3-1 to 3-N connected to the OCS 2, the number A of connection ports between each of the PCIe devices 3-1 to 3-N and the OCS 2, and the number B of connection ports between each WSS 8 and the PCIe devices 3-1 to 3-N is the same as that in Fig. 9. A WSS 8 is provided for each P PCIe devices. The operation of this embodiment is the same as the operation described with reference to Figs. 2 and 7.

[0090] As in the fifth and sixth embodiments, this embodiment also makes it possible to improve the utilization efficiency of devices, power consumption efficiency, and overall computing performance. It also makes it possible to configure a larger system that can handle more workloads, and to improve the robustness of the system.

[0091] Compared to the configuration of the fifth embodiment, this embodiment has the disadvantage of reducing the degree of freedom in access selection because access from the CPU 10b to each of the PCIe devices 3-1 to 3-N is via a set of PCIe devices connected to the WSS 8. However, it also has the advantage of being able to reduce the number of ports on the OCS 2. In other words, depending on the insertion position of the WSS 8 relative to the OCS 2, there is a trade-off between the number of ports on the OCS 2 and the degree of freedom in access selection from the CPU 10b to each of the PCIe devices 3-1 to 3-N. The configuration of the fifth embodiment or the configuration of this embodiment is selected depending on the target workload.

[0092] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0093] (Supplementary Note 1) A computer system of the present invention includes a plurality of PCIe devices, a CPU configured to execute processing according to a program, a switch configured to selectively connect the plurality of PCIe devices to PCIe lanes of the CPU, a first control unit configured to be able to change the number of PCIe lanes on the CPU side, a second control unit configured to control the switch, and a management control system configured to: when the effective bandwidth of a PCIe lane used by a PCIe device connected to the CPU is equal to or less than a first threshold, instruct the first control unit to reduce the number of PCIe lanes used by the device, and to instruct the second control unit to newly connect a PCIe device not connected to the CPU to the PCIe lane made available by the reduction; and the CPU assigns processing to the newly connected PCIe device.

[0094] (Supplementary Note 2) In the computer system according to Supplementary Note 1, the first control unit is a PCIe controller in the CPU.

[0095] (Supplementary Note 3) In the computer system according to Supplementary Note 1, the first control unit is a PCIe bridge chip provided between the CPU and the switch.

[0096] (Supplementary Note 4) In the computer system described in Supplementary Note 3, the switch is an OCS, and further includes an optoelectronic conversion device between the PCIe bridge chip and the switch, and the optoelectronic conversion device converts the electrical signal of the PCIe lane from the PCIe bridge chip into a wavelength-multiplexed optical signal and sends it to the switch, and converts the wavelength-multiplexed optical signal from the switch into the electrical signal of the PCIe lane and sends it to the PCIe bridge chip.

[0097] (Appendix 5) The computer system described in Appendix 4 further includes a wavelength selective switch between the photoelectric conversion device and the switch, or between the switch and the PCIe device, and the second control unit controls the demultiplexing and multiplexing operations of the wavelength selective switch in addition to the switch.

[0098] (Supplementary Note 6) In the computer system described in Supplementary Note 1, when there is a request from the CPU for additional processing by a PCIe device and there is an extra PCIe device not connected to the CPU among the PCIe devices connected to the switch, the management and control system detects the effective bandwidth of the PCIe lane used by the PCIe device connected to the CPU, and when a new PCIe device is connected to a PCIe lane that has become free by reducing the number of PCIe lanes, the CPU assigns additional processing to this PCIe device.

[0099] (Supplementary Note 7) In the computer system described in Supplementary Note 1, when a utilization rate of a PCIe device connected to the CPU is equal to or greater than a second threshold value and there is a remaining PCIe device not connected to the CPU among the PCIe devices connected to the switch, the management and control system detects an effective bandwidth of a PCIe lane used by the PCIe device connected to the CPU, and when a PCIe device is newly connected to a PCIe lane that has become available due to the reduction in the number of PCIe lanes, the CPU allocates part of the processing of the PCIe device whose utilization rate is equal to or greater than the second threshold value to the newly connected PCIe device.

[0100] The present invention can be applied to a computer system that uses PCIe devices.

[0101] 1a to 1d...CPU board, 2...OCS, 3-1 to 3-N, 3c-1 to 3c-4...PCIe device, 4...OCS controller, 5...management control system, 6...electrical switch, 7...controller, 8...wavelength selective switch, 10a, 10b, 40, 50, 70...CPU, 11, 41, 51, 71...memory, 12, 31-1 to 31-N...photoelectric conversion device, 13, 42, 52, 72...NIC, 14...PCIe bridge chip, 20, 21...optical fiber, 22-1 to 22-3...block, 30-1 to 30-N...GPU, 32-1 to 32-4, 100...PCIe controller.

Claims

1. A computer system comprising: a plurality of PCIe devices; a CPU configured to execute processing according to a program; a switch configured to selectively connect the plurality of PCIe devices to a PCIe lane of the CPU; a first control unit configured to be able to change the number of PCIe lanes on the CPU side; a second control unit configured to control the switch; and a management control system configured to instruct the first control unit to reduce the number of PCIe lanes used by a PCIe device connected to the CPU when the effective bandwidth of the PCIe lane used by the device is equal to or less than a first threshold value, and to instruct the second control unit to newly connect a PCIe device not connected to the CPU to the PCIe lane made available by the reduction, wherein the CPU assigns processing to the newly connected PCIe device.

2. A computer system according to claim 1, wherein the first control unit is a PCIe controller within the CPU.

3. A computer system according to claim 1, wherein the first control unit is a PCIe bridge chip provided between the CPU and the switch.

4. A computer system according to claim 3, wherein the switch is an OCS, and further comprising an opto-electrical conversion device between the PCIe bridge chip and the switch, the opto-electrical conversion device converting an electrical signal of a PCIe lane from the PCIe bridge chip into a wavelength-multiplexed optical signal and sending it to the switch, and converting the wavelength-multiplexed optical signal from the switch into an electrical signal of a PCIe lane and sending it to the PCIe bridge chip.

5. A computer system according to claim 4, further comprising a wavelength selective switch between the photoelectric conversion device and the switch, or between the switch and the PCIe device, and the second control unit controls the demultiplexing and multiplexing operations of the wavelength selective switch in addition to the switch.

6. A computer system according to claim 1, wherein the management and control system detects an effective bandwidth of a PCIe lane used by a PCIe device connected to the CPU when there is an additional PCIe device not connected to the CPU among the PCIe devices connected to the switch, and the CPU assigns additional processing to a PCIe device when a new PCIe device is connected to a PCIe lane that has become available by reducing the number of PCIe lanes.

7. A computer system according to claim 1, wherein the management and control system detects an effective bandwidth of a PCIe lane used by a PCIe device connected to the CPU when a utilization rate of the PCIe device connected to the CPU is equal to or greater than a second threshold value and there is an extra PCIe device not connected to the CPU among the PCIe devices connected to the switch, and when a new PCIe device is connected to a PCIe lane that has become vacant by reducing the number of PCIe lanes, the CPU assigns a portion of the processing of the PCIe device whose utilization rate is equal to or greater than the second threshold value to the newly connected PCIe device.