Distributing interrupt requests to be handled by target virtual processors

Abstract

A virtual processor is provided that can be mapped to a plurality of physical processors. An interrupt distributor is responsible for distributing interrupt requests to a subset of the physical processors. An interface communicates with additional interrupt distributors responsible for other physical processors. In response to an interrupt request to be handled by a target virtual processor, the interrupt distributor determines whether to route the interrupt request to one of the subset of physical processors or to one of the additional interrupt distributors based on cached virtual processor mapping information. When a rejection response is received in response to the interrupt request being routed to one of the additional interrupt distributors, an update to the cached virtual processor mapping information is requested based on shared virtual processor mapping information and the re-sent interrupt request is sent to another interrupt distributor determined based on the shared virtual processor mapping information.

Description

Background Technology Technical Field

[0002] This technology relates to the field of data processing. More specifically, this technology relates to allocating interrupt requests. Technical Background

[0004] Virtualization allows multiple operating systems to run simultaneously on the same system hardware. Each operating system can run on a virtual processor (or virtual machine VM), which operates under the impression of having exclusive access to the system hardware's processor, peripherals, system memory, and input / output devices, but in reality, this virtual processor shares the hardware with other virtual processors. Physical hardware can include multiple physical processors that can be mapped to a particular virtual processor. A given virtual processor can frequently move from one physical processor to another. A given physical processor can have more than one virtual processor mapped to it at a given time, but can execute instructions from a specific virtual processor among those mapped to that physical processor. The virtual processor from which instructions are currently being executed is considered a resident virtual processor, while other virtual processors mapped to a given physical processor but whose instructions are not currently being executed are considered non-resident virtual processors. Virtualization can be particularly useful for enterprise server application spaces where a single physical server, including multiple physical processors, can be configured to support many virtual machines, even up to tens, hundreds, or thousands of virtual machines. Such virtualized servers can provide services such as web hosting, cloud services, databases, and storage management. Summary of the Invention

[0005] At least some examples provide an apparatus comprising: an interrupt allocator for receiving an interrupt request to be processed by a target virtual processor selected from a plurality of virtual processors capable of being mapped to a plurality of physical processors, wherein the interrupt allocator is responsible for allocating the interrupt request to a subset of the physical processors; and an interface for communicating with other interrupt allocators responsible for allocating the interrupt request to physical processors other than the subset of physical processors; wherein, in response to the interrupt request, the interrupt allocator is configured to:

[0006] Based on cached virtual processor mapping information, determine whether to route the interrupt request to one of the physical processors in the subset of physical processors, or to one of the other interrupt allocators; and when a rejection response is received in response to the interrupt request being routed to one of the other interrupt allocators: request an update to the cached virtual processor mapping information based on shared virtual processor mapping information obtained from a storage location accessible by the interrupt allocator and the other interrupt allocators; and route the retransmitted interrupt request to the other interrupt allocator determined based on the shared virtual processor mapping information.

[0007] At least some examples provide an apparatus comprising: means for allocating an interrupt request, the means for allocating being configured to receive an interrupt request to be processed by a target virtual processor selected from a plurality of virtual processors capable of being mapped to a plurality of physical processors, wherein the means for allocating is responsible for allocating the interrupt request to a subset of the physical processors; and means for communicating with other means for allocating the interrupt request to physical processors other than the subset of the physical processors; wherein, in response to the interrupt request, the means for allocating is configured to: determine based on cached virtual processor mapping information. The interrupt request is routed to a physical processor in the subset of the physical processors, or to one of the other allocation devices; and when a rejection response is received in response to the interrupt request being routed to one of the other allocation devices: an update to the cached virtual processor mapping information is requested based on shared virtual processor mapping information, which is obtained from a storage location accessible by the allocation device and the other allocation devices; and the retransmitted interrupt request is routed to another allocation device determined based on the shared virtual processor mapping information.

[0008] At least some examples provide a method comprising: receiving at an interrupt allocator an interrupt request to be processed by a target virtual processor selected from a plurality of virtual processors capable of being mapped to a plurality of physical processors, the interrupt allocator being responsible for allocating the interrupt request to a subset of the physical processors, and other interrupt allocators being responsible for allocating the interrupt request to physical processors other than the subset of the physical processors; and in response to the interrupt request, the interrupt allocator: determining, based on cached virtual processor mapping information, whether to route the interrupt request to a physical processor in the subset of the physical processors or to one of the other interrupt allocators; and when a rejection response is received in response to the interrupt request being routed to one of these other interrupt allocators: requesting an update to the cached virtual processor mapping information based on shared virtual processor mapping information obtained from a storage location accessible by the interrupt allocator and the other interrupt allocators; and routing the retransmitted interrupt request to the other interrupt allocator determined based on the shared virtual processor mapping information.

[0009] Further aspects, features, and advantages of this technology will become apparent from the following description, which is taken in conjunction with the accompanying drawings. Attached Figure Description

[0010] Figure 1 An example of a data processing system with multiple interrupt controllers is schematically shown, which in this example are located on different integrated circuits;

[0011] Figure 2 An example of an interrupt controller is shown;

[0012] Figure 3 The components of the processor interface in the physical processor used to receive interrupts from the interrupt controller are shown;

[0013] Figure 4 The illustration schematically shows the handling of an interrupt request that will be processed by the target virtual processor;

[0014] Figure 5 Examples of cached virtual processor mapping information and shared virtual processor mapping information are shown;

[0015] Figure 6 The update of cached virtual processor mapping information and shared virtual processor mapping information in response to virtual processor transfer commands is shown;

[0016] Figure 7 A flowchart illustrating the processing of an interrupt request received at the source interrupt controller;

[0017] Figure 8 A flowchart illustrating the processing of a forwarded interrupt received at the target interrupt controller after being routed from the source interrupt controller;

[0018] Figure 9 This schematically illustrates an example of handling interrupt requests after a virtual processor is transferred from one physical processor to another; and

[0019] Figure 10 An example illustrating the use of monitoring circuitry to detect errors, thereby halting interrupt processing to prevent deadlock. Detailed Implementation

[0020] Virtualized processing systems can have a number of physical processors and support the execution of a number of virtual processors, which can be mapped to different physical processors. For example, the mapping from virtual processors to physical processors can be selected by software based on factors such as the workload required by different virtual processors and the physical resources available at each physical processor. Transferring virtual processors from one physical processor to another can be a relatively common operation in enterprise server environments. This can lead to some complexity in interrupt handling, as some interrupt requests may need to be handled by a specific target virtual processor from the overall set of virtual processors. Therefore, the interrupt allocator used to assign an interrupt request to the appropriate physical processor to handle that interrupt request may need to track the specific physical processor to which a given virtual processor is mapped, so that it can route the interrupt request to the correct physical location. However, modern data processing systems can have a relatively large number of physical processors (in some examples, these physical processors may reside on different integrated circuits or be physically distant on a single integrated circuit). Moreover, to alleviate the burden on the central interrupt allocation point, the interrupt controller can be divided into multiple separate interrupt allocators, each responsible for assigning interrupt requests to a specific subset of physical processors. These factors can make maintaining accurate trace information at each interrupt allocator relatively complex or cumbersome, as this trace precisely indicates the physical processor on which each virtual processor resides. Since virtual processor transfers from one physical processor to another can be relatively common, the overhead of exchanging messages to keep this trace information up-to-date may be unreasonable.

[0021] One approach to recording the location mapped to each virtual processor is to have the interrupt allocator responsible for assigning interrupt requests to the physical processor to which the virtual processor was previously mapped when a move to the virtual processor is indicated. This broadcasts the virtual processor move to all other interrupt allocators in the system, informing them that the move allows them to update their trace information. However, significant latency can occur in message transmissions used to broadcast virtual processor moves because other interrupt allocators may be physically distant from the originating allocator and may even be located on different integrated circuits. When a message transmission is performed to update trace information based on a virtual processor move, an interrupt targeted by that virtual processor and requiring processing may be received. However, doing so during the update process to reflect the virtual processor move may be unsafe, and the broadcast method could cause interrupts to remain effectively unserviced for a considerable period, potentially detrimental to performance. Furthermore, the broadcast method incurs overhead at interrupt allocators not involved in the virtual processor move (those not responsible for the source or destination physical processor to which the virtual processor is moved). At other interrupt allocators, update overhead may be unnecessary because the virtual processor may be moved again before the interrupt allocator receives any interrupts targeting that virtual processor. In such cases, the overhead of responding to broadcast message transmissions may already be redundant. If there are frequent virtual processor transfers but relatively few interrupt requests for a given virtual processor, the overhead of broadcasting each transfer may incur unnecessary power and performance costs.

[0022] In the techniques discussed below, it is counterintuitively recognized that it is not necessary for each interrupt allocator to maintain an exact indication of which physical processor mapping each virtual processor is mapped to, and that it is acceptable for the information held at a given interrupt allocator to be incorrect. A storage location accessible to each interrupt allocator that can store shared virtual processor mapping information, which is updated to indicate the actual location of a given virtual processor, is available. Each interrupt allocator may maintain cached virtual processor mapping information based on the shared virtual processor mapping information, but this cached information may be outdated and therefore may indicate an incorrect mapping for a given virtual processor. Since this provides the possibility that, when an interrupt request is received that is to be processed by a target virtual processor, incorrect cached virtual processor mapping information may cause the interrupt request to be routed to the wrong destination, the present invention provides a rejection mechanism in which a given interrupt allocator receiving an interrupt request forwarded from another interrupt allocator may reject the interrupt request if the interrupt request is not actually targeted at one of the physical processors that the interrupt allocator is responsible for. While it may seem undesirable that interrupt requests might be rejected and then need to be retransmitted if they were previously sent to the wrong destination, in practice, this overhead is far less than the overhead associated with broadcasting every transmission of the virtual processor to other interrupt allocators, and makes the performance cost lower compared to blocking the processing of interrupt requests when updating after a transmission is broadcast, as in the alternative methods discussed above. Therefore, overall, the method of retransmitting incorrectly routed interrupts provides improved performance.

[0023] Therefore, the device may have an interrupt allocator to receive interrupt requests to be processed by a target virtual processor, selected from a plurality of virtual processors capable of being mapped to a number of physical processors. The interrupt allocator is responsible for allocating interrupt requests to a specific subset of those physical processors. This subset may be an appropriate subset of the physical processors, thus there may be at least one physical processor that the interrupt allocator is not responsible for. An interface is provided to communicate with additional interrupt allocators that are responsible for allocating interrupt requests to physical processors other than the subset of physical processors that the interrupt allocator is responsible for. These additional interrupt allocators may reside on the same integrated circuit as the interrupt allocator, or they may reside on a different integrated circuit.

[0024] In response to receiving an interrupt request targeting a given target virtual processor, the interrupt allocator may determine, based on cached virtual processor mapping information (which may be incorrect), whether to route the interrupt request to a physical processor within a subset of physical processors that the interrupt allocator is responsible for, or to an interrupt allocator among other interrupt allocators. If the cached virtual processor mapping information indicates that the target virtual processor is not mapped to a physical processor within the subset of physical processors that the interrupt allocator is responsible for, an interrupt allocator among other interrupt allocators may be selected. If the cached virtual processor mapping information proves to be correct, routing the interrupt request to the selected other interrupt allocator or to a physical processor within a subset of physical processors may then allow the correct physical processor to which the target virtual processor is mapped to to handle the interrupt.

[0025] However, if the cache's virtual processor mapping is incorrect, it is possible that the location to which the interrupt request is routed may reject the interrupt response.

[0026] If a rejection response is received while the interrupt request is being routed to one of the other interrupt allocators, the interrupt allocator can request an update to the cached virtual processor mapping information based on shared virtual processor mapping information obtained from a memory location accessible to the interrupt allocator itself and other interrupt allocators with which it can communicate via an interface. The interrupt allocator can then route the retransmitted interrupt request to another interrupt allocator determined based on the shared virtual processor mapping information. Therefore, once the actual location of the target virtual processor has been determined based on the shared virtual processor mapping information, retransmitting the interrupt request allows it to reach its correct location, enabling the interrupt to be serviced.

[0027] Therefore, counterintuitively, while allowing cached virtual processor mapping information to be incorrect might seem like a drawback, in practice, this can be more efficient than maintaining a more precise indication of the virtual processor mapping at each interrupt allocator. Furthermore, by providing the aforementioned retransmission mechanism, this allows interrupts to be handled correctly even if the initial attempt to allocate an interrupt proves incorrect due to the use of incorrect cached virtual processor mapping information. This generally provides higher performance and lower power consumption.

[0028] When an interrupt allocator determines which physical processor to route an interrupt request to within a subset of the physical processors it is responsible for, based on cached virtual processor mapping information, it is not necessary to route the interrupt request immediately upon receipt. It is possible that the target virtual processor is mapped to a physical processor within a subset of the physical processors, but is not currently resident because its instructions are not currently being executed by the corresponding physical processor, and the target virtual processor may need to wait for a period of time before becoming resident later. Therefore, in some cases, the interrupt allocator may maintain a pending interrupt table for the corresponding virtual processor, which indicates that when the virtual processor next becomes resident, the interrupt request should be routed to the corresponding physical processor of that virtual processor. Thus, the exact timing at which an interrupt request will be routed to its destination may vary depending on the specific implementation.

[0029] The device may include a mapping update circuit responsible for updating shared virtual processor mapping information when a given virtual processor is transferred from a source physical processor to a destination physical processor. In response to a virtual processor transfer command indicating that a virtual processor will be transferred, the mapping update circuit may trigger an update to the shared virtual processor mapping information to indicate that the given virtual processor is now mapped to the destination physical processor. Therefore, the shared virtual processor mapping information remains up-to-date as virtual processors are transferred between physical processors.

[0030] In some examples, the mapping update circuitry can be implemented as a single non-distributed circuit logic unit that responds to virtual processor transfer commands and is shared between interrupt allocators and other interrupt allocators rather than being associated with a specific interrupt allocator within the interrupt allocator.

[0031] However, in another approach, the mapping update circuit can be associated with a specific interrupt allocator, and each of the interrupt allocators and other interrupt allocators can have its own instance of the mapping update circuit. In this case, when the mapping update circuit is associated with an interrupt allocator, it can be arranged to update the shared virtual processor mapping information in response to a virtual processor transmission command when the destination physical processor is a physical processor in a subset of the physical processors that the interrupt allocator is responsible for. Therefore, the mapping update circuit in any interrupt allocator not associated with a destination physical processor does not need to trigger an update to the shared virtual processor mapping information.

[0032] When the source or destination physical processor is a physical processor within a subset of the physical processors handled by the interrupt allocator, the mapping update circuitry at a given interrupt allocator can also update the cached virtual processor mapping information of that allocator. Therefore, when transferring the target virtual processor, the mapping update circuitry associated with the interrupt allocators responsible for both the source and destination physical processors can update the cached virtual processor mapping information of both allocators (or, if both the source and destination physical processors are associated with the same interrupt allocator, only one allocator can update its cached virtual processor mapping information). However, this can be done without broadcasting updates to the shared virtual processor mapping information to any other interrupt allocators not responsible for allocating interrupt requests to the source or destination physical processors. Therefore, any interrupt allocator not associated with the source or destination physical processor can continue to cache expired virtual processor mapping information, which may differ from the updated shared virtual processor mapping information, because their cached information will not be updated in response to the virtual processor transfer command itself. This reduces message passing overhead (by eliminating communication with interrupt allocators associated with processors not involved in the transfer) by limiting updates when transferring virtual processors to interrupt allocators associated with both the source and destination physical processors. By avoiding broadcast windows, this means there is no need to block interrupts during the time broadcasts are pending, which would otherwise be detrimental to performance.

[0033] Interrupt allocators and other interrupt allocators may have corresponding configurations such that each of them can have the same circuit design. Alternatively, there may be minor differences between interrupt allocators, but their general functionality can be similar. Thus, each interrupt allocator may sometimes act as a source interrupt allocator that receives the original interrupt request and then decides how to route it to other interrupt allocators; or at other times it may act as a target interrupt allocator responsible for allocating the interrupt to a physical processor that runs a target virtual processor that will handle the interrupt, where the interrupt request has been forwarded to the physical processor by another interrupt allocator. Thus, the same interrupt allocator may sometimes receive the original interrupt request and at other times receive forwarded interrupt requests (initial attempts or retransmitted interrupt requests) from other interrupt allocators.

[0034] The retransmission mechanism described above describes the function of the interrupt allocator when it directly receives the original interrupt request as the source interrupt allocator, and then decides how to route the original interrupt request to a physical processor in a subset of physical processors or to another interrupt allocator.

[0035] On the other hand, at other times, an interrupt allocator can act as a target interrupt allocator and thus receive received interrupt requests from a given other interrupt allocator, which is the source interrupt allocator for that particular interrupt request. In this case, in response to a received interrupt request received from that given other interrupt allocator, the interrupt allocator acting as the target interrupt allocator can check its cached virtual processor mapping information and determine whether the target virtual processor specified by the received interrupt request is mapped to a physical processor in a subset of the physical processors that the interrupt allocator is responsible for.

[0036] When the target virtual processor for an interrupt request received from a given other interrupt allocator is mapped to a physical processor within a subset of physical processors, the interrupt allocator may pass the received interrupt request to the cached virtual processor mapping information indicating the physical processor to which the target virtual processor is mapped, and may return an acknowledgment response to the given other interrupt allocator. The acknowledgment response indicates that the given other interrupt allocator has indeed correctly transmitted the interrupt request to the other correct interrupt allocator.

[0037] It should be noted that if the above mapping update method is used (where the interrupt allocator responsible for the source and destination physical processors updates its cached virtual processor mapping information when performing transfers involving the source and destination physical processors), it is unlikely that the cached virtual processor mapping information at a particular interrupt allocator might incorrectly indicate that the target virtual processor is mapped to a physical processor within a subset of the physical processors. Errors in the cached virtual processor mapping information are more likely to occur when the cached virtual processor mapping information indicates that the target virtual processor is mapped to a physical processor other than the subset of physical processors handled by that particular interrupt allocator, since this could indicate that a transfer to the target virtual processor was performed, an operation not involving the physical processors handled by that interrupt allocator. Therefore, it is unlikely (in cases where the interrupt allocator forwards received interrupt requests to a physical processor within a subset of the physical processors) that this will be incorrect; however, some implementations may provide additional retry mechanisms to handle the possibility that the indication of the physical processor itself might be incorrect.

[0038] On the other hand, if, in response to a received interrupt request received from a given other interrupt allocator, the cached virtual processor mapping information indicates that the target virtual processor specified by the received interrupt request is not mapped to a physical processor in the subset of physical processors handled by that interrupt allocator, then the interrupt allocator may provide a rejection response to the given other interrupt allocator that routed the interrupt request to it. This rejection response may trigger the given other interrupt allocator to update its cached virtual processor mapping information and retransmit the interrupt request using the retransmission mechanism described above. Therefore, by providing an acknowledgment or rejection response to the given other interrupt allocator, the interrupt allocator signals to the given other interrupt allocator whether its decision to deliver the interrupt request to it was correct.

[0039] When an interrupt allocator routes a given interrupt request to another interrupt allocator and subsequently receives a rejection response in its reply, that allocator resends the interrupt request to the other allocator, which is determined based on a cached virtual processor mapping updated after checking shared virtual processing information. If at least one earlier attempt has been made to send the interrupt request (or the most recent interrupt request) to the same other interrupt allocator, the retransmission request can be sent with a different encoding than the encoding used for the first attempt to send the interrupt request to that other interrupt allocator. This allows the other interrupt allocator to distinguish whether the interrupt request has been incorrectly routed to the same destination repeatedly, which could be an indication that an error has occurred. The encoding of the retransmitted interrupt request can vary in different ways compared to the encoding of the earlier attempts. For example, a different interrupt request type can be specified. Alternatively, the retransmitted interrupt request can be sent with the same interrupt request type for each attempt, but may include bits indicating whether it is the first or subsequent attempt (e.g., a "danger" flag that signals whether an error has occurred). In another example, an interrupt request can be sent with a specified retry count, indicating the number of times the interrupt request has been attempted to be sent to the same destination, such that the retransmitted interrupt request can specify a different retry count value compared to the original interrupt request. Regardless of the specific encoding technique used, it may be useful to distinguish the retransmitted interrupt request from the original interrupt request to allow the system to detect deadlock scenarios where, if multiple attempts to retransmit the interrupt request to the same destination have existed, the final processing of the interrupt is aborted, as this may indicate that some error has occurred.

[0040] For example, an interrupt allocator or other interrupt allocator may include a mechanism to abort processing of an interrupt request after multiple unsuccessful attempts to resend it to the same destination. This can help detect errors caused by corruption of shared virtual processor mapping information or by temporary or permanent hardware failures in the storage circuitry or control logic circuitry within the interrupt allocator. For example, randomly occurring hardware errors (such as bit flips in storage elements due to subatomic particle impacts or radiation) or randomly occurring hardware failures (which cause physical electronic components to degrade, causing some bits to get stuck at zero or in one place) may occur randomly, and such errors may cause the shared virtual processor mapping information to ultimately indicate incorrect values. This can lead to a given interrupt allocator determining that the target virtual processor is located on a different physical processor, even if no transfer involving the target virtual processor has occurred and the target virtual processor actually resides on a physical processor. If no subsequent transfer to the target virtual processor occurs, the error may not be detected, and several interrupt requests may be routed to the wrong destination. By providing a mechanism to abort the processing of interrupt requests after a certain number of attempts, this prevents the system from entering a deadlock scenario where there are unsuccessful attempts to resend the interrupt request.

[0041] In one example, monitoring circuitry may be provided to monitor transfers to a specified virtual processor, maintaining trace information indicating whether the specified virtual processor has received a mapped transfer to or from a physical processor within a subset of physical processors associated with the interrupt allocator. In some examples, each interrupt allocator in the interrupt allocator may have its own separate monitoring circuitry, or shared monitoring circuitry may be provided for the system as a whole. In some examples, the monitoring circuitry may permanently monitor transfers to a particular virtual processor, but this may be less efficient in terms of power; therefore, other examples may selectively enable and disable the monitoring circuitry. For example, monitoring circuitry for a particular interrupt allocator may be enabled to monitor transfers to a single specified virtual processor or a limited number of specified virtual processors, and then disabled once monitoring is no longer needed. For example, for each virtual processor in the monitored virtual processors, a bit may be set if a transfer by that virtual processor to or from a physical processor within a subset of physical processors associated with the corresponding interrupt allocator is detected.

[0042] By providing monitoring circuitry, taking into account whether a transfer involving a specified virtual processor has actually occurred, a more complex decision can then be made regarding whether to continue attempting to resend the interrupt request.

[0043] When an interrupt allocator acts as the target interrupt allocator for processing an interrupt request forwarded from another interrupt allocator, and then receives a retransmitted interrupt request from the given other interrupt allocator, if at least one previous attempt has been made to send the interrupt request (or the retransmitted interrupt request) to the same target interrupt allocator, cached virtual processor mapping information indicates that the target virtual processor specified by the retransmitted interrupt request is not mapped to a physical processor in a subset of physical processors, and the monitoring circuitry has not yet monitored whether the target virtual processor has undergone a mapping transfer, then the interrupt allocator may control the monitoring circuitry to begin monitoring whether the target virtual processor has undergone a mapping transfer. When monitoring begins, trace information may be cleared to indicate that a transfer to the target virtual processor has not yet been detected.

[0044] On the other hand, if a retransmitted interrupt request is received from a given other interrupt allocator (after at least one early attempt to send an interrupt request or a retransmitted interrupt request to the same target interrupt allocator), then when cached virtual processor mapping information indicates that the target virtual processor is not mapped to a physical processor in a subset of physical processors, but the monitoring circuit has been monitoring the transfer of the target virtual processor, the interrupt allocator may check the trace information maintained by the monitoring circuit to check whether the trace information indicates that the mapping transfer of the target virtual processor has been detected.

[0045] If the tracking information indicates that the monitoring circuit has not yet detected the mapping transfer of the target virtual processor, an abort response can be provided to the given additional interrupt allocator in response to the retransmitted interrupt request. This abort response can then trigger the additional interrupt allocator to abort interrupt processing rather than trigger an error handling response, because in this case, the lack of detected mapping transfer may indicate that the shared virtual processor mapping information used by the given additional interrupt allocator to determine where to send the retransmitted interrupt request has been erroneous, or that an error has occurred in the control logic path.

[0046] On the other hand, if the tracking information is checked and indicates that the monitoring circuitry has detected a mapping transfer of the target virtual processor, abort may be premature because the shared virtual processor mapping information may have been updated after the transfer was detected. This would mean that if the given other interrupt allocator subsequently updates its cached virtual processor mapping information and attempts to resend the interrupt request, the interrupt could now reach its correct destination. Therefore, the interrupt allocator can respond to a resent interrupt request by providing a rejection response to the given other interrupt allocator, thus giving the interrupt another chance to be resent and, if possible, trying to avoid the cost of an incorrect response.

[0047] Therefore, this monitoring method allows for more complex decisions about whether it is worthwhile to continue attempting to resend the interrupt or whether the interrupt request should be aborted.

[0048] Error handling response actions (taken by the interrupt allocator upon receiving an abort response from another interrupt allocator in response to a retransmitted interrupt request) can vary. In one example, an error handling response action can be used to generate another type of interrupt, indicating that action must be taken in the software to handle the fact that the original interrupt request could not be routed to its correct destination. For example, the software could trigger a remapping of a given target virtual processor to a new physical processor and a restart, so that any errors in the shared virtual processor mapping information related to that particular target virtual processor can be corrected by updating that shared virtual processor mapping information to specify the new physical location. This can handle transient errors, such as those caused by particle impacts. Alternatively, a processor reset can be triggered, which resets the data processing system to a default state and then allows execution to resume, allowing for the handling of any errors. The specific error handling response action can depend on the nature of the software being executed.

[0049] The above-described techniques can be implemented using a series of interrupt controller designs. In one example, the interrupt distributor may be responsible for a subset of physical processors, including at least two physical processors. Alternatively, the interrupt distributor may be responsible for only a single physical processor. Furthermore, in some examples, at least one of the additional interrupt distributors may reside on a different integrated circuit than the interrupt distributor. These different integrated circuits may be manufactured separately, such that the integrated circuit including the interrupt distributor may be manufactured by one manufacturer, but the additional interrupt distributors may reside on different integrated circuits manufactured by different manufacturers. Therefore, the device protected by the claims, including the interrupt distributor, does not necessarily need to include the additional interrupt distributor itself, but may only include an interface through which signals can then be routed to the additional interrupt distributor. Thus, the additional interrupt distributor does not actually need to be part of the device protected by the claims. However, in other examples, the additional interrupt distributor may be included in the same device as the interrupt distributor. For example, in some implementations, at least one of the additional interrupt distributors may reside on the same integrated circuit as the interrupt distributor.

[0050] Figure 1An example of a data processing system 2 is shown, comprising multiple integrated circuits (chips) 4 communicating with each other. Each chip includes multiple physical processors (CPUs) 6 and has some on-chip memory 8 accessible by the processors on that chip. Each chip may also have one or more device interfaces 10 through which messages are received from a corresponding device communicating with that interface. For example, the device may be a peripheral device, an input / output controller, a network controller, etc. The number of physical processors and device interfaces 10 on each chip may vary from chip to chip. It should be understood that... Figure 1 The specific number of chips shown is merely an example. Other examples may have only a single chip or may have more or fewer chips.

[0051] Each chip 4 has access to the shared memory 12, which can store data accessible by any of the chips 4. Figure 1 In the example, shared memory 12 is off-chip memory; however, in other examples, shared memory may reside on one of the chips 4 but be accessible by other chips 4. Data can be loaded from shared memory 12 into the on-chip memory 8 of a specific chip. Each physical processor 6 may also have a cache that caches data from the on-chip memory 8 or shared memory 12 of that chip 4. It should be understood that... Figure 1 It is a simplified diagram, and although not shown, it provides many other features of the data processing system 2.

[0052] In one example, data processing system 2 could be a virtualized server that provides services such as web hosting, cloud services, database management, or storage management.

[0053] An interrupt controller is provided to handle interrupts that require interrupt handling on one or more physical processors. For example, an interrupt may be based on a message sent from a given device via device interface 10. Interrupt controllers are assigned such that each chip in chip 4 includes a corresponding general-purpose interrupt controller (GIC) 14, which is responsible for relaying interrupts to the physical processor 6 on the corresponding chip. The GICs 14 on each chip communicate with each other via interrupt interconnect 16. The respective chips 4 may also communicate with each other via memory interconnect 18, which also allows data to be read and written in shared memory 12. Although in Figure 1 In this example, interrupt interconnect 16 and memory interconnect 18 are shown as separate, but in other examples, these interconnects may be combined to form a shared interconnect for both interrupt and memory access.

[0054] Data processing system 2 is a virtualized processing system, and therefore capable of executing multiple virtual processors (or virtual machines), each of which can be mapped to different physical processors in the system, including processors on different chips 4. Some virtual processors may require hardware features available only on certain physical processors, which may limit which physical processors can have virtual processors mapped to them, while other virtual processors may not have such limitations. The decision regarding mapping virtual processors to physical processors can be made by a hypervisor or other software components running on data processing system 2.

[0055] Figure 2 An example of a GIC 14 is shown, responsible for allocating interrupts to physical processors 6 on a given chip 4 and virtual processors running on those physical processors 6. GIC 14 includes an interrupt allocator 20, which is responsible for routing decisions about how to route received interrupt requests to the relevant physical processor capable of handling them. Interrupt requests received by allocator 20 may include physical interrupt signals 22 asserted on the wired path of data processing system 2. Interrupt requests may also include device interrupts 24 received from a device via device interface 10. As the complexity of data processing systems increases, the increased number of interrupts on a chip can lead to routing congestion and management complexity (in cases where all interrupts on these chips are associated with physical wired paths). Therefore, for some systems, a more efficient way to manage interrupt handling may be to use message-based interrupts (such as Information Signal Interrupts (MSI) for device interfaces implemented using a PCI Express (RTM) bus). Message-based interrupts may be transactions specifying memory addresses, where the specific memory address indicates the type of interrupt being asserted. This allows a single wired communication channel to be shared among multiple interrupt types distinguished by the corresponding addresses of the messages. For a message-based interrupt received as device interrupt 24 from a given device interface 10, the Interrupt Translation Service (ITS) circuit 26 can translate the message into the appropriate interrupt type by looking up the address specified by the message in a table that defines various interrupt types associated with different addresses. For example, the Interrupt Translation Service can generate an interrupt ID based on the message received from the device and can identify which specific target processor should handle the interrupt, which can be a physical processor or a virtual processor. A virtual processor can also be referred to as a "virtual processing element" or simply a VPE.

[0056] GIC 14 has memory access circuitry 28 that controls the reading of data from or writing to shared memory 12 or the on-chip memory 8 of the corresponding chip 4. GIC 14 may also have internal memory 30 that can be used to cache various tables or other information read from on-chip memory 8 or shared memory 12. Internal memory 30 stores ITS mapping table 32, used by ITS 26 to map message-based interrupts to interrupt IDs and target physical or virtual processors. GIC memory 30 may also cache VPE configuration table 34, which may correspond to the cached virtual processor mapping information as described above and indicate which physical processor a given virtual processor is mapped to throughout data processing system 2. VPE configuration table 34 may be outdated and different from a set of corresponding information stored as shared virtual processor mapping information in off-chip memory 12, accessible by all GICs 14 in the system. For example, this shared information may be a set of Virtual Processor Tables (VPTs) 36, described in more detail below. Although the VPTs primarily reside in shared memory 12, some VPTs or portions thereof may optionally be cached in GIC memory 30. GIC 14 also features mapping update circuitry 38 for controlling updates to the VPE configuration table 34 and VPT 36. (e.g.) Figure 2 As shown, GIC 14 may also have monitoring circuitry 150 for monitoring whether a specified VPE has been transmitted by the physical processor.

[0057] Distributor 20 uses VPE configuration table 34 to determine how to route interrupts 22 and 24 to their respective destinations. If VPE configuration table 34 indicates that a specific VPE targeted by an interrupt is mapped to a CPU 6 under the responsibility of GIC 14, then the interrupt can be routed to the corresponding physical processor 6. If the target virtual processor is mapped to a CPU 6 on a different chip 4 under the responsibility of a different GIC 14, then distributor 20 can route the interrupt to another interrupt controller 14 on another chip 4 via interrupt interconnect 16, and then that other interrupt controller can deliver the interrupt to the desired physical processor.

[0058] In order to distribute interrupt requests to the processors that a given GIC 14 is responsible for, each physical processor 6 may have a corresponding reallocator 42 responsible for routing signals to that physical processor, and communicate with the reallocator to make a more global decision. The reallocator 42 may be responsible for controlling signal exchange with a specific physical processor and may communicate with the CPU interrupt interface 44 within the corresponding physical processor 6.

[0059] Figure 3The components of CPU interface 44 for a specific physical processor 6 are shown in more detail. This specific physical processor is responsible for handling physical and virtual interrupt requests that will be processed by the physical processor or a virtual processor executing on the physical processor. CPU interface component 44 includes a physical CPU interface module 45, a virtual CPU interface 47, and a virtualization control module 46. The physical CPU interface module is used by a hypervisor executing on the physical CPU 6 to interact with physical interrupts. The virtual CPU interface is used by a virtual processor (VPE) 6-v (implemented using a virtual machine executing on the physical CPU 6 under the control of the hypervisor) to handle virtual interrupts. The virtualization control module is used by the hypervisor to configure the operation mode of the virtual CPU interface 47.

[0060] The Interrupt Translation Service (ITS) module 26 is responsible for translating Message-Based Interrupts (MSIs) from peripheral devices into Location-Specific Peripheral Interrupts (LPIs). Location-Specific Peripheral Interrupts (LPIs) are edge-triggered message-based interrupts targeting the reallocator 42 and the connected PE 6. One use case for ITS is translating MSI messages from the PCIe Root Complex (RC). When performing a translation, ITS is provided with a device ID derived from the PCIe requester ID and selects the interrupt namespace associated with that device. This allows two VMs to "live" in their own worlds without having to use mutually exclusive interrupts. For example, VM A can use IRQ number 123, and VM B's IRQ number 123 can be considered completely unrelated.

[0061] The CPU interface registers include physical interface registers, virtual interface registers, and virtualization control registers. The hypervisor executing at EL2 can program the physical CPU interface 45 to handle physical interrupts. The hypervisor has access to additional control registers to perform tasks such as programming the virtual CPU interface 47, accessing virtual register states to enable context switching, and configuring and controlling virtual interrupts.

[0062] When an assertion is made regarding a virtual interrupt targeting a specific guest OS (VPE) 6-v, the exception can be handled by the GIC associated with the physical processor (PE) 6 to which that VPE is mapped, as illustrated in the following follow-up event. Figure 4 As shown:

[0063] 1) Receives MSI from the physical device interface 10 to GIC ITS 26.

[0064] 2) The GIC ITS 26 derives the interrupt ID for the target client and signals the physical interrupt (IRQ) to the relevant CPU via allocator 20 and the redistributor 42 associated with the CPU 6. If the interrupt (IRQ) is configured as virtual (via hypervisor 49), the physical CPU 6 generates a hypervisor (HYP) exception.

[0065] 3) The management program 49 reads the interrupt status from the physical CPU interface 45.

[0066] 4) The hypervisor 49 creates entries in the list register provided in the virtual CPU interface 47. The list register provides data representing a list of incomplete interrupts for the associated VPE 6-v.

[0067] 5) The guest OS running on the target VPE 6-v reads the interrupt status from the virtual CPU interface 47.

[0068] 6) The guest OS handles (virtual) IRQs in a manner similar to that of physical interrupts (by executing the corresponding interrupt handling routine through branches).

[0069] 7) The client OS confirms and clears the virtual CPU interface interrupt.

[0070] 8) The physical CPU interface 45 sends a signal to notify that the corresponding physical interrupt can be cleared at GIC 14.

[0071] Figure 3 and Figure 4 An example of handling interrupts targeted at a virtual processor is shown. It should be understood that this is only one example, and other methods may also be used. The techniques described below for routing interrupts to the target virtual processor can be used in systems that use different signaling mechanisms for a specific physical processor at the interface between the GIC and CPU interface 44.

[0072] Figure 5 Examples of VPE configuration table 34 (cacheable virtual processor mapping information) and Virtual Processor Table (VPT) 36 for a specific virtual processor VPEx are shown in more detail. The VPT for a virtual processor VPEx specifies a pending interrupt mapping 50, which maintains a record of the types of interrupts received that target that specific virtual processor. This can be used to track any interrupts that may have been received while the virtual processor VPEx was not resident. When the virtual processor VPEx later becomes resident again, the pending interrupt mapping 50 can be searched, and the VPEx can be used to service any pending interrupts that are still asserted, as if they had occurred while the virtual processor was running.

[0073] VPT 36 may also specify a VPEx location identifier 52, which identifies the physical location to which the virtual processor is mapped (for example, using a physical processor indicator, which identifies the specific physical processor 6 to which the virtual processor is mapped). Note that this does not necessarily imply that the virtual processor is resident and currently being executed, but simply indicates the physical processor that has been allocated to execute the virtual processor (VPEx can be non-resident).

[0074] It should be understood that VPT 36 may also include Figure 5 Other information not shown in the example.

[0075] Each virtual processor may have a separate VPT 36 that can be stored in shared memory 12. A portion of the VPT for the corresponding virtual processor may be cached in the CPU cache or the on-chip memory 8 of a given chip 4, or cached in the internal GIC memory 30 for a specific interrupt controller 14.

[0076] VPE configuration table 34 represents a set of cache information derived from VPTs for multiple virtual processors. VPE configuration table 34 may have multiple entries 54 for each virtual processor. Each entry is based on a physical location indicator for the corresponding VPT 36 for the corresponding virtual processor. When a virtual processor is transferred away from or to a physical processor handled by the corresponding GIC 14, the mapping update circuit 38 may update the cache entry 54 for a given virtual processor. However, if a transfer involving a VPE for another physical processor not handled by the corresponding GIC has been performed, the cache entry may have expired. Therefore, sometimes the cache entry 54 for a given virtual processor in VPE configuration table 34 may be expired compared to the location indicator 52 held in VPT 36 in shared memory 12.

[0077] It should be noted that the location indicator 52 in the VPT can indicate the specific physical processor 6 to which the VPE is mapped. However, in the VPE configuration table 34, the cache information does not actually need to indicate the specific physical processor 6 to which the corresponding VPE is mapped. While this is one implementation, in other implementations, for a VPE mapped to a physical processor on a different chip than the chip associated with the GIC 14 which has a specific instance of VPE configuration table 34, the entry 54 in the VPE configuration table can simply indicate that the VPE is mapped to a different chip and can specify the identifier of another chip to which interrupts for that VPE should be sent. For example, for a VPE mapped to processor 6 on chip 0, the GIC on chip 0 can specify a specific processor indicator, but for other VPEs mapped to processor 6 on one of chips 1, 2, or 3, the GIC can simply indicate a chip identifier that provides enough information to allow the allocator 20 of GIC 14 to route interrupts for the VPE to the appropriate location. Other examples may actually record the specific processor identifier in entry 54 of the VPE configuration table 34.

[0078] It should be understood that VPE configuration table 34 illustrates one example of how the GIC can maintain cached virtual processor mapping information derived from shared virtual processor mapping information stored in VPT 36 in shared memory 12. However, other specific implementations may represent cached virtual processor mapping information in different formats.

[0079] Figure 6 The diagram illustrates the operations performed when the virtual processor VPEy moves from a first location A to a second location B within the system. The first and second locations can be a source processor 6 and a destination processor 6, which can be on the same chip 4 or different chips 4 throughout the data processing system 2.

[0080] At step 60, the GIC 14 associated with chip 4, including location A, receives a virtual processor mapping instruction indicating that virtual processor VPEy is being moved to location B. In response, the GIC associated with location A updates its VPE configuration table 34 to indicate that VPEy is now mapped to the processor at location B (as described above, this could be a specific processor indication, or an indication of which GIC to send an interrupt to if the processor is on a different chip). Additionally, at step 62, the GIC associated with location A forwards the VPE mapping instruction to indicate that VPEy is being transferred to location B. It should be noted that the actual transfer of VPEs between the processors at locations A and B can be performed separately from the signaling executed by interrupt controller 14, or it can be performed in response to an instruction received at the interrupt controller associated with location A. Therefore, the interrupt transfer responsibility can directly trigger virtual processor remapping, or it can be a separate process performed in parallel with the VPE transfer itself to maintain interrupt count tracking information (VPE configuration table 34, VPT 36).

[0081] When a forwarded VPE mapping instruction is received at interrupt controller 14 associated with location B, in step 64, interrupt controller 14 controls its mapping update circuit 38 to update VPT 36 associated with the virtual processor VPEy to indicate that the physical location mapped to the virtual processor is now the destination processor at location B. Therefore, location indicator 52 is updated at location B by interrupt controller 14 associated with location B. The mapping update circuit 38 associated with source location A does not request any update to VPT 36, as this is being handled by the mapping update circuit 38 associated with location B.

[0082] Additionally, in response to the VPE mapping instruction, both interrupt controllers 14 at location A and location B update their respective VPE configuration tables 34 to indicate that VPEy is now mapped to the processor at location B. If locations A and B are actually processors handled by the same interrupt controller, this will be a single update to the VPE configuration table 34 in only one interrupt controller. However, if A and B are on different chips, two different interrupt controllers 14 can update their mapping information in the VPE configuration table 34. However, this update is not broadcast to any other interrupt controllers 14 associated with a chip that does not contain the source processor A or destination processor B in which the virtual processor VPEy is moving. This avoids the overhead of exchanging numerous messages with other interrupt controllers 14, as well as the associated performance costs of blocking the processing of interrupts for the target virtual processor while performing such message transmission. Therefore, this could cause some of the cached information in the interrupt controller 14 to remain in its VPE configuration table 34, which expires after the transfer of the virtual processor VPEy. Thus, it is possible that when an interrupt is received from the device interface 10 and the ITS 26 determines that the interrupt is targeted at the virtual processor VPEy, the interrupt may end up being routed to the wrong GIC 14 because the VPE configuration table 34 has expired.

[0083] Figure 7 and Figure 8 A flowchart is shown illustrating an interrupt retransmission mechanism that can be used to handle the fact that VPE configuration table 34 may have expired. Figure 7 The operation performed at the source interrupt controller 14, labeled GIC-s, is illustrated. This source interrupt controller refers to the interrupt controller 14 that initially receives the interrupt from the device interface 10. The interrupt request is targeted at a specific virtual processor VPEt, referred to as the target virtual processor. If the interrupt must be forwarded to a different interrupt controller (GIC) 14 on a different chip because the target virtual processor VPEt is mapped to a physical processor not handled by the source interrupt controller GIC-s, the interrupt can be forwarded to another interrupt controller, referred to as the target interrupt controller GIC-t. Figure 8 The operation performed at the target interrupt controller GIC-t is shown.

[0084] It should be noted that any given interrupt controller 14 can sometimes act as, for example Figure 7 The source interrupt controller GIC-s is shown, and at other times it acts as such. Figure 8 The target interrupt controller GIC-t is shown. Each interrupt controller 14 can support both functions, depending on whether the interrupt controller receives interrupts directly from the interrupt source on the chip or receives forwarded interrupts from another chip.

[0085] exist Figure 7At step 100, the source interrupt controller GIC-S receives an interrupt request from a given device interface 10 via ITS 26. This interrupt request has an indication of the target virtual processor VPEt to be processed by. At step 102, the allocator 20 of GIC 14 checks its VPE configuration table 34 to see if the target virtual processor VPEt is mapped to a physical processor in the physical processors 6 associated with the source interrupt controller GIC-S. If so, at step 104, the allocator 20 may forward the interrupt request via the redistributor 42 to the processor interface 44 associated with the specific physical processor 6 to which the VPEt is mapped (e.g., using...). Figure 3 and Figure 4 (The method shown).

[0086] On the other hand, if VPE configuration table 34 indicates that VPEt is mapped to a processor on a different chip that GIC 14 is not responsible for, then at step 106, allocator 20 assigns the interrupt request to the target interrupt controller GIC-t, which is responsible for the specific physical processor to which VPEt is mapped as indicated by VPE configuration table 34. Since this is the first attempt to send the interrupt request to the target interrupt controller GIC-t, the danger flag in the forwarded interrupt request is set to 0 to distinguish it from subsequent attempts to resend the same interrupt to the same GIC 14 (if necessary).

[0087] At step 108, the source interrupt controller GIC determines whether, in response to the forwarded interrupt, the target interrupt controller GIC-t has acknowledged the interrupt request (indicating the physical processor to which it is responsible for VPEt) or has rejected the interrupt request (in the case that the target interrupt controller GIC-t believes VPEt is not mapped to any of the physical processors 6 it is responsible for).

[0088] If an acknowledgment response has been received from GIC-t, then at step 110, the source interrupt controller GIC-s can mark the interrupt as completed and the method as finished.

[0089] However, if a rejection response is received from GIC-t, the method proceeds to step 112 after step 108, where the allocator 20 of GIC-s controls the mapping update circuit 38 to request an update to the VPE configuration table 34 in its internal memory 30 based on the VPT 36 stored in shared memory 12. If the VPT 36 for VPEt has been updated after a corresponding piece of information has been recorded in the VPE configuration table 34 of GIC-s, it can now indicate that different CPUs are mapped to VPEt. Once the relevant information has been retrieved again from the VPT 36 associated with the target virtual processor VPEt, the mapping update circuit 38 of GIC-s checks whether the physical processor location associated with VPEt in the VPT 36 is the same as the location previously indicated in the cache entry 54 of the target virtual processor VPEt in the VPE configuration table 34. If not, at step 114, the mapping update circuit 38 updates the cache entry 54 of VPEt in the VPE configuration table 34 to match the information indicated in the acquired VPT. Then, at step 106, the allocator 20 resends the interrupt request to the target interrupt controller GIC-t associated with the physical processor to which VPEt is mapped, as indicated by the updated cached VPE information. Note that since the VPE configuration table 34 has been updated after the first transmission through step 106, this resent interrupt request may be sent to a different interrupt controller compared to the first transmission. If the interrupt request is resent to a different GIC 14 for the previous attempt, a danger flag can be set to 0 to indicate that this is the first attempt to send an interrupt to that particular GIC 14.

[0090] On the other hand, at step 112, if it is determined that the acquired VPT actually indicates the physical location of the same target virtual processor VPEt as previously cached, this would indicate that no VPEt was transmitted after the previous attempt to forward the interrupt request, which could indicate a possible error. To provide a mechanism to avoid deadlock, at step 116, if the cached information still matches the information acquired from VPT 36, the interrupt request is again sent to the same target interrupt controller GIC-t, but this time, the resent interrupt request specifies a danger flag equal to 1 to distinguish it from the first attempt to send the interrupt request to that interrupt controller at step 106.

[0091] If the interrupt is sent along with the danger flag asserted at step 116, then at step 118, the interrupt allocator 20 at GIC-s determines whether the target interrupt controller GIC-t has responded with an acknowledgment response, a rejection response, or an abort response. If the received response is an acknowledgment response, this indicates that the interrupt has now been successfully routed, and therefore, again at step 110, the interrupt can be marked as completed. If a rejection response is received, the method returns to step 112 to attempt again to retrieve the shared processor mapping information from the relevant VPT 36, and again checks whether this information has changed relative to the cache information, since the rejection response indicates that the target interrupt controller GIC-t believes there may be an opportunity to make further attempts subsequently (e.g., because its monitoring circuitry 150 has detected the transmission of VPEt, as further described below).

[0092] However, if an abort response is received from GIC-t, processing of the current interrupt is aborted at step 120, and an error response action is taken. This error response action could be signaling for a further interrupt to allow software intervention and to address the fact that the interrupt received at step 100 could not be successfully routed, or it could be an action taken to reset the data processing system 2 to its default state or trigger a remapping of the virtual processor VPEt to a different physical processor, with the hope that this would allow any errors in VPT 36 to be corrected.

[0093] Figure 8 This shows when the target controller GIC-t receives in Figure 7 The forwarded interrupt sent at step 106 or 116 is an operation performed at the target controller GIC-t. At step 130, the forwarded interrupt (which may be the original attempt to send the interrupt or a retransmission attempt) is received at GIC-t. The forwarded interrupt request specifies the target virtual processor VPEt that will handle the interrupt request. At step 132, the interrupt allocator 20 in the target interrupt controller GIC-t checks its VPE configuration table 34 and determines whether the target VPEt is mapped to a physical processor that the target interrupt controller GIC-t is responsible for. If so, at step 134, the target interrupt controller GIC-t returns an acknowledgment response to the source interrupt controller GIC-s and passes the interrupt request to the redistributor 42 to forward the interrupt request to the physical processor to which the target virtual processor VPEt is mapped.

[0094] If, at step 132, the target interrupt controller GIC-t (based on its VPE configuration table 34) determines that the target virtual processor VPEt is mapped to a physical processor that is not under the responsibility of the target interrupt controller GIC-t, then at step 136, the allocator 20 of GIC-t checks the danger flag set in the forwarded interrupt received from the source interrupt controller GIC-s. If the danger flag is 0, this is the first attempt to send the interrupt request to that particular target interrupt controller GIC-t, and therefore a rejection response is returned at step 138, allowing the source interrupt controller to attempt to retrieve the information from VPT 36 and resend the interrupt.

[0095] If the danger flag is set to 1, this indicates that a previous attempt has been made to send the same interrupt to the same target interrupt controller GIC-t. At step 140, the allocator checks whether the monitoring circuitry 150 of the target interrupt controller GIC-t has monitored the transfer of the target virtual processor VPEt. For example, the monitoring circuitry 150 may have a register and an enable indicator, which can be updated to specify the virtual processor identifier for a particular virtual processor, and the enable indicator indicates whether monitoring is enabled for that virtual processor. When monitoring is enabled for a particular virtual processor, if the interrupt controller 14 (which acts as the interrupt controller responsible for transferring the corresponding virtual processor from its source physical processor or the virtual processor to its destination processor) receives a virtual processor mapping instruction for that particular virtual processor, the monitoring circuitry 150 may set a "transferred" indicator to indicate that a transfer has been detected. If monitoring is disabled, no update to the "transferred" indicator is performed even if a transfer involving a particular virtual processor occurs. In some implementations, the monitoring circuitry 150 may support monitoring only a single virtual processor at a time, while other implementations may monitor transfers of two or more different virtual processors at a time.

[0096] At step 140, if it is determined that monitoring circuit 150 has monitored the transmission of the target virtual processor VPEt, then at step 142, the allocator 20 of GIC-t checks whether the transmission indication is set to indicate that the target virtual processor VPEt was transmitted after monitoring began. If the virtual processor has been transmitted, it is possible that if a rejection message is sent, the next attempt to retransmit the interrupt can find the correct location, so it is worthwhile to allow interrupt processing of that particular interrupt controller to continue without aborting. Therefore, at step 144, the allocator 20 in GIC-t 14 sends a rejection message to GIC-s.

[0097] On the other hand, if the transmission indicator of the monitoring circuit 150 of GIC-t indicates that VPEt has not been transmitted after monitoring started, then at step 146, an abort message is provided to the source interrupt controller GIC-s to indicate that the processing of the interrupt should be aborted, because an error may have occurred, otherwise a deadlock may occur if a continued attempt to resend the interrupt message is made.

[0098] If, at step 140, it is determined that the monitoring circuit 150 of GIC-t has not yet monitored the transmission of the designated target VPEt, then at step 148, the monitoring circuit 150 may begin monitoring the transmission of the target VPEt, assuming that there are idle resources in the monitoring circuit 150 that have not yet been used to monitor another virtual processor. If there are not sufficient resources to allow monitoring of the target virtual processor to begin, then... Figure 8 As shown by the dashed line, step 148 can be omitted. Regardless of whether step 148 is performed, in the absence of existing monitoring of the target virtual processor VPEt, a rejection message is sent as a response to the source interrupt controller GIC at step 144.

[0099] Figure 9 It shows the use of Figure 6 , Figure 7 and Figure 8 The method shown is used to process Figure 1 The diagram shows an example of interrupt propagation in a system. Figure 9 At step 1, a notification is sent to GIC-0 on chip 0: a transfer is initiated whereby a virtual processor VPEq is moved from physical processor CPU-y on chip 0 to its destination physical processor CPU-z on chip 1. At step 1, the actual transfer of the virtual processor and the communication between the interrupt controller occur in parallel. The responsibility for the actual transfer of the VPE can be managed by software and does not necessarily need to be directly executed by the interrupt controller 14 itself.

[0100] exist Figure 9 In step 2, in response to receiving a VPE mapping instruction notifying GIC-0 of the new location of VPEq, the VPE configuration table 34 in the interrupt controller GIC-0 is updated so that it now indicates that VPEq is mapped to CPU-z on chip 1 (similar to...). Figure 6 Step 60).

[0101] exist Figure 9 In step 3, the VPE mapping instruction is forwarded to the interrupt controller GIC-1 responsible for the physical processor, which is the new location of the virtual processor (see...). Figure 6 Step 62), and in response, in Figure 9At step 4, GIC-1 updates both its cache information in the VPE configuration table and the shared information in VPT36 for the virtual processor VPEq within shared memory 12, to indicate that VPEq is now mapped to the physical processor CPU-z (as in...). Figure 6 (Step 64). Therefore, after processing the VPE mapping instruction, interrupt controllers GIC-0 and GIC-1 have now updated the cache mapping information for VPEq, but interrupt controllers GIC-2 and GIC-3 still have outdated information indicating that VPEq is mapped to CPU-y.

[0102] exist Figure 9 In step 5, the interrupt controller GIC-2 on chip 2 receives an interrupt targeting VPEq from device interface 10. Since its cache mapping information indicates that VPEq is mapped to CPU-y, in step 6, GIC-2 forwards the interrupt request to GIC-0, which is responsible for CPU-y, where GIC-2 determines that the target virtual processor VPEq is currently executing (e.g., ...). Figure 7 (as described in step 106). However, upon examining its cache information, GIC-0 determines that it is not responsible for the physical processor mapped to VPEq, therefore... Figure 9 Step 7 (corresponding to) Figure 8 The response at step 138 is a rejection.

[0103] exist Figure 9 At step 8, in response to the rejection response, GIC-2 retrieves VPT 36 associated with the target processor VPEq from memory and determines, based on the re-retrieved information, that VPEq is now mapped to CPU-z on chip 1. At step 9, GIC-2 updates its local cache information based on the new mapping. At step 10, GIC-2 resends the interrupt request to GIC-1 (corresponding to the CPU-z) responsible for CPU-z. Figure 7 The second transmission occurs via step 106. At this point, when GIC-1 checks the mapping information of its local cache, it determines that it is responsible for the physical processor CPU-z, which is responsible for the target virtual processor VPEq. Therefore, at step 11, GIC-1 routes the interrupt to the physical processor CPU-z to handle the interrupt. Figure 9 At step 12, a confirmation response is returned to GIC-2 to indicate that the interruption has now been serviced (corresponding to...). Figure 8 Step 134).

[0104] Figure 10An example is shown where a corruption of VPT 36 stored in shared memory 12 causes an error. For example, this corruption might be due to a particle impact or other hardware malfunction that causes VPT 36 for VPEq to instruct VPEq to be mapped to the physical processor CPU-k on chip 3 when the target virtual processor VPEq is actually mapped to CPU-z on chip 1. If the error cannot be detected, it can lead to a deadlock. Figure 10 This demonstrates how monitoring mechanisms can be used to detect errors and prevent deadlocks.

[0105] exist Figure 10 In the example, it is assumed that... Figure 9 The steps 1 through 4 were performed as shown, therefore Figure 10 These steps are no longer shown. Subsequently, in Figure 10 At step 5, an interrupt targeting VPEq is received from device interface 10 on chip 2. Figure 10 Steps 5 to 9 in the middle and Figure 9 The same as in the previous example, except that the VPT obtained from memory now indicates that VPEq is mapped to CPU-k, rather than the actual location on CPU-z.

[0106] exist Figure 10 At step 10, GIC-2 retransmits the interrupt, this time targeting GIC-3, as indicated by the updated virtual processor mapping information. The danger flag is set to zero because this is the first attempt to send the interrupt to GIC-3 (which corresponds to...). Figure 7 The second transmission proceeds through step 106, because the first transmission occurred in... Figure 10 (Encountered at step 6). In response to the retransmission interrupt, GIC-3 checks its cached virtual processor mapping information in its local memory 30, which identifies that the target virtual processor VPEq is mapped to CPU-y, which is not a processor in a subset of the processors handled by GIC-3. Therefore, in Figure 10 At step 11, GIC-3 responds to GIC-2 with a rejection response. Figure 8 Step 138). In Figure 10 In step 12, if VPEq has changed, GIC-2 re-acquires the VPT for VPEq. Figure 7 Step 112), but at this point GIC-2 detects that the location of VPEq indicated in VPT 36 is the same as the location indicated in its cached virtual processor mapping information (VPT still indicates that VPEq is mapped to CPU-k). Therefore, at this point in step 13, GIC-2 resends the interrupt specifying that the danger flag is equal to 1 ( Figure 7 Step 116). In Figure 10 At step 14, GIC-3 detects that the danger flag is set to 1 and that it has not yet monitored the transmission of the target virtual processor VPEq, therefore it activates its monitoring circuit 150 to monitor any transmission of the target virtual processor VPEq. Figure 10 At step 15, GIC-3 sends a rejection message to GIC-2 again. Figure 8 Step 144).

[0107] This rejection message triggered another attempt to reacquire the VPT in GIC-2 (in Figure 10 Step 16 corresponds to Figure 7 The second transmission is through step 112), and in Figure 10 At step 17, another retransmission interrupt request is sent, which specifies that the danger flag is equal to 1. Figure 7 The second transmission is through step 116).

[0108] At step 18, GIC-3 detects that the retransmission interrupt specified danger flag is 1, and monitoring circuit 150 has been monitoring the transmission of the target virtual processor VPEq, and VPEq is not indicated as having been transmitted after monitoring began. Therefore, in Figure 10 At step 19, an abort message is returned to GIC-2 (corresponding to...). Figure 8 Step 146). In response, in Figure 10 At step 20, GIC-2 triggers an error handling response action. Figure 7 Step 120).

[0109] Therefore, it can be seen that by monitoring VPEq transmissions and providing abort responses, indefinite deadlocks can be avoided when the GIC continuously attempts to resend failed interruption requests to the same destination and these requests are continuously rejected by the receiver.

[0110] One might ask why monitoring in the monitoring circuit 150 of the GIC-3 only begins with the second retransmitted interrupt request sent to the interrupt controller (at step 13), instead of starting with the first retransmitted interrupt request at step 10. One might argue that it would be more efficient to send only a single instance of the retransmitted interrupt request to the GIC-3 before starting monitoring, thus eliminating the second attempt at steps 13 through 15, and triggering an abort in response to the second attempt to retransmit the interrupt to the GIC-3.

[0111] However, since virtual processor transfers are relatively common, asserting interrupts to a given target processor just before a VPE transfer is initiated may occur reasonably frequently. Therefore, if an interrupt remains pending while a transfer has already occurred, it could lead to... Figure 9 and Figure 10 The first retransmission interruption at step 10 is related to the transmission in... Figure 10 Between step 12 and the subsequent reacquisition of the VPT for the target virtual processor, the VPT has changed. In scenarios where a transfer has occurred after step 10, then if in response to... Figure 10 If monitoring begins upon an interrupt sent at step 10, the monitoring can continue indefinitely because the VPT has been updated based on the virtual processor's transfer. Subsequent attempts to resend the interrupt at step 13 can then be routed to different interrupt controllers of the GIC-3 that initiated monitoring. Therefore, the GIC-3 can continue monitoring, and the event to terminate the monitoring may never occur. This wastes monitoring resources in terms of the overhead of checking if any transfers have occurred and can prevent monitoring resources from being used for other virtual processors where errors may have occurred. Additionally, if there are two consecutive transfers to the same virtual machine, this can lead to more frequent abort triggers. Therefore, the overhead of resending the interrupt and the additional handshake of rejection can be less than... Figure 10 Starting monitoring at step 10 instead of step 14 may incur additional monitoring overhead and potential termination costs. Figures 7 to 10 The method shown ensures that monitoring begins with a second attempt to send the same interrupt to the same interrupt controller, and that interrupt processing can be aborted if a third attempt to resend the same interrupt to the same controller exists.

[0112] Although Figures 7 to 10 The diagram illustrates a mechanism that uses a danger flag to signal when an interrupt is resent to the same interrupt controller (which has been attempted once before), but other methods could use a retry count that counts a specific number of resent interrupt requests, or another technique could be used to provide a different encoding for resent interrupt requests compared to the previous attempts.

[0113] The interrupts allocated between the corresponding allocators 20 in the interrupt controller 14 are described relative to the interrupts. Figures 7 to 10 The retry mechanism described herein involves each interrupt distributor 20 being responsible for routing interrupts to physical processors 6 on different integrated circuits. However, this technique can also be used when a single integrated circuit comprises multiple different interrupt distributors.

[0114] For example, when considering only the GIC-14 on a single chip 4 as described above, the reallocators 42, each associated with a separate CPU, can themselves be regarded as the aforementioned "interrupt allocator" or "another interrupt allocator," and in embodiments where there is a possibility that the "on-chip" indication of which particular physical processor on the same chip as the GIC is mapped to a given virtual processor may have expired, then the same technique can be used to reroute interrupts between the corresponding reallocators 42 on the same chip, such that if a processor rejects an interrupt because the virtual processor to which the interrupt is targeted is not mapped to that processor, the interrupt can be resent to the reallocator 42 responsible for the different processor based on re-acquiring the shared mapping information defining the physical processor on which the given virtual processor is based. Therefore, in some examples, the reallocator 42 can perform... Figure 7 and Figure 8 The method shown can be considered as the interrupt allocator described above and other interrupt allocators.

[0115] In this application, the phrase "configured as..." is used to mean that the elements of the device have a configuration capable of performing the defined operations. In this context, "configuration" means the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operations, or a processor or other processing device may be programmed to perform the function. "Configured as" does not mean that the elements of the device need to be changed in any way to provide the defined operations.

[0116] While exemplary embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to those precise embodiments, and various changes and modifications can be made therein by those skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. An interrupt control device, the device comprising: An interrupt allocator is configured to receive interrupt requests to be processed by a target virtual processor, the target virtual processor being selected from a plurality of virtual processors capable of being mapped to a plurality of physical processors, wherein the interrupt allocator is responsible for allocating interrupt requests to a subset of the physical processors. and An interface for communicating with another interrupt allocator responsible for allocating interrupt requests to physical processors other than the subset of physical processors; wherein: In response to the interrupt request, the interrupt allocator is configured to: Based on cached virtual processor mapping information, it is determined whether to route the interrupt request to a physical processor within the subset of physical processors, or to an interrupt allocator within the other interrupt allocators; and When a rejection response is received in response to an interrupt request being routed to one of the other interrupt allocators: A request for an update to the virtual processor mapping information of the cache is based on shared virtual processor mapping information, which is obtained from a storage location accessible by the interrupt allocator and the other interrupt allocator; and The resent interrupt request is routed to another interrupt allocator determined based on the shared virtual processor mapping information.

2. The apparatus according to claim 1, comprising: A mapping update circuit is configured to update the shared virtual processor mapping information in response to a virtual processor transfer command instructing a given virtual processor to be transferred from a source physical processor to a destination physical processor, thereby indicating that the given virtual processor is mapped to the destination physical processor.

3. The apparatus of claim 2, wherein the mapping update circuitry is associated with the interrupt allocator and is configured to update the shared virtual processor mapping information when the destination physical processor is a physical processor in the subset of physical processors under the responsibility of the interrupt allocator.

4. The apparatus of claim 3, wherein the mapping update circuit is configured to update the cached virtual processor mapping information when the source physical processor or the destination physical processor is a physical processor in the subset of physical processors under the interrupt allocator's responsibility.

5. The apparatus of claim 2, wherein the mapping update circuitry is configured to update the shared virtual processor mapping information without broadcasting the update to the shared virtual processor mapping information to one or more additional interrupt allocators that are not responsible for allocating interrupt requests to the source physical processor or the destination physical processor.

6. The apparatus of claim 1, wherein in response to a received interrupt request received at the interrupt distributor from a given additional interrupt distributor: When the cached virtual processor mapping information indicates that the target virtual processor specified by the received interrupt request is mapped to a physical processor in the subset of physical processors handled by the interrupt allocator, the interrupt allocator is configured to pass the received interrupt request to the physical processor to which the target virtual processor is mapped, and to return an acknowledgment response to the given additional interrupt allocator; and When the cached virtual processor mapping information indicates that the target virtual processor specified by the received interrupt request is not mapped to a physical processor in the subset of the physical processors, the interrupt allocator is configured to provide the rejection response to the given additional interrupt allocator.

7. The apparatus of claim 1, wherein when the retransmitted interrupt request is sent to the same interrupt distributor among the other interrupt distributors after at least one early attempt to send the interrupt request or retransmitted interrupt request to the other interrupt distributor: The interrupt allocator is configured to issue the retransmitted interrupt request using a different code than the code used for the first attempt of sending the interrupt request or the retransmitted interrupt request to the other interrupt allocator.

8. The apparatus of claim 1, further comprising monitoring circuitry for monitoring transmissions to a designated virtual processor to maintain tracking information indicating whether the designated virtual processor has received or received a mapped transmission to or from a physical processor in the subset of physical processors associated with the interrupt distributor.

9. The apparatus of claim 8, wherein in response to receiving a retransmitted interrupt request from a given additional interrupt allocator, when at least one early attempt has been made to send the interrupt request or the retransmitted interrupt request to the interrupt allocator, the cached virtual processor mapping information indicates that the target virtual processor specified by the retransmitted interrupt request is not mapped to a physical processor in the subset of the physical processors, and the monitoring circuit has not yet monitored whether the target virtual processor has been subject to the mapping transfer: The interrupt distributor is configured to control the monitoring circuitry to begin monitoring whether the target virtual processor is undergoing the mapping transfer.

10. The apparatus of claim 8, wherein in response to receiving a retransmitted interrupt request from a given additional interrupt allocator, when at least one early attempt has been made to send the interrupt request or the retransmitted interrupt request to the interrupt allocator, the cached virtual processor mapping information indicates that the target virtual processor specified by the retransmitted interrupt request is not mapped to a physical processor in the subset of the physical processors, the monitoring circuit has been monitoring the transmission of the target virtual processor, and the tracking information indicates that the monitoring circuit has not yet detected the mapping transmission for the target virtual processor: The interrupt distributor is configured to provide an abort response to the given additional interrupt distributor.

11. The apparatus of claim 10, wherein in response to receiving the abort response from another interrupt distributor in response to a retransmission of an interrupt request sent to said other interrupt distributor, the interrupt distributor is configured to trigger an error handling response action.

12. The apparatus of claim 8, wherein in response to receiving a retransmitted interrupt request from a given additional interrupt allocator, when at least one early attempt has been made to send the interrupt request or the retransmitted interrupt request to the interrupt allocator, the cached virtual processor mapping information indicates that the target virtual processor specified by the retransmitted interrupt request is not mapped to a physical processor in the subset of the physical processors, the monitoring circuit has monitored the transmission of the target virtual processor, and the tracking information indicates that the monitoring circuit has detected the mapping transmission for the target virtual processor, the interrupt allocator is configured to provide the rejection response to the given additional interrupt allocator.

13. The apparatus of claim 1, wherein the subset of the physical processors comprises at least two physical processors.

14. The apparatus of claim 1, wherein the subset of the physical processors comprises a single physical processor.

15. The apparatus of claim 1, wherein at least one of the additional interrupt distributors is located on a different integrated circuit than the interrupt distributor.

16. The apparatus of claim 1, wherein at least one of the additional interrupt distributors is located on the same integrated circuit as the interrupt distributor.

17. The apparatus of claim 1, further comprising the additional interrupt distributor.

18. An interrupt control device, the device comprising: A means for allocating interrupt requests, the means for allocating interrupt requests being configured to receive interrupt requests to be processed by a target virtual processor, the target virtual processor being selected from a plurality of virtual processors capable of being mapped to a plurality of physical processors, wherein the means for allocating is responsible for allocating interrupt requests to a subset of the physical processors. as well as A means for communicating with other devices, the means for communicating with other devices being used to distribute interrupt requests to physical processors other than the subset of said physical processors; wherein: In response to the interruption request, the allocation device is configured to: Based on cached virtual processor mapping information, it is determined whether to route the interrupt request to a physical processor within the subset of physical processors, or to one of the other allocation devices; and When a rejection response is received in response to the interrupt request being routed to one of the other devices used for allocation: A request for an update to the cached virtual processor mapping information is based on shared virtual processor mapping information, which is obtained from a storage location accessible by the allocation device and the other allocation device; and The resent interrupt request is routed to another allocation device determined based on the shared virtual processor mapping information.

19. An interrupt control method, the method comprising: An interrupt request to be processed by a target virtual processor is received at an interrupt allocator, the target virtual processor being selected from a plurality of virtual processors capable of being mapped to a plurality of physical processors, wherein the interrupt allocator is responsible for allocating the interrupt request to a subset of the physical processors, and a further interrupt allocator is responsible for allocating the interrupt request to physical processors other than the subset of the physical processors. as well as In response to the interrupt request, the interrupt allocator: The virtual processor mapping information based on the cache determines whether to route the interrupt request to one of the physical processors in the subset of physical processors, or to one of the other interrupt allocators. as well as When a rejection response is received in response to an interrupt request being routed to one of the other interrupt allocators: The request is based on shared virtual processor mapping information to update the virtual processor mapping information of the cache, which is obtained from a storage location accessible by the interrupt allocator and the other interrupt allocator. as well as The resent interrupt request is routed to another interrupt allocator determined based on the shared virtual processor mapping information.

Images