System and method for monitoring and responding to bus bit error ratio events

Abstract

A computer system includes a bus interface having error correction capabilities. The bus interface includes an error register configured to provide error information related to correctable errors. System software within the computer system is configured to obtain the error information from the error register and calculate a bit error metric based on the error information. A baseboard management controller within the computer system is configured to take action in response to obtaining the bit error metric from the system software and determining that a condition related to the bit error metric has been met.

Description

Background Art

[0001] The present invention generally relates to a bus interface in a computer system. The term "bus interface" refers to any entity or combination of entities that facilitates the transmission of data between components within a computer or between computers. In the context of the technology disclosed herein, a bus interface is alternatively referred to as an interconnection. The present disclosure specifically relates to a bus interface having the ability to detect (multiple) errors in the transmission of combined data and to correct or retry (multiple) transactions that generate (multiple) errors. Some examples of such bus interfaces include: Peripheral Component Interconnect (PCIe), Ultra Path Interconnect (UPI), Quick Path Interconnect (QPI), XGMI, XGMI2, Ethernet, Serial AT Attachment (SATA), Serial Attached Small Computer System Interface (SAS), and any computer bus operating using double data rate (DDR) technology. Of course, the bus interfaces listed above are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure. The technology disclosed herein can be used in conjunction with other bus interfaces not included in the above list.

[0002] When data is transmitted across a bus interface, errors may sometimes occur. Generally speaking, errors can be classified as correctable or uncorrectable. As the name implies, correctable errors can be corrected. Such errors are usually corrected by hardware and do not require software intervention. Although correctable errors may have an impact on performance (e.g., delay, bandwidth), no data / information is lost and the communication link remains reliable. Uncorrectable errors are errors that cannot be corrected, so data / information is lost.

[0003] Some bus interfaces have error detection and correction capabilities. For example, some bus interfaces have error correction codes (ECC), which are mechanisms that can quickly correct single-bit errors. Depending on the architecture, multi-bit errors can sometimes also be corrected. However, generally speaking, the greater the number of errors, the less likely they are to be correctable. Error correction in bus interfaces can involve retransmission of data. If a receiving device detects an error in data already received by the receiving device, the receiving device typically notifies the sending device of the error, and the sending device then retransmits the data.

[0004] Correctable error handling has previously been addressed in a variety of ways. One approach simply involves masking and ignoring errors. However, this is not an effective approach because high levels of correctable errors can degrade performance and ultimately lead to uncorrectable errors and system crashes.

[0005] Another approach involves asynchronous, interrupt-based error handling to log and clear errors. In one implementation of this approach, when an error occurs, a system management interface (SMI) interrupt is generated to the basic input / output system (BIOS). The SMI handler then scans the error status of the input / output (I / O) registers. For any errors found, the SMI handler tells the baseboard management controller (BMC) to log the error occurrence in the event log. However, the error handler may miss errors that occur faster than the time it takes to run the handler code. In addition, logging capacity is sometimes limited, in which case a high error rate may fill or overflow the event log. This can prevent the ability to see other useful log information. To prevent this, logging limits are sometimes implemented. In these cases, once the logging limit is reached, additional errors are simply ignored (as mentioned above, this is not a desirable approach). Even if all errors are logged, it is difficult to use the log data from this approach to implement meaningful warnings and take preemptive actions.

[0006] Some systems utilize a periodic interrupt or polling approach that logs and clears errors. In addition to suffering from many of the same drawbacks discussed previously, this approach typically has a higher latency in responding to errors. If this approach is used to handle all error types, uncorrectable errors may not be handled in a timely manner.

[0007] Some systems combine one of the above-mentioned error handling techniques with a "leaky bucket" approach. With this approach, once a certain number of errors is reached, the errors are logged. For example, in one implementation, the operating system logs each error and notifies the BIOS when N errors are seen (where N can be a relatively large number, such as 500). The BIOS then logs an error event to the BMC event log for every N errors seen by the operating system (OS). While the leaky bucket approach alleviates the problem of filling up the BMC event log and helps provide more actionable events, a drawback of the leaky bucket approach is that it appears as if fewer errors have occurred than have actually occurred (because N errors are logged as a single event). Even though it is known that a single event corresponds to N errors and that the absence of events does not necessarily mean that no errors have occurred, this approach still loses granularity in terms of what can be determined from the BMC event log, necessitating reference to the OS event log. If the OS cannot respond to the error for some reason (as would be the case with an OS-first approach), there may be no indication that an error existed at all. Furthermore, because the leaky bucket method is used in conjunction with one of the above-mentioned error handling techniques, the method suffers from the same drawbacks as previously discussed.

[0008] In view of the foregoing, benefits may be realized through improved techniques for correctable error handling in conjunction with bus interfaces. Summary of the Invention

[0009] According to one aspect of the present disclosure, a computer system is disclosed, comprising a bus interface having error correction capabilities and an error register within the bus interface. The error register is configured to provide error information associated with correctable errors. The computer system also includes system software configured to obtain error information from the error register and a baseboard management controller (BMC), wherein the system software is configured to obtain error information from the error register and the baseboard management controller is configured to take action in response to determining that a condition associated with a bit error metric has been met. The bit error metric is calculated based on error data.

[0010] The bit error metric may comprise a ratio of the number of correctable bit errors that have occurred during a time interval to the total number of bits that have been transmitted across the bus interface during the time interval.

[0011] The error register may be configured to provide an error status of the bus interface. The system software may be configured to run an interrupt service routine in response to a generated interrupt. The interrupt service routine may be configured to scan the bus interface to determine the error status and increment a counter when the error status indicates that a correctable error has occurred.

[0012] The error register may be configured to provide an error counter for the bus interface. System software may be configured to periodically poll the error register to determine the current value of the error counter.

[0013] The system software may be selected from the group consisting of: the computer system's host operating system, the Basic Input / Output System (BIOS), BMC firmware, agent code running on the host operating system, or firmware or another operating system located on an add-in card or other subsystem of the computer system.

[0014] The action may include at least one of logging an indication that a condition related to the bit error metric has been met, notifying the entity that a condition related to the bit error metric has been met, or directly supplying the bit error metric to the entity.

[0015] Determining that the condition has been met may include determining that the bit error metric is not better than a threshold.

[0016] A plurality of thresholds may be defined. The BMC may be configured to take a first action in response to determining that the bit error metric is not better than a first threshold, and to take a second action in response to determining that the bit error metric is not better than a second threshold. The second action may be different from the first action.

[0017] Determining that the condition has been met may include detecting that a statistical parameter has changed by more than a threshold amount within a defined time period, the statistical parameter being related to the historical bit error metric.

[0018] Calculating the bit error metric may include: calculating (E 结束 -E 开始 ) / ΔT ratio. E 开始 The first error counter value at the beginning of the time interval may be indicated. E 结束 A second error counter value at the end of the time interval may be indicated. ΔT may indicate the total number of bits that have been transmitted across the bus interface during the time interval.

[0019] According to another aspect of the present disclosure, a baseboard management controller for a computer system is disclosed. The baseboard management controller includes one or more processors, a memory, and instructions, the memory being in electronic communication with the one or more processors, the instructions stored in the memory. The instructions are executable by the one or more processors to determine whether a condition associated with a bit error metric has been met. The bit error metric is associated with a bus interface within the computer system. The bus interface includes error detection and correction capabilities. The bit error metric depends on how many correctable bit errors have occurred on the bus interface during a time interval. The instructions are also executable by the one or more processors to take an action in response to determining that the condition has been met.

[0020] The bit error metric may comprise a ratio of the number of correctable bit errors that have occurred during a time interval to the total number of bits that have been transmitted across the bus interface during the time interval.

[0021] The bit error metrics may be obtained from a process running within an operating system (OS) or basic input / output system (BIOS) or generated by the BMC based on error data obtained from the OS or BIOS.

[0022] The action may include recording the current value of the bit error metric in a log, or sending a notification message to another entity.

[0023] Determining that the condition has been met may include determining that the bit error metric is not better than a threshold.

[0024] The baseboard management controller may further include additional instructions executable by the one or more processors to take a first action in response to determining that the bit error metric is not better than a first threshold, and to take a second action in response to determining that the bit error metric is not better than a second threshold. The second action may be different from the first action.

[0025] According to another aspect of the present disclosure, a computer-implemented method is disclosed. The method includes receiving a notification related to a bit error metric for a bus interface within a computer system. The bus interface includes error correction capabilities. The bit error metric depends on how many correctable bit errors have occurred on the bus interface during a time interval. The method also includes determining, based on the notification, that a condition related to the bit error metric has been met. The method also includes taking an action in response to determining that the condition has been met.

[0026] The condition may include a migration condition indicating that the computer system may experience a system crash. The computer system may include a workload. The action may include migrating the workload to a different computer system.

[0027] Determining that the condition has been met may include interpreting the notification as an indication that the migration condition has been met.

[0028] The notification may include a bit error metric.Determining that the condition has been met may include determining that the bit error metric is not better than a threshold.

[0029] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0030] Additional features and advantages will be set forth in the description that follows. The features and advantages of the present disclosure may be realized and obtained by the systems and methods particularly pointed out in the appended claims. The features of the present disclosure will become more fully apparent from the following description and the appended claims, or may be learned by practice of the disclosed subject matter as set forth below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To describe the manner in which the above-described and other features of the present disclosure can be obtained, a more detailed description will be given with reference to specific embodiments illustrated in the accompanying drawings. For better understanding, like elements are represented by like reference numerals in the various drawings. Understanding that the drawings depict some example embodiments, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0032] Figure 1 Aspects of a computer system configured to compute and use a bit error metric according to the techniques disclosed herein are shown.

[0033] Figure 1A Aspects of a computer system are shown in which error information provided by an error register takes the form of an error counter.

[0034] Figure 2 An example of a method performed to determine and maintain an error counter for a bus interface is shown.

[0035] Figure 3 An example of a method that may be performed to determine a bit error metric is shown.

[0036] Figure 4 Aspects of a computer system are shown that define multiple conditions related to a bit error metric.

[0037] Figure 5 An example of a cloud computing system is shown in which the techniques disclosed herein may be utilized.

[0038] Figure 6 Aspects of a computer system are shown in which historical information regarding bit error metrics is maintained. DETAILED DESCRIPTION

[0039] The present invention provides a bit error metric for use in conjunction with a bus interface in a computer system. The bit error metric can take the form of a bit error rate. In some embodiments, the bit error metric can be defined as the ratio of the number of correctable bit errors that have occurred during a time interval to the total number of bits that have been transmitted across the bus interface during the time interval.

[0040] According to the present invention, existing device-specific registers that indicate error status and / or counts can be used to calculate bit error metrics. In this context, the term "register" can refer to a hardware register that can be used to store one or more bits of information. An "error register" can refer to a hardware register used to store error information. As an example, a bus interface can include an error register that provides the error status of the bus interface. When a system interrupt is generated, system software (e.g., BIOS or OS) can run an interrupt service routine that scans the bus interface to determine the error status. When the error status indicates that a correctable error has occurred, a counter can be incremented. Alternatively, the bus interface can include an error register that provides an error counter (rather than just an error status). The system software can be configured to periodically poll the error register to determine the number of errors that have occurred over a specific time interval.

[0041] The bit error metric can be used in a variety of ways. Generally speaking, rather than monitoring individual correctable error events, the bit error metric can be monitored. This will make it easier to implement meaningful warnings and take preventive measures on bus interfaces in a computer system.

[0042] In some embodiments, system software (e.g., BIOS or OS) may communicate the bit error metric to the BMC or make it available in a location accessible to the BMC. Alternatively, instead of obtaining the bit error metric from the system software, the BMC may calculate the bit error metric based on error data received by the BMC from the system software. For example, the system software may be designed or capable of indicating only one error at a time to the BMC, and the BMC may be configured to maintain a count of these reported errors along with the time interval and data rate to calculate the bit error metric.

[0043] The BMC can be configured to take one or more actions in response to determining that a condition associated with a bit error metric has been met. In this context, the term "condition" can refer to one or more defined events (such as a bit error metric exceeding a defined threshold). In this context, a condition is met when the defined event(s) associated with the condition have occurred. The term "action" can broadly refer to any type of activity or operation performed. Some specific examples of actions that can be taken in response to determining that a condition associated with a bit error metric has been met (such as notifying one or more other entities that the condition has been met) are described below.

[0044] For example, determining that a condition related to the bit error metric has been met can involve determining that the bit error metric is not better than a threshold (e.g., determining that the bit error metric exceeds a threshold). Actions that can be taken by the BMC include logging an indication that the condition related to the bit error metric has been met, notifying one or more other entities that the condition related to the bit error metric has been met, and directly supplying the bit error metric to one or more other entities. Thus, the BMC can process the bit error metric like other management sensors and implement thresholds for actions.

[0045] In some embodiments, conditions related to historical information regarding bit error metrics may be defined. For example, if a statistical parameter (e.g., mean, standard deviation) that is related to historical bit error metric values ​​has changed by more than a threshold amount within a defined time period, a condition may be defined that specifies that one or more actions should be taken.

[0046] Using the bit error metric in the manner described enables preventive action to be taken with respect to bus interfaces in a computer system. For example, a trend in the bit error metric values ​​may indicate that a failure of a particular bus interface may occur soon. In such a case, preventive action may be taken, such as moving any workload running on the computer system to another computer system.

[0047] In addition to facilitating preventative measures, the techniques disclosed herein can also make the analysis of past failures more meaningful. For example, when a computer system crashes due to a fault in a bus interface, a log created using the techniques disclosed herein can indicate trends in bit error metrics that can be used to predict future failures.

[0048] Figure 1 Aspects of a computer system 100 according to the present disclosure are shown. The computer system 100 includes a bus interface 102. The bus interface 102 includes an error correction component 110. The error correction component 110 includes an error register 104 that provides information about correctable errors.

[0049] Error information may refer to any information or data related to a correctable error. In the described example, the error information takes the form of an error flag 108, which is binary in nature. In other words, the error flag 108 has two possible values: a first value (an "error value") indicating that an error has occurred, and a second value (a "non-error value") indicating that no error has occurred. When the bus interface 102 is initially placed into operation, the error flag 108 may be initialized to the non-error value. Each time an error occurs, the error flag 108 may be changed to the error value. At some point thereafter (e.g., after the error has been appropriately processed), the error flag 108 may be changed back to the non-error value. As another example, as will be described in more detail below, the error information may take the form of an error counter.

[0050] Computer system 100 includes system software 106. System software 106 can be any software component (or group of software components) that performs the functions described herein with respect to system software 106. For example, system software 106 can be the operating system of computer system 100, the basic input / output system (BIOS) of computer system 100, firmware within baseboard management controller (BMC) 116, agent code running on the operating system of computer system 100, or firmware or another operating system located on an add-in card or other subsystem of computer system 100.

[0051] System software 106 includes a component that uses error flags 108 provided by error registers 104 to provide an error counter 134 for bus interface 102 and calculates a bit error metric 112 based on error counter 134. This component may be referred to herein as a bit error metric calculator 114.

[0052] To provide the error counter 134, an interrupt service routine 111 may be utilized. The system software 106 may be configured to execute the interrupt service routine 111 in response to a generated system interrupt. When an error is detected, a system interrupt may be generated. The system interrupt may be generated by a device that detects an error (e.g., the bus interface 102). In response to the generated system interrupt, the interrupt service routine 111 may be configured to scan the bus interface 102 to determine the error status based on the error flag 108 in the error register 104. When the error flag 108 has an error value (indicating that an error has been detected), the interrupt service routine 111 may increment the error counter 134. Thus, the value of the error flag 108 may be captured asynchronously to update the error counter 134.

[0053] In some embodiments, the bit error metric 112 is a ratio of the number of bit errors that have occurred to the total number of bits that have been transmitted. In an implementation of the bit error metric 112 defined in this manner, the bit error metric 112 may be determined relative to a particular time interval 138. For example, determining the bit error metric 112 may involve both determining the number of bit errors that have occurred during the particular time interval 138 and also determining the total number of bits that have been transmitted across the bus interface 102 during the time interval 138.

[0054] The time interval 138 over which the bit error metric 112 is determined may be a defined parameter within the computer system 100. In some embodiments, the time interval 138 may indicate how often the bit error metric 112 is determined. For example, if the time interval 138 is x seconds, the bit error metric 112 may be determined every x seconds. In such an embodiment, if the bit error metric 112 is determined at time t, the bit error metric 112 may next be determined at time t+x. The bit error metric 112 at time t+x may be determined as the ratio of: (i) the number of bit errors that have occurred between time t and time t+x, to (ii) the total number of bits that have been transmitted across the bus interface 102 between time t and time t+x.

[0055] Certain information can be used in conjunction with determining the bit error metric 112. For example, the bit error metric calculator 114 can determine the total number of bits that have been transmitted across the bus interface 102 during a particular time interval 138. This information can be obtained from the bus interface 102 itself. The total number of bits that have been transmitted across the bus interface 102 during a particular time interval 138 can be calculated as the time interval (x in the above example) multiplied by the data rate of the bus interface 102 (i.e., the maximum data rate at which the bus interface 102 is capable of transmitting data). This method assumes that the bus interface 102 does not enter a low-power state (where no data is transmitted) during any measurement interval, and this is a good assumption for many server systems. However, even with the bus low-power state enabled, there is still a risk that the bit error metric 112 may be erroneous on the low end, as the number of bits used in the calculation will always be greater than or equal to the actual number of bits transmitted. Therefore, this method is less likely to register a false positive when a threshold is exceeded. Although it is possible to miss logging or reacting to a true threshold violation using this method, the risk of this occurring should be low for various reasons.

[0056] Certain variables may also be used in conjunction with determining the bit error metric 112, including E 开始 (the value of error counter 134 at the beginning of time interval 138) and E 结束 (The value of error counter 134 at the end of time interval 138). Figure 3 The illustrated method 300 discusses these variables in greater detail.

[0057] The bit error metric 112 may be defined in other ways according to the techniques disclosed herein. For example, in some embodiments, the bit error metric 112 may be defined as the ratio of the total number of bits that have been transmitted to the number of bit errors that have occurred (i.e., the inverse of the definition provided above).

[0058] Alternatively, in some embodiments, the bit error metric 112 may be defined as the number of bit errors per unit time. In implementations where the bit error metric 112 is defined in this manner, determining the bit error metric 112 may simply involve determining the number of bit errors that have occurred during a particular time interval.

[0059] The baseboard management controller (BMC) 116 may be configured to obtain the bit error metric 112 from the system software 106. Alternatively, instead of obtaining the bit error metric 112 from the system software 106, the BMC 116 may calculate the bit error metric 112. For example, the system software 106 (e.g., BIOS) may be designed or capable of indicating only one error at a time to the BMC 116, and the BMC 116 may be configured to keep a count of these reported errors along with the time interval and data rate to calculate the bit error metric 112. Although the system software 106 and the BMC 116 may communicate with each other in a manner similar to the embodiment of the present invention, the BMC 116 may communicate with each other in a manner similar to the embodiment of the present invention. Figure 1 112 resides in the BMC 116, while the portion of the system software 106 that reports data to calculate the bit error metric 112 resides elsewhere (such as in the BIOS).

[0060] BMC 116 may include a component configured to determine whether any action should be taken in response to bit error metric 112. This component may be referred to herein as bit error metric evaluator 118. Specifically, bit error metric evaluator 118 may be configured to determine whether a condition 120 associated with bit error metric 112 has been met. In the depicted example, condition 120 includes a threshold 122, and bit error metric evaluator 118 compares bit error metric 112 to threshold 122. If bit error metric 112 is not better than threshold 122, this means that some action should be taken.

[0061] Depending on the characteristics of bit error metric 112, bit error metric 112 may be superior to threshold 122 by being above or below threshold 122. As described above, in some embodiments, bit error metric 112 may depend on both the number of bit errors that have occurred and the total number of bits that have been transmitted. In embodiments where the value of bit error metric 112 increases as the percentage of bit errors relative to the total number of bits transmitted increases (e.g., if bit error metric 112 is the ratio of the number of bit errors that have occurred to the total number of bits transmitted), then bit error metric 112 is superior to threshold 122 by being below the threshold. However, in embodiments where the value of bit error metric 112 decreases as the percentage of bit errors relative to the total number of bits transmitted increases (e.g., if bit error metric 112 is the ratio of the total number of bits transmitted to the number of bit errors that have occurred), then bit error metric 112 is superior to threshold 122 by being above threshold 122.

[0062] If a condition 120 associated with the bit error metric 112 is satisfied, the BMC 116 may take one or more actions. Many different types of actions may be taken according to the present disclosure. For example, the BMC 116 may include a logging component 126, and the bit error metric evaluator 118 may cause the logging component 126 to record an indication that the condition 120 has been satisfied. Other information, such as a timestamp (indicating the current date and time), may also be recorded in the log 124 along with the indication.

[0063] As another example, the BMC 116 may include a notification component 128. When a condition 120 associated with the bit error metric 112 has been met, the bit error metric evaluator 118 may cause the notification component 128 to send a notification message 130 to one or more other entities, which may be located within or outside the computer system 100. The notification message 130 may include the value of the bit error metric 112. Sending the notification message 130 to another entity allows the entity to track, analyze, and make decisions regarding appropriate actions.

[0064] Figure 1A Aspects of another computer system 100A according to the present disclosure are shown. Figure 1A The computer system 100A shown is similar in many respects to Figure 1 The computer system 100 shown. However, Figure 1A In the illustrated computer system 100A, the error information provided by the error register 104 takes the form of an error counter 108 A. The system software 106 may be configured to periodically poll the error register 104 to determine the current value of the error counter 108 A.

[0065] exist Figure 1A In the embodiment shown, E 开始 refers to the value of error counter 108A at the beginning of time interval 138, and E 结束 Refers to the value of error counter 108A at the end of time interval 138.

[0066] Figure 2 An example of a method 200 that may be performed to provide an error counter 134 for a bus interface 102 is shown. The method 200 may be combined with Figure 1 , in which the error information provided by the error register 104 takes the form of the error flag 108. The method 200 may be executed by the system software 106.

[0067] Method 200 begins when a system interrupt is detected 202. As indicated above, a system interrupt can be generated when an error is detected, and the system interrupt can be generated by the device that detects the error (e.g., bus interface 102). In response to detecting 202 the system interrupt, interrupt service routine 111 scans 204 bus interface 102 to determine an error status based on error flags 108 in error register 104.

[0068] A determination is made 206 as to whether the error flag 108 has an error value or a non-error value. If the error flag 108 has a non-error value, the method 200 ends. However, if the error flag 108 has an error value, the error counter 134 is incremented 208 and the error flag may be cleared 210 to prepare for another error event. The method 200 then ends.

[0069] Figure 3 An example of a method 300 is shown that may be performed (eg, by the bit error metric calculator 114) to determine the bit error metric 112. The method 300 may be combined with Figure 1 The embodiment shown (where the error information provided by the error register 104 takes the form of an error flag 108) or Figure 1A The embodiment shown in which the error information provided by the error register 104 takes the form of an error counter 108A is implemented. In both embodiments, the error counter value is determined at different points in time. Figure 1 In the embodiment shown, the error counter takes the form of an error counter 134 maintained by the system software 106 (and which may be combined with the above Figure 2 Determined and maintained in the manner described). Figure 1A In the embodiment shown, the error counter takes the form of an error counter 108A, which is part of the error register 104 .

[0070] The method 300 includes determining 302 a current value of the error counter at the beginning of the time interval 138 during which the bit error metric 112 is determined. This value may be referred to herein as E 开始 If combined with Figure 1 In the embodiment shown, the method 300 is performed, and the determination 302E is made. 开始 This includes determining the current value of the error counter 134 maintained by the system software 106. Figure 1A In the embodiment shown, the method 300 is performed, and the determination 302E is made. 开始 This includes polling the error register 104 to determine the current value of the error counter 108A.

[0071] Confirm 302E 开始The actions may include: setting an error counter (e.g., Figure 1 In the illustrated embodiment, the error counter 134 is maintained by the system software 106, or in Figure 1A In the embodiment shown, an error counter 108A), which is part of the error register 104, is initialized to a known value (such as zero).

[0072] After waiting 304 until the end of the time interval 138, the method 300 then includes determining 306 the current value of the error counter at the end of the time interval 138. This value may be referred to herein as E 结束 If combined with Figure 1 In the embodiment shown, the method 300 is performed, and the determination 306E is made. 结束 This includes determining the current value of the error counter 134 maintained by the system software 106 at the end of the time interval 138. Figure 1A In the embodiment shown, the method 300 is performed, and the determination 306E is made. 结束 This includes polling the error register 104 to determine the current value of the error counter 108A at the end of the time interval 138 .

[0073] The method 300 also includes determining 308 the number of bits that have been transmitted across the bus interface 102 during the time interval 138. This value may be referred to as ΔT. The bit error metric 112 for the time interval 138 may then be calculated 314 according to equation (1) below:

[0074] (E 结束 –E 开始 ) / ΔT (1)

[0075] exist Figure 1A In the illustrated embodiment, where error counter 108A is part of error register 104, error counter 108A may be configured such that each time an error occurs, error counter 108A increments by a single value. When error counter 108A reaches its maximum value, error counter 108A may "roll over," meaning the value of error counter 108A may return to zero. For example, if error counter 108A includes N bits, the maximum value of error counter 108A may indicate 2. N –1. If the value of error counter 108A is 2 N -1 and another error occurs, the value of error counter 108A may "roll over" to zero (ie, from 2 N –1 changes to zero). When considering E 结束 and E 开始 When the difference between the two is greater, the possibility of rollover should be considered.

[0076] exist Figure 1In the example shown, the bit error metric 112 is compared to a single threshold 122, and action is taken if the bit error metric 112 exceeds the threshold 122. Alternatively, in some implementations, multiple thresholds may be defined, and different actions may be taken depending on which threshold has been crossed.

[0077] Figure 4 4. Aspects of a computer system 400 are shown in which a plurality of conditions 420 associated with a bit error metric 412 have been defined. The conditions 420 include a plurality of thresholds 422a-422b. In the depicted example, two thresholds 422a-422b are defined: an upper non-critical threshold 422a and an upper critical threshold 422b. These thresholds 422a-422b can be defined such that the upper critical threshold 422b is higher than the upper non-critical threshold 422a. Of course, the specific number of thresholds 422a-422b in the depicted computer system 400 is provided for illustrative purposes only and should not be construed as limiting the scope of the present invention.

[0078] In addition to defining multiple thresholds 422a-422b, condition 420 may also define one or more actions to be taken when bit error metric 412 is not better than one or more of thresholds 422a-422b. For example, logging action 440 may be defined in conjunction with each threshold 422a-422b. This means that when bit error metric 412 is not better than any of thresholds 422a-422b (e.g., when bit error metric 412 exceeds any of thresholds 422a-422b), logging component 426 may create an entry in log 424.

[0079] Figure 4 The demonstration can be directed to a bus interface (e.g. Figure 1 4. The example of a log 424 created by the bus interface 102 shown in FIG. 4. The log 424 includes a plurality of entries 446a to 466c. Each of the entries 446a to 466c may correspond to a specific point in time at which the bit error metric evaluator 418 receives the bit error metric 412 and compares the bit error metric 412 to defined thresholds 422a to 422b.

[0080] For the purposes of this example, assume that if the bit error metric 412 exceeds a certain threshold, then the bit error metric 412 is not better than the threshold.

[0081] The first entry 446a in the log 424 includes an interface ID 448 corresponding to the bus interface for which the bit error metric 412 has been calculated. The first entry 446a also includes an indication of the threshold that has been crossed, in this case, the upper non-critical threshold 422a. The first entry 446a also includes a timestamp 450a. The timestamp 450a can correspond to the time when the first entry 446a was created, which should be reasonably close to the time when the bit error metric 412 was determined, resulting in the creation of the first entry 446a. Optionally, in addition to the threshold, each entry can include the actual value of the bit error metric.

[0082] The second entry 446b in the log 424 also includes an interface ID 448 corresponding to the bus interface and an indication of the threshold that has been crossed, which is again the upper non-critical threshold 422a. The second entry 446b also includes a timestamp 450b corresponding to the time when the second entry 446b was created.

[0083] The third entry 446c in the log 424 also includes: an interface ID 448 corresponding to the bus interface and an indication of a threshold value that has been crossed, in this case, the upper critical threshold value 422b. The third entry 446c also includes a timestamp 450c, which corresponds to the time when the third entry 446c was created. A log entry can be created when a threshold indicating that the bit error metric is unfavorably compared to the threshold is crossed, and when the bit error metric is again better than the metric. This can be indicated in another field of each log entry called an "assertion". If the assertion is true, the threshold value is crossed in the positive direction indicating an unfavorable condition. If the assertion is false, the assertion indicates that the threshold value has been crossed in the negative direction, indicating that the condition is again preferred.

[0084] In addition to the logging action 440, a reporting action can also be defined in conjunction with at least some of the thresholds. In the depicted example, the reporting action is not defined in conjunction with the upper non-critical threshold 422a. However, the reporting action 444 is defined in conjunction with the upper critical threshold 422b. As part of the reporting action 444, a notification message 452 can be sent to one or more entities.

[0085] In some embodiments, the techniques disclosed herein may be implemented in the context of a cloud computing system. Figure 5An example of a cloud computing system 500 in which the techniques disclosed herein can be utilized is shown. The system 500 includes multiple data centers 560a through 560c. A first data center 560a is shown having multiple hosts 562a through 562c and a data center manager 564. Each of the hosts 562a through 562c can be used to run zero or more virtual machines at any given time. In the depicted example, the first host 562a is shown having three virtual machines 566a through 566c. The first host 562a is also shown having a virtualization layer 568, which is alternatively referred to as a hypervisor layer. The virtualization layer 568 can be configured to keep the virtual machines 566a through 566c isolated from each other on the first host 562a.

[0086] For simplicity, only three data centers 560a to 560c are shown in the system 500, and only three hosts 562a to 562c are shown in the first data center 560a. However, those skilled in the art will understand that a cloud computing system according to the present disclosure may include more than three data centers, and a data center may include far more than three host machines (e.g., hundreds or thousands of host machines). Similarly, for simplicity, Figure 5 Only the contents of the first data center 560a are shown in FIG. However, the other data centers 560b to 560c may be configured similarly to the first data center 560a. In other words, the other data centers 560b to 560c may also include a data center manager and multiple host machines (and Figure 5 Other components not shown in the simplified diagram). In the first data center 560a, Figure 5 Only the contents of the first host 562a are shown in FIG. However, the other hosts 562b through 560c may be configured similarly to the first host 562a.

[0087] The system 500 further includes a system controller 570 configured to manage the data centers 560a to 560c and the hosts 562a to 562c contained in the data centers 560a to 560c. To enable the system controller 570 to perform various actions related to the hosts 562a to 562c in the system 500, each of the hosts 562a to 562c may include one or more agents configured to communicate with the system controller 570 and perform various actions on its behalf. Figure 5 An agent 572 is shown on a first host 562a, and similar agents may be running on other hosts 562b through 562c.

[0088] The first host 562a includes a bus interface 502 that includes an error register 504. The error register 504 may be in a manner similar to Figure 1 The error register 504 may be configured by the error register 104 in the bus interface 102 shown. For example, the error register 504 may provide information about correctable errors that have occurred on the bus interface 502. The error register 504 may take the form of a binary error flag or an error counter that provides error status.

[0089] The first host 562a also includes a bit error metric calculator 514, which may be implemented in system software 506, such as an operating system (OS) or basic input / output system (BIOS) of the first host 562a. Figure 1 The bit error metric calculator 114 in the illustrated computer system 100 configures the bit error metric calculator 514. For example, the bit error metric calculator 514 may be configured to calculate the bit error metric 512 using any information provided by the error register 504 (eg, error status or error counter).

[0090] The first host 562a also includes a baseboard management controller (BMC) 516, which includes a bit error metric evaluator 518. Figure 1 The bit error metric evaluator 118 in the illustrated BMC 116 may be configured to configure the bit error metric evaluator 518. For example, the bit error metric evaluator 518 may be configured to periodically obtain the bit error metric 512 from the system software 506 and determine whether one or more conditions related to the bit error metric 512 are satisfied. The BMC 516 may take one or more actions in response to determining that certain conditions have been satisfied.

[0091] As described above, in response to determining that a condition associated with the bit error metric 512 has been met, a number of different types of actions can be taken, including recording an indication that the condition has been met in the log 524 and / or notifying one or more other entities that the condition has been met. In some embodiments, the BMC 516 can be configured to notify one or more entities other than the first host 562a that the condition has been met. For example, the BMC 516 can be configured to notify the data center manager 564 that the condition has been met.

[0092] In some embodiments, a migration condition 574 may be defined in relation to the bit error metric 512. The migration condition 574 may define a situation in which a particular host is at risk of system failure, such that the workload on the host should be migrated to another host. In some embodiments, the migration condition 574 may be that the value of the bit error metric 512 is not greater than the threshold 522.

[0093] The BMC 516 can be configured to obtain the bit error metric 512 from the system software 506 (e.g., the BIOS and / or OS of the first host 562a). In response to obtaining the current value of the bit error metric 512, the bit error metric evaluator 518 can be configured to evaluate the migration condition 574 based on the current value of the bit error metric 512. For example, the bit error metric evaluator 518 can be configured to determine whether the current value of the bit error metric 512 is better than the threshold 522. In response to determining that the migration condition 574 has been met (e.g., the current value of the bit error metric 512 is not better than the threshold 522), the BMC 516 can send a notification message 552 to the data center manager 564. Sending the notification message 552 to the data center manager 564 can involve sending the notification message 552a to the agent 572 running on the first host 562a, which can then forward the notification message 552 to the data center manager 564. The data center manager 564 can interpret the receipt of the notification message 552 as an indication that the migration condition 574 has been met. In response, the datacenter manager 564 can migrate one or more workloads on the first host 562a to another host (eg, the second host 562b).

[0094] In this context, the term "workload" can refer to any set of instructions running on a computer system. For example, a workload can include one or more processes, applications, virtual machines, containers, etc. Figure 5 In the context of the illustrated example, the workload may include one or more of virtual machines 566a through 566c running on a first host 562a.

[0095] In an alternative embodiment, the data center manager 564 can evaluate the migration condition 574 instead of (or possibly in addition to) the BMC 516. For example, the BMC 516 can send a notification message to the data center manager 564 that includes the value of the bit error metric 512. In response, the data center manager 564 can determine whether the migration condition 574 has been met (e.g., by determining whether the current value of the bit error metric 512 is better than the threshold 522). If the migration condition 574 has been met, the data center manager 564 can proceed with migrating one or more workloads on the first host 562a to another host.

[0096] In general, the data center manager 564 can be configured to receive a notification related to the bit error metric 512 and, based on the notification, determine whether the migration condition 574 has been met. In some embodiments, the data center manager 564 can interpret the notification itself as an indication that the migration condition 574 has been met. Alternatively, the notification can include the bit error metric 512, and the data center manager 564 can evaluate the migration condition 574 (based on the value of the bit error metric 512) to see whether the migration condition 574 has been met.

[0097] In some embodiments, the actions described above in conjunction with data center manager 564 may alternatively be performed by system controller 570. In such embodiments, notification message 552 may be sent to system controller 570 instead of (or possibly in addition to) data center manager 564.

[0098] Figure 5 The data center manager 564 and the system controller 570 shown are examples of entities that can be notified when a condition related to the bit error metric 512 is met, and one or more actions can be taken in response to such notification. According to the techniques disclosed herein, other entities besides or in addition to the data center manager 564 and the system controller 570 can be notified. For example, in some embodiments, a cabinet manager (i.e., an entity that manages server cabinets within a data center) can be notified when a condition related to a bit error metric corresponding to one of the servers in the cabinet is met.

[0099] In the examples described previously, a single value of a bit error metric (e.g., a current value) has been compared to one or more thresholds to determine whether one or more conditions have been met. However, in some embodiments, conditions involving historical information about the bit error metric may be defined. In other words, conditions involving determining the value of the bit error metric over a certain time period may be defined. For example, conditions involving trend analysis related to the bit error metric may be defined.

[0100] Figure 6 Aspects of a computer system 600 are shown in which historical information 676 is maintained regarding bit error metrics 612. In the depicted example, the computer system 600 includes a component that maintains historical information 676 regarding the bit error metrics 612. This component may be referred to herein as a metric analyzer 684.

[0101] The historical information 676 may include previous values ​​of the bit error metric 612 that have been determined over a certain period of time (e.g., the previous j days / weeks / months, where j may be a configurable parameter). These previous values ​​may be referred to herein as historical bit error metric values ​​678. The historical information 676 may also include statistical parameters that have been calculated relative to the historical bit error metric values ​​678, such as a statistical mean 680 and a standard deviation value 682.

[0102] When a new bit error metric 612 value is generated, the value may be stored with the other historical bit error metric values ​​678. Metric analyzer 684 may occasionally calculate statistical parameters associated with some or all of the historical bit error metric values ​​678.

[0103] The bit error metric evaluator 618 may periodically determine whether any condition 620 related to the bit error metric 612 is satisfied. In some embodiments, the condition 620 may depend, at least in part, on historical information 676 related to the bit error metric 612, rather than or in addition to the current value of the bit error metric 612.

[0104] Metric analyzer 684 can analyze the trend of historical bit error metric values ​​678 and / or statistical parameters related to historical bit error metric values ​​678, such as statistical mean value 680 and standard deviation value 682. Metric analyzer 684 can provide information about the trend of these values ​​to bit error metric evaluator 618. This trend information 686 can be used to determine whether one or more actions should be performed. For example, if trend information 686 indicates that a fault of the bus interface may occur quite quickly, one or more other entities can be notified so that preventative action can be taken.

[0105] In some embodiments, a condition 620a may be defined that is related to a change in a particular statistical parameter 688 (e.g., statistical mean, standard deviation) associated with the historical bit error metric value 678. For example, if the statistical parameter 688 associated with the historical bit error metric value 678 changes by more than a threshold amount 690 within a defined time period 692 (assuming the change indicates the presence of more errors), the condition 620a may specify that one or more actions should be taken.

[0106] In some embodiments, the above combination Figure 6 The components and functions described may be implemented in a baseboard management controller (BMC). Figure 6The components and functions described may be implemented within another entity (such as a data center manager or system controller). In general, any entity that determines whether a condition related to a bit error metric has been satisfied may include the above combined Figure 6 Components and functions described.

[0107] For simplicity and clarity, some aspects of the present disclosure have been described with respect to a single bus interface. However, the techniques disclosed herein can be applied to computer systems that include multiple bus interfaces. In such a computer system, a bit error metric can be determined for multiple bus interfaces in the system.

[0108] As described above, examples of bus interfaces include Peripheral Component Interconnect (PCIe), Ultra Path Interconnect (UPI), Quick Path Interconnect (QPI), XGMI, XGMI2, any computer bus operating using double data rate (DDR) technology, Ethernet, Serial AT Attachment (SATA), and Serial Attached Small Computer System Interface (SAS).

[0109] PCIe is a high-speed serial computer expansion bus standard. PCIe is commonly used as a motherboard interface for various components in a computer system, such as graphics cards, hard drives, solid-state drives (SSDs), wireless communication interfaces (e.g., interfaces that support the IEEE 802.11 standard, also known as Wi-Fi), and wired communication interfaces (e.g., Ethernet). A PCIe system includes a processor and memory, a switch fabric, and a root complex device. The switch fabric includes one or more switch devices, and the root complex device connects the processor and memory to the switch fabric. PCIe endpoints (i.e., devices connected to a computer system via a PCIe system) can be connected to the root complex device and / or the switch fabric.

[0110] UPI and QPI are high-speed, point-to-point interconnects used in certain processors developed by Intel. UPI and QPI provide high-speed, point-to-point links within and outside the processor. These links facilitate data transfer by connecting components within the processor (such as distributed shared memory, internal processor cores, I / O hubs, etc.). UPI and QPI links within a processor can also provide connections to other processors.

[0111] XGMI and XGMI2 are interconnects developed by Advanced Micro Devices. XGMI and XGMI2 are commonly used to connect graphics processing units (GPUs). XGMI and XGMI2 allow GPUs to be clustered to increase processing power.

[0112] Computer buses that operate with DDR transfer data on both the rising and falling edges of the clock signal. This technology has been used in a variety of systems with high data transfer speed requirements, including microprocessor front-side buses, processor interconnects, main memory, and graphics memory.

[0113] SATA and SAS are computer bus interfaces commonly used to connect host bus adapters to mass storage devices such as hard drives, SSDs, and optical drives.

[0114] Ethernet is a family of network technologies commonly used in computer networks, including local area networks (LANs). Unshielded twisted pair (UTP) cables and coaxial cables are commonly used for Ethernet LANs. Current Ethernet networks are capable of data transmission rates of hundreds of gigabits per second.

[0115] Some aspects of the present disclosure relate to the basic input / output system (BIOS) and / or operating system (OS) of a computer system. BIOS is non-volatile firmware in a computer system. BIOS performs hardware initialization during the process of booting (or starting) a computer system. In typical operation, the BIOS initializes and tests the system hardware components, then loads the boot loader from a mass storage device, which initializes the operating system. BIOS may also be referred to as the Unified Extensible Firmware Interface (UEFI). An operating system (OS) is system software that manages computer hardware and software resources and provides common services for computer programs.

[0116] Some aspects of the present disclosure relate to a baseboard management controller (BMC), which is a dedicated microcontroller embedded on the motherboard of a computer system. Different types of sensors built into the computer system report parameters such as temperature, cooling fan speed, power supply status, operating system (OS) status, etc. to the BMC. The BMC monitors the sensors and can send an alert to another entity (e.g., another computer system operated by a system administrator) if any parameter does not remain within preset limits. The administrator can also communicate with the BMC remotely to take certain corrective actions (such as resetting or restarting the system).

[0117] Unless explicitly described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules, components, etc. may also be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory computer-readable medium having computer-executable instructions stored thereon, which, when executed by at least one processor, perform some or all of the steps, operations, actions, or other functions disclosed herein. Instructions may be organized into routines, programs, objects, components, data structures, etc., which may perform specific tasks and / or implement specific data types, and which may be combined or distributed as needed in various embodiments.

[0118] The processor may be a general-purpose single-chip or multi-chip microprocessor (e.g., Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a special-purpose microprocessor (e.g., Digital Signal Processor (DSP)), a microcontroller, a programmable gate array, etc. The processor may be referred to as a central processing unit (CPU).

[0119] As used herein, memory can be any electronic component capable of storing electronic information. For example, memory can be embodied as random access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included in a processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) memory, registers, and the like, including combinations thereof.

[0120] Instructions and data may be stored in memory. Instructions may be executed by a processor to implement some or all of the methods, steps, operations, actions, or other functions disclosed herein. Executing instructions may involve using data stored in memory. Unless otherwise specified, any of the various examples of modules and components described herein may be implemented in part or in whole as instructions stored in memory and executed by one or more processors. Any of the various data examples described herein may be among the data stored in memory and used during the execution of instructions by the processor.

[0121] The steps, operations, and / or actions of the methods described herein may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps, operations, and / or actions is required for proper functioning of the method being described, the order and / or use of specific steps, operations, and / or actions may be modified without departing from the scope of the claims.

[0122] In one example, the term "determining" (and its grammatical variations) encompasses a variety of actions, and thus, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), ascertaining, etc. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0123] The terms "comprising," "including," and "having" are intended to be inclusive and mean that additional elements may be present in addition to the listed elements. Furthermore, it should be understood that references to "one embodiment" or "an embodiment" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. For example, any element or feature described with respect to an embodiment herein may be combined with any element or feature of any other embodiment described herein, where compatible.

[0124] The present disclosure may be embodied in other specific forms without departing from the spirit or features of the present disclosure. The described embodiments are to be considered illustrative rather than restrictive. Therefore, the scope of the present disclosure is indicated by the appended claims rather than by the foregoing description. Changes within the meaning and range of equivalence of the claims are to be included within the scope of the claims.

Claims

1. A computer system comprising: Bus interface with error correction capability; an error register within the bus interface, wherein the error register is configured to provide error information associated with a correctable error; system software configured to obtain the error information from the error register; as well as a baseboard management controller (BMC) configured to take action in response to determining that a condition related to a bit error metric has been satisfied, wherein the bit error metric is calculated based on the error information and comprises a ratio of a number of correctable bit errors that have occurred during a time interval to a total number of bits that have been transmitted across the bus interface during the time interval.

2. The computer system of claim 1 , wherein: The error register is configured to provide an error status of the bus interface; and The system software is configured to execute an interrupt service routine in response to a generated interrupt, the interrupt service routine being configured to scan the bus interface to determine the error status and to increment a counter when the error status indicates that a correctable error has occurred.

3. The computer system of claim 1 , wherein: The error register is configured to provide an error counter for the bus interface; and The system software is configured to periodically poll the error register to determine the current value of the error counter.

4. The computer system of claim 1 , wherein the system software is selected from the group consisting of: a host operating system of the computer system, a basic input / output system (BIOS), BMC firmware, agent code running on the host operating system, or firmware or another operating system located on an add-in card or other subsystem of the computer system.

5. The computer system of claim 1 , wherein the action comprises at least one of: logging an indication that the condition associated with the bit error metric has been met; notifying an entity that said condition associated with said bit error metric has been met; or The bit error metric is supplied directly to the entity.

6. The computer system of claim 1 , wherein determining that the condition has been met comprises: It is determined that the bit error metric is not better than a threshold.

7. The computer system of claim 1 , wherein a plurality of thresholds are defined, and wherein the BMC is configured to: taking a first action in response to determining that the bit error metric is not better than a first threshold; and A second action is taken in response to determining that the bit error metric is not better than a second threshold, wherein the second action is different from the first action.

8. The computer system of claim 1 , wherein determining that the condition has been met comprises: A statistical parameter is detected to have changed by more than a threshold amount within a defined time period, the statistical parameter being related to the historical bit error metric.

9. The computer system of claim 1 , wherein: Calculating the bit error metric includes: calculating (E 结束 -E 开始 ) / ΔT ratio; E 开始 indicating a first error counter value at the beginning of a time interval; E 结束 indicating a second error counter value at the end of the time interval; and ΔT indicates the total number of bits that have been transmitted across the bus interface during the time interval.

10. A baseboard management controller for a computer system, comprising: one or more processors; a memory in electronic communication with the one or more processors; as well as instructions stored in the memory, the instructions being executable by the one or more processors to: determining that a condition associated with a bit error metric has been satisfied, wherein the bit error metric is associated with a bus interface within a computer system, wherein the bus interface includes error detection and correction capabilities, and wherein the bit error metric comprises: a ratio of a number of correctable bit errors that have occurred on the bus interface during a time interval to a total number of bits that have been transmitted across the bus interface during the time interval; as well as An action is taken in response to determining that the condition has been met.

11. The baseboard management controller of claim 10, wherein the bit error metric is: From a process running within the operating system (OS) or basic input / output system (BIOS); or Generated by the baseboard management controller based on error data acquired from the OS or the BIOS.

12. The baseboard management controller according to claim 10, wherein the action comprises: Recording the current value of the bit error metric in a log; or Sends a notification message to another entity.

13. The baseboard management controller of claim 10, wherein determining that the condition has been met comprises: It is determined that the bit error metric is not better than a threshold.

14. The baseboard management controller according to claim 10, further comprising: Additional instructions executable by the one or more processors to: taking a first action in response to determining that the bit error metric is not better than a first threshold; as well as A second action is taken in response to determining that the bit error metric is not better than a second threshold, wherein the second action is different from the first action.

15. A computer-implemented method comprising: receiving a notification related to a bit error metric for a bus interface within a computer system, wherein the bus interface includes an error correction capability, and wherein the bit error metric comprises a ratio of a number of correctable bit errors that have occurred on the bus interface during a time interval to a total number of bits that have been transmitted across the bus interface during the time interval; determining, based on the notification, that a condition associated with the bit error metric has been satisfied; as well as An action is taken in response to determining that the condition has been met.

16. The method according to claim 15, wherein: The conditions include a migration condition indicating that the computer system may experience a system crash; The computer system includes a workload; and The action includes migrating the workload to a different computer system.

17. The method of claim 16, wherein determining that the condition has been met comprises: The notification is interpreted as an indication that the migration condition has been met.

18. The method of claim 17, wherein the notification includes the bit error metric, and wherein determining that the condition has been met includes determining that the bit error metric is not better than a threshold.

Images