Virtual fault domain isolation and recovery method suitable for spirit and qi interconnection system

Abstract

The present application relates to the technical field of chip interconnection, and particularly relates to a virtual fault domain isolation and recovery method suitable for a flexible interconnection system, which comprises the following steps: when a fault event occurs, a state machine is switched to a QUIESCE mode, and a transaction shadow table is maintained; uncompleted transactions in the transaction shadow table are classified and processed, and the state of an access object is listened to; after reconnection, task replay is performed according to the transaction shadow table, and the state machine is switched back to an ACTIVE state. In view of the problem that a third-party node with a fault in the existing flexible interconnection system is prone to cause a chain reaction, an isolation mechanism is added to an IO interconnection chip used for accessing the third-party chip, the state machine of the fault domain is switched to the QUIESCE mode, and a transaction shadow table of uncompleted transactions is established, so that the IO interconnection chip performs proxy processing on part of the transactions, the fault of other devices in the interconnection system is avoided in the fault reconnection stage, and replay is performed after the access object is recovered, so that the risk isolation and recovery process is realized.

Description

Technical Field

[0001] This invention relates to the field of chip interconnect technology, and specifically to a virtual fault domain isolation and recovery method applicable to the Lingqu interconnect system. Background Technology

[0002] UnifiedBus is an interconnection protocol developed by Huawei for supernodes, designed to solve the interconnection challenges of large-scale computing resources. It achieves microsecond-level latency by globally unifying the addressing of all computing, memory, and storage resources into a single logical address space and eliminating the traditional master-slave scheduling architecture.

[0003] Based on this interconnection system, third-party components, chips, and modules can be supported to access the same interconnection domain. The physical carrier form of the above-mentioned access objects can include in-package component interconnection, board-level or system-level chip-to-chip interconnection (C2C), and modular access via bridging, switching, or network extension.

[0004] For example, patent application number PCT / CN2023 / 099530 discloses a data processing method, apparatus, device, and system, relating to the field of data processing. The method includes a scheduler acquiring a job to be processed, controlling at least one supernode according to the resource requirements of the job, and processing the job based on the supernode's global memory pool. The job to be processed is a processing request related to a distributed application. Thus, since the global memory pool is a shared resource among nodes within the supernode, constructed from the storage media of the nodes within the supernode through unified addressing, nodes connected via high-speed interconnect technology within the supernode can share access to this global memory pool to process the job. This avoids MPI-based communication between nodes within the supernode, simplifies the programming model of the application running on the node, effectively reduces I / O communication between nodes, and fully utilizes the performance of the supernode. This effectively shortens data processing time, reduces system energy consumption, and improves system performance.

[0005] For example, patent application CN202511172887.7 discloses a supernode system including an intelligent computing resource pool and a general computing resource pool. The intelligent computing resource pool includes a first pool intra-pool switching module, a first pool inter-pool switching module, and multiple GPUs. The general computing resource pool includes a second pool intra-pool switching module, a second pool inter-pool switching module, and multiple CPUs, thus decoupling heterogeneous computing resources. GPUs within the same intelligent computing resource pool and different intelligent computing resource pools can communicate through the first pool intra-pool switching module. Similarly, CPUs within the same general computing resource pool and different general computing resource pools can communicate through the second pool intra-pool switching module. The system can simultaneously provide intelligent and general computing resources and can expand both the intelligent and general computing resource pools separately, achieving flexible resource allocation and improving utilization. Pooling and deploying intelligent and general computing resources separately in different racks decouples the deployment of heterogeneous resources, increases the GPU density per rack in the intelligent computing resource rack, reduces the impact of single points of failure, and facilitates maintenance.

[0006] However, in actual implementation, the inventors found that if existing third-party chips experience anomalies such as partial reset, power failure, controller failure, heartbeat timeout, protocol violation, link degradation, or context loss after being connected to the system, the main system still faces risks such as transaction suspension, credit blockage, disordered ordering, shared memory pollution, interruption suspension, and service stoppage. Summary of the Invention

[0007] To address the aforementioned problems in existing technologies, a virtual fault domain isolation and recovery method suitable for the Lingqu Interconnection system is provided.

[0008] The specific technical solution is as follows: A virtual fault domain isolation and recovery method applicable to the Lingqu Interconnect system, applicable to the IO interconnect chip under the Lingqu Interconnect system, including: Step S1: Assigning virtual fault domain identifiers to access objects connected to the IO interconnect chip, and then detecting fault events of the access objects; Step S2: When it is determined that the fault event has occurred, the IO interconnect chip switches the state machine of the access object to QUIESCE mode and maintains the transaction shadow table of the access object; the state machine is used by the upper-level interconnect system to confirm the state of the access object; Step S3: Classifying and processing the incomplete transactions in the transaction shadow table, and monitoring the state of the access object; Step S4: When the access object reconnects, replaying the task according to the transaction shadow table, and switching the state machine back to ACTIVE state.

[0009] On the other hand, step S1 includes: step S11: when a new access object is accessed, interactive authentication is performed with the access object and capability information is obtained; step S12: a virtual fault domain ID is assigned to the access object; step S13: multiple logical fault domains are mapped for the virtual fault domain ID, and then the fault event is detected; in step S2, the access object is processed by proxy according to the logical fault domain.

[0010] On the other hand, when performing step S2, a freeze timer is set for the fault event; during the execution of step S3, if the freeze timer is triggered, the execution of step S4 is stopped, and the state machine is switched to FROZEN.

[0011] On the other hand, the fault events include at least one of the following: third-party object active reset indication, abnormal power supply status, clock loss, link degradation exceeding the threshold, heartbeat timeout, protocol message verification failure, context fingerprint inconsistency, response timeout, credit not being recovered for a long time, interruption suspending, or management plane read failure.

[0012] On the other hand, step S2 includes: step S21: when the fault event is determined to occur, switch the state machine to QUIESCE mode; step S22: obtain the incomplete transactions that the access object is running and extract the transaction information; step S23: establish the transaction shadow table according to the transaction information.

[0013] On the other hand, the transaction information includes at least one of the following: transaction identifier, transaction type, address range, queue / channel to which it belongs, transaction stage, completion progress, dependencies, whether proxy completion is allowed, and whether replay after recovery is allowed.

[0014] On the other hand, during the execution of step S21, the IO interconnect chip also performs the following operations on the access object: freezes new credit issuance, reclaims unused credit, marks dangling transaction identifiers, blocks new write requests, closes the writable address window, retains the read-only diagnostic window, freezes queue head and tail updates, locks ownership of the shared buffer, and restricts at least one of the doorbell, interrupt, or event from continuing to be injected into the main system.

[0015] On the other hand, step S3 includes: step S31: classifying the incomplete transactions into three categories according to the transaction shadow table: those that can be waited to be completed, those that can be proxied to be completed, and those that can be replayed; step S32: the IO interconnect chip performs proxy processing on the incomplete transactions of the categories that can be waited to be completed and those that can be proxied to be completed, and suspends the incomplete transactions of the categories that can be replayed to wait for task replay; step S33: during the proxy processing, the incomplete transactions are classified into read and write categories and processed separately.

[0016] On the other hand, step S4 includes: step S41: after the access object reconnects, switch the state machine to the RECOVERING state; step S42: verify the identity of the access object and rebuild the context; step S43: replay the transaction according to the transaction shadow table; step S44: restore the access object and switch the state machine to the REJOIN state to the ACTIVE state in sequence.

[0017] A storage medium includes computer instructions that, when executed by a computer device, perform the virtual fault domain isolation and recovery method described above.

[0018] The above technical solution has the following advantages or beneficial effects: Addressing the problem that faulty third-party nodes in the existing Lingqu interconnection system can easily trigger chain reactions, an isolation mechanism is added to the IO interconnection chip used for accessing third-party chips. This switches the state machine of the fault domain to QUIESCE mode and establishes a transaction shadow table for incomplete transactions, which is then delegated to the IO interconnection chip for partial transaction proxy processing. This avoids triggering faults in other devices within the interconnection system during the fault reconnection phase and allows for replay after the access object recovers, thereby achieving risk isolation and recovery. Attached Figure Description

[0019] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0020] Figure 1 This is an overall schematic diagram of an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of step S1 in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of step S2 in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of step S3 in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of step S4 in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0028] This invention includes: a virtual fault domain isolation and recovery method applicable to the Lingqu Interconnect system, suitable for I / O interconnect chips in the Lingqu Interconnect system, such as... Figure 1 As shown, the process includes: Step S1: Assigning virtual fault domain identifiers to the access objects connected to the IO interconnect chip, and then detecting fault events of the access objects; Step S2: When a fault event is detected, the IO interconnect chip switches the state machine of the access object to QUIESCE mode and maintains the transaction shadow table of the access object; the state machine is used by the upper-level interconnect system to confirm the status of the access object; Step S3: Classifying and processing incomplete transactions in the transaction shadow table, and monitoring the status of the access objects; Step S4: When the access object reconnects, replaying the task according to the transaction shadow table, and switching the state machine back to the ACTIVE state.

[0029] Specifically, to address the issue that faulty third-party nodes in the existing Lingqu Interconnect system can easily trigger chain reactions, an isolation mechanism is added to the IO interconnect chip used to access the third-party chip. This mechanism switches the state machine of the fault domain to QUIESCE mode and establishes a transaction shadow table for incomplete transactions. The IO interconnect chip then handles some of the transactions on behalf of the other devices, thus preventing other devices in the interconnect system from failing during the reconnection phase. The system then replays the data after the accessed object recovers, thereby achieving risk isolation and recovery.

[0030] Specifically, the aforementioned virtual fault domain isolation and recovery method is deployed in the IO interconnect chip under the Lingqu Interconnect system in practical applications. The IO interconnect chip is an interface component used to connect third-party chips and UB interconnect domains, and is also used to complete the aforementioned risk isolation and transaction control process.

[0031] The IO interconnect chip is located at the interconnect entry point of the UB interconnect domain, providing one or more access ports and undertaking virtual fault domain management, transaction takeover, resource isolation, and recovery orchestration functions. Access objects refer to third-party intervening chips that access the UB interconnect domain through the IO interconnect chip, including but not limited to GPUs, NICs, switches, dedicated accelerators, storage controllers, board-level expansion modules, or other heterogeneous objects that can be accessed via UB-related interfaces.

[0032] The UB interconnect domain provides unified data exchange, message interaction, shared resource access, or memory pool access capabilities for objects controlled through access, including shared DDR / HBM, on-chip buffers, message queues, doorbell registers, credit pools, or other resources that are accessed by multiple access objects.

[0033] When a newly added access object interacts with the IO interconnect chip, the two will establish a connection, negotiate capabilities, and conduct access control according to the protocol. This allows the access object to join the UB interconnect domain for operations such as computing resource scheduling and container generation. Simultaneously, the IO interconnect chip assigns a virtual fault domain identifier to the access object. This virtual fault domain identifier is used to manage the access object and specify the logical domain to be used for isolation in case of subsequent failures. The chip then listens for various messages sent by the access object, such as heartbeat messages and response messages, and determines the access object's status based on preset monitoring measurements.

[0034] When a failure event is detected in the access object, the IO interconnect chip switches the state machine of the access object to QUIESCE mode and maintains the transaction shadow table of the access object.

[0035] The state machine includes the following states: ACTIVE, QUIESCE, DRAIN, FROZEN, ISOLATED, RECOVERING, and REJOIN. Specifically: ACTIVE indicates that the access object is providing normal service; QUIESCE indicates that new transactions are no longer accepted, only necessary management access is retained; DRAIN indicates that the access object is waiting for existing transactions that can be naturally closed to complete; FROZEN indicates that the boundary state is frozen, necessary context is retained, and no new credits are issued or new requests are opened to the fault domain; ISOLATED indicates that the address window, message path, and high-risk transactions are completely isolated; RECOVERING indicates that the access object has met the recovery conditions and is performing capability verification and context reconstruction; REJOIN indicates that the access object is reopening data paths, credit quotas, and interrupted paths in stages.

[0036] Based on the above settings, when a fault event occurs, the state of the access object is suspended by switching the state machine of the corresponding fault domain to QUIESCE mode, instead of immediately announcing that the device has disappeared. This allows other devices in the interconnect system to continue to execute the originally agreed access transactions normally. For example, during hardware scheduling of the computing cluster, this mode allows the container to obtain the state of the access object, but will not directly trigger fault handling branches such as container disconnection, thereby reducing the impact on other devices in the UB interconnect domain.

[0037] Meanwhile, for incomplete transactions that are currently in operation, a transaction shadow table is maintained to handle them. For example, some transactions that need to receive responses from the interconnection system, and transactions that have not yet been read or written, are classified and processed by the IO interconnection chip, and the status of the access object is monitored synchronously.

[0038] After the access object completes the fault reconnection, the IO interconnect chip will independently complete the chain establishment, capability negotiation and access control, but the relevant messages will not be sent to the interconnection system, and will only be processed independently by the IO interconnect chip.

[0039] After reconnection is completed, the task is replayed according to the transaction shadow table so that the access object returns to the interruption point, and the state machine is switched back to the ACTIVE state, so that the access object can execute the normal interaction process. In the above process, the perception of the interconnection system itself is weak and the transactions involving the faulty access object will not be interrupted.

[0040] In one embodiment, such as Figure 2 As shown, step S1 includes: step S11: when a new access object appears, perform interactive authentication with the access object and obtain capability information; step S12: assign a virtual fault domain ID to the access object; step S13: map multiple logical fault domains for the virtual fault domain ID, and then detect fault events; in step S2, perform transaction proxy on the access object according to the logical fault domain.

[0041] Specifically, in order to reduce the impact of faulty access objects on the interconnection system, in this embodiment, when a new access object appears, interactive authentication is performed with the access object to complete the chain establishment, capability negotiation and access, and then the capability information of the access object is obtained.

[0042] Capability information is used to establish capability fingerprints, allowed resource sets, recovery policy sets, and isolation policy sets associated with access objects while assigning virtual fault domain IDs to those objects.

[0043] Among them, a set of virtual fault domain IDs are semantically mapped to multiple logical fault domains, including: control plane fault domains: used for configuration, management, status reading and debugging access; data plane fault domains: used for DMA, messages, read and write transactions and data flow paths; interrupt / event fault domains: used for doorbell, MSI, interrupt vector or event notification paths; shared resource visibility fault domains: used for address windows, cache consistency domains, shared memory pools and queue ownership.

[0044] During step S2, the detected fault events are classified to determine the logical fault domains where the faults actually occurred. Freezing, isolation, proxying, and recovery are then performed only on portions of the logical fault domains to maintain the availability of the overall service.

[0045] In one embodiment, such as Figure 3 As shown, step S2 includes: step S21: when a fault event is determined to occur, switch the state machine to QUIESCE mode; step S22: obtain the incomplete transactions that the access object is running and extract the transaction information; step S23: establish a transaction shadow table based on the transaction information.

[0046] The fault events include at least one of the following: third-party object active reset indication, abnormal power supply status, clock loss, link degradation exceeding the threshold, heartbeat timeout, protocol message verification failure, inconsistent context fingerprint, response timeout, credit not being reclaimed for a long time, interruption suspending, or management plane read failure.

[0047] Transaction information includes at least one of the following: transaction identifier, transaction type, address range, queue / channel to which it belongs, transaction stage, completion progress, dependencies, whether proxy completion is allowed, and whether replay after recovery is allowed.

[0048] Specifically, based on the above settings, the I / O interconnect chip obtains the capability fingerprint and allowed resource set of the access object by allocating logical fault domains. Then, according to the access object's capability declaration, it can pre-configure a message monitoring strategy for that access object, such as the chip's internal watchdog, the chip's heartbeat messages, and the access object's responses to the UB interconnect domain. When a monitoring rule is triggered and points to the aforementioned fault event, the I / O interconnect chip disconnects according to the logical fault domain corresponding to the fault event and switches its state machine to QUIESCE mode to suspend the access object, preventing the generation of new transactions.

[0049] When switching the state machine to the QUIESCE, FROZEN, or ISOLATED state, the I / O interconnect chip synchronously performs the following operations on the access object: freezes new credit issuance, reclaims unused credits, marks dangling transaction identifiers, blocks new write requests, closes the writable address window, retains the read-only diagnostic window, freezes queue head and tail updates, locks ownership of the shared buffer, and restricts at least one of the doorbell, interrupt, or event injections into the master system.

[0050] The above operation is used to temporarily freeze access objects that are in a faulty state to avoid inconsistencies in data synchronization.

[0051] In addition, for some access objects that have declared support for cache consistency or shared memory pools during the capability resource acquisition process, the IO interconnect chip can remove them from the consistency domain in case of an anomaly, or downgrade the consistent access to a controlled non-consistent access, in order to prevent dirty data from continuing to spread to the UB interconnect domain or shared resource pool.

[0052] At the same time, it obtains the incomplete transactions that are currently running on the access object and extracts the transaction information. It then establishes a transaction shadow table around the transaction information, which is used by the subsequent IO interconnect chip to proxy the transaction or suspend it and wait for reconnection before replaying it.

[0053] In one embodiment, a freeze timer is set for the fault event during step S2; if the freeze timer is triggered during step S3, step S4 is stopped and the state machine is switched to FROZEN.

[0054] Furthermore, to achieve better equipment control, when the state machine is switched to the QUIESCE state in step S2, a freeze timer is also established simultaneously. The duration of the freeze timer is determined based on the revisit time of most services in the interconnection system. If the access object has not recovered by the time this timer expires, the logical fault domain is switched to FROZEN to freeze its boundary state and prevent any impact.

[0055] At the same time, while reading incomplete transactions, it further determines whether there is dirty data in the read and write data that may pollute the shared resource pool. If so, it further switches the state to ISOLATED state, thereby freezing the address window, message path and high-risk transactions to avoid polluting the shared resources.

[0056] In one embodiment, such as Figure 4As shown, step S3 includes: Step S31: Divide incomplete transactions into three categories according to the transaction shadow table: those that can be waited to be completed, those that can be proxied to be completed, and those that can be recovered and replayed; Step S32: The IO interconnect chip proxied the incomplete transactions of the categories that can be waited to be completed and those that can be proxied to be completed, and suspended the incomplete transactions of the category that can be recovered and replayed to wait for task replay; Step S33: During the proxy processing, incomplete transactions are classified into read and write categories and processed separately.

[0057] Specifically, based on the records in the transaction shadow table, information such as "whether proxy completion is allowed", "whether replay after recovery is allowed", "transaction stage", and "completion progress" can be read from the incomplete transactions. Based on the above key values, incomplete transactions can be divided into three categories according to the matching rules: Waiting-for-completion type: Transactions that have reached the final completion stage or only have a acknowledgment response remaining, and are kept in the DRAIN state until completion information is received; Proxy completion type: Controlled responses, controlled error codes, temporary unavailable states, or default completion states are generated by the IO interconnect chip to prevent upstream master devices or buses from being suspended for a long time; Recoverable replay type: Under the premise of not violating memory consistency and transaction idempotency, the transaction can be re-initiated based on the shadow information after the third-party object is recovered.

[0058] Among them, the wait-to-complete and proxy-complete classes are handled by the IO interconnect chip to generate response messages so that the process can continue, while the resumable replay class is suspended and replayed to the access object after reconnection.

[0059] In the proxy processing, incomplete transactions involving read and write operations are further categorized and handled according to the read and write operations. For write transactions, if the data has not yet been committed to the shared resource, it can be directly discarded or marked as ineffective during isolation; if it has been partially committed, the IO interconnect chip records the visibility boundary and prohibits subsequent writes. For read transactions, if the peer response is unavailable, the proxy module can return a controlled exception or retry after recovery. For transactions related to barriers, fences, atomic operations, or consistency, they can be downgraded, rejected, or confirmed by the local proxy according to the policy.

[0060] In one embodiment, such as Figure 5 As shown, step S4 includes: step S41: after the access object reconnects, switch the state machine to the RECOVERING state; step S42: verify the identity of the access object and rebuild the context; step S43: replay the transaction according to the transaction shadow table; step S44: restore the access object and switch the state machine to the REJOIN state to the ACTIVE state in sequence.

[0061] Specifically, based on the above settings, the service status of the access object itself by the devices in the upper-level interconnection system is controlled downgraded, while in the reconnection process, it is necessary to restore the state switching process of the access object.

[0062] Specifically, when the access object restores its power supply, clock, link, and internal control capabilities, the IO interconnect chip does not immediately release the isolation, but instead enters the RECOVERING state.

[0063] Then, the access objects are verified and their context is reconstructed, including: identity and capability verification: verifying whether the device identifier, capability summary, version fingerprint or recovery token match the state before the anomaly; context reconstruction: restoring the necessary port context, queue mapping, address window, credit budget and service policy; and then replaying the transaction according to the transaction shadow table, including: filtering and replaying transactions marked as replayable; and maintaining the historical state of transactions that have been proxied or terminated and not repeating them.

[0064] After the transaction processing state synchronization is complete, the access objects are restored in stages, including first restoring the management plane, then the low-risk data plane, and finally the interruption plane and high-bandwidth paths. Once restoration is complete, the state machine first transitions from the RECOVERING state to the REJOIN state, and then switches back to the ACTIVE state to complete the state machine loop. In the above process, if an exception occurs, the system re-enters the FROZEN or ISOLATED state without affecting the continued operation of other normally accessed objects.

[0065] Example 1:

[0066] In this embodiment, the third-party AI acceleration chip connects to the I / O interconnect chip via the relevant ports of the UB within the package, and accesses shared HBM resources and message queues through the I / O interconnect chip. During normal system operation, the I / O interconnect chip establishes control plane fault domains, data plane fault domains, and shared resource visibility fault domains for the acceleration chip.

[0067] When the watchdog timer inside the accelerator chip triggers a partial reset, the fault event detection module first detects a heartbeat timeout and a management plane read failure. The virtual fault domain controller switches its state from ACTIVE to QUIESCE, stopping new DMA requests and freezing new credit issuance. Subsequently, the IO interconnect chip enters the DRAIN state, waiting for the read completion message, which has entered the final stage, to be terminated.

[0068] For transactions that are not yet completed, the I / O interconnect chip classifies and processes them according to the transaction shadow table: read transactions that are close to completion are allowed to complete naturally; write transactions that cannot continue to complete and are not suitable for replay are returned with a controlled error; and idempotent and recoverable data retrieval transactions are marked for replay after recovery. At the same time, the address window isolation module closes the write address window of this acceleration chip, while retaining the read-only status window for diagnostics.

[0069] Once the accelerated core has completed its reset and regained management responsiveness, the recovery orchestration module verifies its capability fingerprint and version information, restores queue mappings and credit budgets, reissues transactions marked as replayable in sequence, and then restores services in the order of "management plane—low-risk data plane—high-bandwidth data plane," ultimately returning the state to ACTIVE. Throughout this process, other third-party access objects and shared HBM services remain unaffected.

[0070] Example 2:

[0071] In this embodiment, a third-party board-level module connects to the I / O interconnect chip via a C2C link and is authorized to access the shared DDR memory pool and doorbell register. During operation, the board-level module experiences a momentary power outage, causing its control plane and data plane to fail simultaneously, but the main system requires the shared DDR to continue providing services to other objects.

[0072] After detecting an abnormal power supply status, the IO interconnect chip directly switches the relevant virtual fault domain from ACTIVE to FROZEN, and further into ISOLATED. The credit and sorting freeze module stops assigning new transaction identifiers and credits to this module; the shared resource isolation module closes its address window written to DDR, locks ownership of its unreleased buffer, and freezes its associated doorbell updates to prevent continuous injection of error interrupts.

[0073] If some write transactions are detected as not yet reaching a final visible state on the shared DDR, they are judged as uncommitted and abandoned. If some metadata has been written but the data body is not fully written, it is marked as invalid or rolled back at the metadata layer to prevent other access parties from reading incomplete data. Externally, the main system only perceives that this module is in a "service isolation / recovery pending" state, while the shared DDR remains continuously available to other access objects.

[0074] Example 3:

[0075] In this embodiment, a third-party network acceleration chip is connected to the I / O interconnect chip via a board-level interface. Although its physical link remains in a training-complete state, due to an internal controller malfunction, logical failures such as prolonged failure to reclaim credits, mismatch between completion messages and transaction identifiers, and abnormal interruption storms begin to occur.

[0076] The fault event detection module of the IO interconnect chip determines that the access object has entered a logical failure state based on the number of protocol violations, timeout thresholds, and credit anomaly statistics. The virtual fault domain controller first enters QUIESCE and then FROZEN, directly blocking new high-risk transactions; it performs separate isolation for interrupt / event fault domains, reserving only the management channel for reading error counters.

[0077] Since the PHY remains online in this scenario, traditional "port disconnection" mechanisms struggle to mitigate risks promptly. This invention, however, transforms logical failures into a controllable service state through virtual fault domain boundaries, thereby preventing erroneous completion messages, interruptions, and dangling credits from further polluting the UB interconnect domain. Once the third-party network acceleration chip recovers its internal control logic, its services are restored according to the controlled reentry process.

[0078] A storage medium includes computer instructions that, when executed by a computer device, perform the aforementioned virtual fault domain isolation and recovery method.

[0079] Those skilled in the art will understand that various aspects, or possible implementations of various aspects, of the present invention can be embodied as systems, methods, or computer program products. Therefore, various aspects, or possible implementations of various aspects, of the present invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, etc.), or embodiments combining software and hardware aspects, all collectively referred to herein as "circuit," "module," or "system." Furthermore, various aspects, or possible implementations of various aspects, of the present invention can take the form of computer program products, which are computer instructions stored in memory.

[0080] The memory can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination thereof, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, and portable read-only memory (CD-ROM).

[0081] A processor in a computer reads computer instructions stored in memory, enabling the processor to execute the functional actions specified in each step or combination of steps in a flowchart; and to generate means for implementing the functional actions specified in each block or combination of blocks in a flowchart.

[0082] It should be understood that a processor in a computer can be understood as one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components used to execute the aforementioned computer instructions.

[0083] Computer instructions may be executed entirely on the user's local computer, partially on the user's local computer, as a separate software package, partially on the user's local computer and partially on a remote computer, or entirely on a remote computer or server. It should also be noted that in some alternative implementations, the functions indicated by the steps in the flowchart or the blocks in the block diagram may not occur in the order shown in the diagram. For example, depending on the functions involved, two consecutive steps or blocks may actually be executed approximately simultaneously, or these blocks may sometimes be executed in reverse order.

[0084] Of course, in practical applications, the various components of a computer system are coupled together through a bus system. The bus system is used to enable communication and connection between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0085] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A virtual fault domain isolation and recovery method suitable for a flexible interconnection system, suitable for an IO interconnection chip under a flexible interconnection system, characterized in that, include: Step S1: Assign virtual fault domain identifiers to the access objects connected to the IO interconnect chip, and then detect fault events of the access objects; Step S2: When the fault event is detected, the IO interconnect chip switches the state machine of the access object to QUIESCE mode and maintains the transaction shadow table of the access object; the state machine is used by the upper-level interconnect system to confirm the state of the access object; Step S3: Incomplete transactions in the transaction shadow table are classified and processed, and the state of the access object is monitored; Step S4: When the access object reconnects, the task is replayed according to the transaction shadow table, and the state machine is switched back to ACTIVE state.

2. The method of claim 1, wherein, Step S1 includes: Step S11: When a new access object is accessed, interactive authentication is performed with the access object and capability information is obtained; Step S12: A virtual fault domain ID is assigned to the access object; Step S13: Multiple logical fault domains are mapped to the virtual fault domain ID, and then the fault event is detected; In Step S2, the access object is processed by proxy according to the logical fault domains.

3. The method of claim 1, wherein, When performing step S2, a freeze timer is set for the fault event; if the freeze timer is triggered during the execution of step S3, the execution of step S4 is stopped, and the state machine is switched to FROZEN.

4. The method of claim 1, wherein, The fault events include at least one of the following: third-party object active reset indication, abnormal power supply status, clock loss, link degradation exceeding the threshold, heartbeat timeout, protocol message verification failure, inconsistent context fingerprint, response timeout, credit not being reclaimed for a long time, interruption suspending, or management plane read failure.

5. The method of claim 1, wherein, Step S2 includes: Step S21: When the fault event is determined to occur, switch the state machine to QUIESCE mode; Step S22: Obtain the incomplete transactions that the access object is running and extract the transaction information; Step S23: Establish the transaction shadow table based on the transaction information.

6. The virtual fault domain isolation and recovery method according to claim 5, characterized in that, The transaction information includes at least one of the following: transaction identifier, transaction type, address range, queue / channel, transaction stage, completion progress, dependencies, whether proxy completion is allowed, and whether replay after recovery is allowed.

7. The virtual fault domain isolation and recovery method according to claim 5, characterized in that, During the execution of step S21, the IO interconnect chip also performs the following operations on the access object: freezes new credit issuance, reclaims unused credit, marks dangling transaction identifiers, blocks new write requests, closes the writable address window, retains the read-only diagnostic window, freezes queue head and tail updates, locks ownership of the shared buffer, and restricts at least one of the doorbell, interrupt, or event injections into the main system.

8. The virtual fault domain isolation and recovery method according to claim 1, characterized in that, Step S3 includes: Step S31: Dividing the incomplete transactions into three categories according to the transaction shadow table: those that can be waited to be completed, those that can be proxied to be completed, and those that can be recovered and replayed; Step S32: The IO interconnect chip performs proxy processing on the incomplete transactions of the categories that can be waited to be completed and those that can be proxied to be completed, and suspends the incomplete transactions of the category that can be recovered and replayed for task replay; Step S33: During the proxy processing, the incomplete transactions are classified into read and write categories and processed separately.

9. The virtual fault domain isolation and recovery method according to claim 1, characterized in that, Step S4 includes: Step S41: After the access object reconnects, switch the state machine to the RECOVERING state; Step S42: Verify the identity of the access object and rebuild the context; Step S43: Replay the transaction according to the transaction shadow table; Step S44: Restore the access object and switch the state machine sequentially to the REJOIN state and then to the ACTIVE state.

10. A storage medium comprising computer instructions, characterized in that... When the computer device executes the computer instructions, it performs the virtual fault domain isolation and recovery method as described in any one of claims 1-9.

Images