Queue-level automatic answering method and device, equipment and storage medium

Abstract

The invention discloses a queue-level fault automatic response method and device, equipment and a storage medium. The method comprises the following steps: recording obtained queue state information into a queue pair state record table; polling the queue pair state record table through a queue pair automatic response state machine; recording a queue identifier of which the running state is fault timeout to a timeout queue record table, generating an automatic response for the fault timeout queue, and feeding back the automatic response to a near-end chip requester; and when the received actual response fed back by the far-end chip corresponds to the fault timeout queue, filtering and discarding the actual response. When the request sent by the near-end chip requester to the far-end chip has the queue fault, the automatic response is generated aiming at the fault queue and fed back to the near-end chip requester, and the normal queue is not interfered, so that the network fault does not influence the normal work in the near-end chip, the normal operation of other chips connected with the near-end chip is ensured, and the service life of the near-end chip is prolonged. Therefore, the normal operation of the whole system is realized.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a queue-level automatic answering method, apparatus, device and storage medium. Background Art

[0002] In large-scale AI training clusters, there are a large number of GPU chip computing cards, which can reach 100,000. Each GPU chip computing card is interconnected with other GPU chip computing cards in the server cabinet through 16 expansion ports. The number of interconnected cables can be as many as 1600,000. As the system scale and complexity increase, the number of failed GPU links per unit time increases significantly.

[0003] Currently, when a single chip link fails in an AI training system, it usually affects the normal operation of other chips. However, AI training systems generally require high reliability and high availability. Therefore, when a single chip fails in an AI system, how to ensure fault convergence and isolation of the AI training system to maintain normal operation is an urgent problem that needs to be solved. Summary of the Invention

[0004] The present invention provides a queue-level fault automatic response method, so as to realize automatic response to queue-level fault when a fault occurs in a queue sent by a near-end chip requester.

[0005] According to a first aspect of the present invention, a queue-level fault automatic response method is provided, comprising: monitoring a queue sent by a proximal chip requester to a remote chip based on distributed memory semantic communication, and recording the acquired queue status information in a queue pair status record table, wherein the queue status information includes a queue timestamp and a number of flights;

[0006] Polling the queue pair status record table through the queue pair automatic answering state machine, and determining the operation status of each queue according to the queue status information, wherein the operation status includes fault timeout or normal;

[0007] Recording the queue identifier of the operation state of the fault timeout queue into the timeout queue record table, and generating an automatic response for the fault timeout queue and feeding it back to the proximal chip requester;

[0008] When the actual response fed back by the remote chip is received and it is determined by referring to the timeout queue record table that the actual response corresponds to a queue with fault timeout, the actual response is filtered and discarded.

[0009] According to another aspect of the present invention, a queue-level fault automatic response device is provided, comprising: a queue monitoring module, configured to monitor a queue sent by a proximal chip requester to a remote chip based on distributed memory semantic communication, and record the acquired queue status information in a queue pair status record table, wherein the queue status information includes a queue timestamp and a number of flights;

[0010] An operation status determination module, configured to poll the queue pair status record table through the queue pair automatic answering state machine, and determine the operation status of each queue according to the queue status information, wherein the operation status includes fault timeout or normal;

[0011] An automatic response module, configured to record the queue identifier of the fault timeout queue in a timeout queue record table, and generate an automatic response for the fault timeout queue and feed it back to the proximal chip requester;

[0012] The actual response discarding module is used to filter and discard the actual response when receiving the actual response fed back by the remote chip and determining with reference to the timeout queue record table that the actual response corresponds to the queue of fault timeout.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method described in any embodiment of the present invention when executed.

[0018] According to the technical solution of the embodiment of the present invention, when a queue failure occurs in the request sent to the remote chip by the proximal chip requester due to a port failure, an automatic response will be generated for the fault queue and fed back to the proximal chip requester, and the fault queue will not interfere with other normal queues. Therefore, the network failure will not affect the normal operation of the proximal chip, and fault convergence and isolation are achieved. In addition, the proximal chip can be kept alive through automatic response, which can ensure the normal operation of other chips connected to the proximal chip, so as to achieve the normal operation of the entire system, thereby improving the reliability and availability of the system.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flowchart of a queue-level fault automatic response method provided in accordance with the first embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of fault types of a network system provided according to the first embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of an application framework provided according to embodiment 1 of the present invention;

[0024] Figure 4 This is a flowchart of another queue-level fault automatic response method provided according to an embodiment of the present invention;

[0025] Figure 5 This is a flowchart of a queue-level fault automatic response method provided in accordance with the second embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of a queue-level fault automatic response device provided according to the third embodiment of the present invention;

[0027] Figure 7 It is a structural diagram of an electronic device provided by the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Example 1

[0031] Figure 1 A flowchart of a queue-level fault automatic response method is provided for the first embodiment of the present invention. This embodiment is applicable to automatically responding to a queue with a request transmission failure of a proximal chip. The method can be executed by a queue-level fault automatic response device, which can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0032] Step S101 : monitoring a queue sent by a proximal chip requester to a remote chip based on distributed memory semantic communication, and recording the obtained queue status information into a queue pair status recording table.

[0033] Specifically, this application is mainly aimed at the situation where chips in the same system use queue pairs (QP) to perform RDMA communication in the Remote Direct Memory Access (RDMA) scenario, such as Figure 2 The figure shows a schematic diagram of the failure types of the network system, where the solid line represents the physical circuit connection and the dotted line represents the queue pair (QP) logical connection. Therefore, the failure types of the network system mainly include port failure and QP failure. When the connection of the entire system port is disconnected, all QPs cannot function normally. When the port is normal and a QP has an abnormal connection, in the networking scenario, a port failure will inevitably cause the QP failure of other logical connection chips. The queue failure mentioned in this application mainly refers to the QP failure.

[0034] Optionally, the queue sent by the proximal chip requester to the remote chip based on distributed memory semantic communication is monitored, and the obtained queue status information is recorded in the queue pair status record table, including: real-time monitoring of the queue pairs transmitted by the proximal chip requester to the remote chip through the sending data path based on distributed memory semantic communication, wherein each queue pair is marked with a queue identifier; obtaining the queue identifier and detection timestamp of each queue pair, and determining the target queue based on the queue identifier; updating the queue timestamp of the target queue in the queue pair status record table based on the detection timestamp, and incrementally updating the flight quantity of the target queue in the queue pair status record table based on the number of queue pairs subordinate to the target queue.

[0035] Among them, such as Figure 3 The figure shows a schematic diagram of the application framework involved in this embodiment, and the application framework can be a structure located in the chip. For example, when the structure is located in the proximal chip, when the proximal chip fails to send a request message in the form of a queue to the remote chip, the structure can be used to automatically respond within the proximal chip to avoid the proximal chip being unable to obtain a response from the remote chip due to a request message failure, causing the proximal chip to fail. Figure 3 As shown, the application framework mainly includes a queue pair status record table, a queue pair automatic response state machine, a multi-queue first-in-first-out component, a timeout queue record table, and a response filter. Of course, this embodiment is only an example and does not limit the specific content contained in the application framework. In this embodiment, when the proximal chip and the remote chip transmit data based on distributed memory semantic communication, the proximal chip will send a request to the remote chip in the form of a queue, and when the proximal chip sends a request data stream to the cloud chip through the sending data path, Figure 3The application framework shown will monitor the queue pairs flowing through the sending data path in real time. Each queue pair is marked with a queue identifier, so the queue pairs with the same queue identifier belong to the same queue request data. This implementation does not limit the specific number of queues contained in the request data sent by the proximal chip to the distal chip. The application framework obtains the queue identifier and detection timestamp of each queue pair flowing through the sending data path through monitoring, determines the target queue to which the queue pair belongs based on the queue identifier, and updates the queue timestamp of the target queue recorded in the queue pair status record table based on the detection timestamp. For example, when it is determined that the queue identifier marked by the queue pair a that has recently flown through the sending data path is 1, and the detection time is T1, the queue timestamp TimeStamp of queue 1 is updated to T1 in the queue pair status record table, and the outstanding counter of queue 1 is increased by 1. The outstanding counter refers to the number of unfinished items for which the near-end chip requester sent a request to the far-end chip but did not receive a response from the far-end chip. Therefore, the queue timestamp and the outstanding counter of each queue are specifically recorded in the queue pair status record table, and the queue timestamp and the outstanding counter of each queue are constantly changing during the monitoring period. The following Table 1 shows an example of the queue pair status record table at time t:

[0036] Table 1

[0037] queue Queue timestamp Number of flights Queue 1 1：1：1：00 2 Queue 2 1：1：1：10 1 Queue 3 1：1：1：04 4 … … …

[0038] Table 1 does not limit the number of queues, and the status record table shown in Table 1 is continuously updated over time.

[0039] Optionally, after recording the obtained queue status information in the queue pair status record table, the method further includes: obtaining key information and queue identifiers of the queue pair; and saving the corresponding relationship between the key information and the queue identifiers in a multi-queue first-in-first-out component.

[0040] Specifically, after the application framework in this embodiment records the obtained queue status information into the queue pair status record, it will also obtain the key information of each queue pair, such as meta, and queue identification, and save the correspondence between the key information and the queue identification into the multi-queue first-in-first-out component. The multi-queue first-in-first-out component in this embodiment can realize logically multiple queues, share a static random access memory (SRAM) buffer, and record the order of each element of the multiple queues to realize the queue first-in-first-out function. Since a queue can include multiple queue pairs, the multi-queue first-in-first-out component records all the key information contained in each queue in units of queues.

[0041] In a specific implementation, the queue pair status record table already records queue 1's queue timestamp as 1:1:1:00 and its in-flight quantity as 2; queue 2's queue timestamp as 1:1:1:10 and its in-flight quantity as 1; and queue 3's queue timestamp as 1:1:1:04 and its in-flight quantity as 4. When queue pair b, labeled queue 1, is detected flowing through the transmit data path at detection time 1:1:1:10, the queue timestamp of queue 1 is correspondingly updated to 1:1:1:10 and its in-flight quantity is updated to 3 in the queue pair status record table. In addition, after the queue pair status record table is updated, the key information contained in queue pair b will be extracted and saved in queue 1 of the multi-queue first-in-first-out component. Of course, in this embodiment, only the update method of the queue pair status record table and the multi-queue first-in-first-out component when the sending data path passes through queue pair a is used as an example to illustrate. When other queue pairs are passed through the sending data path, the corresponding queue pair status record table and the multi-queue first-in-first-out component are updated in roughly the same way, and will not be repeated in this embodiment.

[0042] Step S102: polling the queue pair status record table through the queue pair automatic answering state machine, and determining the running status of each queue according to the queue status information.

[0043] Optionally, the queue pair status record table is polled by the queue pair automatic answering state machine, and the operating status of each queue is determined according to the queue status information, including: obtaining the queue status information of each queue by polling the queue pair status record table through the queue pair automatic answering state machine; judging for each queue whether the number of flights in the queue status information is 0, if so, directly determining that the operating status of the currently polled queue is normal, otherwise, determining the operating status of the currently polled queue according to the current polling time and the queue timestamp in the queue status information.

[0044] Optionally, the operating status of the currently polled queue is determined based on the current polling time and the queue timestamp in the queue status information, including: obtaining a pre-configured time threshold and the time difference between the current polling time and the queue timestamp; judging whether the time difference is greater than the time threshold; if so, determining that the operating status of the currently polled queue is a fault timeout; otherwise, determining that the operating status of the currently polled queue is normal.

[0045] Specifically, in this embodiment, while the application framework updates the queue pair status record table based on the monitoring status of the transmit data path, the queue pair auto-response state machine polls the obtained queue pair status record table to obtain queue status information for each queue recorded in the queue pair status record table. Based on the status information obtained from the polling, the machine determines the operating status of each queue, where the operating status includes a fault timeout or normal. For example, if the flight count included in the status information of queue 1 is determined to be 0, it means that the request information sent by the near-end chip requester to the remote chip, namely, the queue pairs in queue 1, have all received responses from the remote chip. At this point, the operating status of queue 1 is directly determined to be normal. When it is determined that the number of flights included in the running status information of queue 2 is non-zero, such as 2, the running status of queue 2 will be further judged based on the queue timestamp in the status information. When it is determined that the pre-configured time threshold is 1 millisecond, the current polling time is 1:1:2:20, and the queue timestamp of queue 2 is 1:1:1:05, it is determined by calculation that the time difference between the current polling time and the queue timestamp is greater than the time threshold of 1 millisecond, that is, after the near-end chip request direction sends a request for queue 2 to the far-end chip, no response has been received from the far-end chip. At this time, it is determined that the running status of the currently polled queue 2 is a fault timeout. Only when the time difference is less than the threshold can the running status of queue 2 be determined to be normal. Of course, this embodiment only takes obtaining the running status of queue 2 as an example for explanation. The method for obtaining the running status of other queues is roughly the same and will not be repeated in this embodiment.

[0046] Step S103: record the queue identifier of the queue whose operation status is fault timeout into the timeout queue record table, and generate an automatic response for the fault timeout queue and feed it back to the near-end chip requester.

[0047] Optionally, before generating an automatic response for a queue that has timed out due to a fault and feeding it back to the proximal chip requester, the process also includes: obtaining a queue identifier marked on a queue pair passing through the data transmission path; querying a timeout queue record table based on the queue identifier, and when it is determined that the same queue identifier is contained in the timeout queue record table, filtering and discarding the queue pair transmitted to the remote chip, and triggering the start of an automatic response program.

[0048] Optionally, an automatic response is generated for the fault timed-out queue and fed back to the proximal chip requester, including: generating an automatic response for the fault timed-out queue based on an automatic response program with reference to key information; adding an interrupt mark to the automatic response, and feeding back the automatic response with the interrupt mark to the proximal chip requester.

[0049] Specifically, in this embodiment, after determining the operating status of each queue, the queue pair automatic response status records the queue identifier of the queue whose operating status is determined to be fault timeout in the timeout queue record table. Since the queue pair status record table is updated in real time during the process of the near-end chip requesting data to send to the far-end chip, the corresponding timeout queue record table is also updated in real time. The following Table 2 shows an example of the timeout queue pair record table obtained at time t:

[0050] Table 2

[0051] queue Operational failure Queue 2 Fault Timeout Queue 3 Fault Timeout Queue 6 Fault Timeout … …

[0052] Among them, Table 2 does not limit the number of queues, and the timeout queue record table shown in Table 2 is continuously updated over time. When the timeout queue record table is updated, when the queue pair a sent by the proximal chip requester to the remote chip flows through the sending data path, it is necessary to query the timeout queue record table. When it is determined that the queue pair a is marked as queue 2, and when queue 2 is included in the timeout queue record table, it can be determined that queue 2 to which queue pair a belongs has a fault, so queue pair a will be filtered and discarded, and the automatic operation program will be triggered to start. In this embodiment, the faulty queue pairs are filtered and discarded, thereby achieving isolation between the proximal chip and the external network.

[0053] It should be noted that when the automatic response program is determined to be activated, the queue-to-automatic response state machine will generate an automatic response for the aforementioned queue 2 that has timed out, and feedback it to the near-end chip requester. This is because if a fault occurs in queue 2 and is filtered and discarded, the near-end chip will remain in a waiting state if it cannot receive a response from the remote chip. Other chips that rely on the near-end chip's execution results will also be unable to operate normally. Since the execution results of a single queue have little impact on the overall execution results of the entire near-end chip, an automatic response is generated for the single faulty queue. Furthermore, while generating an automatic response for the faulty queue, it does not affect the remote chip's response to the normal queues. That is, the normal queues can still be accessed normally, allowing the near-end chip to obtain execution results based on the responses obtained from all queues. This keeps the near-end chip in a keepalive state, preventing the entire chip, or even the entire system, from being interrupted due to a single queue failure within the near-end chip. This isolates the internal chip fault from the external network. In addition, when the queue pair automatic response state machine determines that the faulty queue is queue 2 and the automatic response program is activated, it will obtain key information associated with queue 2 from the multi-queue first-in-first-out component. Since queue 2 can include multiple queue pairs, the amount of key information obtained can be multiple. After obtaining all the key information of queue 2, the automatic response program will generate an automatic response for queue 2 based on all the key information. In addition, in order to facilitate the near-end requester to distinguish the automatic response from the actual response fed back by the remote chip, an interrupt mark will be added to the automatic response, and the automatic response with the interrupt mark will be fed back to the near-end chip requester. Therefore, in this embodiment, when a normally operating chip suddenly fails at a port, the computing core's access to the port will not cause a program error. The automatic response machine can automatically respond to each access, including abnormal interrupts, to ensure the normal operation of the chip's internal components and programs.

[0054] Step S104: When the actual response fed back by the remote chip is received and it is determined by referring to the timeout queue record table that the actual response should correspond to the queue with fault timeout, the actual response is filtered and discarded.

[0055] Optionally, when an actual response is received from the remote chip and it is determined with reference to the timeout queue record table that the actual response corresponds to a queue with a fault timeout, the actual response is filtered and discarded, including: receiving the actual response from the remote chip through a response filter, wherein the actual response is marked with a queue identifier; querying from the timeout queue record table according to the queue identifier marked in the actual response; when it is determined that a queue identifier identical to the actual response is found in the timeout queue record table, it is determined that the actual response corresponds to a queue with a fault timeout, and the actual response corresponding to the queue with the fault timeout is filtered and discarded.

[0056] Specifically, if Figure 4 The figure shows a flow chart of another queue-level fault automatic response method. Figure 4 Step S104 is described in detail, and mainly includes the following steps:

[0057] Step S1041: Receive the actual response fed back by the remote chip through the response filter.

[0058] Specifically, the application framework also includes a response filter. In this embodiment, the actual response fed back by the remote chip is received through the response filter. The actual response is marked with a queue identifier, and the actual response corresponds to a queue pair.

[0059] Step S1042: query the timeout queue record table according to the queue identifier marked in the actual acknowledgement response.

[0060] Specifically, after obtaining the actual response, this embodiment will query the timeout queue record table based on the queue identifier, specifically to check whether the same queue identifier exists in the timeout queue record table. This embodiment does not limit the query results. In addition, the actual response in this embodiment corresponds to the queue pair. The actual response may correspond to a queue pair in a normal queue or a normal queue pair in a faulty queue. For example, when it is determined that queue 1 contains queue pair a and queue pair b, when the near-end chip requester sends queue pair a, queue 1 is still in a normal state and successfully sends queue pair a as a remote request to the remote chip. However, when queue pair b is sent, the state of queue 1 changes to a faulty timeout. In this case, the remote chip will still generate a remote response for queue pair a and feed it back to the near-end chip.

[0061] Step S1043: When it is determined that the queue ID identical to the actual response is found in the timeout queue record table, it is determined that the actual response corresponds to a fault timeout queue, and the actual response corresponding to the fault timeout queue is filtered and discarded.

[0062] Among them, when it is determined that the queue identifier that is the same as the actual response is queried in the timeout queue record table, for example, the actual response is the above-mentioned response to queue pair a, and queue 1 is marked, and queue 1 is included in the timeout queue record table, then it is determined that the actual response corresponds to a fault timeout queue. Since queue 1 has been determined as a fault timeout queue when transmitting queue pair b, and an automatic response has been generated for queue 1 and fed back to the proximal chip requester, in order to avoid repeated sending of responses to queue 1, the actual response currently obtained will be filtered and discarded through the response filter.

[0063] In the implementation mode of the present application, when a queue failure occurs in the request sent by the proximal chip requester to the remote chip due to a port failure, an automatic response will be generated for the fault queue and fed back to the proximal chip requester, and the fault queue will not interfere with other normal queues. Therefore, the network failure will not affect the normal operation of the proximal chip, and fault convergence and isolation are achieved. In addition, the proximal chip can be kept alive through automatic response, which can ensure the normal operation of other chips connected to the proximal chip, so as to achieve the normal operation of the entire system, thereby improving the reliability and availability of the system.

[0064] Example 2

[0065] Figure 5 A flowchart of a queue-level fault automatic response method is provided in a second embodiment of the present invention. This embodiment is based on the above embodiment and further includes: when it is determined that the queue identifier marked in the actual response is not found in the timeout queue record table, the processing process of the actual response is specifically described, such as Figure 5 As shown, the method includes steps S201 to S209, wherein steps S201 to S207 have been specifically described in the above embodiment and will not be repeated in this embodiment.

[0066] Step S201 : monitoring the queue sent by the proximal chip requester to the remote chip based on distributed memory semantic communication, and recording the obtained queue status information into a queue pair status recording table.

[0067] Optionally, the queue sent by the proximal chip requester to the remote chip based on distributed memory semantic communication is monitored, and the obtained queue status information is recorded in the queue pair status record table, including: real-time monitoring of the queue pairs transmitted by the proximal chip requester to the remote chip through the sending data path based on distributed memory semantic communication, wherein each queue pair is marked with a queue identifier; obtaining the queue identifier and detection timestamp of each queue pair, and determining the target queue based on the queue identifier; updating the queue timestamp of the target queue in the queue pair status record table based on the detection timestamp, and incrementally updating the flight quantity of the target queue in the queue pair status record table based on the number of queue pairs subordinate to the target queue.

[0068] Optionally, after recording the obtained queue status information in the queue pair status record table, the method further includes: obtaining key information and queue identifiers of the queue pair; and saving the corresponding relationship between the key information and the queue identifiers in a multi-queue first-in-first-out component.

[0069] Step S202: polling the queue pair status record table through the queue pair automatic answering state machine, and determining the running status of each queue according to the queue status information.

[0070] Optionally, the queue pair status record table is polled by the queue pair automatic answering state machine, and the operating status of each queue is determined according to the queue status information, including: obtaining the queue status information of each queue by polling the queue pair status record table through the queue pair automatic answering state machine; judging for each queue whether the number of flights in the queue status information is 0, if so, directly determining that the operating status of the currently polled queue is normal, otherwise, determining the operating status of the currently polled queue according to the current polling time and the queue timestamp in the queue status information.

[0071] Optionally, the operating status of the currently polled queue is determined based on the current polling time and the queue timestamp in the queue status information, including: obtaining a pre-configured time threshold and the time difference between the current polling time and the queue timestamp; judging whether the time difference is greater than the time threshold; if so, determining that the operating status of the currently polled queue is a fault timeout; otherwise, determining that the operating status of the currently polled queue is normal.

[0072] Step S203: record the queue identifier of the queue whose operation status is fault timeout into the timeout queue record table, and generate an automatic response for the fault timeout queue and feed it back to the near-end chip requester.

[0073] Optionally, before generating an automatic response for a queue that has timed out due to a fault and feeding it back to the proximal chip requester, the process also includes: obtaining a queue identifier marked on a queue pair passing through the data transmission path; querying a timeout queue record table based on the queue identifier, and when it is determined that the same queue identifier is contained in the timeout queue record table, filtering and discarding the queue pair transmitted to the remote chip, and triggering the start of an automatic response program.

[0074] Optionally, an automatic response is generated for the fault timeout queue and fed back to the proximal chip requester, including: generating an automatic response for the fault timeout queue based on an automatic response program with reference to key information; adding an interrupt mark to the automatic response response, and feeding back the automatic response response with the interrupt mark to the proximal chip requester.

[0075] Step S204: Receive the actual response fed back by the remote chip through the response filter.

[0076] Step S205: query the timeout queue record table according to the queue identifier marked in the actual acknowledgement response.

[0077] Step S206, determining whether a queue identifier identical to the actual acknowledgement response is found in the timeout queue record table; if so, executing step S207; otherwise, executing step S208.

[0078] Step S207: determine whether the actual response corresponds to a fault timeout queue, and filter and discard the actual response corresponding to the fault timeout queue.

[0079] Step S208: Determine whether the actual response corresponds to a normal queue, and send the actual response to the queue automatic response state machine through the response filter.

[0080] Specifically, when the queue ID that is identical to the actual response is not found in the timeout queue record table, it means that the actual response queue is a normal queue and specifically corresponds to a queue pair in the normal queue. At this time, the response filter will not filter and discard the actual response, but will send the actual response to the queue pair automatic response state machine.

[0081] Step S209: Send the actual response to the proximal chip requester through the automatic response state machine, and update the queue pair state record table according to the actual response.

[0082] Optionally, the actual response is sent to the proximal chip requester through the automatic response state machine, and the queue pair status record table is updated according to the actual response response, including: deleting the key information that matches the queue identifier marked in the actual response from the multi-queue first-in-first-out component through the automatic response state machine; sending the actual response to the proximal chip requester through the multi-queue first-in-first-out component after the key information deletion is completed; determining the status information in the queue pair status record table that matches the queue identifier marked in the actual response through the automatic response state machine, and updating the flight quantity contained in the matching status information decrementally.

[0083] Among them, after receiving the actual response, and the actual response is marked as queue 1, the automatic response state machine mainly performs two actions. The first action is to reduce the flight number corresponding to queue 1 in the above Table 1 by 1. For example, the flight number 2 corresponding to queue 1 in Table 1 is updated to 1. Since the flight number refers to the number of unfinished matters in which the proximal chip requester sent a request to the remote chip but did not receive a response from the remote chip, when it is determined that there is a response corresponding to a request, it means that one matter has been completed, so the unfinished matter will be reduced by one. In addition, the second action will also delete the key information matching queue 1 in the multi-queue first-in-first-out component to reduce the storage pressure of the multi-queue first-in-first-out component, and the multi-queue first-in-first-out component that completes the deletion of key information will send the actual response fed back by the remote chip to the proximal chip requester.

[0084] It should be noted that in this embodiment, in the multi-queue pair first-in-first-out component, the key information of each queue pair is saved in order for each queue, and when the queue is in a normal state, each time an actual response for a queue pair in the queue is fed back by a remote chip, the key information of the queue pair corresponding to the actual response in the multi-queue pair first-in-first-out component will be deleted. When the queue is updated to a fault timeout state, the queue pair automatic response state machine will automatically generate a response based on the remaining key information under the fault timeout queue in the multi-queue first-in-first-out component. For example, queue 1 includes queue pair a, queue pair b, and queue pair c. When the near-end chip requester transmits queue pair a to the remote chip, it extracts the key information 1 corresponding to a and saves it under the queue 1 identifier of the multi-queue pair first-in-first-out component, and determines through judgment that queue 1 is in a normal state at this time. Therefore, when receiving the actual response 1 fed back by the remote chip and feeding back the actual response to the near-end chip requester, the key information 1 under the queue 1 identifier in the multi-queue first-in-first-out component will be deleted; when the near-end chip requester transmits queue pair b to the remote chip, it extracts the key information 2 corresponding to b and saves it under the queue 1 identifier of the multi-queue pair first-in-first-out component, and determines through judgment that queue 2 has been updated to a fault timeout state at this time. Since queue pair b will be filtered and discarded after flowing through the sending data path and not sent to the remote chip, the queue pair automatic response state machine will generate an automatic response response for queue 1 based on the key information 2 under the queue identifier 1 in the multi-queue first-in-first-out component and feed it back to the near-end chip requester. Of course, in this embodiment, only the case where queue 1 includes two queue pairs is used as an example for explanation. The automatic response methods for other queues are roughly the same and will not be described in detail in this embodiment.

[0085] In the implementation mode of the present application, when a queue failure occurs in the request sent by the proximal chip requester to the remote chip due to a port failure, an automatic response will be generated for the fault queue and fed back to the proximal chip requester, and the fault queue will not interfere with other normal queues. Therefore, the network failure will not affect the normal operation of the proximal chip, and fault convergence and isolation are achieved. In addition, the proximal chip can be kept alive through automatic response, which can ensure the normal operation of other chips connected to the proximal chip, so as to achieve the normal operation of the entire system, thereby improving the reliability and availability of the system.

[0086] Example 4

[0087] Figure 6 A schematic diagram of the structure of a queue-level fault automatic response device provided by the fourth embodiment of the present invention. Figure 6 As shown, the device includes: a queue monitoring module 310, an operation status determination module 320, an automatic answering module 330 and an actual answer discarding module 340.

[0088] The queue monitoring module 310 is configured to monitor the queue sent by the proximal chip requester to the remote chip based on distributed memory semantic communication, and record the obtained queue status information in a queue pair status record table, wherein the queue status information includes a queue timestamp and a number of flights;

[0089] An operation status determination module 320 is configured to poll a queue pair status record table through a queue pair automatic answering state machine and determine the operation status of each queue based on the queue status information, wherein the operation status includes fault timeout or normal;

[0090] The automatic response module 330 is used to record the queue identifier of the fault timeout queue in the timeout queue record table, and generate an automatic response for the fault timeout queue and feed it back to the near-end chip requester;

[0091] The actual response discarding module 340 is configured to filter and discard the actual response when receiving the actual response fed back by the remote chip and determining with reference to the timeout queue record table that the actual response corresponds to a queue with fault timeout.

[0092] Optionally, the device further includes a key information storage module for obtaining key information and queue identifiers of the queue pair;

[0093] The correspondence between key information and queue identifiers is saved in the multi-queue first-in-first-out component.

[0094] Optionally, an automatic response module, configured to generate an automatic response based on the automatic response program for the fault timed-out queue with reference to the key information;

[0095] An interruption mark is added to the automatic acknowledgement response, and the automatic acknowledgement response with the interruption mark added is fed back to the near-end chip requester.

[0096] Optionally, a queue monitoring module is used to monitor in real time the queue pairs transmitted by the proximal chip requester to the remote chip via the send data path based on distributed memory semantic communication, wherein each queue pair is marked with a queue identifier;

[0097] Get the queue ID and detection timestamp of each queue pair, and determine the target queue based on the queue ID;

[0098] The queue timestamp of the target queue in the queue pair status record table is updated according to the detection timestamp, and the flight quantity of the target queue in the queue pair status record table is incrementally updated according to the number of queue pairs subordinate to the target queue.

[0099] Optionally, the apparatus further comprises an automatic answering program starting module, configured to obtain a queue identifier marked on a queue pair passing through a data transmission path;

[0100] The timeout queue record table is queried according to the queue identifier. When it is determined that the same queue identifier is contained in the timeout queue record table, the queue pair transmitted to the remote chip is filtered and discarded, and the automatic response program is triggered to start.

[0101] Optionally, an operation status determination module is used to obtain queue status information of each queue by polling the queue pair status record table through the queue pair automatic answering state machine;

[0102] For each queue, determine whether the number of flights in the queue status information is 0. If so, directly determine that the running status of the currently polled queue is normal. Otherwise, determine the running status of the currently polled queue based on the current polling time and the queue timestamp in the queue status information.

[0103] Optionally, the running status determination module is further configured to obtain a pre-configured time threshold and a time difference between a current polling time and a queue timestamp;

[0104] It is determined whether the time difference is greater than a time threshold. If so, it is determined that the running state of the currently polled queue is fault timeout; otherwise, it is determined that the running state of the currently polled queue is normal.

[0105] Optionally, an actual response discarding module is configured to receive an actual response fed back by a remote chip through a response filter, wherein the actual response is marked with a queue identifier;

[0106] Query the timeout queue record table based on the queue ID marked in the actual response;

[0107] When it is determined that the queue ID identical to the actual response is found in the timeout queue record table, it is determined that the actual response corresponds to the fault timeout queue, and the actual response corresponding to the fault timeout queue is filtered and discarded.

[0108] Optionally, the apparatus further comprises a queue pair status record table updating module, configured to determine that the actual response corresponds to a normal queue when it is determined that no queue identifier identical to the actual response is found in the timeout queue record table;

[0109] Send the actual reply response to the queue through the response filter to the automatic reply state machine;

[0110] The actual response is sent to the near-end chip requester through the automatic response state machine, and the queue pair state record table is updated according to the actual response.

[0111] Optionally, the queue pair state record table update module is further configured to delete key information matching the queue identifier marked in the actual answer response from the multi-queue first-in-first-out component through the automatic answer state machine;

[0112] The multi-queue first-in-first-out component complete with key information deletion sends the actual acknowledgement response to the near-end chip requester;

[0113] The automatic answering state machine determines the state information in the queue pair state record table that matches the queue identifier marked in the actual answering response, and updates the flight quantity contained in the matching state information decrementally.

[0114] The queue-level fault automatic response device provided by the embodiment of the present invention can execute the queue-level fault automatic response method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Example 5

[0116] Figure 7 The present invention is a block diagram of an electronic device 10 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0117] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the queue-level fault automatic response method.

[0120] In some embodiments, the queue-level fault automatic response method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the queue-level fault automatic response method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the queue-level fault automatic response method in any other appropriate manner (e.g., via firmware).

[0121] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0126] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0127] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0128] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A queue-level fault automatic response method, characterized in that: include: Monitor the queue sent by the proximal chip requester to the remote chip based on distributed memory semantic communication, and record the obtained queue status information in a queue pair status record table, wherein the queue status information includes a queue timestamp and a number of flights; Polling the queue pair status record table through the queue pair automatic answering state machine, and determining the operation status of each queue according to the queue status information, wherein the operation status includes fault timeout or normal; Recording the queue identifier of the operation state of the fault timeout queue into the timeout queue record table, and generating an automatic response for the fault timeout queue and feeding it back to the proximal chip requester; When the actual response fed back by the remote chip is received and it is determined by referring to the timeout queue record table that the actual response corresponds to a queue with fault timeout, the actual response is filtered and discarded.

2. The method according to claim 1, characterized in that After recording the acquired queue status information into the queue pair status record table, the method further includes: Get the key information and queue ID of the queue pair; The corresponding relationship between the key information and the queue identifier is saved in a multi-queue first-in-first-out component.

3. The method according to claim 2, characterized in that The generating of an automatic response for a fault timeout queue and feeding back the response to the proximal chip requester includes: generating an automatic response based on the automatic response program for the queue with timeout of the fault, with reference to the key information; An interruption mark is added to the automatic acknowledgement response, and the automatic acknowledgement response with the interruption mark added is fed back to the proximal chip requester.

4. The method according to claim 1, wherein The monitoring of the queue sent by the proximal chip requester to the remote chip based on the distributed memory semantic communication and recording the obtained queue status information in the queue pair status record table includes: Real-time monitoring is performed on queue pairs transmitted by the proximal chip requester to the remote chip through a sending data path based on distributed memory semantic communication, wherein each queue pair is marked with a queue identifier; Obtaining a queue identifier and a detection timestamp of each queue pair, and determining a target queue according to the queue identifier; The queue timestamp of the target queue in the queue pair status record table is updated according to the detection timestamp, and the flight quantity of the target queue in the queue pair status record table is incrementally updated according to the number of queue pairs subordinate to the target queue.

5. The method according to claim 3, characterized in that Before the automatic response generated by the queue with fault timeout is fed back to the proximal chip requester, the method further includes: Obtaining a queue identifier marked on the queue pair passing through the data transmission path; The timeout queue record table is queried according to the queue identifier. When it is determined that the same queue identifier is contained in the timeout queue record table, the queue pair transmitted to the remote chip is filtered and discarded, and the automatic response program is triggered to start.

6. The method according to claim 1, characterized in that The polling of the queue pair status record table by the queue pair automatic answering state machine and determining the operating status of each queue according to the queue status information includes: Polling the queue pair status record table through the queue pair automatic answering state machine to obtain the queue status information of each queue; For each queue, determine whether the number of flights in the queue status information is 0. If so, directly determine that the operating status of the currently polled queue is normal. Otherwise, determine the operating status of the currently polled queue based on the current polling time and the queue timestamp in the queue status information.

7. The method according to claim 6, characterized in that Determining the running status of the currently polled queue according to the current polling time and the queue timestamp in the queue status information includes: Obtaining a pre-configured time threshold and a time difference between the current polling time and the queue timestamp; It is determined whether the time difference is greater than the time threshold; if so, it is determined that the running state of the currently polled queue is fault timeout; otherwise, it is determined that the running state of the currently polled queue is normal.

8. The method according to claim 2, characterized in that When receiving the actual response fed back by the remote chip and determining, with reference to the timeout queue record table, that the actual response corresponds to a queue with a fault timeout, filtering and discarding the actual response includes: receiving, through a response filter, an actual response fed back by the remote chip, wherein the actual response is marked with a queue identifier; Querying the timeout queue record table according to the queue identifier marked in the actual response; When it is determined that the queue identifier identical to the actual response is found in the timeout queue record table, it is determined that the actual response corresponds to the fault timeout queue, and the actual response corresponding to the fault timeout queue is filtered and discarded.

9. The method according to claim 8, characterized in that After querying the timeout queue record table according to the queue identifier marked in the actual response, the method further includes: When it is determined that the queue identifier identical to the actual response is not found in the timeout queue record table, it is determined that the actual response corresponds to a normal queue; sending the actual reply response to the queue pair automatic reply state machine through the response filter; The actual response is sent to the proximal chip requester through the automatic response state machine, and the queue pair state record table is updated according to the actual response.

10. The method according to claim 9, characterized in that The step of sending the actual response to the proximal chip requester through the automatic response state machine and updating the queue pair status record table according to the actual response includes: Deleting, from the multi-queue first-in-first-out component, the key information matching the queue identifier marked in the actual answer response by the automatic answer state machine; The multi-queue first-in-first-out component after completing the key information deletion sends the actual acknowledgement response to the proximal chip requester; The automatic answering state machine determines the state information in the queue pair state record table that matches the queue identifier marked in the actual answer response, and updates the flight quantity included in the matching state information decrementally.

11. A queue-level fault automatic response device, characterized in that: include: A queue monitoring module is used to monitor the queue sent by the proximal chip requester to the remote chip based on distributed memory semantic communication, and record the obtained queue status information in a queue pair status record table, wherein the queue status information includes a queue timestamp and a number of flights; An operation status determination module, configured to poll the queue pair status record table through the queue pair automatic answering state machine, and determine the operation status of each queue according to the queue status information, wherein the operation status includes fault timeout or normal; An automatic response module, configured to record the queue identifier of the fault timeout queue in a timeout queue record table, and generate an automatic response for the fault timeout queue and feed it back to the proximal chip requester; The actual response discarding module is used to filter and discard the actual response when receiving the actual response fed back by the remote chip and referring to the timeout queue record table to determine that the actual response corresponds to the queue of fault timeout.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program to be executed by the at least one processor, where the computer program is executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 10 when executed.

Images