Communication exception processing method and device, medium and program product

Abstract

The invention discloses a communication exception processing method and device, a medium and a program product, and the method comprises the steps: calculating the response waiting time of a communication request in real time when the communication request sent by a local programmable multiprocessor is received and the corresponding request response information is not received; and when the response waiting time is greater than or equal to the preset waiting time threshold value, the virtual response information is sent to the local programmable multiprocessor, and the interrupt information is reported to the interrupt controller, so that the local programmable multiprocessor can end the current waiting response state. According to the method, the phenomenon of downtime of the computing equipment due to long-time response waiting is effectively avoided, so that diffusion of the downtime phenomenon in the cluster computing system can be avoided, resetting of the whole cluster computing system is avoided, and meanwhile normal operation of the cluster computing system can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for processing communication anomalies, electronic equipment, a computer-readable storage medium, and a computer program product. Background Art

[0002] Cluster computing systems are widely used in today's high-performance computing and large-scale data processing environments. In cluster computing systems, processors on multiple computing devices can collaborate to complete processing tasks that require high computing power, such as large model training, by communicating data with each other.

[0003] In the existing technology, when an abnormality occurs in a computing device, processor, or the communication link or switch between processors, the related computing devices that have a communication connection relationship with the computing device, processor, communication link or switch may crash. Due to the current lack of a timely processing mechanism for such communication abnormalities, it is easy to cause the computing devices of the entire cluster computing system to crash. At this time, the only way is to reset the computing devices of the entire cluster computing system, resulting in the interruption of the ongoing task, and all task-related data and calculation results may be lost, which is very costly. Summary of the Invention

[0004] The present invention provides a method, device, medium and program product for processing communication anomalies to solve the technical problem in existing cluster computing systems that when an anomaly occurs in a computing device, processor or the communication link or switch between processors, the computing device of the entire cluster computing system may easily crash.

[0005] In order to solve the above technical problems, a first aspect of an embodiment of the present invention provides a method for handling communication anomalies, which is executed by a communication control module in a processor, wherein the communication control module is used to perform data transmission and reception operations between the processor and other processors. The method includes:

[0006] When a communication request sent by a local programmable multiprocessor is received and no request response information corresponding to the communication request is received, calculating the response waiting time of the communication request in real time;

[0007] When the response waiting time is greater than or equal to a preset waiting time threshold, virtual response information corresponding to the communication request is sent to the local programmable multiprocessor, and interrupt information is reported to an interrupt controller.

[0008] As a preferred solution, the method further comprises:

[0009] When receiving the communication request, determining a downstream processor corresponding to the communication request;

[0010] When it is detected that the downstream processor has computing resources capable of processing the communication request and the current response waiting time is less than the waiting time threshold, the communication request is sent to the downstream processor and the response waiting time is reset.

[0011] As a preferred solution, the method further comprises:

[0012] When receiving the communication request, generating an identification number corresponding to the communication request;

[0013] The identification number and the request information corresponding to the communication request are written into a preset request queue; wherein the request queue also stores time characterization information corresponding to the communication request and used to represent the response waiting time.

[0014] As a preferred solution, the method further comprises:

[0015] When the communication request is sent to the downstream processor corresponding to the communication request, identification indication information is sent to the downstream processor; wherein the identification indication information includes the identification number and return indication information, and the return indication information is used to instruct the downstream processor to return request response information carrying the identification number.

[0016] As a preferred solution, the method further comprises:

[0017] When receiving the request response information from the downstream processor, determining whether there is a target identification number identical to the identification number carried in the request response information in the request queue;

[0018] When the target identification number does not exist in the request queue, the request response information is discarded and the response waiting time is continued to be calculated.

[0019] As a preferred solution, the method further comprises:

[0020] When the target identification number exists in the request queue and the response waiting time corresponding to the target identification number is less than the waiting time threshold, the request response information is sent to the local programmable multiprocessor, and the calculation of the response waiting time of the target communication request corresponding to the target identification number is stopped;

[0021] The target identification number, request information, and time representation information corresponding to the target communication request are removed from the request queue.

[0022] As a preferred solution, the method further comprises:

[0023] When the response waiting time is greater than or equal to the waiting time threshold, the identification number, the request information, and the time representation information corresponding to the communication request are removed from the request queue.

[0024] As a preferred solution, when a communication request sent by a local programmable multiprocessor is received and request response information corresponding to the communication request is not received, calculating the response waiting time of the communication request in real time specifically includes:

[0025] When the communication request is received, a timing trigger signal is received from a timing trigger in real time; wherein the timing trigger is used to send the timing trigger signal based on a preset timing trigger accuracy;

[0026] In a process in which the request response information is not received, in response to the timing trigger signal, updating the time representation information corresponding to the communication request in the request queue;

[0027] The response waiting time is calculated in real time based on the time characterization information and the timing trigger accuracy.

[0028] As a preferred solution, the timing trigger is started at the moment when the request queue is empty and the communication control module receives the communication request;

[0029] The stop time of the timing trigger is the time when the request queue is transformed into an empty queue.

[0030] As a preferred solution, when the response waiting time is greater than or equal to a preset waiting time threshold, sending virtual response information corresponding to the communication request to the local programmable multiprocessor specifically includes:

[0031] When the response waiting time is greater than or equal to the waiting time threshold, obtaining a request type of the communication request;

[0032] Based on preset virtual response information corresponding to different request types, virtual response information corresponding to the request type of the communication request is obtained and sent to the local programmable multiprocessor.

[0033] As a preferred solution, the interruption information includes: the identification number corresponding to the communication request, the request information, and the time representation information.

[0034] A second aspect of an embodiment of the present invention provides a method for handling communication anomalies, which is executed by a host and includes:

[0035] When receiving interrupt information reported by any communication control module from the interrupt controller, determining abnormal information that the communication request is delayed in response based on the interrupt information; wherein the interrupt information is reported by the communication control module when the response waiting time of the communication request is greater than or equal to a preset waiting time threshold;

[0036] Interruption processing is performed based on the interruption information and the abnormality information.

[0037] As a preferred solution, the exception information includes exception handler information and exception cause.

[0038] As a preferred solution, the interrupt processing based on the interrupt information and the exception information specifically includes:

[0039] determining, based on the interrupt information, a target processor to which the interrupt information is to be reported;

[0040] Determining a current exception handler based on the exception handler information;

[0041] Disconnect the communication connection between the target processor and the exception processor, and establish a communication connection between the target processor and the remaining downstream processors; wherein the remaining downstream processors are processors other than the exception processor that have computing power resources capable of processing the communication request.

[0042] As a preferred solution, the method further comprises:

[0043] Based on the preset timing trigger accuracy, generating timing trigger indication information carrying the timing trigger accuracy;

[0044] The timing trigger indication information is sent to a timing trigger; wherein the timing trigger indication information is used to instruct the timing trigger to send a timing trigger signal to the communication control module based on the timing trigger accuracy.

[0045] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the method for handling communication anomalies described in any one of the first aspect or the method for handling communication anomalies described in any one of the second aspect is implemented.

[0046] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the communication anomaly processing method described in any one of the first aspects, or implement the communication anomaly processing method described in any one of the second aspects.

[0047] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for handling communication anomalies described in any one of the first aspects, or implements the method for handling communication anomalies described in any one of the second aspects.

[0048] Compared with the prior art, the beneficial effect of the embodiments of the present invention lies in that, when a communication request sent by a local programmable multiprocessor is received and no corresponding request response information is received, the response waiting time of the communication request is calculated in real time, and when the response waiting time is greater than or equal to the preset waiting time threshold, a virtual response information is sent to the local programmable multiprocessor, and the interrupt information is reported to the interrupt controller, so that the local programmable multiprocessor ends the current waiting response state, effectively avoiding the phenomenon of the computing device crashing due to a long wait for a response, thereby avoiding the spread of the crash phenomenon in the cluster computing system, avoiding the reset of the entire cluster computing system, and ensuring the normal operation of the cluster computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a method for handling communication anomalies executed by a communication control module in a processor in an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of communication interconnection between two GPUs in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of a request queue in an embodiment of the present invention;

[0052] Figure 4 1 is a flow chart of a method for handling communication anomalies executed by a host in an embodiment of the present invention;

[0053] Figure 5 is a schematic structural diagram of an electronic device in an embodiment of the present invention;

[0054] Figure 6 1 is a schematic structural diagram of a communication anomaly processing device according to an embodiment of the present invention;

[0055] Figure 7 It is a structural diagram of another communication abnormality processing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] See Figure 1 According to a first aspect of an embodiment of the present invention, a method for handling communication anomalies is provided. The method is executed by a communication control module in a processor, wherein the communication control module is configured to perform data transmission and reception operations between the processor and other processors. The method includes the following steps S11 and S12:

[0058] Step S11, when a communication request sent by a local programmable multiprocessor is received and no request response information corresponding to the communication request is received, calculating the response waiting time of the communication request in real time;

[0059] Step S12: When the response waiting time is greater than or equal to a preset waiting time threshold, virtual response information corresponding to the communication request is sent to the local programmable multiprocessor, and interrupt information is reported to an interrupt controller.

[0060] It is worth noting that for each computing device in the cluster computing system, the processor in this embodiment can be a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processor (DPU), an accelerated processing unit (APU) or a general-purpose graphics processing unit (GPGPU), etc., and can be implemented as a chip, which is not specifically limited in this embodiment. In addition, the processor in this embodiment includes one or more programmable multiprocessors, a network on chip (NOC) circuit and a communication control module. Optionally, the programmable multiprocessor can be a streaming processor cluster (SPC) or other programmable multi-computing core group. For different chips, the corresponding programmable multiprocessors can be the same or different. This embodiment is not specifically limited here. Each programmable multiprocessor can include multiple computing units (CUs), such as tensor cores, vector cores and other computing cores, and other computing cores such as integer computing cores or floating-point computing cores, and these computing units can be connected to an on-chip network circuit. In addition, when there are multiple programmable multiprocessors, each of the multiple programmable multiprocessors can be connected to an on-chip network circuit to utilize the communication resources of the on-chip network circuit to send data streams to the communication control module via the on-chip network circuit.

[0061] Furthermore, the communication control module in this embodiment is used to perform data transmission and reception operations between the processor and the remaining processors to achieve communication interconnection between the processor and the remaining processors. It is worth noting that the communication control module includes a control circuit and one or more communication interconnection ports. Each communication interconnection port is used to establish a communication link with the communication interconnection port of the corresponding communication control module in the remaining different processors, thereby achieving communication interconnection between the local processor and the remaining processors. The control circuit is used to transmit the communication request received from the programmable multiprocessor to the opposite processor with a communication interconnection relationship using the communication interconnection port, and to feed back the communication request response data received through the communication interconnection port to the programmable multiprocessor, thereby achieving data transmission and reception operations between the local processor and the remaining processors. In addition, the other processors mentioned here can be a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processor (DPU), an accelerated processing unit (APU) or a general-purpose graphics processing unit (GPGPU), etc., and this embodiment does not make specific limitations here. Taking the processor in this embodiment as a GP U and the programmable multiprocessor as a streaming processor cluster as an example, as shown in FIG. Figure 2As shown, it is a schematic diagram of the communication interconnection between two GPUs. Optionally, in this embodiment, the communication interconnection port of the communication control module includes a sending port tx and a receiving port rx. When GPU0 needs to send a communication request to GPU1, the programmable multiprocessor in GPU0 first sends the communication request to the local communication control module, and sends the communication request to the receiving port rx of the communication control module in GPU1 via the sending port tx of the communication control module, and transmits the communication request to the programmable multiprocessor in GPU1 via the communication control module in GPU1; when GPU1 needs to return the request response information corresponding to the communication request to GPU0, the programmable multiprocessor in GPU1 first sends the request response information to the local communication control module, and sends the request response information to the receiving port rx of the communication control module in GPU0 via the sending port tx of the communication control module, and transmits the request response information to the programmable multiprocessor in GPU0 via the communication control module in GPU0. It is worth noting that in the process of data interaction between processors, there may also be the following Figure 2 In the case where a receiving port rx in the same communication control module transmits data to a transmitting port tx, for example, since the number of communication interconnection ports in the communication control module is limited, there may be a situation where there is no communication interconnection between the target processors corresponding to the current communication request, while one of the processors that has established a communication interconnection relationship with the target processor has a communication interconnection relationship. In this case, the processor that has established a communication interconnection can be used as an intermediate node to send the communication request to be sent to the processor serving as the intermediate node. For the processor serving as the intermediate node, the communication request received by the receiving port rx in the current communication control module is not a communication request for the local programmable multiprocessor. Therefore, the communication control module will transmit the communication request from the receiving port rx to the transmitting port tx based on the target processor information corresponding to the communication request, and then send the communication request to the target processor via the transmitting port tx.

[0062] During the operation of a cluster computing system, if a computing device, processor, or a communication link or switch between processors experiences an abnormality, then related computing devices that have a communication connection relationship with the computing device, processor, communication link, or switch may not receive a request response message after sending a communication request, resulting in a system downtime. Since the computing devices in the cluster computing system have a close communication connection relationship, when one computing device experiences a system downtime, the system downtime may gradually spread to the entire cluster computing system. To address this problem, this embodiment, when receiving a communication request sent by a local programmable multiprocessor and not receiving a request response message corresponding to the communication request, indicates that the system is currently in a response waiting state, and therefore calculates the response waiting time of the communication request in real time. During the process of waiting for the request response message to be returned, if the response waiting time is greater than or equal to a preset waiting time threshold, in order to terminate the current waiting response state of the local programmable multiprocessor, a virtual response message corresponding to the communication request is sent to the local programmable multiprocessor, so that the local programmable multiprocessor can normally exit the current task process after receiving the virtual response message. Furthermore, this embodiment also reports interrupt information to the interrupt controller, and the interrupt information is used to indicate the abnormal information that the communication request is delayed in response, so that the relevant operation and maintenance personnel or the host can subsequently repair or replace the abnormal processor based on the reported interrupt information to ensure the normal operation of the cluster computing system. It can be understood that the virtual response information returned in this embodiment is not the actual response information generated by processing the communication request, that is, the data contained in the virtual response information is not the correct data obtained by processing the current task process. It is only to avoid the local programmable multiprocessor from being in a down state. Therefore, in order to ensure the accuracy of the calculation results of the final task process, it is necessary to report the interrupt information to the interrupt controller.

[0063] The method for handling communication anomalies provided by an embodiment of the present invention, when receiving a communication request sent by a local programmable multiprocessor and not receiving the corresponding request response information, calculates the response waiting time of the communication request in real time, and when the response waiting time is greater than or equal to the preset waiting time threshold, sends virtual response information to the local programmable multiprocessor, and reports interrupt information to the interrupt controller, so as to enable the local programmable multiprocessor to end the current waiting response state, effectively avoiding the phenomenon of computing equipment crashing due to long waiting time for response, thereby avoiding the spread of the crash phenomenon in the cluster computing system, avoiding the reset of the entire cluster computing system, and ensuring the normal operation of the cluster computing system.

[0064] As a preferred solution, the method further comprises:

[0065] When receiving the communication request, determining a downstream processor corresponding to the communication request;

[0066] When it is detected that the downstream processor has computing resources capable of processing the communication request and the current response waiting time is less than the waiting time threshold, the communication request is sent to the downstream processor and the response waiting time is reset.

[0067] Specifically, generally speaking, the communication request will include recipient information, so that the downstream processor corresponding to the current communication request can be determined. Furthermore, when sending a communication request, it is necessary to evaluate the computing power resources of the downstream processor. It can be understood that this embodiment can determine whether its current computing power resources can process the communication request based on the computing power resource notification of the downstream processor received in advance. Optionally, the computing power resource notification of the downstream processor can be sent periodically or in response to a computing power resource information request. The computing power resource notification can include the amount of data that the downstream processor can currently process or the number of communication requests that can be received. This embodiment does not make specific limitations here. When it is detected that the downstream processor has computing power resources that can process the communication request and the current response waiting time is less than the waiting time threshold, the communication request is sent to the downstream processor, and the response waiting time needs to be reset to avoid misjudging the abnormal processor.

[0068] As a preferred solution, the method further comprises:

[0069] When receiving the communication request, generating an identification number corresponding to the communication request;

[0070] The identification number and the request information corresponding to the communication request are written into a preset request queue; wherein the request queue also stores time characterization information corresponding to the communication request and used to represent the response waiting time.

[0071] Specifically, considering that the various computing devices in the cluster computing system cooperate with each other to complete different computing tasks, a large number of communication requests will be generated during the task processing process. Therefore, this embodiment sets up a request queue to uniformly manage the communication requests issued by the local programmable multiprocessor that are waiting for response. In order to facilitate the rapid matching of the request response information with the communication request in the request queue when it is returned, this embodiment first generates an identification number corresponding to the communication request when receiving the communication request. Exemplarily, the identification number in this embodiment can be randomly generated, or generated according to the order of writing to the request queue within a preset time period. For example, within the preset time period, the current communication request is the third communication request written to the request queue, then the identification number "003" is generated. This embodiment does not make specific limitations here, and it only needs to ensure that the identification number of the issued communication request and the identification number of the returned request response information can match. Furthermore, the generated identification number and the request information corresponding to the communication request are written into the request queue, and the request queue also stores the time representation information corresponding to the communication request for indicating the current response waiting time. It can be understood that when the identification number and request information corresponding to the communication request are written into the request queue, the initial value of the corresponding time representation information is 0, and the time representation information is continuously updated during the response waiting period, thereby establishing a corresponding relationship between the identification number, request information and real-time time representation information corresponding to a communication request waiting for a response.

[0072] In addition, when it is detected that the downstream processor has computing resources capable of processing the communication request and the current response waiting time is less than the waiting time threshold, after sending the communication request to the downstream processor, the time representation information corresponding to the communication request in the request queue needs to be reset.

[0073] Optionally, the request queue in this embodiment stores the identification number, request information and real-time response waiting time corresponding to each communication request in the form of a table, such as Figure 3 As shown, request outstanding represents a request queue, id represents an identification number, request info represents request information, and time info represents time representation information.

[0074] As a preferred solution, the method further comprises:

[0075] When the communication request is sent to the downstream processor corresponding to the communication request, identification indication information is sent to the downstream processor; wherein the identification indication information includes the identification number and return indication information, and the return indication information is used to instruct the downstream processor to return request response information carrying the identification number.

[0076] Specifically, in order to quickly match the returned request response information with the communication request in the request queue, this embodiment sends identification indication information to the downstream processor at the same time when sending the communication request to its corresponding downstream processor. The identification indication information includes the identification number corresponding to the generated communication request and the return indication information for instructing the downstream processor to return the request response information carrying the identification number.

[0077] As a preferred solution, the method further comprises:

[0078] When receiving the request response information from the downstream processor, determining whether there is a target identification number identical to the identification number carried in the request response information in the request queue;

[0079] When the target identification number does not exist in the request queue, the request response information is discarded and the response waiting time is continued to be calculated.

[0080] Specifically, when a request response message is received from a downstream processor, it is necessary to obtain the identification number carried by the request response message and determine whether there is a target identification number identical to this identification number in the current request queue. If the target identification number exists, it indicates that the current request response message is the request response message corresponding to one of the communication requests in the request queue; if the target identification number does not exist, it indicates that the current request response message is not the request response message corresponding to any of the communication requests in the request queue, or the identification number carried by the current request response message is an incorrect identification number. Therefore, when the target identification number does not exist in the request queue, the currently received request response message is discarded, that is, at this time, each communication request in the request queue is still in the response waiting state, and the response waiting time continues to be calculated.

[0081] As a preferred solution, the method further comprises:

[0082] When the target identification number exists in the request queue and the response waiting time corresponding to the target identification number is less than the waiting time threshold, the request response information is sent to the local programmable multiprocessor, and the calculation of the response waiting time of the target communication request corresponding to the target identification number is stopped;

[0083] The target identification number corresponding to the target communication request, the request information corresponding to the target identification number, and the time representation information are removed from the request queue.

[0084] Specifically, when there is a target identification number in the request queue that is the same as the identification number carried by the currently received request response information, and the response waiting time corresponding to the target identification number is less than the waiting time threshold, it indicates that the currently received request response information is valid request response information, and there is no timeout response in the downstream processor that returns the request response information. Therefore, the currently received request response information is returned to the local programmable multiprocessor, and since the communication request corresponding to the request response information has been effectively responded to at this time, the calculation of the response waiting time of the target communication request corresponding to the target identification number is stopped, and the target identification number, request information and time representation information corresponding to the target communication request are removed from the request queue.

[0085] As a preferred solution, the method further comprises:

[0086] When the response waiting time is greater than or equal to the waiting time threshold, the identification number, the request information, and the time representation information corresponding to the communication request are removed from the request queue.

[0087] Specifically, when the response waiting time of a communication request is greater than or equal to the waiting time threshold, it indicates that the downstream processor corresponding to the communication request has a timeout response. Therefore, the identification number, request information and current time representation information corresponding to the communication request are removed from the request queue. Therefore, even if the request response information returned by the downstream processor is received at this time, the corresponding communication request cannot be found in the request queue. It is still determined that the downstream processor has a timeout response, and the interrupt information is reported to the interrupt controller.

[0088] As a preferred solution, when a communication request sent by a local programmable multiprocessor is received and request response information corresponding to the communication request is not received, calculating the response waiting time of the communication request in real time specifically includes:

[0089] When the communication request is received, a timing trigger signal is received from a timing trigger in real time; wherein the timing trigger is used to send the timing trigger signal based on a preset timing trigger accuracy;

[0090] In a process in which the request response information is not received, in response to the timing trigger signal, updating the time representation information corresponding to the communication request in the request queue;

[0091] The response waiting time is calculated in real time based on the time characterization information and the timing trigger accuracy.

[0092] Specifically, this embodiment receives a timing trigger signal from a timing trigger in real time starting from the moment a communication request is received. It can be understood that since the request queue needs to store multiple communication requests waiting for response, in order to improve the timing accuracy of the response waiting time, if nanoseconds are used as the accuracy, 1000ns requires at least 10 bits of binary data, and if 1ms of response waiting time needs to be counted, at least 20 bits of binary data are required, which will cause the request queue to occupy a large amount of resources. In order to reduce the resource consumption of the request queue, this embodiment uses a timing trigger to time the current response waiting time as usual with a timing accuracy, that is, allocates larger resources to the timing trigger, and then sets a timing trigger accuracy. For example, the timing trigger accuracy can be 1000ns, 1200ns, 1500ns, etc. This embodiment is not specifically limited here. In the process of not receiving the request response information, the time representation information corresponding to the communication request in the request queue is updated in response to the timing trigger signal received from the timing trigger. For example, each time a timing trigger signal is received, the value of the time representation information is increased by one, thereby greatly reducing the resource consumption of the request queue. For example, assuming that the timing trigger accuracy is set to 1000ns, that is, the timing trigger sends a timing trigger signal every 1000ns, and when a timing trigger signal is received, the time representation information only needs to be increased by one, which effectively reduces resource overhead.

[0093] Furthermore, based on the time characterization information and the timing trigger accuracy, the current response waiting time can be determined. For example, assuming that the timing trigger accuracy is set to 1000ns and the time characterization information is "0010", the current response waiting time is 2000ns.

[0094] Preferably, the time representation information in this embodiment consists of a binary number with a data size of 4 bits, that is, the initial value of the time representation information is "0000", and the waiting time threshold can be set to the response waiting time corresponding to when the time representation information is "1111".

[0095] As a preferred solution, the timing trigger is started at the moment when the request queue is empty and the communication control module receives the communication request;

[0096] The stop time of the timing trigger is the time when the request queue is transformed into an empty queue.

[0097] It's worth noting that when the request queue is empty, there are no communication requests currently awaiting a response. Therefore, there's no need to calculate the response wait time, and the timer trigger doesn't need to be activated. However, when the communication control module receives a communication request, it indicates that there is a communication request currently awaiting a response, and the response wait time needs to be calculated. Therefore, when the request queue is empty, the moment the communication control module receives the communication request is the moment the timer trigger is activated. When the request queue becomes empty, all communication requests currently awaiting a response have been processed, and there's no need to calculate the response wait time. This is the moment the timer trigger stops.

[0098] As a preferred solution, when the response waiting time is greater than or equal to a preset waiting time threshold, sending virtual response information corresponding to the communication request to the local programmable multiprocessor specifically includes:

[0099] When the response waiting time is greater than or equal to the waiting time threshold, obtaining a request type of the communication request;

[0100] Based on preset virtual response information corresponding to different request types, virtual response information corresponding to the request type of the communication request is obtained and sent to the local programmable multiprocessor.

[0101] Specifically, since the request types of different communication requests have different forms of corresponding virtual response information, this embodiment pre-sets corresponding virtual response information carrying error data or empty data for different request types, so that when it is detected that the response waiting time is greater than or equal to the waiting time threshold, the currently required virtual response information can be directly determined based on the request type of the communication request and sent to the local programmable multiprocessor.

[0102] Optionally, the request types of the communication request in this embodiment include a write request, an atomic request, and a read request.

[0103] As a preferred solution, the interruption information includes: the identification number corresponding to the communication request, the request information, and the time representation information.

[0104] See Figure 4 A second aspect of an embodiment of the present invention provides a method for handling communication anomalies, which is executed by a host and includes the following steps S21 and S22:

[0105] Step S21, upon receiving interrupt information reported by any communication control module from the interrupt controller, determining abnormal information indicating a delayed response to a communication request based on the interrupt information; wherein the interrupt information is reported by the communication control module when the response waiting time of the communication request is greater than or equal to a preset waiting time threshold;

[0106] Step S22: performing interrupt processing based on the interrupt information and the abnormality information.

[0107] Specifically, the host in this embodiment can be a central processing unit (CPU), a microprocessor, etc. When an interrupt message reported by any communication control module is received from the interrupt controller, it is necessary to determine the abnormal information of the delayed response of the communication request based on the interrupt message, for example, using an interrupt service routine to locate the specific abnormal processor information and the abnormal cause. In this embodiment, the abnormal cause can be a failure of the processor connected to the target processor that reports the current interrupt message, or a failure of the communication link between the target processor and the other processors, or a failure of the switch connected between the target processor and the other processors, etc. This embodiment will not be described in detail here. Further, interrupt processing is performed based on the interrupt message and the abnormal information. It can be understood that since the interrupt message is reported by the communication control module when the response waiting time of the communication request is greater than or equal to the preset waiting time threshold, that is, the moment when the current communication request is delayed in response, interrupt processing based on the interrupt message and the abnormal information determined using the interrupt message can effectively prevent the downtime phenomenon from spreading in the cluster computing system, avoid resetting the entire cluster computing system, and ensure that the cluster computing system can resume normal operation.

[0108] As a preferred solution, the interrupt processing based on the interrupt information and the exception information specifically includes:

[0109] determining, based on the interrupt information, a target processor to which the interrupt information is to be reported;

[0110] Determining a current exception handler based on the exception handler information;

[0111] Disconnect the communication connection between the target processor and the exception processor, and establish a communication connection between the target processor and the remaining downstream processors; wherein the remaining downstream processors are processors other than the exception processor that have computing power resources capable of processing the communication request.

[0112] Specifically, since the exception processor that is in communication connection with the target processor cannot normally return the request response information of the communication request, in order to ensure that the communication request of the target processor can be processed normally, this embodiment first disconnects the communication connection between the target processor and the exception processor, and establishes a communication connection between the target processor and the remaining downstream processors that have computing power resources capable of processing the communication request of the target processor. The downstream processor that has re-established the communication connection continues to process the communication request of the target processor to ensure the normal progress of the task process.

[0113] Optionally, before disconnecting the communication connection between the target processor and the exception processor, the exception processor may be checked again through a self-check program to see if there is a fault. If it is determined that there is a fault, the communication connection between the target processor and the exception processor is disconnected.

[0114] As a preferred solution, the method further comprises:

[0115] Based on the preset timing trigger accuracy, generating timing trigger indication information carrying the timing trigger accuracy;

[0116] The timing trigger indication information is sent to a timing trigger; wherein the timing trigger indication information is used to instruct the timing trigger to send a timing trigger signal to the communication control module based on the timing trigger accuracy.

[0117] The method for handling communication anomalies provided by an embodiment of the present invention determines the abnormal information of the delayed response to the communication request based on the received interruption information, and further performs interruption processing based on the interruption information and the abnormal information. Since the interruption information is reported by the communication control module when the response waiting time of the communication request is greater than or equal to the preset waiting time threshold, the fault point in the cluster computing system can be processed in time, effectively avoiding the spread of the downtime phenomenon in the cluster computing system, avoiding the reset of the entire cluster computing system, and ensuring that the cluster computing system can resume normal operation.

[0118] See Figure 5 A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101. When the processor 101 executes the computer program, it implements the method for handling communication anomalies described in any embodiment of the first aspect, or the method for handling communication anomalies described in any embodiment of the second aspect.

[0119] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 102 and executed by the processor 101 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0120] The electronic device may include, but is not limited to, a processor 101 and a memory 102. Those skilled in the art will appreciate that the schematic diagram is merely an example of an electronic device and does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or different components. For example, the electronic device may also include input and output devices, network access devices, buses, and the like.

[0121] The processor 101 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 101 may be any conventional processor, etc. The processor 101 is the control center of the electronic device, connecting various parts of the entire electronic device using various interfaces and lines.

[0122] The memory 102 can be used to store the computer programs and / or modules. The processor 101 implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory 102 and calling the data stored in the memory 102. The memory 102 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 102 can include a high-speed random access memory and a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0123] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the communication anomaly processing method described in any embodiment of the first aspect, or implement the communication anomaly processing method described in any embodiment of the second aspect.

[0124] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for handling communication anomalies described in any embodiment of the first aspect, or implements the method for handling communication anomalies described in any embodiment of the second aspect.

[0125] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0126] See Figure 6 A sixth aspect of an embodiment of the present invention provides a device for processing communication anomalies, including:

[0127] The request response monitoring module 201 is configured to calculate the response waiting time of the communication request in real time when the communication request sent by the local programmable multiprocessor is received and the request response information corresponding to the communication request is not received;

[0128] The interrupt reporting module 202 is configured to send virtual response information corresponding to the communication request to the local programmable multiprocessor and report interrupt information to the interrupt controller when the response waiting time is greater than or equal to a preset waiting time threshold.

[0129] As a preferred solution, the request response monitoring module 201 is further configured to:

[0130] When receiving the communication request, determining a downstream processor corresponding to the communication request;

[0131] When it is detected that the downstream processor has computing resources capable of processing the communication request and the current response waiting time is less than the waiting time threshold, the communication request is sent to the downstream processor and the response waiting time is reset.

[0132] As a preferred solution, the request response monitoring module 201 is further configured to:

[0133] When receiving the communication request, generating an identification number corresponding to the communication request;

[0134] The identification number and the request information corresponding to the communication request are written into a preset request queue; wherein the request queue also stores time characterization information corresponding to the communication request and used to represent the response waiting time.

[0135] As a preferred solution, the request response monitoring module 201 is further configured to:

[0136] When the communication request is sent to the downstream processor corresponding to the communication request, identification indication information is sent to the downstream processor; wherein the identification indication information includes the identification number and return indication information, and the return indication information is used to instruct the downstream processor to return request response information carrying the identification number.

[0137] As a preferred solution, the request response monitoring module 201 is further configured to:

[0138] When receiving the request response information from the downstream processor, determining whether there is a target identification number identical to the identification number carried in the request response information in the request queue;

[0139] When the target identification number does not exist in the request queue, the request response information is discarded and the response waiting time is continued to be calculated.

[0140] As a preferred solution, the request response monitoring module 201 is further configured to:

[0141] When the target identification number exists in the request queue and the response waiting time corresponding to the target identification number is less than the waiting time threshold, the request response information is sent to the local programmable multiprocessor, and the calculation of the response waiting time of the target communication request corresponding to the target identification number is stopped;

[0142] The target identification number, request information, and time representation information corresponding to the target communication request are removed from the request queue.

[0143] As a preferred solution, the interrupt reporting module 202 is further configured to:

[0144] When the response waiting time is greater than or equal to the waiting time threshold, the identification number, the request information, and the time representation information corresponding to the communication request are removed from the request queue.

[0145] As a preferred solution, the request response monitoring module 201 is configured to calculate the response waiting time of the communication request in real time when receiving a communication request sent by the local programmable multiprocessor and not receiving the request response information corresponding to the communication request, specifically including:

[0146] When the communication request is received, a timing trigger signal is received from a timing trigger in real time; wherein the timing trigger is used to send the timing trigger signal based on a preset timing trigger accuracy;

[0147] In a process in which the request response information is not received, in response to the timing trigger signal, updating the time representation information corresponding to the communication request in the request queue;

[0148] The response waiting time is calculated in real time based on the time characterization information and the timing trigger accuracy.

[0149] As a preferred solution, the timing trigger is started at the moment when the request queue is empty and the communication control module receives the communication request;

[0150] The stop time of the timing trigger is the time when the request queue is transformed into an empty queue.

[0151] As a preferred solution, the interrupt reporting module 202 is configured to send virtual response information corresponding to the communication request to the local programmable multiprocessor when the response waiting time is greater than or equal to a preset waiting time threshold, specifically including:

[0152] When the response waiting time is greater than or equal to the waiting time threshold, obtaining a request type of the communication request;

[0153] Based on preset virtual response information corresponding to different request types, virtual response information corresponding to the request type of the communication request is obtained and sent to the local programmable multiprocessor.

[0154] As a preferred solution, the interruption information includes: the identification number corresponding to the communication request, the request information, and the time representation information.

[0155] The communication anomaly processing device provided by an embodiment of the present invention, when receiving a communication request sent by a local programmable multiprocessor and not receiving the corresponding request response information, calculates the response waiting time of the communication request in real time, and when the response waiting time is greater than or equal to the preset waiting time threshold, sends a virtual response information to the local programmable multiprocessor, and reports the interrupt information to the interrupt controller, so as to enable the local programmable multiprocessor to end the current waiting response state, effectively avoiding the phenomenon of the computing device crashing due to a long wait for a response, thereby avoiding the spread of the crash phenomenon in the cluster computing system, avoiding the reset of the entire cluster computing system, and ensuring the normal operation of the cluster computing system.

[0156] See Figure 7 A fourth aspect of an embodiment of the present invention provides a device for processing communication anomalies, including:

[0157] The abnormality information acquisition module 301 is configured to, upon receiving interruption information reported by any communication control module from the interruption controller, determine abnormality information indicating a delayed response to a communication request based on the interruption information; wherein the interruption information is reported by the communication control module when the response waiting time of the communication request is greater than or equal to a preset waiting time threshold;

[0158] The interrupt processing module 302 is configured to perform interrupt processing based on the interrupt information and the exception information.

[0159] As a preferred solution, the exception information includes exception handler information and exception cause.

[0160] As a preferred solution, the interrupt processing module 302 is used to perform interrupt processing based on the interrupt information and the exception information, specifically including:

[0161] determining, based on the interrupt information, a target processor to which the interrupt information is to be reported;

[0162] Determining a current exception handler based on the exception handler information;

[0163] Disconnect the communication connection between the target processor and the exception processor, and establish a communication connection between the target processor and the remaining downstream processors; wherein the remaining downstream processors are processors other than the exception processor that have computing power resources capable of processing the communication request.

[0164] As a preferred solution, the device further includes a timing trigger indication information sending module, which is configured to:

[0165] Based on the preset timing trigger accuracy, generating timing trigger indication information carrying the timing trigger accuracy;

[0166] The timing trigger indication information is sent to a timing trigger; wherein the timing trigger indication information is used to instruct the timing trigger to send a timing trigger signal to the communication control module based on the timing trigger accuracy.

[0167] The communication anomaly processing device provided by an embodiment of the present invention determines the abnormal information of the delayed response to the communication request based on the received interruption information, and further performs interruption processing based on the interruption information and the abnormal information. Since the interruption information is reported by the communication control module when the response waiting time of the communication request is greater than or equal to the preset waiting time threshold, the fault point in the cluster computing system can be processed in time, effectively avoiding the spread of the downtime phenomenon in the cluster computing system, avoiding the reset of the entire cluster computing system, and ensuring that the cluster computing system can resume normal operation.

[0168] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for handling communication anomalies, executed by a communication control module in a processor, characterized in that: The communication control module is used to perform data transmission and reception operations between the processor and other processors, and the method includes: When a communication request sent by a local programmable multiprocessor is received and no request response information corresponding to the communication request is received, calculating the response waiting time of the communication request in real time; When the response waiting time is greater than or equal to a preset waiting time threshold, virtual response information corresponding to the communication request is sent to the local programmable multiprocessor, and interrupt information is reported to an interrupt controller.

2. The method for handling communication anomalies according to claim 1, wherein: The method further comprises: When receiving the communication request, determining a downstream processor corresponding to the communication request; When it is detected that the downstream processor has computing resources capable of processing the communication request and the current response waiting time is less than the waiting time threshold, the communication request is sent to the downstream processor and the response waiting time is reset.

3. The method for handling communication anomalies according to claim 1, wherein: The method further comprises: When receiving the communication request, generating an identification number corresponding to the communication request; The identification number and the request information corresponding to the communication request are written into a preset request queue; wherein the request queue also stores time characterization information corresponding to the communication request and used to represent the response waiting time.

4. The method for handling communication anomalies according to claim 3, wherein: The method further comprises: When the communication request is sent to the downstream processor corresponding to the communication request, identification indication information is sent to the downstream processor; wherein the identification indication information includes the identification number and return indication information, and the return indication information is used to instruct the downstream processor to return request response information carrying the identification number.

5. The method for handling communication anomalies according to claim 4, wherein: The method further comprises: When receiving the request response information from the downstream processor, determining whether there is a target identification number identical to the identification number carried in the request response information in the request queue; When the target identification number does not exist in the request queue, the request response information is discarded and the response waiting time is continued to be calculated.

6. The method for handling communication anomalies according to claim 5, wherein: The method further comprises: When the target identification number exists in the request queue and the response waiting time corresponding to the target identification number is less than the waiting time threshold, the request response information is sent to the local programmable multiprocessor, and the calculation of the response waiting time of the target communication request corresponding to the target identification number is stopped; The target identification number, request information, and time representation information corresponding to the target communication request are removed from the request queue.

7. The method for handling communication anomalies according to claim 3, wherein: The method further comprises: When the response waiting time is greater than or equal to the waiting time threshold, the identification number, the request information, and the time representation information corresponding to the communication request are removed from the request queue.

8. The method for handling communication anomalies according to claim 3, wherein: The method of calculating the response waiting time of the communication request in real time when the communication request sent by the local programmable multiprocessor is received and the request response information corresponding to the communication request is not received specifically includes: When the communication request is received, a timing trigger signal is received from a timing trigger in real time; wherein the timing trigger is used to send the timing trigger signal based on a preset timing trigger accuracy; In a process in which the request response information is not received, in response to the timing trigger signal, updating the time representation information corresponding to the communication request in the request queue; The response waiting time is calculated in real time based on the time characterization information and the timing trigger accuracy.

9. The method for handling communication anomalies according to claim 8, wherein: The timing trigger is started at the moment when the request queue is empty and the communication control module receives the communication request; The stop time of the timing trigger is the time when the request queue is transformed into an empty queue.

10. The method for handling communication anomalies according to claim 1, wherein: When the response waiting time is greater than or equal to a preset waiting time threshold, sending virtual response information corresponding to the communication request to the local programmable multiprocessor specifically includes: When the response waiting time is greater than or equal to the waiting time threshold, obtaining a request type of the communication request; Based on preset virtual response information corresponding to different request types, virtual response information corresponding to the request type of the communication request is obtained and sent to the local programmable multiprocessor.

11. The method for handling communication anomalies according to claim 3, wherein: The interruption information includes: the identification number corresponding to the communication request, the request information, and the time representation information.

12. A method for handling communication anomalies, executed by a host, characterized in that: include: When receiving interrupt information reported by any communication control module from the interrupt controller, determining abnormal information that the communication request is delayed in response based on the interrupt information; wherein the interrupt information is reported by the communication control module when the response waiting time of the communication request is greater than or equal to a preset waiting time threshold; Interruption processing is performed based on the interruption information and the abnormality information.

13. The method for handling communication anomalies according to claim 12, wherein: The exception information includes exception handler information and exception cause.

14. The method for handling communication anomalies according to claim 13, wherein: The interrupt processing based on the interrupt information and the exception information specifically includes: determining, based on the interrupt information, a target processor to which the interrupt information is to be reported; Determining a current exception handler based on the exception handler information; Disconnect the communication connection between the target processor and the exception processor, and establish a communication connection between the target processor and the remaining downstream processors; wherein the remaining downstream processors are processors other than the exception processor that have computing power resources capable of processing the communication request.

15. The method for handling communication anomalies according to claim 12, wherein: The method further comprises: Based on the preset timing trigger accuracy, generating timing trigger indication information carrying the timing trigger accuracy; The timing trigger indication information is sent to a timing trigger; wherein the timing trigger indication information is used to instruct the timing trigger to send a timing trigger signal to the communication control module based on the timing trigger accuracy.

16. An electronic device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for handling communication anomalies according to any one of claims 1 to 11 is implemented, or the method for handling communication anomalies according to any one of claims 12 to 15 is implemented.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the communication anomaly processing method described in any one of claims 1 to 11, or implement the communication anomaly processing method described in any one of claims 12 to 15.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for handling a communication anomaly according to any one of claims 1 to 11 is implemented, or the method for handling a communication anomaly according to any one of claims 12 to 15 is implemented.

Images