Link detection system, method, electronic device

Abstract

The application discloses a link detection system and method and an electronic device, relates to the technical field of graphic processing units, and comprises a master device, a slave device and a processor. The master device is used for receiving a test instruction sent by the processor, randomly generating a plurality of test data packets, generating and sending a first test stream based on a first identifier of a target device, a second identifier of the master device and the plurality of test data packets, and sending a sending timestamp of the first test stream to the slave device. The slave device is used for generating response information and feeding back the response information to the master device according to a receiving timestamp of a second test stream, the second test stream and the sending timestamp. The master device is used for recording a return timestamp of the response information when the response information is received, and determining whether a communication link between the master device and the slave device is abnormal according to at least one of a bit error rate, a link rate, a receiving timestamp, a sending timestamp and a return timestamp included in the response information. The application can solve the problem of poor detection effect of the stability of the exclusive link between GPUs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of a graphics processing unit, and particularly relates to a link detection system and method and an electronic device. BACKGROUND

[0002] With the development of the field of artificial intelligence and high-performance computing, multi-GPU servers and GPU clusters have become mainstream computing architectures. GPUs interact with each other through high-speed links to perform tasks such as tensor parallel computing, model parameter synchronization, and memory pool sharing in deep learning. As the computing performance of GPUs improves, the demand for link bandwidth is also increasing, and the stability of the link corresponding to the GPU has an increasingly significant impact on the overall performance of the system.

[0003] In related technologies, only the Peripheral Component Interconnect Express (PCIE) link between a GPU and a host can be detected, or the network layer connectivity of the link between GPUs can be verified by using a traditional network detection tool, but the handshake failure problem of the physical layer and the protocol layer of the exclusive link between GPUs cannot be detected in depth, resulting in poor stability detection effect of the exclusive link between GPUs. SUMMARY

[0004] The present application provides a link detection system, method, and electronic device to at least solve the problem of poor stability detection effect of the exclusive link between GPUs in related technologies.

[0005] The present application provides a link detection system, which comprises:

[0006] A master device configured to receive a test instruction sent from a processor, the test instruction carrying a first identifier of a target slave device, randomly generate a plurality of test data packets, generate a first test stream based on the first identifier, a second identifier of the master device, and the plurality of test data packets, and send the first test stream and a sending timestamp of the first test stream to a slave device.

[0007] A slave device configured to generate response information and feed back to the master device according to a receiving timestamp of a received second test stream, the second test stream, and a sending timestamp, wherein the response information comprises a receiving timestamp of the slave device for receiving the second test stream, a bit error rate, and a link rate.

[0008] The master device is further configured to record a return timestamp of the response information when the response information is received, and determine whether a communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp.

[0009] The application provides a link detection method applied to a master device in a link detection system; the method comprises the following steps: receiving a test instruction sent from a processor, wherein the test instruction carries a first identifier of a target end device; generating a plurality of test data packets randomly; generating a first test stream based on the first identifier, a second identifier of the master device and the plurality of test data packets; sending the first test stream and a sending time stamp of the first test stream to a slave device of the link detection system, so that the slave device generates response information and feeds back to the master device, wherein the response information comprises a receiving time stamp of the slave device when receiving the second test stream, a bit error rate and a link rate; when the response information is received, recording a return time stamp of the response information, and determining whether a communication link between the master device and the slave device is abnormal according to at least one of the bit error rate, the link rate, the receiving time stamp, the sending time stamp and the return time stamp.

[0010] The application further provides a link detection device, which comprises the following parts: a first transceiving module, which is used for receiving a test instruction sent from a processor, wherein the test instruction carries a first identifier of a target end device; a first generating module, which is used for generating a plurality of test data packets randomly, and generating a first test stream based on the first identifier, a second identifier of the master device and the plurality of test data packets; the first transceiving module, which is used for sending the first test stream and a sending time stamp of the first test stream to a slave device of the link detection system, so that the slave device generates response information and feeds back to the master device, wherein the response information comprises a receiving time stamp of the slave device when receiving the second test stream, a bit error rate and a link rate; and a first processing module, which is used for recording a return time stamp of the response information when the response information is received, and determining whether a communication link between the master device and the slave device is abnormal according to at least one of the bit error rate, the link rate, the receiving time stamp, the sending time stamp and the return time stamp.

[0011] The application provides a link detection method applied to a slave device in a link detection system; the method comprises the following steps: receiving a second test stream, verifying whether a first identifier in a packet header included in a jth original test data sequence matches the slave device, wherein the second test stream is a test stream sent by a master device to the slave device, and the test stream is received by the slave device; the first test stream is generated by the master device based on a first identifier of a target end device carried in a test instruction sent by a processor and a second identifier of the master device; if it is verified that the first identifier matches the slave device, generating a target check code included in a jth target test data sequence according to a packet header and a test data packet included in the jth target test data sequence; and comparing whether the target check code included in the jth target test data sequence is consistent with an original check code included in an original test data sequence corresponding to the jth target test data sequence.

[0012] If the target check code included in the jth target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the jth target test data sequence, the number of error codes of the target test data sequence of the second frame number is counted. According to the number of error codes and the total amount of data of the original test data sequence of the first frame number, the error rate is determined. If the error rate is less than or equal to the first threshold value, the link rate of the communication link is calculated according to the receiving time stamp, the sending time stamp and the total amount of data. According to the receiving time stamp, the error rate and the link rate, the response information is generated.

[0013] The application further provides a link detection device, which comprises: a second transceiving module, configured to receive a second test stream and check whether a first identifier included in a packet header of a jth original test data sequence matches a slave device. The second test stream is a test stream received by the slave device after the master device sends a first test stream to the slave device. The first test stream is generated by the master device based on a first identifier of a target device and a second identifier of the master device carried in a test instruction sent by a processor.

[0014] A second generating module is configured to generate a target check code included in a jth target test data sequence according to a packet header and test data packets included in the jth target test data sequence if it is checked that the first identifier matches the slave device.

[0015] A comparing module is configured to compare whether the target check code included in the jth target test data sequence is consistent with an original check code included in an original test data sequence corresponding to the jth target test data sequence.

[0016] A second processing module is configured to count the number of error codes of the target test data sequence of the second frame number if the target check code included in the jth target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the jth target test data sequence. According to the number of error codes and the total amount of data of the original test data sequence of the first frame number, the error rate is determined. If the error rate is less than or equal to the first threshold value, the link rate of the communication link is calculated according to the receiving time stamp, the sending time stamp and the total amount of data.

[0017] The second generating module is further configured to generate the response information according to the receiving time stamp, the error rate and the link rate.

[0018] The application further provides an electronic device, which comprises: a memory configured to store a computer program; and a processor configured to execute the computer program to implement the steps of any one of the link detection methods.

[0019] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the link detection methods.

[0020] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the above link detection methods.

[0021] According to the application, the bit error rate can be used to detect the physical layer of the communication link, the link rate can be used to detect the protocol layer of the communication link, the receiving time stamp, the sending time stamp and the return time stamp can be used to detect the data link layer of the communication link, that is, the link detection system can deeply detect the problems of the physical layer, the protocol layer and the data link layer of the dedicated link between GPUs, improve the stability detection effect of the dedicated link between GPUs, avoid the interruption or failure of data transmission and the risk of data integrity of the link between GPUs, and improve the system performance stability and accurate fault positioning. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A topological structure diagram of a link detection system provided for the embodiments of the application;

[0024] Figure 2 A flowchart of a link detection method provided for the embodiments of the application;

[0025] Figure 3 A flowchart of another link detection method provided for the embodiments of the application;

[0026] Figure 4 A flowchart of another link detection method provided for the embodiments of the application;

[0027] Figure 5 A device structure block diagram of a link detection device provided for the embodiments of the application;

[0028] Figure 6 A device structure block diagram of another link detection device provided for the embodiments of the application;

[0029] Figure 7 A hardware structure diagram of an electronic device provided for the embodiments of the application. DETAILED DESCRIPTION

[0030] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0031] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0032] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] The embodiments of the present application are applied to the scene of detecting the exclusive link between GPUs. The exclusive link between GPUs can be a high-speed interconnection technology link, a Peripheral Component Interconnect Express (PCIE) link, or a Compute Express Link (CXL).

[0034] In the related art, only the PCIE link between the GPU and the host can be detected; or the network layer connectivity of the link between the GPUs is verified by a traditional network detection tool, and the handshake failure problem of the physical layer and the protocol layer of the exclusive link between the GPUs cannot be detected in depth, resulting in poor stability detection effect of the exclusive link between the GPUs.

[0035] To solve the above technical problems, the embodiments of the present application provide a link detection system, as shown in Figure 1 Figure 1 is a topological structure diagram of the link detection system provided by the embodiments of the present application; the link detection system 100 can include a master device 101 and a slave device 102; optionally, the link detection system 100 further includes a processor 103 and a bridge 104.

[0036] ​The host device 101 can be a host graphics processing unit. The host device 101 integrates a detection control module, which is configured to call a Compute Unified Device Architecture Application Programming Interface (CUDAAPI) and a high-speed interconnection technology debugging interface, to implement test stream generation, sending and result analysis.

[0037] The slave device 102 can be a slave graphics processing unit. The slave device 102 integrates a response processing module, which is embedded in firmware of the slave device, to parse a test stream based on an integrated circuit to integrated circuit protocol and record an error number recorded by an error counter and a link speed.

[0038] The processor 103 can be a central processing unit. The processor 103 is configured to send a test instruction to the host device 101.

[0039] The bridge 104 is configured to connect the host device 101 and the slave device 102.

[0040] The link between the host device 101 and the slave device 102 is preset as a preset link speed. For example, the preset link speed can be 200 GB / s. The link between the host device 101 and the slave device 102 enables a cyclic redundancy check (CRC) mechanism and a retry mechanism.

[0041] The host device is configured to receive a test instruction sent by the processor, randomly generate a plurality of test data packets, generate a first test stream based on a first identifier, a second identifier of the host device and the plurality of test data packets, and send the first test stream and a sending time stamp of the first test stream to the slave device.

[0042] The test instruction carries a first identifier of a target device. The test instruction is used to test a communication link between the host device and the slave device.

[0043] The first test stream includes a first frame number of original test data sequences. Each original test data sequence includes a packet header, a test data packet and an original check code. The test data packet is also referred to as a random load number. The original check code is a CRC check code. The first frame number can be 10,000 frames.

[0044] The sending time stamp can be in nanoseconds. The sending time stamp is used for subsequent calculation of link delay.

[0045] Specifically, the master device is configured to generate a packet header included in an ith frame of original test data sequence according to the first identifier and the second identifier; generate an ith original check code included in the ith frame of original test data sequence according to the ith test data packet and the packet header corresponding to the ith frame of original test data sequence; generate the ith frame of original test data sequence according to the ith packet header, the ith test data packet and the ith original check code; and obtain the first test stream after obtaining a same number of original test sequences as the test data packets.

[0046] wherein i is a positive integer.

[0047] In an example, the master device is further configured to send the first frame number of original test data sequences to the slave device frame by frame at a preset interval length.

[0048] The preset interval length can be set according to actual needs. For example, the preset interval length can be 10 microseconds.

[0049] It can be understood that, taking 128k as an example for each test data packet, the total amount of data of the first test stream is 1.25 GB.

[0050] The slave device 102 is configured to generate response information according to the receiving time stamp of the received second test stream, the second test stream and the sending time stamp, and feed back the response information to the master device 101.

[0051] The second test stream includes a second frame number of target test data sequences. The second frame number is less than or equal to the first frame number. It can be understood that the slave device is not sure whether the first test stream can be completely received, and therefore the second frame number of the second test stream received by the slave device is less than the first frame number.

[0052] The response information includes the receiving time stamp of the slave device receiving the second test stream, the bit error rate and the link rate.

[0053] Specifically, the slave device is specifically configured to: check whether the first identifier in the packet header included in the jth original test data sequence matches the slave device; if the first identifier is checked to match the slave device, generate a target check code included in the jth target test data sequence according to the packet header and the test data packet included in the jth target test data sequence; compare whether the target check code included in the jth target test data sequence and an original check code included in an original test data sequence corresponding to the jth target test data sequence are consistent; if the target check code included in the jth target test data sequence and the original check code included in the original test data sequence corresponding to the jth target test data sequence are consistent, count an error code number of the target test data sequence of the second frame number; determine an error code rate according to the error code number and a total amount of data of the original test data sequence of the first frame number; if the error code rate is less than or equal to a first threshold value, calculate a link rate of the communication link according to the receiving time stamp, the sending time stamp and the total amount of data; and generate response information according to the receiving time stamp, the error code rate and the link rate.

[0054] wherein j is a positive integer.

[0055] The first threshold value can be set according to actual needs, and is not limited. For example, the first threshold value can be set to 0.001%.

[0056] Specifically, the slave device is specifically configured to calculate a ratio between the error code number and the total amount of data of the original test data sequence of the first frame number, to obtain the error code rate.

[0057] Specifically, the slave device is specifically configured to calculate a difference between the receiving time stamp and the sending time stamp, to obtain a transmission duration; and calculate a ratio between the total amount of data and the transmission duration, to obtain the link rate of the communication link between the master device and the slave device.

[0058] Optionally, the slave device is specifically configured to return response information to the master device if it is checked that the first identifier does not match the slave device, the response information being used to indicate that the slave device incorrectly receives the second test stream, and the master device needs to resend the first test stream to the target end device.

[0059] Optionally, the slave device is specifically configured to return check failure information to the master device if it is checked that the target check code included in the jth target test data sequence and the original check code included in the original test data sequence corresponding to the jth target test data sequence are inconsistent. The master device determines a data link layer exception of the communication link based on the check failure information. It can be understood that the data link layer exception can be a driver configuration error or a protocol stack exception of the communication link.

[0060] The slave device is specifically configured to obtain a retry number of the master device sending the first test stream if the error code rate is greater than the first threshold value; and generate response information according to the receiving time stamp, the error code rate, the link rate and the retry number.

[0061] The master device 101 is further configured to record a return timestamp of the response information when the response information is received, and determine whether the communication link between the master device 101 and the slave device 102 is abnormal according to at least one of the error code rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp.

[0062] The communication link includes an uplink and a downlink between the master device and the slave device. The uplink is a transmission link from the master device to the slave device; and the downlink is a transmission link from the slave device to the master device.

[0063] The master device is specifically configured to determine that the physical layer of the communication link is abnormal if the error code rate is greater than the first threshold value.

[0064] It can be understood that if the error code rate between the master device and the slave device is greater than the first threshold value, it indicates that the signal of the physical layer of the communication link is attenuated, and the uplink fault of the communication link is located. At this time, the poor contact of the bridge may cause the signal attenuation of the physical layer, and the connection of the bridge can be checked to restore the normal communication link.

[0065] The master device is specifically configured to determine that the protocol layer of the communication link is abnormal if the link rate is less than the second threshold value.

[0066] The second threshold value can be set according to actual conditions and is not limited. For example, the second threshold value can be three-quarters of the preset link rate.

[0067] It can be understood that if the link rate between the master device and the slave device is less than the second threshold value, it indicates that the protocol layer negotiation of the communication link fails, and the downlink fault of the communication link is located. At this time, the equalization value configuration error between the master device and the slave device may cause the protocol layer negotiation to fail, and the equalization value of the communication link can be modified to restore the normal communication link.

[0068] The master device is specifically configured to calculate a delay difference between the uplink and the downlink according to the receiving timestamp, the sending timestamp, and the return timestamp; and determine that the symmetry of the communication link is poor if the delay difference is greater than a third threshold value.

[0069] The third threshold value can be set according to actual conditions and is not limited. The third threshold value can be 50ns.

[0070] Specifically, the master device is specifically configured to calculate a difference between the receiving timestamp and the sending timestamp to obtain a first transmission duration of the uplink; calculate a difference between the receiving timestamp and the return timestamp to obtain a second transmission duration of the downlink; and calculate a difference between the first transmission duration and the second transmission duration to obtain the delay difference between the uplink and the downlink.

[0071] It can be understood that the delay difference between the uplink and the downlink between the master device and the slave device is greater than the third threshold, indicating that there is a significant asymmetric anomaly in the communication link, and at this time, there is an anomaly in the protocol layer negotiation and the physical layer of the communication link between the master device and the slave device.

[0072] The master device is specifically configured to determine that the line on the hardware circuit board corresponding to the communication link is subject to electromagnetic interference if the bit error rate is greater than the first threshold and the link rate is greater than or equal to the second threshold.

[0073] It can be understood that the link rate greater than or equal to the second threshold excludes the existence of physical damage, poor contact, protocol errors and other faults in the communication link; and a high bit error rate directly points to "signal distortion caused by interference", and such interference can only be significantly triggered under a high-speed link, and the interference path is strongly related to the line on the hardware circuit board, so it is determined that the line on the hardware circuit board corresponding to the communication link is subject to electromagnetic interference.

[0074] When the bit error rate received by the master device is greater than the first threshold, and the response information further includes the number of retries, if the received number of retries is less than or equal to a preset number, the first test stream is retransmitted to the slave device. If the bit error rate returned by the slave device again is less than or equal to the first threshold, the first test stream is no longer retransmitted to the slave device. If the bit error rate returned by the slave device again is greater than the first threshold, the first test stream is continuously retransmitted to the slave device until the bit error rate returned by the slave device again is greater than the first threshold, or the number of retries is greater than the preset number.

[0075] The preset number can be set according to actual needs, and is not limited. For example, the preset number can be 3 times.

[0076] It can be understood that there are often transient and occasional interferences in the communication link, and such interferences will only cause a single bit error rate to exceed the standard, and are not persistent faults of the hardware. By allowing retries within a preset number of times, if the bit error rate returns to normal after retrying, the misjudgment of hardware failure can be directly excluded, without triggering subsequent complex troubleshooting processes (such as shutdown maintenance and replacement of hardware), reducing invalid operation and maintenance costs. In addition, when the bit error is caused by occasional problems, the retry mechanism can quickly restore normal signal transmission by retransmitting the test stream, avoiding interruption of communication or determination of link unavailability due to a single bit error exceeding the standard. When the number of retries reaches the preset number, subsequent processing such as fault alarm or link switching can be triggered in time, improving the resource utilization efficiency of the overall system.

[0077] In the embodiment of the present application, the master device records the return timestamp of the response information only when the response information is received within the preset time period, and determines whether the communication link between the master device and the slave device is abnormal according to at least one of the error code rate, the link rate, the receiving timestamp, the sending timestamp and the return timestamp. The master device receives the response information within the preset time period, indicating that the data link layer of the communication link between the master device and the slave device is normal.

[0078] The preset time period can be determined according to the link rate and the total amount of transmitted data. For example, the preset time period can be 500 ns.

[0079] Optionally, if the master device does not receive the response information within the preset time period, it is determined that the communication link is interrupted or the slave device is faulty.

[0080] Optionally, if the master device fails to send the first test stream, the master device sends an alarm information to the processor, and the prompt information is used to indicate that the driver of the master device is abnormal or the sending buffer is overflowed. Optionally, the master device can also push the alarm information to the baseboard management controller through the system management bus.

[0081] The link detection system, the master device, of the embodiment of the present application is used to receive a test instruction sent from the processor, randomly generate a plurality of test data packets, generate a first test stream based on the first identifier, the second identifier of the master device and the plurality of test data packets, and send the first test stream and the sending timestamp of the first test stream to the slave device. The slave device is used to generate a response information according to the receiving timestamp of the received second test stream, the second test stream and the sending timestamp, and feed back to the master device. The master device is also used to record the return timestamp of the response information when the response information is received, and determine whether the communication link between the master device and the slave device is abnormal according to at least one of the error code rate, the link rate, the receiving timestamp, the sending timestamp and the return timestamp.

[0082] Since the error code rate can detect the physical layer of the communication link, the link rate can detect the protocol layer of the communication link, and the receiving timestamp, the sending timestamp and the return timestamp can detect the data link layer of the communication link, that is, the link detection system can deeply detect the problems of the physical layer, the protocol layer and the data link layer of the dedicated link between GPUs, improve the stability detection effect of the dedicated link between GPUs, avoid the interruption or failure of data transmission and the risk of data integrity of the link between GPUs, and improve the system performance stability; through the bidirectional verification of the uplink sending test stream and the downlink receiving response information, the master GPU sending failure, the slave GPU receiving failure and the link medium failure can be distinguished, and the failure can be accurately located.

[0083] The embodiment of the present application provides a link detection method, which is applied to Figure 1The link detection system comprises a master device and a slave device; as shown in Figure 2 As shown in Figure 2 A flowchart of a link detection method provided by an embodiment of the application is shown in the figure. The link detection method comprises the following steps:

[0084] S201, the master device, configured to receive a test instruction sent by a processor; randomly generate a plurality of test data packets; generate a first test stream based on a first identifier, a second identifier of the master device and the plurality of test data packets, and send the first test stream and a sending timestamp of the first test stream to the slave device.

[0085] The test instruction carries a first identifier of a target device.

[0086] S202, the slave device, configured to generate response information based on a receiving timestamp of the received second test stream, the second test stream and the sending timestamp, and feed back to the master device.

[0087] The response information comprises a receiving timestamp of the slave device receiving the second test stream, a bit error rate and a link rate.

[0088] S203, the master device, further configured to record a return timestamp of the response information when the response information is received, and determine whether a communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp and the return timestamp.

[0089] For details of S201-S203, please refer to the embodiment shown in Figure 1 which will not be described here again.

[0090] An embodiment of the application provides another link detection method, which is applied to Figure 1 The master device of the link detection system is shown in Figure 3 As shown in Figure 3 A flowchart of another link detection method provided by an embodiment of the application is shown in the figure. The link detection method comprises the following steps:

[0091] S301, receiving a test instruction sent by a processor.

[0092] S302, randomly generating a plurality of test data packets.

[0093] S303, generating a first test stream based on a first identifier, a second identifier of the master device and the plurality of test data packets.

[0094] S304, sending the first test stream and a sending timestamp of the first test stream to a slave device of the link detection system, so that the slave device generates response information and feeds back to the master device.

[0095] S305, upon receiving the response information, recording a return timestamp of the response information, determining whether the communication link between the master device and the slave device is abnormal according to at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp.

[0096] For details of S301-S305, please refer to Figure 1 The embodiments are not described here again.

[0097] In some optional embodiments, the first test stream includes a first number of original test data sequences, the number of test data packets is the same as the number of original test data sequences; the i th original test data sequence includes a packet header generated according to the first identifier and the second identifier; the i th original test data sequence includes an i th original check code generated according to the i th test data packet and the packet header corresponding to the i th original test data sequence; the i th original test data sequence is generated according to the i th packet header, the i th test data packet, and the i th original check code; and the first test stream is obtained after a same number of original test sequences as the test data packets are acquired.

[0098] In some optional embodiments, if the bit error rate is greater than the first threshold, it is determined that the physical layer of the communication link is abnormal.

[0099] In some optional embodiments, if the link rate is less than the second threshold, it is determined that the protocol layer of the communication link is abnormal.

[0100] In some optional embodiments, the communication link includes an uplink and a downlink between the master device and the slave device; a delay difference between the uplink and the downlink is calculated according to the receiving timestamp, the sending timestamp, and the return timestamp; and if the delay difference is greater than a third threshold, it is determined that the symmetry of the communication link is poor.

[0101] In some optional embodiments, if the bit error rate is greater than the first threshold and the link rate is greater than or equal to the second threshold, it is determined that the line on the hardware circuit board corresponding to the communication link is subject to electromagnetic interference.

[0102] Based on Figure 3The method shown, the master device can generate a first test stream based on the test instruction, and the error code rate can detect the physical layer of the communication link, the link rate can detect the protocol layer of the communication link, the receiving time stamp, the sending time stamp, and the return time stamp can detect the data link layer of the communication link, that is, the link detection system can deeply detect the problems of the physical layer, the protocol layer, and the data link layer of the exclusive link between GPUs, improve the stability detection effect of the exclusive link between GPUs, avoid the data transmission interruption or failure and the data integrity risk of the link between GPUs, and improve the system performance stability, accurate fault positioning.

[0103] Embodiments of the present application provide another link detection method, which is applied to Figure 1 The slave device of the link detection system shown; as Figure 4 As shown, Figure 4 The flowchart of another link detection method provided by the embodiments of the present application, the link detection method comprises the following steps:

[0104] S401, receiving a second test stream, verifying whether the first identifier in the packet header included in the jth frame of original test data sequence matches the slave device.

[0105] The second test stream is the test stream received by the slave device when the master device sends the first test stream to the slave device. The first test stream is generated by the master device based on the first identifier of the target device carried in the test instruction sent by the processor and the second identifier of the master device.

[0106] S402, if it is verified that the first identifier matches the slave device, the target check code included in the jth target test data sequence is generated according to the packet header and the test data packet included in the jth target test data sequence.

[0107] S403, comparing whether the target check code included in the jth target test data sequence and the original check code included in the original test data sequence corresponding to the jth target test data sequence are consistent.

[0108] S404, if the target check code included in the jth target test data sequence and the original check code included in the original test data sequence corresponding to the jth target test data sequence are consistent, the error code number of the target test data sequence of the second frame number is counted.

[0109] S405, determining the error code rate according to the error code number and the total amount of data of the original test data sequence of the first frame number.

[0110] S406, if the error code rate is less than or equal to the first threshold value, the link rate of the communication link is calculated according to the receiving time stamp, the sending time stamp, and the total amount of data.

[0111] S407, generating response information according to the receiving time stamp, the error code rate and the link rate.

[0112] The response information is used for the master device to determine whether the communication link is abnormal based on the sending time stamp, the returning time stamp, and at least one of the error code rate, the link rate and the receiving time stamp included in the response information.

[0113] For details of S401-S407, please refer to Figure 1 The embodiments are not described here again.

[0114] In some optional embodiments, if the error code rate is greater than the first threshold, the retry number of the master device sending the first test stream is acquired; and the response information is generated according to the receiving time stamp, the error code rate, the link rate and the retry number.

[0115] Based on Figure 4 The method shown in the figure, the slave device can generate response information based on the receiving time stamp of the received second test stream, the second test stream and the sending time stamp, and feed back to the master device, since the error code rate included in the response information can be used to detect the physical layer of the communication link, the link rate can be used to detect the protocol layer of the communication link, the receiving time stamp, the sending time stamp and the returning time stamp can be used to detect the data link layer of the communication link, that is, the link detection system can deeply detect the problems of the physical layer, the protocol layer and the data link layer of the dedicated link between GPUs, improve the stability detection effect of the dedicated link between GPUs, avoid the interruption or failure of data transmission and the risk of data integrity of the link between GPUs, and improve the system performance stability and accurate fault positioning.

[0116] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0117] Embodiments of the present application also provide a link detection device, as shown in Figure 5 The figure shows a device structure block diagram of a link detection device provided by the embodiments of the present application; the device is applied to a master device in a link detection system; the link detection device comprises: Figure 5 The figure shows a device structure block diagram of a link detection device provided by the embodiments of the present application; the device is applied to a master device in a link detection system; the link detection device comprises:

[0118] The first transceiver module 501 is used for receiving a test instruction sent by the processor, and the test instruction carries a first identifier of a target device.

[0119] The first generation module 502 is used for randomly generating a plurality of test data packets; and generating a first test stream based on the first identifier, a second identifier of the master device and the plurality of test data packets.

[0120] The first transceiving module 501 is configured to send a first test stream and a sending time stamp of the first test stream to a slave device of a link detection system, and the first test stream and the sending time stamp of the first test stream are used to generate response information by the slave device and feed back to the master device, wherein the response information comprises a receiving time stamp of the slave device receiving a second test stream, a bit error rate and a link rate.

[0121] The first processing module 503 is configured to record a returning time stamp of the response information when the response information is received, and determine whether a communication link between the master device and the slave device is abnormal according to at least one of the bit error rate, the link rate, the receiving time stamp, the sending time stamp and the returning time stamp.

[0122] In some optional embodiments, the first test stream comprises a first number of original test data sequences, and the number of test data packets is the same as the number of the original test data sequences; the first generating module 502 is specifically configured to generate an i-th original test data sequence according to the first identifier and the second identifier; generate an i-th original check code included in the i-th original test data sequence according to an i-th test data packet and a packet header corresponding to the i-th original test data sequence; generate the i-th original test data sequence according to the i-th packet header, the i-th test data packet and the i-th original check code; and obtain the first test stream after obtaining a same number of original test sequences as the test data packets.

[0123] In some optional embodiments, the first processing module 503 is specifically configured to determine that a physical layer of the communication link is abnormal if the bit error rate is greater than a first threshold.

[0124] In some optional embodiments, the first processing module 503 is specifically configured to determine that a protocol layer of the communication link is abnormal if the link rate is less than a second threshold.

[0125] In some optional embodiments, the communication link comprises an uplink and a downlink between the master device and the slave device; and the first processing module 503 is specifically configured to calculate a delay difference between the uplink and the downlink according to the receiving time stamp, the sending time stamp and the returning time stamp; and determine that the communication link is asymmetric if the delay difference is greater than a third threshold.

[0126] In some optional embodiments, the first processing module 503 is specifically configured to determine that a line on a hardware circuit board corresponding to the communication link is subjected to electromagnetic interference if the bit error rate is greater than the first threshold and the link rate is greater than or equal to the second threshold.

[0127] Embodiments of the present application also provide another link detection device, as shown in Figure 6 Figure 6 ​A device structure block diagram of another link detection device provided by an embodiment of the present application; a slave device applied in a link detection system; the link detection device comprises:

[0128] The second transceiving module 601 is configured to receive a second test stream, and check whether a first identifier in a packet header included in the jth original test data sequence matches the slave device. The second test stream is a test stream received by the slave device from the master device. The first test stream is generated by the master device based on a first identifier of a target device and a second identifier of the master device carried in a test instruction sent by the processor.

[0129] The second generation module 602 is configured to, if it is checked that the first identifier matches the slave device, generate a target check code included in the jth target test data sequence according to the packet header and the test data packet included in the jth target test data sequence.

[0130] The comparison module 603 is configured to compare whether the target check code included in the jth target test data sequence is consistent with an original check code included in an original test data sequence corresponding to the jth target test data sequence.

[0131] The second processing module 604 is configured to, if the target check code included in the jth target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the jth target test data sequence, count an error code number of the target test data sequence of the second frame number. According to the error code number and a total amount of data of the original test data sequence of the first frame number, a bit error rate is determined. If the bit error rate is less than or equal to a first threshold value, a link rate of the communication link is calculated according to a receiving time stamp, a sending time stamp and the total amount of data.

[0132] The second generation module 602 is further configured to generate response information according to the receiving time stamp, the bit error rate and the link rate.

[0133] In some optional embodiments, the second processing module 604 is further configured to, if the bit error rate is greater than the first threshold value, acquire a retry number of the master device sending the first test stream; and the second generation module 602 is further configured to generate the response information according to the receiving time stamp, the bit error rate, the link rate and the retry number.

[0134] The features of the embodiments of the link detection device can be referred to the related descriptions of the embodiments of the link detection method, which will not be repeated here.

[0135] Embodiments of the present application also provide an electronic device, as shown in Figure 7 Figure 7 ​A hardware structure schematic diagram of an electronic device is provided in the embodiments of the present application. The electronic device comprises a processor 10 and a memory 20, the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to perform the steps in any of the above link detection method embodiments.

[0136] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is configured to perform the steps in any of the above link detection method embodiments when running.

[0137] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0138] The embodiments of the present application further provide a computer program product, which comprises a computer program. The computer program is executed by a processor to implement the steps in any of the above link detection method embodiments.

[0139] The embodiments of the present application further provide another computer program product, which comprises a non-volatile computer readable storage medium. The non-volatile computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps in any of the above link detection method embodiments.

[0140] The skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0141] The above provides a detailed introduction to the link detection system, method and electronic device provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the examples is only applicable to help understand the method and its core idea of the present application. It should be noted that for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways. These improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A link detection system, characterized in that, The system includes: The master device is configured to receive test instructions sent from the processor, the test instructions carrying a first identifier of the target device; randomly generate multiple test data packets; generate a first test stream based on the first identifier, the second identifier of the master device and the multiple test data packets, and send the first test stream and the sending timestamp of the first test stream to the slave device. The slave device is configured to generate response information and feed it back to the master device based on the received timestamp of the received second test stream, the second test stream, and the sent timestamp. The response information includes the received timestamp of the slave device completing the second test stream, the bit error rate, and the link rate. The master device is further configured to record the return timestamp of the response information when it receives the response information, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp. The first test stream includes a sequence of raw test data for a first number of frames, wherein the number of test data packets is the same as the number of raw test data sequences; the master device is specifically used for: Based on the first identifier and the second identifier, generate the packet header of the original test data sequence of the i-th frame, where i is a positive integer; Based on the packet header corresponding to the i-th test data packet and the original test data sequence of the i-th frame, generate the i-th original checksum included in the i-th original test data sequence; Based on the i-th packet header, the i-th test data packet, and the i-th original checksum, generate the original test data sequence of the i-th frame; After obtaining an equal number of original test sequences to the test data packet, the first test stream is obtained; The second test stream includes a target test data sequence with a second number of frames; the second number of frames is less than or equal to the first number of frames; the slave device is specifically used for: Receive the second test stream and check whether the first identifier in the packet header of the original test data sequence in the j-th frame matches the slave device, where j is a positive integer; If the first identifier is found to match the slave device, then the target check code included in the j-th target test data sequence is generated according to the packet header and test data packet included in the j-th frame of the target test data sequence; Compare whether the target check code included in the j-th target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the j-th target test data sequence; If the target checksum included in the j-th target test data sequence is consistent with the original checksum included in the original test data sequence corresponding to the j-th target test data sequence, then the number of bit errors in the target test data sequence of the second frame number is counted. The bit error rate is determined based on the number of bit errors and the total amount of data in the original test data sequence of the first frame number; If the bit error rate is less than or equal to the first threshold, then the link rate of the communication link is calculated based on the receiving timestamp, the sending timestamp, and the total amount of data. The response information is generated based on the received timestamp, the bit error rate, and the link rate; The slave device is specifically used to calculate the difference between the received timestamp and the sent timestamp to obtain the transmission duration; and to calculate the ratio between the total data volume and the transmission duration to obtain the link rate of the communication link between the master device and the slave device.

2. The system according to claim 1, characterized in that, The slave device is specifically used for: If the bit error rate is greater than the first threshold, obtain the number of retries the master device will make when sending the first test stream; The response information is generated based on the received timestamp, the bit error rate, the link rate, and the number of retries.

3. The system according to any one of claims 1-2, characterized in that, The master device is specifically used to determine that the physical layer of the communication link is abnormal if the bit error rate is greater than a first threshold.

4. The system according to claim 1 or 2, characterized in that, The master device is specifically used to determine that the protocol layer of the communication link is abnormal if the link rate is less than a second threshold.

5. The system according to any one of claims 1-2, characterized in that, The communication link includes an uplink and a downlink between the master device and the slave device; the master device is specifically used for: Calculate the delay difference between the uplink and the downlink based on the received timestamp, the sent timestamp, and the returned timestamp; If the delay difference is greater than the third threshold, then the symmetry difference of the communication link is determined.

6. The system according to claim 4, characterized in that, The main device is specifically configured to determine that electromagnetic interference has occurred on the lines of the hardware circuit board corresponding to the communication link if the bit error rate is greater than the first threshold and the link rate is greater than or equal to the second threshold.

7. A link detection method, characterized in that, The method is applied to the main device in the link detection system; the method includes: Receive a test command sent from the processor, the test command carrying a first identifier of the target device; Multiple test data packets are generated randomly; A first test stream is generated based on the first identifier, the second identifier of the master device, and multiple test data packets; The first test stream and its transmission timestamp are sent to the slave device of the link detection system. The first test stream and its transmission timestamp are used by the slave device to generate response information and feed it back to the master device. The response information includes the slave device's reception timestamp of the completed second test stream, bit error rate, and link rate. Upon receiving the response information, the return timestamp of the response information is recorded. Based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp, it is determined whether the communication link between the master device and the slave device is abnormal. The first test stream includes a sequence of raw test data for a first number of frames, and the number of test data packets is the same as the number of raw test data packets; generating the first test stream based on the first identifier, the second identifier of the master device, and multiple test data packets includes: Based on the first identifier and the second identifier, generate the packet header of the original test data sequence of the i-th frame, where i is a positive integer; Based on the packet header corresponding to the i-th test data packet and the original test data sequence of the i-th frame, generate the i-th original checksum included in the i-th original test data sequence; Based on the i-th packet header, the i-th test data packet, and the i-th original checksum, generate the original test data sequence of the i-th frame; After obtaining an equal number of original test sequences to the test data packet, the first test stream is obtained; The second test stream includes a target test data sequence of a second frame number; the second frame number is less than or equal to the first frame number; the slave device generating response information includes: receiving the second test stream, checking whether the first identifier in the header of the original test data sequence of the j-th frame matches the slave device, where j is a positive integer; if the first identifier matches the slave device, then generating a target checksum included in the j-th target test data sequence based on the header and test data packet included in the target test data sequence of the j-th frame; comparing the target checksum included in the j-th target test data sequence with the original test data sequence corresponding to the j-th target test data sequence. The test data sequence includes whether the original checksums are consistent; if the target checksum included in the j-th target test data sequence is consistent with the original checksum included in the original test data sequence corresponding to the j-th target test data sequence, then the number of bit errors in the target test data sequence of the second frame number is counted; the bit error rate is determined based on the number of bit errors and the total amount of data in the original test data sequence of the first frame number; if the bit error rate is less than or equal to a first threshold, then the link rate of the communication link is calculated based on the receiving timestamp, the sending timestamp, and the total amount of data; the response information is generated based on the receiving timestamp, the bit error rate, and the link rate. The slave device calculates the link rate of the communication link based on the received timestamp, the sent timestamp, and the total data volume, including: calculating the difference between the received timestamp and the sent timestamp to obtain the transmission duration; and calculating the ratio between the total data volume and the transmission duration to obtain the link rate of the communication link between the master device and the slave device.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the link detection method as described in claim 7.

Images