Link detection system and method, and electronic equipment

Abstract

The invention discloses a link detection system and method and electronic equipment, and relates to the technical field of graphic processing units, and the system comprises main equipment which is used for receiving a test instruction sent by a processor; randomly generating a plurality of test data packets; and generating and sending a sending timestamp of the first test flow to the slave device based on the first identifier of the target end device, the second identifier of the master device and the plurality of test data packets. And the slave device is used for generating response information and feeding back the response information to the master device according to the received receiving timestamp of the second test flow, the second test flow and the sending timestamp. And the master device is used for recording a return timestamp of the response information when receiving the response information, and determining whether a communication link between the master device and the slave device is abnormal or not according to at least one of an error rate, a link rate, a receiving timestamp, a sending timestamp and the return timestamp included in the response information. The problem that the stability detection effect of the exclusive link between the GPUs is poor can be solved.

Description

Technical Field

[0001] The present application relates to the field of graphics processing unit technology, and in particular to a link detection system, method, and electronic device. Background Art

[0002] With the advancement of artificial intelligence and high-performance computing, servers and GPU clusters with multiple graphics processing units (GPUs) have become mainstream computing architectures. GPUs exchange data via high-speed links. For example, in deep learning, GPUs perform tasks such as tensor parallel computing, model parameter synchronization, and memory pool sharing. As GPU computing performance improves, link bandwidth requirements are also increasing, and the stability of the corresponding GPU links has an increasingly significant impact on overall system performance.

[0003] Related technologies typically only detect the Peripheral Component Interconnect Express (PCIE) link between the GPU and the host. Alternatively, traditional network detection tools are used to verify the network layer connectivity of the link between GPUs. These tools fail to deeply detect handshake failures at the physical and protocol layers of the dedicated link between GPUs, resulting in poor stability detection of the dedicated link between GPUs. Summary of the Invention

[0004] The present application provides a link detection system, method, and electronic device to at least solve the problem of poor stability detection of exclusive links between GPUs in the related art.

[0005] The present application provides a link detection system, which includes: A master device is configured to receive a test instruction sent from a processor, the test instruction carrying a first identifier of a target device; randomly generate multiple test data packets; generate a first test stream based on the first identifier, a second identifier of the master device, and the multiple test data packets, and send the first test stream and a sending timestamp of the first test stream to a slave device.

[0006] The slave device is used to generate response information and feed it back to the master device based on the received reception timestamp of the second test stream, the second test stream and the sending timestamp, wherein the response information includes the reception timestamp, bit error rate and link rate of the second test stream received by the slave device.

[0007] The master device is also used to record the return timestamp of the response information when receiving the response information, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, link rate, receiving timestamp, sending timestamp, and return timestamp.

[0008] The present application provides a link detection method, which is applied to a master device in a link detection system; the method includes: receiving a test instruction sent from a processor, the test instruction carrying a first identifier of a target end device; randomly generating multiple test data packets; generating a first test stream based on the first identifier, a second identifier of the master device and multiple test data packets; sending the first test stream and the sending timestamp of the first test stream to a slave device of the link detection system, the first test stream and the sending timestamp of the first test stream are used to generate response information from the slave device and feed it back to the master device, the response information including a receiving timestamp, a bit error rate, and a link rate when the slave device completes receiving the second test stream; when the response information is received, recording the return timestamp of the response information, and determining whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the returning timestamp.

[0009] The present application also provides a link detection device, which includes: a first transceiver module for receiving a test instruction sent from a processor, the test instruction carrying a first identifier of a target end device. A first generation module for randomly generating multiple test data packets; generating a first test stream based on the first identifier, the second identifier of the master device and multiple test data packets. The first transceiver module is used to send the first test stream and the sending timestamp of the first test stream to a slave device of the link detection system, the first test stream and the sending timestamp of the first test stream are used to generate a response message from the slave device and feed it back to the master device, the response message including the receiving timestamp, bit error rate, and link rate of the second test stream received by the slave device. A first processing module is used to record the return timestamp of the response message when receiving the response message, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, link rate, receiving timestamp, sending timestamp, and return timestamp.

[0010] The present application provides a link detection method, which is applied to a slave device in a link detection system; the method includes: receiving a second test stream, checking whether the first identifier in the packet header included in the original test data sequence of the jth frame matches the slave device, the second test stream is a test stream sent by the master device to the slave device, and the slave device receives the test stream. The first test stream is generated by the master device based on the first identifier of the target end device and the second identifier of the master device carried in the test instruction sent by the processor. If it is checked that the first identifier matches the slave device, then the target check code included in the jth target test data sequence is generated according to the packet header and test data packet included in the target test data sequence of the jth frame. Compare the target check code included in the jth target test data sequence and the original check code included in the original test data sequence corresponding to the jth target test data sequence to see if they are consistent.

[0011] If the target check code included in the jth target test data sequence and the original check code included in the original test data sequence corresponding to the jth target test data sequence are consistent, then the number of bit errors in the second number of target test data sequences is counted. A bit error rate is determined based on the number of bit errors and the total amount of data in the first number of original test data sequences. If the bit error rate is less than or equal to a first threshold, a link rate of the communication link is calculated based on the receive timestamp, the transmit timestamp, and the total amount of data. A response message is generated based on the receive timestamp, the bit error rate, and the link rate.

[0012] The present application also provides a link detection device, which includes: a second transceiver module, configured to receive a second test stream and verify whether a first identifier in a packet header included in a j-th frame of the original test data sequence matches a slave device. The second test stream is a test stream generated by the master device based on a first identifier of a target device and a second identifier of the master device carried in a test instruction sent by a processor.

[0013] The second generating module is configured to generate a target check code included in the j-th target test data sequence according to the packet header and the test data packet included in the j-th frame target test data sequence if it is verified that the first identifier matches the slave device.

[0014] The comparison module is used to compare whether the target check code included in the j-th target test data sequence and the original check code included in the original test data sequence corresponding to the j-th target test data sequence are consistent.

[0015] The second processing module is configured to, if the target check code included in the j-th target test data sequence and the original check code included in the original test data sequence corresponding to the j-th target test data sequence are consistent, count the number of bit errors in the second number of target test data sequences, determine a bit error rate based on the number of bit errors and the total amount of data in the first number of original test data sequences, and calculate a link rate of the communication link based on the receive timestamp, the transmit timestamp, and the total amount of data if the bit error rate is less than or equal to a first threshold.

[0016] The second generating module is further configured to generate response information according to the receiving timestamp, the bit error rate and the link rate.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned link detection methods when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned link detection methods are implemented.

[0019] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned link detection methods when executed by a processor.

[0020] Through this application, since the bit error rate can detect the physical layer of the communication link, the link rate can detect the protocol layer of the communication link, and the receive timestamp, send timestamp, and return timestamp can detect the data link layer of the communication link, the link detection system can deeply detect problems in the physical layer, protocol layer, and data link layer of the exclusive link between GPUs, improve the stability detection effect of the exclusive link between GPUs, avoid data transmission interruption or failure and data integrity risks of the link between GPUs, improve system performance stability, and accurately locate faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A topological diagram of a link detection system provided in an embodiment of the present application; Figure 2 A flowchart of a link detection method provided in an embodiment of the present application; Figure 3 A flowchart of another link detection method provided in an embodiment of the present application; Figure 4 A flowchart of another link detection method provided in an embodiment of the present application; Figure 5 A block diagram of the structure of a link detection device provided in an embodiment of the present application; Figure 6 A block diagram of the structure of another link detection device provided in an embodiment of the present application; Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0025] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] The embodiments of the present application are applied to the scenario of detecting a dedicated link between GPUs. The dedicated link between GPUs can be a high-speed interconnect technology link, a Peripheral Component Interconnect Express (PCIE) link, or a Compute Express Link (CXL).

[0027] Related technologies can usually only detect the PCIE link between the GPU and the host; or, using traditional network detection tools to verify the network layer connectivity of the link between the GPUs, it is impossible to deeply detect the physical layer and protocol layer handshake failure problems of the dedicated link between the GPUs, resulting in poor stability detection of the dedicated link between the GPUs.

[0028] In order to solve the above technical problems, the present application provides a link detection system, such as Figure 1 As shown, Figure 1 A topological structure diagram of a link detection system provided in an embodiment of the present application; the link detection system 100 may include a master device 101 and a slave device 102; optionally, the link detection system 100 further includes a processor 103 and a bridge 104.

[0029] The master device 101 may be a master graphics processing unit (GPU). The master device 101 integrates a test control module that is used to call the CUDA API (Compute Unified Device Architecture Application Programming Interface) and a high-speed interconnect technology debugging interface to generate and send test streams and analyze test results.

[0030] The slave device 102 may be a slave graphics processing unit. The slave device 102 may include an integrated response processing module embedded in the slave device's firmware, which parses the test flow based on the integrated circuit to integrated circuit protocol and records the number of bit errors recorded by the error counter and the link speed.

[0031] The processor 103 may be a central processing unit and is configured to send a test instruction to the main device 101 .

[0032] The bridge 104 is used to connect the master device 101 and the slave device 102 .

[0033] The link rate between the master device 101 and the slave device 102 is pre-set to a preset link rate. For example, the preset link rate may be 200 GB / s. A cyclic redundancy check (CRC) mechanism and a retry mechanism are enabled on the link between the master device 101 and the slave device 102.

[0034] A master device is configured to receive a test instruction sent from a processor; randomly generate a plurality of test data packets; generate a first test stream based on a first identifier, a second identifier of the master device, and the plurality of test data packets, and send the first test stream and a sending timestamp of the first test stream to a slave device.

[0035] The test instruction carries the first identifier of the target device and is used to test the communication link between the master device and the slave device.

[0036] The first test stream includes a first number of frames of original test data sequence. Each original test data sequence includes a packet header, a test data packet, and an original check code. The test data packet is also called a random payload. The original check code is a CRC check code. The first number of frames can be 10,000 frames.

[0037] The accuracy of the sending timestamp can be in the nanosecond level. The sending timestamp is used for subsequent calculation of link delay.

[0038] Specifically, the main device is specifically used to generate a packet header included in the i-th frame original test data sequence based on the first identifier and the second identifier; generate an i-th original check code included in the i-th original test data sequence based on the packet header corresponding to the i-th test data packet and the i-th frame original test data sequence; generate an i-th frame original test data sequence based on the i-th packet header, the i-th test data packet and the i-th original check code; when the same number of original test sequences as the test data packets are obtained, the first test stream is obtained.

[0039] Wherein, i is a positive integer.

[0040] In one example, the master device is further configured to send a first number of frames of original test data sequence to the slave device frame by frame according to a preset interval duration.

[0041] The preset interval length can be set according to actual needs. For example, the preset interval length can be 10 microseconds.

[0042] It can be understood that, taking each test data packet as 128k as an example, the total amount of data in the first test flow is 1.25GB.

[0043] The slave device 102 is configured to generate response information according to the received reception timestamp of the second test stream, the second test stream, and the transmission timestamp, and feed the response information back to the master device 101 .

[0044] The second test stream includes a second number of frames of target test data sequence. The second number of frames is less than or equal to the first number of frames. It is understandable that the slave device is uncertain whether it can fully receive the first test stream. Therefore, the second number of frames of the second test stream received by the slave device is less than the first number of frames.

[0045] The response information includes a reception timestamp, a bit error rate, and a link rate of the second test stream received from the slave device.

[0046] Specifically, the slave device is specifically used to: check whether the first identifier in the packet header included in the j-th frame original test data sequence matches the slave device; if it is checked that the first identifier matches the slave device, then generate the target check code included in the j-th target test data sequence according to the packet header and test data packet included in the j-th frame target test data sequence; compare whether the target check code included in the j-th target test data sequence and the original check code included in the original test data sequence corresponding to the j-th target test data sequence are consistent; if the target check code included in the j-th target test data sequence and the original check code included in the original test data sequence corresponding to the j-th target test data sequence are consistent, then count the number of bit errors of the target test data sequence of the second frame number; determine the bit error rate according to the number of bit errors and the total amount of data of the original test data sequence of the first frame number; if the bit error rate is less than or equal to the first threshold, then calculate the link rate of the communication link according to the receiving timestamp, the sending timestamp and the total amount of data; generate response information according to the receiving timestamp, the bit error rate and the link rate.

[0047] Wherein, j is a positive integer.

[0048] The first threshold value can be set according to actual needs and is not limited. For example, the first threshold value can be set to 0.001%.

[0049] Specifically, the slave device is specifically configured to calculate the ratio between the number of bit errors and the total amount of data in the original test data sequence of the first frame number to obtain the bit error rate.

[0050] Specifically, the slave device is used to calculate the difference between the receiving timestamp and the sending timestamp to obtain the transmission duration; calculate the ratio between the total amount of data and the transmission duration to obtain the link rate of the communication link between the master device and the slave device.

[0051] Optionally, the slave device is specifically configured to return a response message to the master device if it is detected that the first identifier does not match the slave device. The response message is used to indicate that the slave device has incorrectly received the second test stream and the master device needs to resend the first test stream to the target device.

[0052] Optionally, the slave device is specifically configured to return a verification failure message to the master device if it is detected that the target verification code included in the j-th target test data sequence is inconsistent with the original verification code included in the original test data sequence corresponding to the j-th target test data sequence. The master device determines, based on the verification failure message, that a data link layer anomaly of the communication link is present. It is understood that the data link layer anomaly may be a driver configuration error or a protocol stack anomaly of the communication link.

[0053] The slave device is specifically configured to obtain the number of retries of the master device sending the first test stream if the bit error rate is greater than the first threshold; and generate response information according to the receiving timestamp, bit error rate, link rate and number of retries.

[0054] The master device 101 is also used to record the return timestamp of the response information when receiving the response information, and determine whether the communication link between the master device 101 and the slave device 102 is abnormal based on at least one of the bit error rate, link rate, receiving timestamp, sending timestamp, and return timestamp.

[0055] The communication link includes the uplink and downlink between the master device and the slave device. The uplink is the transmission link from the master device to the slave device; the downlink is the transmission link from the slave device to the master device.

[0056] The master device is specifically configured to determine that a physical layer of the communication link is abnormal if the bit error rate is greater than a first threshold.

[0057] As can be understood, a bit error rate between the master and slave devices greater than the first threshold indicates signal attenuation at the physical layer of the communication link, locating an uplink fault in the communication link. In this case, the attenuation at the physical layer may be caused by poor contact in the bridge. Checking the bridge connection can help restore the communication link.

[0058] The master device is specifically configured to determine that a protocol layer of the communication link is abnormal if the link rate is less than a second threshold.

[0059] The second threshold value can be set according to actual conditions and is not limited. For example, the second threshold value can be three quarters of the preset link rate.

[0060] As can be understood, if the link rate between the master and slave devices is less than the second threshold, it indicates that the protocol layer negotiation of the communication link has failed, and a downlink fault in the communication link has been located. In this case, the protocol layer negotiation failure may be caused by an incorrect configuration of the balance value of the communication link between the master and slave devices. The balance value of the communication link can be modified to restore normal communication link operation.

[0061] The master device is specifically configured to calculate a delay difference between an uplink and a downlink according to a receiving timestamp, a sending timestamp, and a returning timestamp; if the delay difference is greater than a third threshold, determine that the communication link has a poor symmetry.

[0062] The third threshold value can be set according to actual conditions and is not limited. The third threshold value can be 50ns.

[0063] Specifically, the master device is used to calculate the difference between the receiving timestamp and the sending timestamp to obtain the first transmission duration of the uplink; calculate the difference between the receiving timestamp and the returning timestamp to obtain the second transmission duration of the downlink; calculate the difference between the first transmission duration and the second transmission duration to obtain the delay difference between the uplink and the downlink.

[0064] It can be understood that the delay difference between the uplink and downlink between the master device and the slave device is greater than the third threshold, indicating that there is a significant asymmetric abnormality in the communication link. At this time, there are abnormalities in the protocol layer negotiation and physical layer of the communication link between the master device and the slave device.

[0065] The master device is specifically configured to determine that electromagnetic interference occurs in a line on a hardware circuit board corresponding to the communication link if the bit error rate is greater than a first threshold and the link rate is greater than or equal to a second threshold.

[0066] It can be understood that the link rate is greater than or equal to the second threshold, which means that physical damage, poor contact, protocol errors and other faults in the communication link are ruled out; and the high bit error rate directly points to "signal distortion caused by interference", and this interference will only be significantly triggered under high-speed links, and the interference path is strongly correlated with the hardware circuit board line. Therefore, it is determined that electromagnetic interference occurs in the lines on the hardware circuit board corresponding to the communication link.

[0067] When the bit error rate received by the master device is greater than a first threshold and the response information also includes a retry count, if the number of retries received is less than or equal to a preset number, the first test stream is resent to the slave device. If the bit error rate returned by the slave device is less than or equal to the first threshold again, the first test stream is no longer resent to the slave device. If the bit error rate returned by the slave device is greater than the first threshold again, the first test stream is continued to be resent to the slave device until the bit error rate returned by the slave device is greater than the first threshold again, or the number of retries exceeds the preset number.

[0068] The preset number of times can be set according to actual needs and is not limited. For example, the preset number of times can be 3 times.

[0069] Understandably, communication links often experience transient and occasional interference. This type of interference only causes a single bit error rate violation, not a persistent hardware failure. By allowing retries within a preset number of times, if the bit error rate returns to normal after retries, the misjudgment of a hardware failure can be eliminated without triggering subsequent complex troubleshooting processes (such as downtime for maintenance or hardware replacement), reducing ineffective operation and maintenance costs. Furthermore, when bit errors are caused by sporadic issues, the retry mechanism can quickly restore normal signal transmission by resending the test stream, avoiding communication interruption or link unavailability judgments due to a single bit error violation. When the number of retries reaches the preset number, subsequent processing such as fault alarms or link switching can be triggered in a timely manner, improving the resource utilization efficiency of the entire system.

[0070] In this embodiment of the present application, the master device records the return timestamp of the response message only when it receives the response message within a preset time period. Based on at least one of the bit error rate, link rate, receive timestamp, transmit timestamp, and return timestamp, the master device determines whether the communication link between the master and slave devices is abnormal. Receiving the response message within the preset time period indicates that the data link layer of the communication link between the master and slave devices is normal.

[0071] The preset time period can be determined based on the link rate and the total amount of transmitted data. For example, the preset time period can be 500 ns.

[0072] Optionally, if the master device does not receive a response message within a preset time period, it is determined that the communication link is interrupted or the slave device is faulty.

[0073] Optionally, if the master device fails to send the first test stream, the master device sends an alarm message to the processor, indicating that the master device's driver is abnormal or the send buffer overflows. Optionally, the master device can also push the alarm message to the baseboard management controller via the system management bus.

[0074] In the link detection system of an embodiment of the present application, a master device is configured to receive a test instruction sent from a processor; randomly generate multiple test data packets; generate a first test stream based on a first identifier, a second identifier of the master device, and multiple test data packets, and send the first test stream and the send timestamp of the first test stream to a slave device. The slave device is configured to generate a response message based on the received receive timestamp of the second test stream, the second test stream, and the send timestamp, and feed it back to the master device; the master device is further configured to record the return timestamp of the response message upon receipt, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, link rate, receive timestamp, send timestamp, and return timestamp.

[0075] Since the bit error rate can detect the physical layer of the communication link, the link rate can detect the protocol layer of the communication link, and the receive timestamp, send timestamp, and return timestamp can detect the data link layer of the communication link, the link detection system can deeply detect problems in the physical layer, protocol layer, and data link layer of the exclusive link between GPUs, improve the stability detection effect of the exclusive link between GPUs, avoid data transmission interruption or failure and data integrity risks in the link between GPUs, and improve system performance stability; through the two-way verification of the uplink sending test flow and the downlink receiving response information, it can distinguish between the main GPU sending failure, the slave GPU receiving failure and the link medium failure, and accurately locate the fault.

[0076] The embodiment of the present application provides a link detection method, which is applied to Figure 1 The link detection system shown in FIG. 1 includes a master device and a slave device; Figure 2 As shown, Figure 2 A schematic diagram of a link detection method provided in an embodiment of the present application is provided, wherein the link detection method includes the following steps: S201, a master device, is configured to receive a test instruction sent from a processor; randomly generate multiple test data packets; generate a first test stream based on a first identifier, a second identifier of the master device, and the multiple test data packets, and send the first test stream and a sending timestamp of the first test stream to a slave device.

[0077] The test instruction carries the first identifier of the target device.

[0078] S202: The slave device is configured to generate response information according to the received receiving timestamp of the second test stream, the second test stream, and the sending timestamp, and feed the response information back to the master device.

[0079] The response information includes a reception timestamp, a bit error rate, and a link rate of the second test stream received from the slave device.

[0080] S203, the master device is further used to record the return timestamp of the response information when receiving the response information, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, link rate, receiving timestamp, sending timestamp, and return timestamp.

[0081] For details of S201-S203 above, please refer to Figure 1 The embodiments shown will not be described in detail here.

[0082] The embodiment of the present application provides another link detection method, which is applied to Figure 1 The main device of the link detection system shown; Figure 3 As shown, Figure 3 A flowchart of another link detection method provided in an embodiment of the present application is provided, wherein the link detection method includes the following steps: S301, receiving a test instruction sent from a processor.

[0083] S302: Randomly generate multiple test data packets.

[0084] S303: Generate a first test flow based on the first identifier, the second identifier of the master device, and a plurality of test data packets.

[0085] S304: Send the first test flow and the sending timestamp of the first test flow to the slave device of the link detection system. The first test flow and the sending timestamp of the first test flow are used by the slave device to generate response information and feed it back to the master device.

[0086] S305, when receiving the response information, record the return timestamp of the response information, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, link rate, receiving timestamp, sending timestamp, and returning timestamp.

[0087] For details of S301-S305 above, please refer to Figure 1 The embodiments shown will not be described in detail here.

[0088] In some optional embodiments, the first test stream includes a first number of frames of original test data sequences, and the number of test data packets is the same as the number of original test data sequences; based on the first identifier and the second identifier, a packet header included in the i-th frame original test data sequence is generated; based on the packet header corresponding to the i-th test data packet and the i-th frame original test data sequence, an i-th original check code included in the i-th original test data sequence is generated; based on the i-th packet header, the i-th test data packet and the i-th original check code, an i-th frame original test data sequence is generated; when the same number of original test sequences as the test data packets are obtained, the first test stream is obtained.

[0089] In some optional implementations, if the bit error rate is greater than a first threshold, it is determined that the physical layer of the communication link is abnormal.

[0090] In some optional implementations, if the link rate is less than a second threshold, it is determined that the protocol layer of the communication link is abnormal.

[0091] In some optional embodiments, the communication link includes an uplink and a downlink between the master device and the slave device; the delay difference between the uplink and the downlink is calculated based on the receive timestamp, the send timestamp and the return timestamp; if the delay difference is greater than a third threshold, the symmetry difference of the communication link is determined.

[0092] In some optional implementations, if the bit error rate is greater than a first threshold and the link rate is greater than or equal to a second threshold, it is determined that electromagnetic interference occurs in the line on the hardware circuit board corresponding to the communication link.

[0093] based on Figure 3 According to the method shown, the master device can generate a first test stream based on the test instruction, and detect the physical layer of the communication link according to the bit error rate in the response information received and returned by the slave device, detect the protocol layer of the communication link according to the link rate, and detect the data link layer of the communication link according to the receiving timestamp, sending timestamp, and returning timestamp. That is, the link detection system can deeply detect problems in the physical layer, protocol layer, and data link layer of the exclusive link between GPUs, improve the stability detection effect of the exclusive link between GPUs, avoid data transmission interruption or failure and data integrity risks of the link between GPUs, improve system performance stability, and accurately locate faults.

[0094] The embodiment of the present application provides another link detection method, which is applied to Figure 1 The slave device of the link detection system shown; Figure 4 As shown, Figure 4 A flowchart of another link detection method provided in an embodiment of the present application, the link detection method includes the following steps: S401: Receive a second test stream and check whether a first identifier in a packet header included in a j-th frame original test data sequence matches a slave device.

[0095] The second test stream is a test stream generated by the master device based on the first identifier of the target device and the second identifier of the master device carried in the test instruction sent by the processor.

[0096] S402: If it is verified that the first identifier matches the slave device, a target check code included in the j-th target test data sequence is generated according to the packet header and the test data packet included in the j-th frame target test data sequence.

[0097] S403 : Compare the target check code included in the j th target test data sequence and the original check code included in the original test data sequence corresponding to the j th target test data sequence to see whether they are consistent.

[0098] S404 , if the target check code included in the j th target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the j th target test data sequence, then counting the number of bit errors of the target test data sequence of the second number of frames.

[0099] S405 , determining a bit error rate according to the number of bit errors and the total amount of data in the original test data sequence of the first number of frames.

[0100] S406: If the bit error rate is less than or equal to the first threshold, calculate the link rate of the communication link according to the receiving timestamp, the sending timestamp and the total amount of data.

[0101] S407: Generate response information according to the receiving timestamp, bit error rate and link rate.

[0102] The response information is used by the master device to determine whether the communication link is abnormal based on the sending timestamp, the returning timestamp, and at least one of the bit error rate, the link rate, and the receiving timestamp included in the response information.

[0103] For details of S401-S407 above, please refer to Figure 1 The embodiments shown will not be described in detail here.

[0104] In some optional implementations, if the bit error rate is greater than a first threshold, the number of retries of the master device sending the first test stream is obtained; and response information is generated based on the receiving timestamp, bit error rate, link rate and number of retries.

[0105] based on Figure 4 According to the method shown, the slave device can generate a response message and feed it back to the master device based on the received receive timestamp of the second test stream, the second test stream and the send timestamp. Since the bit error rate included in the response information can be used to detect the physical layer of the communication link, the link rate can be used to detect the protocol layer of the communication link, the receive timestamp, the send timestamp and the return timestamp can be used to detect the data link layer of the communication link, that is, the link detection system can deeply detect the problems of the physical layer, protocol layer and data link layer of the exclusive link between GPUs, improve the stability detection effect of the exclusive link between GPUs, avoid data transmission interruption or failure and data integrity risks of the link between GPUs, improve system performance stability and accurately locate faults.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0107] The embodiment of the present application also provides a link detection device, such as Figure 5 As shown, Figure 5 A block diagram of the structure of a link detection device provided in an embodiment of the present application; a main device used in a link detection system; the link detection device includes: The first transceiver module 501 is configured to receive a test instruction sent from a processor, where the test instruction carries a first identifier of a target device.

[0108] The first generating module 502 is configured to randomly generate a plurality of test data packets; and generate a first test flow based on the first identifier, the second identifier of the master device, and the plurality of test data packets.

[0109] The first transceiver module 501 is used to send the first test stream and the sending timestamp of the first test stream to the slave device of the link detection system. The first test stream and the sending timestamp of the first test stream are used to generate response information from the slave device and feed it back to the master device. The response information includes the receiving timestamp, bit error rate, and link rate of the second test stream received by the slave device.

[0110] The first processing module 503 is used to record the return timestamp of the response information when receiving the response information, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, link rate, receiving timestamp, sending timestamp, and returning timestamp.

[0111] In some optional embodiments, the first test stream includes a first number of frames of original test data sequences, and the number of test data packets is the same as the number of original test data sequences; the first generation module 502 is specifically used to generate a packet header included in the i-th frame original test data sequence based on the first identifier and the second identifier; generate an i-th original check code included in the i-th original test data sequence based on the packet header corresponding to the i-th test data packet and the i-th frame original test data sequence; generate an i-th frame original test data sequence based on the i-th packet header, the i-th test data packet and the i-th original check code; when the same number of original test sequences as the test data packets are obtained, the first test stream is obtained.

[0112] In some optional implementations, the first processing module 503 is specifically configured to determine that a physical layer of the communication link is abnormal if the bit error rate is greater than a first threshold.

[0113] In some optional implementations, the first processing module 503 is specifically configured to determine that a protocol layer of the communication link is abnormal if the link rate is less than a second threshold.

[0114] In some optional embodiments, the communication link includes an uplink and a downlink between the master device and the slave device; the first processing module 503 is specifically used to calculate the delay difference between the uplink and the downlink based on the receiving timestamp, the sending timestamp and the return timestamp; if the delay difference is greater than a third threshold, the symmetry difference of the communication link is determined.

[0115] In some optional implementations, the first processing module 503 is specifically configured to determine that electromagnetic interference occurs in a line on a hardware circuit board corresponding to the communication link if the bit error rate is greater than a first threshold and the link rate is greater than or equal to a second threshold.

[0116] The embodiment of the present application also provides another link detection device, such as Figure 6 As shown, Figure 6 A block diagram of the structure of another link detection device provided in an embodiment of the present application; applied to a slave device in a link detection system; the link detection device includes: The second transceiver module 601 is configured to receive a second test stream and verify whether the first identifier in the packet header of the j-th frame of the original test data sequence matches the first identifier of the slave device. The second test stream is generated by the master device based on the first identifier of the target device and the second identifier of the master device carried in the test instruction sent by the processor.

[0117] The second generating module 602 is configured to generate a target check code included in the jth target test data sequence according to the header and test data packet included in the jth frame target test data sequence if it is verified that the first identifier matches the slave device.

[0118] The comparison module 603 is configured to compare the target check code included in the j-th target test data sequence and the original check code included in the original test data sequence corresponding to the j-th target test data sequence to determine whether they are consistent.

[0119] The second processing module 604 is configured to count the number of bit errors in the second number of target test data sequences if the target check code included in the j-th target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the j-th target test data sequence. The second processing module 604 is configured to determine the bit error rate based on the number of bit errors and the total amount of data in the first number of original test data sequences. If the bit error rate is less than or equal to the first threshold, the link rate of the communication link is calculated based on the receive timestamp, the transmit timestamp, and the total amount of data.

[0120] The second generating module 602 is further configured to generate response information according to the receiving timestamp, the bit error rate and the link rate.

[0121] In some optional embodiments, the second processing module 604 is further used to obtain the number of retries of the master device to send the first test stream if the bit error rate is greater than the first threshold; the second generation module 602 is further used to generate response information based on the receiving timestamp, bit error rate, link rate and number of retries.

[0122] For the description of the features in the embodiment corresponding to the link detection device, reference can be made to the relevant description of the embodiment corresponding to the link detection method, and no further details will be given here.

[0123] The embodiment of the present application also provides an electronic device, such as Figure 7 As shown, Figure 7The electronic device includes a processor 10 and a memory 20, wherein the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to perform the steps of any of the above-mentioned link detection method embodiments.

[0124] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above-mentioned link detection method embodiments when running.

[0125] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0126] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned link detection method embodiments are implemented.

[0127] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned link detection method embodiments are implemented.

[0128] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] The above is a detailed introduction to the link detection system, method, and electronic device provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A link detection system, characterized in that: The system comprises: A master device is configured to receive a test instruction sent from a processor, the test instruction carrying a first identifier of a target device; randomly generate a plurality of test data packets; generate a first test stream based on the first identifier, a second identifier of the master device, and the plurality of test data packets, and send the first test stream and a sending timestamp of the first test stream to a slave device; The slave device is configured to generate response information based on the received reception timestamp of the second test stream, the second test stream, and the transmission timestamp, and feed the response information back to the master device, wherein the response information includes the reception timestamp, bit error rate, and link rate of the second test stream when the slave device completes reception; The master device is further used to record the return timestamp of the response information when receiving the response information, and determine whether the communication link between the master device and the slave device is abnormal based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp.

2. The system according to claim 1, wherein: The first test stream includes a first number of frames of original test data sequences, and the number of the test data packets is the same as the number of the original test data sequences; the master device is specifically configured to: Generate a packet header included in the original test data sequence of the i-th frame according to the first identifier and the second identifier, where i is a positive integer; generating, according to the packet header corresponding to the i-th test data packet and the i-th frame of the original test data sequence, an i-th original check code included in the i-th original test data sequence; Generate the i-th frame of the original test data sequence according to the i-th packet header, the i-th test data packet and the i-th original check code; After obtaining original test sequences with a number equal to the test data packets, the first test stream is obtained.

3. The system according to claim 2, characterized in that The second test stream includes a target test data sequence of a second number of frames; the second number of frames is less than or equal to the first number of frames; and the slave device is specifically configured to: receiving the second test stream, and checking whether the first identifier in the packet header included in the original test data sequence of the jth frame matches the slave device, where j is a positive integer; If it is verified that the first identifier matches the slave device, generating a target check code included in the j-th target test data sequence according to the packet header and the test data packet included in the j-th target test data sequence; comparing a target check code included in the j-th target test data sequence and an original check code included in an original test data sequence corresponding to the j-th target test data sequence to determine whether they are consistent; If the target check code included in the j-th target test data sequence is consistent with the original check code included in the original test data sequence corresponding to the j-th target test data sequence, then counting the number of bit errors of the target test data sequence of the second number of frames; determining the bit error rate according to the number of bit errors and the total amount of data in the original test data sequence of the first number of frames; If the bit error rate is less than or equal to a first threshold, calculating the link rate of the communication link according to the receiving timestamp, the sending timestamp and the total amount of data; The response information is generated according to the receiving timestamp, the bit error rate and the link rate.

4. The system according to claim 3, characterized in that The slave device is specifically configured to: If the bit error rate is greater than the first threshold, obtaining the number of retries of the master device to send the first test stream; The response information is generated according to the receiving timestamp, the bit error rate, the link rate and the number of retries.

5. The system according to any one of claims 1 to 4, characterized in that: The master device is specifically configured to determine that a physical layer of the communication link is abnormal if the bit error rate is greater than a first threshold.

6. The system according to any one of claims 3 or 4, characterized in that The master device is specifically configured to determine that a protocol layer of the communication link is abnormal if the link rate is less than a second threshold.

7. The system according to any one of claims 1 to 4, characterized in that: The communication link includes an uplink and a downlink between the master device and the slave device; the master device is specifically configured to: Calculating a delay difference between the uplink and the downlink according to the receiving timestamp, the sending timestamp, and the return timestamp; If the delay difference is greater than a third threshold, it is determined that the communication link has a symmetry problem.

8. The system according to claim 6, wherein: The master device is specifically configured to determine that electromagnetic interference occurs in a line on a hardware circuit board corresponding to the communication link if the bit error rate is greater than the first threshold and the link rate is greater than or equal to the second threshold.

9. A link detection method, characterized in that: A master device applied to a link detection system; the method comprising: receiving a test instruction sent from a processor, wherein the test instruction carries a first identifier of a target end device; Randomly generate multiple test data packets; generating a first test flow based on the first identifier, the second identifier of the master device, and the plurality of test data packets; Sending the first test stream and the sending timestamp of the first test stream to a slave device of the link detection system, wherein the first test stream and the sending timestamp of the first test stream are used by the slave device to generate response information and feed it back to the master device, wherein the response information includes the receiving timestamp, bit error rate, and link rate of the second test stream received by the slave device; When the response information is received, the return timestamp of the response information is recorded, and whether the communication link between the master device and the slave device is abnormal is determined based on at least one of the bit error rate, the link rate, the receiving timestamp, the sending timestamp, and the return timestamp.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the link detection method according to claim 9 when executing the computer program.

Images