Computing power server detection method and device, computer equipment and storage medium

Abstract

The invention relates to a computing power server detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a sequence set; for each sequence, determining a communication bandwidth between computing power cards in an estimated server corresponding to the sequence according to a query bandwidth between every two connected computing power cards in the sequence; traversing the sequence, determining a pre-estimated inter-server communication bandwidth corresponding to the current traversed sequence, stopping traversing when a traversing stopping conditional expression is met, and screening out a sample sequence from the traversed sequence; determining a communication bandwidth between standard servers according to the sample sequence; and acquiring a measured value of communication bandwidth between the computing power cards in the server and a measured value of communication bandwidth between the servers, and generating a computing power server detection report according to the standard communication bandwidth between the servers, the measured value of the communication bandwidth between the computing power cards in the server and the measured value of the communication bandwidth between the servers. By adopting the method, whether the computing power server has problems or not can be detected.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a computing power server detection method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] In the field of supercomputing, it is very common for multiple computing cards to complete calculations together, and multiple computing servers to complete distributed computing together. A computing card is a hardware device specifically used to improve the computing power of a server. It usually includes high-performance processing units such as GPU (graphics processing unit) or TPU (tensor processing unit) to accelerate the computing tasks of the server. A computing server refers to a server deployed with a computing card.

[0003] The collaborative work between computing power cards has complex requirements for the interconnection between computing power cards from the hardware to the software level. In multi-card collaboration, especially in distributed computing scenarios, multiple computing power cards will form a ring communication. In order to avoid the barrel effect in communication, the installation topology of the cards on the server must be guaranteed first, and the hardware ratio must be balanced. This hardware ratio and topology design is complex and involves multiple collaborative links. Errors in each link lead to high configuration and low performance of the computing power server. The existing technology lacks automatic detection tools to check the assembled computing power server to determine whether there are any problems with the computing power server. Summary of the invention

[0004] Based on this, it is necessary to provide a computing power server detection method, device, computer equipment, computer-readable storage medium and computer program product that can check problems on the computing power server in response to the above technical problems.

[0005] In a first aspect, the present application provides a computing power server detection method. The method comprises:

[0006] Obtain a sequence set corresponding to each computing server in the computing server cluster; each sequence in each of the sequence sets includes a computing card and a communication card deployed on the corresponding computing server;

[0007] For each of the sequences, determining the estimated communication bandwidth between computing power cards in the server corresponding to the sequence according to the query bandwidth between each two connected computing power cards in the sequence;

[0008] Traversing the sequence in descending order of the estimated communication bandwidth between computing power cards in the server, determining the estimated communication bandwidth between servers corresponding to the currently traversed sequence, and stopping the traversal when the traversal stop condition is met, and filtering out the sample sequence from the traversed sequence;

[0009] Determine a standard inter-server communication bandwidth according to the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth corresponding to the sample sequence;

[0010] The actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers obtained by performing stress testing are obtained, and a computing power server detection report is generated according to the standard communication bandwidth between servers, the actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers.

[0011] In a second aspect, the present application also provides a computing power server detection device. The device includes:

[0012] The module for estimating the communication bandwidth between computing power cards in the server is used to obtain a sequence set corresponding to each computing power server in the computing power server cluster; each sequence in each sequence set includes a computing power card and a communication card deployed on the corresponding computing power server; for each sequence, the estimated communication bandwidth between computing power cards in the server corresponding to the sequence is determined according to the query bandwidth between each two connected computing power cards in the sequence;

[0013] The standard inter-server communication bandwidth determination module is used to traverse the sequence in descending order of the estimated communication bandwidth between computing power cards in the server, determine the estimated communication bandwidth between servers corresponding to the currently traversed sequence, stop traversing when the traversal stop condition is met, and filter out a sample sequence from the traversed sequence; determine the standard inter-server communication bandwidth according to the estimated communication bandwidth between computing power cards in the server and the estimated communication bandwidth between servers corresponding to the sample sequence;

[0014] The report generation module is used to obtain the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers obtained by stress testing, and generate a computing power server detection report based on the standard communication bandwidth between servers, the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers.

[0015] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any one of the computing power server detection methods in the embodiments of the present application is implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any one of the computing power server detection methods in the embodiments of the present application is implemented.

[0017] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, any computing power server detection method in the embodiments of the present application is implemented.

[0018] The above-mentioned computing power server detection method, device, computer equipment, storage medium and computer program product can determine the estimated communication bandwidth between computing power cards in the server corresponding to each sequence by determining multiple sequences. By determining the estimated communication bandwidth between computing power cards in the server, each sequence can be traversed in the order of the estimated communication bandwidth between computing power cards in the server from large to small to determine the estimated communication bandwidth between servers corresponding to the traversed sequence. When the sequence currently traversed meets the traversal stop condition, the traversal is stopped, and a sample sequence is obtained in which the communication card bandwidth performance can be fully utilized when the communication bandwidth between computing power cards in the server is as large as possible. By determining the sample sequence, the smaller value of the estimated communication bandwidth between computing power cards in the server and the estimated communication bandwidth between servers corresponding to the sample sequence can be used as the standard communication bandwidth between servers for subsequent bandwidth comparison. The standard communication bandwidth between servers is a bandwidth that a computing power server cluster can achieve under normal circumstances in theory. Therefore, by obtaining the actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers based on the stress test, the actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers can be compared with the standard communication bandwidth between servers to determine whether there is a problem with the computing power server cluster based on the comparison results, and generate a server detection report marked with the comparison results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a diagram of an application environment of a computing power server detection method in one embodiment;

[0020] FIG2 is a schematic diagram of a flow chart of a computing power server detection method in one embodiment;

[0021] FIG3 is a schematic diagram of determining a sample sequence in one embodiment;

[0022] FIG4 is a schematic diagram of the overall process of computing power server detection in one embodiment;

[0023] FIG5 is a structural block diagram of a computing power server detection device in one embodiment;

[0024] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The computing power server detection method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the detection server 102 communicates with the computing power server 104 through the network. The data storage system can store the data that the computing power server 104 needs to process. The data storage system can be integrated on the computing power server 104, or it can be placed on the cloud or other servers. The detection server 102 can generate a sequence set corresponding to each computing power server 104, and process based on the sequence set to obtain a processing result, and generate a computing power server detection report based on the processing result. Among them, the computing power server 104 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0027] In one embodiment, as shown in FIG2 , a computing power server detection method is provided, and the method is applied to the detection server in FIG1 as an example for explanation, wherein the detection server may be one of the computing power servers in the computing power server cluster, or may be another server other than the computing power server cluster. The computing power server detection method includes the following steps:

[0028] Step 202, obtain the sequence set corresponding to each computing server in the computing server cluster; each sequence in each sequence set includes the computing power card and communication card deployed on the corresponding computing power server.

[0029] Specifically, the detection server can determine the computing power cards and communication cards deployed on each computing power server, and collect the topological relationship of the computing power cards on the computing power server, as well as the hardware connection information between the computing power cards on the computing power server and other computing power servers. The topological relationship of the computing power cards on the computing power server may include the hardware connection information between the computing power cards. The communication card refers to the hardware device used for communication between servers.

[0030] Furthermore, the topology analyzer in the detection server can enumerate the combination relationship of all topological sequences between the computing power cards based on the collected information, and obtain the sequence set corresponding to each computing power server in the computing power server cluster. For example, for computing power server 1, 8 computing power cards GPU0 to GPU7 are deployed on computing power server 1, and 2 communication cards NIC0 to NIC1 are also deployed. Then the detection server can enumerate the series connection sequence of these 8 computing power cards to obtain multiple computing power card strings, and the detection server connects the computing power card string and the communication card in series to obtain a sequence. Exemplarily, the computing power card string can be GPU0-GPU1-GPU2-GPU3-GPU4-GPU5-GPU6-GPU7, to indicate that data flows from GPU0 to GPU1, then from GPU1 to GPU2, and so on, all the way to GPU7. Among them, GPUi-GPUj indicates the hardware connection between GPUi and GPUj. The hardware connection types include PCIE (peripheral component interconnect express, a high-speed serial computer expansion bus standard) and SMP (Symmetric multiprocessing) across NUMA (Non Uniform Memory Access). Since the data flows into another computing server through a communication card, the communication card of the computing server can be spliced ​​to the computing card string to obtain a sequence GPU0-GPU1-GPU2-GPU3-GPU4-GPU5-GPU6-GPU7-NIC0-NIC1. For computing server 1, another sequence generated can also be, for example, GPU0-GPU1-GPU2-GPU4-GPU5-GPU3-GPU6-GPU7-NIC0-NIC1. It is easy to understand that a computing server can have multiple sequences, and the multiple sequences are obtained by enumerating the serial connection order of the computing cards on the computing server.

[0031] Step 204: for each sequence, according to the query bandwidth between each two connected computing power cards in the sequence, determine the estimated communication bandwidth between computing power cards in the server corresponding to the sequence.

[0032] Specifically, for each sequence, the detection server performs the following operations. The detection server determines the query bandwidth between each two hardware-connected computing power cards in the sequence. For example, for the sequence "GPU0-GPU1-GPU2-GPU3-GPU4-GPU5-GPU6-GPU7-NIC0-NIC1", the detection server can determine the query bandwidth between GPU0 and GPU1, determine the query bandwidth between GPU1 and GPU2, and so on. The detection server determines the estimated communication bandwidth between computing power cards in the server corresponding to the sequence based on each query bandwidth.

[0033] In one embodiment, the short board effect, also known as the wooden barrel principle or water bucket effect, means that how much water a water bucket can hold does not depend on the longest board, but on the shortest board. Due to the existence of the wooden barrel effect, in a data transmission line, the bandwidth rate of a data transmission line does not depend on the maximum bandwidth rate, but on the minimum bandwidth rate. Therefore, the minimum query bandwidth is used as the estimated communication bandwidth between computing power cards in the server corresponding to the sequence.

[0034] In one embodiment, the query bandwidth between two connected computing power cards can be obtained through a preset instruction. The server's instruction to read the bandwidth can be implemented through a variety of command line tools, such as the nload instruction, etc.

[0035] Step 206, traverse the sequence in descending order of the estimated communication bandwidth between computing power cards in the server, determine the estimated communication bandwidth between servers corresponding to the currently traversed sequence, stop traversal when the traversal stop condition is met, and filter out sample sequences from the traversed sequences.

[0036] Specifically, after obtaining the estimated communication bandwidth between computing power cards in the server corresponding to each sequence, the sequences can be sorted in descending order according to the estimated communication bandwidth between computing power cards in the server to obtain a sequence group. Starting from the first sequence in the sequence group, the sequences in the sequence group are traversed, and for the sequence currently traversed, the estimated communication bandwidth between servers corresponding to the sequence currently traversed is determined according to the bandwidth rate between the head computing power card and the communication card in the sequence currently traversed, and the bandwidth rate between the tail computing power card and the communication card. The detection server determines whether the sequence currently traversed meets the traversal stop condition, and when the traversal stop condition is met, a sample sequence is selected from the traversed sequence. Among them, the sample sequence can be a sequence corresponding to a correct data transmission path, or a sequence corresponding to a data transmission path with the least error. The correct data transmission path is a data transmission path that fully utilizes the bandwidth performance of the communication card.

[0037] Step 208, determining the standard inter-server communication bandwidth according to the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth corresponding to the sample sequence.

[0038] Specifically, since the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth of the sample sequence have been obtained, the detection server can determine the standard inter-server communication bandwidth based on the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth of the sample sequence. For example, since the bandwidth rate of data flowing in the same computing power server is generally greater than the bandwidth rate of data transmitted from one computing power server to another computing power server, the smaller value of the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth is used as the standard inter-server communication bandwidth.

[0039] Step 210, obtain the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers obtained by the stress test, and generate a computing power server detection report based on the standard communication bandwidth between servers, the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers.

[0040] Specifically, for each computing power server, the stress testing tool can determine an optimal data transmission path within the server based on the computing power cards within the computing power server, and perform stress testing on the optimal data transmission path within the server to obtain the actual measured value of the communication bandwidth between the computing power cards within the server corresponding to the computing power server. For a computing power server cluster, the stress testing tool can connect each computing power card and communication card in the computing power server cluster in series to obtain an optimal data transmission path between servers, and perform stress testing based on the optimal data transmission path between servers to obtain the actual measured value of the communication bandwidth between servers corresponding to the computing power server cluster. The detection server compares the actual measured value of the communication bandwidth between the computing power cards in each server with the standard communication bandwidth between servers. If the actual value of the communication bandwidth between the computing power cards in the server is less than the standard communication bandwidth between servers, and is more than 80% lower than the standard communication bandwidth between servers, the computing power server corresponding to the actual value of the communication bandwidth between the computing power cards in the server is marked as an error, and it is considered that there is a problem with the computing power server corresponding to the actual value of the communication bandwidth between the computing power cards in the server, such as unreasonable configuration of the computing power card and communication card, or the phenomenon of high configuration and low performance of the computing power server. If the actual value of the communication bandwidth between servers is less than the standard communication bandwidth between servers, and is more than 80% lower than the standard communication bandwidth between servers, it is considered that there is a problem with the computing power server cluster, such as possible driver installation errors, or hardware not properly plugged in.

[0041] Furthermore, the detection server generates a computing power server detection report based on the standard inter-server communication bandwidth, the measured value of the communication bandwidth between computing power cards in the server, and the measured value of the communication bandwidth between servers. For example, the standard inter-server communication bandwidth, the measured value of the communication bandwidth between computing power cards in the server, and the measured value of the communication bandwidth between servers are written into the detection report, and when it is determined that there is a problem, the problem is stated in the detection report.

[0042] In the above-mentioned computing power server detection method, by determining multiple sequences, the estimated communication bandwidth between computing power cards in the server corresponding to each sequence can be determined. By determining the estimated communication bandwidth between computing power cards in the server, each sequence can be traversed in the order of the estimated communication bandwidth between computing power cards in the server from large to small to determine the estimated communication bandwidth between servers corresponding to the traversed sequence. When the sequence currently traversed meets the traversal stop condition, the traversal is stopped, and a sample sequence is obtained in which the communication card bandwidth performance can be fully utilized when the communication bandwidth between computing power cards in the server is as large as possible. By determining the sample sequence, the smaller value of the estimated communication bandwidth between computing power cards in the server and the estimated communication bandwidth between servers corresponding to the sample sequence can be used as the standard communication bandwidth between servers for subsequent bandwidth comparison. The standard communication bandwidth between servers is a bandwidth that a computing power server cluster can achieve under normal circumstances in theory. Therefore, by obtaining the actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers based on the stress test, the actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers can be compared with the standard communication bandwidth between servers to determine whether there is a problem with the computing power server cluster based on the comparison results, and generate a server detection report marked with the comparison results.

[0043] In one of the embodiments, an estimated communication bandwidth between computing power cards in a server corresponding to the sequence is determined based on the query bandwidth between every two connected computing power cards in the sequence, including: traversing every two connected computing power cards in the sequence; querying the query bandwidth between the two computing power cards currently traversed; determining the nominal bandwidth between the two computing power cards currently traversed based on the first hardware connection information between the two computing power cards currently traversed; when the query bandwidth between the two computing power cards currently traversed is less than the nominal bandwidth, marking the two computing power cards currently traversed as errors; when the traversal is completed, using the minimum query bandwidth as the estimated communication bandwidth between the computing power cards in the server corresponding to the sequence.

[0044] Specifically, the detection server can traverse every two connected computing power cards in the sequence. For the two computing power cards currently traversed, the detection server can determine the first hardware connection information between the two computing power cards currently traversed, for example, determine the hardware connection information between GPU0 and GPU1. Based on the first hardware connection information, determine the nominal bandwidth between the two computing power cards currently traversed. For example, the hardware connection information between GPU0 and GPU1 is SYS, SYS indicates that it is necessary to cross numa and Pcle switching chips, and the nominal bandwidth corresponding to SYS is 2GB / s. The hardware connection information between GPU2 and GPU3 is PXB, PXB indicates that it is necessary to cross Pcle switching chips, and the nominal bandwidth corresponding to PXB is 16GB / s, and so on.

[0045] Furthermore, for the two computing power cards currently traversed, the detection server can also obtain the query bandwidth between the two computing power cards currently traversed. The query bandwidth is the bandwidth queried through the instruction. Generally speaking, the query bandwidth is equal to the nominal bandwidth. If the query bandwidth of the two computing power cards currently traversed is less than the nominal bandwidth, it can be considered that there is a problem with the two computing power cards currently traversed, and the two computing power cards currently traversed are marked as errors. The marked error can be subsequently recorded in the computing power server detection report.

[0046] Furthermore, when the traversal is completed, the minimum query bandwidth is used as the estimated communication bandwidth between computing power cards in the server corresponding to the sequence.

[0047] In this embodiment, by comparing the nominal bandwidth with the query bandwidth, it is possible to determine whether there is a problem with the computing power card in the computing power server based on the comparison result, and if there is a problem, the computing power card with possible problems is marked as an error, so that engineers can troubleshoot the computing power card with possible errors through the marked errors in the computing power server detection report. And due to the existence of the short board effect, the minimum query bandwidth is used as the estimated communication bandwidth between computing power cards in the server corresponding to the sequence, so that the estimated communication bandwidth between computing power cards in the server is more accurate.

[0048] In one of the embodiments, determining the estimated inter-server communication bandwidth corresponding to the currently traversed sequence includes: determining the second hardware connection information between the head computing power card in the currently traversed sequence and the communication card in the currently traversed sequence, and determining the estimated communication bandwidth between the head computing power card and the communication card according to the second hardware connection information; determining the third hardware connection information between the tail computing power card in the currently traversed sequence and the communication card in the currently traversed sequence, and determining the estimated communication bandwidth between the tail computing power card and the communication card according to the third hardware connection information; determining the estimated inter-server communication bandwidth corresponding to the currently traversed sequence according to the estimated communication bandwidth between the head computing power card and the communication card and the estimated communication bandwidth between the tail computing power card and the communication card.

[0049] Specifically, each sequence can be traversed in the order of the estimated communication bandwidth between the computing power cards in the server from large to small. For the sequence currently traversed, the detection server determines the second hardware connection information between the head computing power card in the sequence currently traversed and the communication card in the sequence currently traversed. Among them, the head computing power card refers to the first computing power card in the sequence. The detection server can determine the estimated communication bandwidth between the head computing power card and the communication card based on the second hardware connection information. Furthermore, the detection server determines the third hardware connection information between the tail computing power card in the sequence currently traversed and the communication card in the sequence currently traversed, and determines the estimated communication bandwidth between the tail computing power card and the communication card based on the third hardware connection information. Among them, the tail computing power card refers to the last computing power card in the sequence.

[0050] When the estimated communication bandwidth between the head computing power card and the communication card and the estimated communication bandwidth between the tail computing power card and the communication card are determined, the estimated communication bandwidth between servers corresponding to the currently traversed sequence can be determined based on the estimated communication bandwidth between the head computing power card and the communication card and the estimated communication bandwidth between the tail computing power card and the communication card. For example, the smaller value of the estimated communication bandwidth between the head computing power card and the communication card and the estimated communication bandwidth between the tail computing power card and the communication card is used as the estimated communication bandwidth between servers corresponding to the currently traversed sequence.

[0051] In one of the embodiments, an estimated communication bandwidth between a head computing power card and a communication card is determined based on the second hardware connection information, including: for each communication card in the currently traversed sequence, determining the hardware bandwidth between the head computing power card and the targeted communication card based on the second hardware connection information between the head computing power card and the targeted communication card; determining the communication card bandwidth of the targeted communication card, and marking the head computing power card and the targeted communication card as errors when the hardware bandwidth is less than the communication card bandwidth; accumulating the communication card bandwidth of each communication card in the currently traversed sequence to obtain an estimated communication bandwidth between the head computing power card and the communication card.

[0052] Specifically, the detection server takes out the head computing power card in the sequence currently traversed, queries the second hardware connection information between the head computing power card and each communication card in the sequence currently traversed; for each communication card in the sequence currently traversed, the hardware bandwidth between the head computing power card and the targeted communication card is determined according to the second hardware connection information between the head computing power card and the targeted communication card. For example, when the second hardware connection information between the head computing power card and the targeted communication card is SYS, it is determined that the hardware bandwidth between the head computing power card and the targeted communication card is 2GB / S. The detection server determines the communication card bandwidth of the targeted communication card. For example, the communication card will have a nominal bandwidth, and the nominal bandwidth is used as the communication card bandwidth of the communication card. The detection server determines whether the hardware bandwidth between the head computing power card and the targeted communication card is less than the communication card bandwidth of the targeted communication card. If it is less than, the head computing power card and the targeted communication card are marked as errors, and it is considered that the performance of the targeted communication card is not fully utilized.

[0053] Furthermore, the detection server accumulates the communication card bandwidth of each communication card in the currently traversed sequence to obtain an estimated communication bandwidth between the head computing power card and the communication card.

[0054] In this embodiment, by comparing the hardware bandwidth with the communication bandwidth, it is possible to determine whether there is a problem between the head computing power card and the communication card based on the comparison result, and if there is a problem, mark the problematic head computing power card and communication card as an error, so that the marked errors can be recorded in the computing power server detection report later, and engineers can troubleshoot the head computing power card and communication card that may have problems through the marked errors in the computing power server detection report. And because data generally flows into the head computing power card in the sequence through the communication card, and then flows out to the communication card through the tail computing power card in the sequence, this embodiment estimates the communication bandwidth between the head computing power card and the communication card, and obtains the estimated communication bandwidth between the head computing power card and the communication card.

[0055] In one embodiment, the detection server takes out the tail computing power card in the sequence currently traversed, queries the third hardware connection information between the tail computing power card and each communication card in the sequence currently traversed; for each communication card in the sequence currently traversed, the hardware bandwidth between the tail computing power card and the targeted communication card is determined according to the third hardware connection information between the tail computing power card and the targeted communication card. For example, when the third hardware connection information between the tail computing power card and the targeted communication card is SYS, it is determined that the hardware bandwidth between the tail computing power card and the targeted communication card is 2GB / S. The detection server determines the communication card bandwidth of the targeted communication card. The detection server determines whether the hardware bandwidth between the tail computing power card and the targeted communication card is less than the communication card bandwidth of the targeted communication card. If it is less than, the tail computing power card and the targeted communication card are marked as wrong, and it is considered that the performance of the targeted communication card is not fully used. Further, the detection server accumulates the communication card bandwidth of each communication card in the sequence currently traversed to obtain the estimated communication bandwidth between the tail computing power card and the communication card.

[0056] In one of the embodiments, the traversal is stopped when the traversal stop condition is met, and a sample sequence is selected from the traversed sequence, including: when there is no marking error between the head computing power card and each communication card in the currently traversed sequence, and there is no marking error between the tail computing power card and each communication card in the currently traversed sequence, the traversal is stopped and the currently traversed sequence is used as the sample sequence; or, after all sequences have been traversed, each sequence has a marking error between the head computing power card and the communication card, or each sequence has a marking error between the tail computing power card and the communication card, the sequence with the least marking error is used as the sample sequence.

[0057] Specifically, when there is no marking error between the head computing power card and each communication card in the sequence currently traversed, and there is no marking error between the tail computing power card and each communication card in the sequence currently traversed, the traversal is stopped, and the sequence currently traversed is used as the sample sequence. If all sequences are traversed and each sequence is marked with errors, the sequence with the least marking errors is used as the sample sequence. For example, referring to Figure 3, in sequence 1, the head computing power card and communication card 1 and communication card 2 all have marking errors, and the tail computing power card and communication card 1 have marking errors; in sequence 2, the tail computing power card and communication card 3, communication card 4 and communication card 6 all have marking errors; in sequence 3, the head computing power card and communication card 7 have marking errors, and the tail computing power card and communication card 9 have marking errors. Since sequence 3 has only 2 marking errors, and other sequences have 3 marking errors, sequence 3 is used as the sample sequence. Figure 3 shows a schematic diagram of determining a sample sequence in an embodiment.

[0058] In this embodiment, a sequence with no marking errors or with the least marking errors is used as a sample sequence, so that the sample sequence can be an optimal path for data transmission. In this way, a bandwidth that the computing power server cluster can achieve under normal circumstances can be obtained based on the sample sequence, and the bandwidth can be used as a standard value for subsequent comparison with the actual measured value.

[0059] In one of the embodiments, the standard inter-server communication bandwidth is determined based on the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth corresponding to the sample sequence, including: when the estimated intra-server computing power card communication bandwidth of the sample sequence is less than the estimated inter-server communication bandwidth of the sample sequence, and the difference between the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth of the sample sequence is greater than the communication bandwidth of any communication card in the sample sequence, the sample sequence is marked as an error; and the smaller value of the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth of the sample sequence is used as the standard inter-server communication bandwidth.

[0060] Specifically, for the sample sequence, when the estimated communication bandwidth between computing power cards within the server of the sample sequence is less than the estimated communication bandwidth between servers of the sample sequence, and the difference is greater than the communication bandwidth of any communication card in the sample sequence, it can be considered that the difference between the estimated communication bandwidth between computing power cards within the server and the estimated communication bandwidth between servers is too large. At this time, at least one of the estimated communication bandwidth between computing power cards within the server and the estimated communication bandwidth between servers has a problem. Therefore, the sample sequence can be marked as an error, and the marked error can be recorded in the computing power server detection report.

[0061] Furthermore, the detection server may use the smaller value of the estimated communication bandwidth between computing power cards within the server and the estimated communication bandwidth between servers of the sample sequence as the standard communication bandwidth between servers.

[0062] In this embodiment, compared with data transmission within the server, the bandwidth for data transmission out of the server will be smaller. Therefore, the smaller value of the estimated communication bandwidth between computing power cards within the server and the estimated communication bandwidth between servers is used as the standard communication bandwidth between servers, making the standard communication bandwidth between servers more accurate.

[0063] In one of the embodiments, a computing power server detection report is generated based on the standard inter-server communication bandwidth, the actual measured value of the communication bandwidth between computing power cards in the server, and the actual measured value of the communication bandwidth between servers, including: for the actual measured value of the communication bandwidth between computing power cards in the server corresponding to each computing power server, when the actual measured value of the communication bandwidth between computing power cards in the server is less than the standard inter-server communication bandwidth, and the difference between the actual measured value of the communication bandwidth between computing power cards in the server and the standard inter-server communication bandwidth is greater than a preset difference, the computing power server corresponding to the communication bandwidth between computing power cards in the server is marked as an error; when the actual measured value of the communication bandwidth between servers is less than the standard inter-server communication bandwidth, and the difference between the actual measured value of the communication bandwidth between servers and the standard inter-server communication bandwidth is greater than a preset difference, the computing power server cluster is marked as an error; and a computing power server detection report is generated based on each marked error.

[0064] Specifically, each computing power server can be stress-tested to obtain the actual measured value of the communication bandwidth between computing power cards in the server corresponding to each computing power server. For each actual measured value of the communication bandwidth between computing power cards in the server, the actual measured value of the communication bandwidth between computing power cards in the server is compared with the standard communication bandwidth between servers. When the actual measured value of the communication bandwidth between computing power cards in the server is less than the standard communication bandwidth between servers, and the difference between the two is greater than the preset difference, for example, when the actual measured value of the communication bandwidth between computing power cards in the server is less than 80% of the standard communication bandwidth between servers, it is determined that the computing power server corresponding to the communication bandwidth between computing power cards in the server may have a problem. At this time, the computing power server corresponding to the communication bandwidth between computing power cards in the server is marked as an error.

[0065] The computing server cluster can be stress-tested to obtain the measured value of the inter-server communication bandwidth. The detection server compares the measured value of the inter-server communication bandwidth with the standard inter-server communication bandwidth. When the measured value of the inter-server communication bandwidth is less than the standard inter-server communication bandwidth, and the difference between the measured value of the inter-server communication bandwidth and the standard inter-server communication bandwidth is greater than the preset difference, for example, when the measured value of the inter-server communication bandwidth is less than 80% of the standard inter-server communication bandwidth, it is determined that there may be a problem with the computing server cluster, and the computing server cluster is marked as an error.

[0066] When a marking error is determined, each marking error can be recorded in the computing power server detection report, so that engineers can troubleshoot the problem based on the marking errors in the computing power server detection report.

[0067] In one of the embodiments, refer to Figure 4, which shows a schematic diagram of the overall process of computing power server detection in one embodiment. Step 1: The user configures the names of the communication cards between computing power servers, NIC0 and NIC1. The collector completes the collection of computing power card topology information on the server, including the computing power card name NVIDIA 4090, the number of computing power cards 8, the hardware connection topology between each computing power card, and the hardware connection topology between the communication card between the server and the computing power card. The hardware connection types are PCIE and SMP with quanuma, and PCIE is 3.0 16GB / s bandwidth. Step 2: The topology analyzer traverses all sequences of computing power cards, and compares the bandwidth rates of the hardware connections between the two cards one by one for the traversed sequences, and calculates the minimum query bandwidth as the estimated communication bandwidth between the computing power cards in the server corresponding to the traversed sequence. For example, for the sequence GPU0-GPU1-GPU2-GPU3-GPU4-GPU5-GPU6-GPU7, the estimated minimum cross-NUMA communication rate between GPU3 and GPU4 is 10GB / s, and the PCIE3.0 rate between the other two is 16GB / s. The 10GB / s rate is used as the estimated communication bandwidth between computing power cards in the server corresponding to this sequence, and the same applies to other sequences.

[0068] Step 3: Starting from the group of sequences with the largest estimated communication bandwidth between computing power cards in the server, traverse each sequence in descending order of the communication bandwidth between computing power cards in the server, and calculate the estimated communication bandwidth between computing power cards in the server. Step 4: Take out the head computing power card in the sequence currently traversed, query the connection hardware information between the head computing power card and each communication card, query the hardware bandwidth and communication card bandwidth recorded in the data storage device, and mark an error if the hardware bandwidth is less than the communication card bandwidth. For example, the performance of both communication cards is 25GB / s, but there is a problem with the hardware connection between the computing power cards, which is 10GB / s. Then mark two errors, and finally accumulate the communication card bandwidth of 50GB / s as the estimated communication bandwidth between the head computing power card and the communication card for the sequence currently traversed. Step 5: Take out the tail computing power card in the sequence currently traversed, repeat the four-step calculation method, and the estimated communication bandwidth between the tail computing power card and the communication card is also 50GB / s, but there are also 2 errors. Compare the two groups of communication bandwidths in the fourth and fifth steps. Since the two groups of communication bandwidths are the same in the above example, 50GB / s is taken as the standard inter-server communication bandwidth. Step 6: Determine the sample sequence. Since in the above example, there are error marks after the traversal is completed, and the number of error marks is the same, the first sequence can be used as the sample sequence. Step 7: According to the sample sequence obtained in the sixth step, the estimated communication bandwidth between the computing power cards in the server, 10GB / s, obtained in the second step, and the estimated communication bandwidth between servers, 50GB / s, obtained in the fifth step, are compared. Since the estimated communication bandwidth between the computing power cards in the server, 10GB / s, is much smaller than the estimated communication bandwidth between servers, 50GB / s, and the difference between the two exceeds the bandwidth of one of the communication cards, the mark is wrong; take the smaller value of 10GB / s as the standard inter-server communication bandwidth.

[0069] Step 8: Stress test the computing power card in the computing power server to obtain the measured value of the communication bandwidth between the computing power cards in the server, for example, 9.7GB / s. Stress test the computing power server cluster to obtain the measured value of the communication bandwidth between servers, for example, 9GB / s. Since the two measured values ​​are close to the standard communication bandwidth between servers of 10GB / s, no error is marked. Step 9: Generate a computing power server detection report and indicate the computing power card topology information in the computing power server detection report. It is estimated that the communication bandwidth between the computing power cards in the server is 10GB / s, the standard communication bandwidth between servers is 10GB / s, the measured value of the communication bandwidth between servers is 9GB / s, and the measured value of the communication bandwidth between the computing power cards in the server is 9.7GB / s. You can also indicate the change suggestions in the computing power server detection report. For example, if there is a marking error in the second step above, the error message, nominal rate and query rate are output, and it is recommended to check the server startup parameters; if the sixth step above obtains a sequence with a marking error, the error message, hardware bandwidth and communication card bandwidth are output, and it is recommended to adjust the communication card slot; if there is a marking error in the seventh step above, the error message is output, and it is recommended to reduce the communication card or increase the communication bandwidth between computing power cards; if there is a problem in the eighth step above, the error message, measured value and estimated value are output, and it is recommended to check the server startup parameters, whether the hardware is properly plugged in, and whether the driver is installed correctly.

[0070] It can be seen from the above embodiments that this method can quickly analyze the communication bandwidth of the server topology and compare the estimated value with the measured value, thereby improving the detection efficiency, improving the consistency of the detection results, and improving the reliability of the server assembly; by outputting a report to clarify the problem link and providing troubleshooting suggestions, the complexity of manual troubleshooting is reduced, thereby improving the efficiency of problem troubleshooting.

[0071] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0072] This application greatly improves detection efficiency, improves consistency of detection results, and improves reliability of server assembly by comparing theoretical values ​​and measured values ​​of communication bandwidth after automatic algorithmic analysis of server topology. It reduces the complexity of manual troubleshooting and improves efficiency of problem troubleshooting by outputting reports and providing guiding troubleshooting suggestions for various issues. By comparing theoretical values ​​and measured values ​​of communication bandwidth after automatic algorithmic analysis of server topology, detection efficiency is improved, improves consistency of detection results, and improves reliability of server assembly. By outputting reports and providing guiding troubleshooting suggestions for various issues, it reduces the complexity of manual troubleshooting and improves efficiency of problem troubleshooting.

[0073] Based on the same inventive concept, the embodiment of the present application also provides a computing power server detection device for implementing the computing power server detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more computing power server detection device embodiments provided below can refer to the limitations of the computing power server detection method above, and will not be repeated here.

[0074] In one embodiment, as shown in FIG. 5 , a computing power server detection device 500 is provided, including: a server computing power card communication bandwidth estimation module 502, a standard server communication bandwidth determination module 504 and a report generation module 506, wherein:

[0075] The module 502 for estimating the communication bandwidth between computing power cards in the server is used to obtain a sequence set corresponding to each computing power server in the computing power server cluster; each sequence in each sequence set includes a computing power card and a communication card deployed on the corresponding computing power server; for each sequence, according to the query bandwidth between each two connected computing power cards in the sequence, the estimated communication bandwidth between computing power cards in the server corresponding to the sequence is determined;

[0076] The standard inter-server communication bandwidth determination module 504 is used to traverse the sequence in descending order of the estimated communication bandwidth between the computing power cards in the server, determine the estimated communication bandwidth between the servers corresponding to the currently traversed sequence, stop traversing when the traversal stop condition is met, and select a sample sequence from the traversed sequence; determine the standard inter-server communication bandwidth according to the estimated communication bandwidth between the computing power cards in the server and the estimated communication bandwidth between servers corresponding to the sample sequence;

[0077] The report generation module 506 is used to obtain the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers obtained by the stress test, and generate a computing power server detection report based on the standard communication bandwidth between servers, the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers.

[0078] In one of the embodiments, the communication bandwidth estimation module 502 between computing power cards in the server is also used to traverse every two connected computing power cards in the sequence, query the query bandwidth between the two computing power cards currently traversed; determine the nominal bandwidth between the two computing power cards currently traversed according to the first hardware connection information between the two computing power cards currently traversed; when the query bandwidth between the two computing power cards currently traversed is less than the nominal bandwidth, mark the two computing power cards currently traversed as errors; when the traversal is completed, use the minimum query bandwidth as the estimated communication bandwidth between the computing power cards in the server corresponding to the sequence.

[0079] In one of the embodiments, the standard server-to-server communication bandwidth determination module 504 is also used to determine the second hardware connection information between the head computing power card in the currently traversed sequence and the communication card in the currently traversed sequence, and determine the estimated communication bandwidth between the head computing power card and the communication card based on the second hardware connection information; determine the third hardware connection information between the tail computing power card in the currently traversed sequence and the communication card in the currently traversed sequence, and determine the estimated communication bandwidth between the tail computing power card and the communication card based on the third hardware connection information; determine the estimated server-to-server communication bandwidth corresponding to the currently traversed sequence based on the estimated communication bandwidth between the head computing power card and the communication card and the estimated communication bandwidth between the tail computing power card and the communication card.

[0080] In one of the embodiments, the standard server-to-server communication bandwidth determination module 504 is also used to determine, for each communication card in the currently traversed sequence, the hardware bandwidth between the head computing power card and the targeted communication card according to the second hardware connection information between the head computing power card and the targeted communication card; query the communication card bandwidth of the targeted communication card, and mark the head computing power card and the targeted communication card as errors when the hardware bandwidth is less than the communication card bandwidth; accumulate the communication card bandwidth of each communication card in the currently traversed sequence to obtain the estimated communication bandwidth between the head computing power card and the communication card.

[0081] In one of the embodiments, the standard server-to-server communication bandwidth determination module 504 is further used to stop traversal and use the currently traversed sequence as a sample sequence when there is no marking error between the head computing power card and each communication card in the currently traversed sequence, and there is no marking error between the tail computing power card and each communication card in the currently traversed sequence; or, after all sequences have been traversed, each sequence has a marking error between the head computing power card and the communication card, or each sequence has a marking error between the tail computing power card and the communication card, the sequence with the least marking error is used as the sample sequence.

[0082] In one of the embodiments, the standard server-to-server communication bandwidth determination module 504 is also used to mark the sample sequence as wrong when the estimated intra-server computing power card communication bandwidth of the sample sequence is less than the estimated inter-server communication bandwidth of the sample sequence, and the difference between the estimated intra-server computing power card communication bandwidth of the sample sequence and the estimated inter-server communication bandwidth is greater than the communication bandwidth of any communication card in the sample sequence; and the smaller value of the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth of the sample sequence is used as the standard inter-server communication bandwidth.

[0083] In one of the embodiments, the report generation module 506 is also used to measure the communication bandwidth between computing power cards in the server corresponding to each computing power server. When the measured value of the communication bandwidth between computing power cards in the server is less than the standard communication bandwidth between servers, and the difference between the measured value of the communication bandwidth between computing power cards in the server and the standard communication bandwidth between servers is greater than a preset difference, mark the computing power server corresponding to the communication bandwidth between computing power cards in the server as an error; when the measured value of the communication bandwidth between servers is less than the standard communication bandwidth between servers, and the difference between the measured value of the communication bandwidth between servers and the standard communication bandwidth between servers is greater than a preset difference, mark the computing power server cluster as an error; and generate a computing power server detection report based on each marked error.

[0084] Each module in the above computing power server detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0085] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store computing power server detection data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a computing power server detection method is implemented.

[0086] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the computing power server detection methods in the embodiments of the present application is implemented.

[0088] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any computing power server detection method in the embodiments of the present application is implemented.

[0089] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements any one of the computing power server detection methods in the embodiments of the present application.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A computing power server detection method, characterized in that: The method comprises: Obtain a sequence set corresponding to each computing server in the computing server cluster; each sequence in each of the sequence sets includes a computing card and a communication card deployed on the corresponding computing server; For each of the sequences, determining the estimated communication bandwidth between computing power cards in the server corresponding to the sequence according to the query bandwidth between each two connected computing power cards in the sequence; Traversing the sequence in descending order of the estimated communication bandwidth between computing power cards in the server, determining the estimated communication bandwidth between servers corresponding to the currently traversed sequence, and stopping the traversal when the traversal stop condition is met, and filtering out the sample sequence from the traversed sequence; Determine a standard inter-server communication bandwidth according to the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth corresponding to the sample sequence; The actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers obtained by performing stress testing are obtained, and a computing power server detection report is generated according to the standard communication bandwidth between servers, the actual measured values ​​of the communication bandwidth between the computing power cards in the server and the actual measured values ​​of the communication bandwidth between servers.

2. The method according to claim 1, characterized in that The step of determining the estimated communication bandwidth between computing power cards in the server corresponding to the sequence according to the query bandwidth between each two connected computing power cards in the sequence includes: Traversing every two connected computing power cards in the sequence, and querying the query bandwidth between the two computing power cards currently traversed; Determine the nominal bandwidth between the two computing power cards currently traversed according to the first hardware connection information between the two computing power cards currently traversed; When the query bandwidth between the two computing power cards currently traversed is less than the nominal bandwidth, marking the two computing power cards currently traversed as errors; When the traversal is completed, the minimum query bandwidth is used as the estimated communication bandwidth between computing power cards in the server corresponding to the sequence.

3. The method according to claim 1, characterized in that The step of determining an estimated inter-server communication bandwidth corresponding to the currently traversed sequence includes: Determine second hardware connection information between a head computing power card in a currently traversed sequence and a communication card in the currently traversed sequence, and determine an estimated communication bandwidth between the head computing power card and the communication card according to the second hardware connection information; Determine third hardware connection information between the tail computing power card in the currently traversed sequence and the communication card in the currently traversed sequence, and determine an estimated communication bandwidth between the tail computing power card and the communication card according to the third hardware connection information; According to the estimated communication bandwidth between the head computing power card and the communication card and the estimated communication bandwidth between the tail computing power card and the communication card, the estimated inter-server communication bandwidth corresponding to the currently traversed sequence is determined.

4. The method according to claim 3, characterized in that The step of determining the estimated communication bandwidth between the head computing power card and the communication card according to the second hardware connection information includes: For each communication card in the currently traversed sequence, determine the hardware bandwidth between the head computing power card and the targeted communication card according to the second hardware connection information between the head computing power card and the targeted communication card; Querying the communication card bandwidth of the targeted communication card, and marking the head computing power card and the targeted communication card as errors when the hardware bandwidth is less than the communication card bandwidth; The communication card bandwidth of each communication card in the currently traversed sequence is accumulated to obtain an estimated communication bandwidth between the head computing power card and the communication card.

5. The method according to claim 4, characterized in that The traversal is stopped when the traversal stop condition is met, and a sample sequence is selected from the traversed sequence, including: When there is no marking error between the head computing power card and each communication card in the currently traversed sequence, and there is no marking error between the tail computing power card and each communication card in the currently traversed sequence, the traversal is stopped, and the currently traversed sequence is used as a sample sequence; or After all sequences have been traversed, when each of the sequences has a marking error between the head computing power card and the communication card, or when each of the sequences has a marking error between the tail computing power card and the communication card, the sequence with the least marking error is used as the sample sequence.

6. The method according to claim 1, characterized in that The determining the standard inter-server communication bandwidth according to the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth corresponding to the sample sequence includes: When the estimated intra-server computing power card communication bandwidth of the sample sequence is less than the estimated inter-server communication bandwidth of the sample sequence, and the difference between the estimated intra-server computing power card communication bandwidth and the estimated inter-server communication bandwidth of the sample sequence is greater than the communication bandwidth of any communication card in the sample sequence, the sample sequence is marked as an error; The smaller value of the estimated communication bandwidth between computing power cards within the server and the estimated communication bandwidth between servers of the sample sequence is used as the standard communication bandwidth between servers.

7. The method according to claim 1, characterized in that The generating a computing power server detection report according to the standard inter-server communication bandwidth, the measured value of the communication bandwidth between computing power cards in the server and the measured value of the communication bandwidth between servers includes: For each of the computing power servers, the actual measured value of the communication bandwidth between computing power cards in the server is less than the standard communication bandwidth between servers, and the difference between the actual measured value of the communication bandwidth between computing power cards in the server and the standard communication bandwidth between servers is greater than a preset difference, marking the computing power server corresponding to the communication bandwidth between computing power cards in the server as an error; When the actual measured value of the inter-server communication bandwidth is less than the standard inter-server communication bandwidth, and the difference between the actual measured value of the inter-server communication bandwidth and the standard inter-server communication bandwidth is greater than a preset difference, marking the computing power server cluster as an error; Generate a computing power server detection report based on the marked errors.

8. A computing power server detection device, characterized in that: The device comprises: The module for estimating the communication bandwidth between computing power cards in the server is used to obtain a sequence set corresponding to each computing power server in the computing power server cluster; each sequence in each sequence set includes a computing power card and a communication card deployed on the corresponding computing power server; for each sequence, the estimated communication bandwidth between computing power cards in the server corresponding to the sequence is determined according to the query bandwidth between each two connected computing power cards in the sequence; The standard inter-server communication bandwidth determination module is used to traverse the sequence in descending order of the estimated communication bandwidth between computing power cards in the server, determine the estimated communication bandwidth between servers corresponding to the currently traversed sequence, stop traversing when the traversal stop condition is met, and filter out a sample sequence from the traversed sequence; determine the standard inter-server communication bandwidth according to the estimated communication bandwidth between computing power cards in the server and the estimated communication bandwidth between servers corresponding to the sample sequence; The report generation module is used to obtain the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers obtained by stress testing, and generate a computing power server detection report based on the standard communication bandwidth between servers, the actual measured value of the communication bandwidth between the computing power cards in the server and the actual measured value of the communication bandwidth between servers.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images