Gpu cluster silicon pre- validation method, simulation platform, storage medium and program product

Abstract

The application discloses a GPU cluster silicon pre-verification method, a simulation platform, a storage medium and a program product. When the silicon pre-verification of the GPU cluster is performed, at least one network card is mapped as a simulation GPU card, and all network cards and at least one virtual GPU card generated by a hardware simulator are respectively connected with a switch to control each network card to send simulation traffic to the switch according to a preset message sending rule. Based on a target port in the switch, messages sent by all virtual GPU cards are mirrored to at least one target network card, and analysis software is used to capture and analyze the mirrored messages received by the target network card. According to the embodiment of the application, a large-scale GPU interconnection scene can be quickly built through a common PC network card, convenient message debugging is realized, and the efficiency and safety of the silicon pre-verification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic simulation, in particular to a GPU cluster pre-silicon validation method, a simulation platform, a storage medium and a program product. BACKGROUND

[0002] Ethernet-based GPU (Graphics Processing Unit) technology is entering a period of rapid development. The computing power of a single GPU has been difficult to meet the demand for massive data processing and complex task parallel operation. Through the combination of network interface cards and Ethernet switches, a large-scale GPU cluster can be flexibly expanded, and with its distributed computing mode, it can release super computing power and become the core infrastructure supporting large model training, scientific computing simulation and other computing-intensive scenarios. Before the application of the GPU cluster, pre-silicon validation is a key link to ensure product stability and reliability, and the interaction test between the GPU card and the switch is particularly important.

[0003] The existing mainstream GPU cluster interconnection pre-silicon simulation technology mainly relies on an emu (Emulator, hardware emulator) system to achieve modeling of the interconnection interface of the GPU chip and the forwarding logic of the switch through a hardware description language, then mapping the model to a special simulation hardware, building a virtual multi-card interconnection topology, finally injecting test traffic to the virtual GPU card through the simulation platform to simulate the actual cluster data stream transmission, and capturing the signal and packet information in the transmission process through the monitoring module of the platform to complete the channel validation and problem troubleshooting. Although the existing special hardware simulation technology can meet the basic verification needs, the deployment cost is high, and the hardware resources of the emu are related to the simulation scale. To meet the verification needs of the largest scale, the resources configured in the small and medium-sized verification scenarios are redundant, and the hardware resources cannot be dynamically allocated like software resources, but can only be passively idle, resulting in low resource utilization. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a GPU cluster pre-silicon validation method, a simulation platform, a storage medium and a program product, which can quickly build a large-scale GPU interconnection scene through a common PC network card, reduce the deployment cost and improve the resource utilization.

[0005] To achieve the above purpose, the embodiments of the present application provide a GPU cluster pre-silicon validation method, comprising:

[0006] mapping at least one network card as a simulation GPU card;

[0007] establishing a communication connection between all network cards and at least one virtual GPU card generated by the hardware emulator and the switch, respectively.

[0008] Each network card sends the simulation traffic to the switch according to the preset message sending rule;

[0009] Based on the target port in the switch, mirror all the messages sent by the virtual GPU cards to at least one target network card;

[0010] Capture and analyze the mirrored messages received by the target network card by using the analysis software.

[0011] As an improvement of the above scheme, the message sending rule is a preset rule matching the traffic characteristics and data interaction behavior of the virtual GPU card.

[0012] As an improvement of the above scheme, the mirroring of all the messages sent by the virtual GPU cards to at least one target network card based on the target port in the switch comprises:

[0013] Set the access ports corresponding to all the virtual GPU cards in the switch as the mirror source ports, and set the target port in the switch as the mirror destination port;

[0014] Establish a communication connection between the target port and at least one target network card, so that the switch copies the original messages sent by the virtual GPU cards received by all the mirror source ports, generates mirrored messages, and forwards the mirrored messages to the corresponding target network card through the target port.

[0015] As an improvement of the above scheme, the mapping of at least one network card as a simulation GPU card comprises:

[0016] Map at least one network card as a simulation GPU card by using a TCP traffic replay tool.

[0017] As an improvement of the above scheme, the capturing and analyzing of the mirrored messages received by the target network card by using the analysis software comprises:

[0018] Start the analysis software, and select the target network card receiving the mirrored messages as the capture port to obtain the mirrored messages;

[0019] Load a preset analysis plug-in through the analysis software; wherein the analysis plug-in is defined according to a preset private protocol field;

[0020] Analyze the mirrored messages by using the analysis plug-in to obtain field information;

[0021] Analyze the mirrored messages based on the field information.

[0022] As an improvement of the above scheme, the analyzing of the mirrored messages based on the field information comprises:

[0023] verify the mirror packet, wherein the verifying operation comprises at least one of format integrity verification, field legality verification and transmission timing rationality verification;

[0024] generate an analysis report based on the verification result.

[0025] As an improvement of the above scheme, the traffic feature comprises at least one of packet sending rate, packet length and traffic load type, and the data interaction behavior comprises at least one of packet sending timing, retransmission mechanism and congestion response strategy.

[0026] To achieve the above object, the embodiment of the present application further provides a simulation platform, comprising at least one network card, a hardware simulator, a switch and a processor, wherein the processor is configured to:

[0027] map the at least one network card as a simulation GPU card;

[0028] generate at least one virtual GPU card by all network cards and the hardware simulator, and respectively establish a communication connection between the at least one virtual GPU card and the switch;

[0029] control each network card to send simulation traffic to the switch according to a preset packet sending rule;

[0030] mirror the packets sent by all virtual GPU cards to at least one target network card based on a target port in the switch;

[0031] capture and analyze the mirrored packets received by the target network card by using analysis software.

[0032] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, comprising a stored computer program, wherein when the computer program runs, the computer readable storage medium controls the device where the computer readable storage medium is located to execute the GPU cluster pre-silicon verification method as any one of the above embodiments.

[0033] To achieve the above object, the embodiment of the present application further provides a computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the GPU cluster pre-silicon verification method as any one of the above embodiments is realized.

[0034] Compared with the prior art, the GPU cluster pre-silicon verification method, the simulation platform, the storage medium and the program product disclosed by the application can quickly build a large-scale GPU interconnection scene through a common PC network card, reduce deployment cost and improve resource utilization. In addition, the application maps the network card as a simulation GPU card, expands the cluster verification scale, does not need to completely rely on a hardware simulator to generate a large-scale virtual GPU card, reduces verification cost and hardware deployment threshold, and through the construction of the interconnection environment of the virtual GPU card and the simulation GPU card, combined with the message sending rules matching the characteristics of the real device, can accurately simulate the actual communication scene of the large-scale GPU cluster, guarantee the authenticity and effectiveness of the verification. Moreover, with the help of the mirroring function of the target port of the switch, the complete capture of the virtual GPU card message can be realized without interfering with the transmission of the core business traffic, and multi-dimensional message analysis can be completed by cooperating with the analysis software and analysis capability, which can discover potential problems such as protocol compatibility anomalies, transmission timing deviation and path integrity defects in the cluster interconnection in advance, effectively shorten the pre-silicon verification period. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is an architecture schematic diagram of a GPU cluster pre-silicon verification method provided by the prior art;

[0036] Figure 2 is an architecture schematic diagram of a GPU cluster pre-silicon verification method provided by the embodiment of the application;

[0037] Figure 3 is a flowchart of a GPU cluster pre-silicon verification method provided by the embodiment of the application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0039] Reference is made to Figure 1 , Figure 1is an architecture schematic diagram of a GPU cluster pre-silicon verification method provided by the prior art. A CPU (Central Processing Unit) establishes an internal communication link with emu, GPU0, GPU1, GPU2, and GPU3 through a PCIe Switch (Peripheral Component Interconnect Express Switch), and all GPUs access a network switch through independent ports to realize external interconnection between multiple GPUs. In actual verification, the physical deployment of the GPU, the PCIe Switch, and the network switch is required, instructions are issued by the CPU to control each GPU to send test traffic, the network switch is used to complete message forwarding within the cluster, and finally the transmission data is captured by analysis software to realize path verification and problem troubleshooting. Although this traditional method can realize pre-silicon verification of a GPU cluster, it completely relies on a special emu simulation platform, resulting in high verification cost and low hardware resource utilization.

[0040] The emu system is a customized special simulation hardware, which can be a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), or a Zebu (Zero Bug Utility), etc. The hardware procurement, production, and maintenance cost of the emu system is high, and the GPU chip interconnection interface and switch forwarding logic need to be modeled in detail through a hardware description language, and then the model is mapped to the special hardware. The modeling, mapping, and simulation environment debugging process requires professional technicians to invest several weeks or even months of time, and the labor cost is significantly increased. In addition, if the simulation scale needs to be expanded, such as from 4 cards to 16 cards, higher-specification emu devices or hardware expansion modules need to be replaced, further increasing the cost threshold. Since the hardware resources of the emu, such as FPGA logic units, high-speed interfaces, and storage bandwidth, are fixed configurations preset to adapt to the maximum expected simulation scale, but the actual verification process needs to cover the whole scene from small-scale prototype verification of 2 cards, 4 cards, etc. to large-scale deployment verification of 16 cards and above, when carrying out small-scale verification, a large amount of hardware resources will be in an idle state. For example, the FPGA logic units configured to support 16-card cluster simulation are not called when verifying a 2-card interconnection scene, and only a few high-speed interfaces need to be enabled, resulting in long-term low-load operation of the emu and resource waste.

[0041] To solve the problems of high cost and low resource utilization caused by the dependence on emu in the prior art, the embodiments of the present application provide a new GPU cluster pre-silicon verification architecture, as shown in Figure 2, Figure 2 is the architecture schematic diagram of the GPU cluster pre-silicon verification method provided by the embodiment of the application, and the architecture includes a CPU, a PCIe Switch, a network card, a virtual GPU card (including GPU0 and GPU1) emulated by emu, and a switch.

[0042] The CPU serves as the control center of the entire verification system and is responsible for issuing instructions to complete the configuration of the network card mapped to the emulated GPU card, controlling the network card to send emulated traffic, coordinating the packet capture and analysis of the analysis software, and the like.

[0043] The PCIe Switch realizes the internal communication link extension between the CPU and the network card and the virtual GPU card, and supports the high-speed data interaction between multiple devices and the CPU.

[0044] The network card refers to a general Ethernet network card at the PC (Personal Computer, personal computer) end, such as a common gigabit or gigabit RJ45 interface network card, a fiber network card, and the like, and is a standardized computer network device that has basic functions such as data transmission and reception and network protocol analysis. In the embodiment of the application, the network card is mapped to an emulated GPU card after being configured by software, replaces the virtual GPU card function of the traditional emu, and simultaneously accesses the switch through the physical network port (such as the 0 and 1 network ports in the figure) to realize the functions of emulated traffic sending and mirror packet receiving. It should be noted that, Figure 2 In the figure, only the case where one network card has four physical network ports (labeled 0, 1, 2, and 3) is shown. In fact, the application can use a network card integrated with multiple network ports, such as a 4-port, 8-port, or even more network port PCIe network card, or multiple single / dual network port network cards stacked through the PC mainboard expansion slot. Such an integrated network card can be connected to the CPU and the PCIe Switch through the PCIe high-speed interface, can provide multiple independent physical network ports on a single hardware, and each network port can be individually mapped to an emulated GPU card without occupying excessive hardware space. This integrated design not only simplifies the hardware deployment, but also quickly expands the number of simulation nodes through a single network card, further improves the flexibility of cluster scale expansion, and simultaneously reduces the cable complexity and maintenance cost of multiple device deployment.

[0045] GPU0 and GPU1 are basic virtual GPU nodes generated by emu and jointly form a hybrid GPU cluster with the emulated GPU card mapped by the network card to simulate the multi-card interconnection scenario of an actual cluster.

[0046] The switch (network switch) undertakes the external interconnection function of the cluster. The numbers in the figure are port numbers of the switch, which are used to distinguish the access ports of different devices, such as ports 0 and 1 connected to virtual GPU cards, ports 2 and 3 connected to physical network ports of network cards, respectively, to realize the traffic forwarding of the simulated GPU cards, and ports 4 and 5 connected to the additional links of the network cards, which can be used to receive the mirror messages of the virtual GPU cards. In addition to being able to realize the traffic forwarding between the virtual GPU cards and the simulated GPU cards, the switch can also copy and forward the messages of the virtual GPU cards to the network card through the port mirroring function, providing a debugging channel for message analysis.

[0047] It should be noted that the traditional method relies on emu to realize large-scale virtual GPU cards, while the present application maps a general PC network card to a simulated GPU card. The cost of a general PC network card is relatively low, and it does not require a large amount of high-cost manpower for model mapping and hardware debugging. In addition, the network card and the switch are general network devices that can be directly reused in other network verification and testing scenarios, without the need to customize hardware for single GPU cluster verification like emu, further reducing the hardware investment cost. The hardware resources of traditional emu are fixedly configured, and a large amount of resources are idle during small and medium-scale verification, while the present application can flexibly increase or decrease the number of network cards according to the verification scale, such as using a small number of network cards for small-scale verification and adding network cards for large-scale verification, realizing on-demand allocation of hardware resources, avoiding resource redundancy, and improving resource utilization.

[0048] Referring to Figure 3 , Figure 3 is a flowchart of a GPU cluster pre-silicon verification method provided by an embodiment of the present application. The GPU cluster pre-silicon verification method is implemented by a controller in a simulation platform. The GPU cluster pre-silicon verification method comprises steps S1-S5.

[0049] S1, mapping at least one network card to a simulated GPU card.

[0050] For example, at least one network card is mapped to a simulated GPU card by a TCP (Transmission Control Protocol) traffic replay tool. For example, the TCP traffic replay tool is Tcpreplay. Through the configuration module of Tcpreplay, the IP (Internet Protocol) address and MAC (Media Access Control) address of the network card are set as parameters of the same communication network segment as the virtual GPU card, for example, the IP of the virtual GPU card is 192.168.1.1 / 24, and the IP of the network card is also configured as 192.168.1.3 / 24, so that the network card can be recognized by the switch as a legal node in the cluster. Then, the physical network port of the network card is specified by Tcpreplay, such asFigure 2 The 0th and 1st ports of the network card are used as the output interfaces of the simulated traffic, so that the generated messages can be sent to the switch through the network port, and the communication messages of the virtual GPU card captured in advance are introduced into the Tcpreplay, the file containing the traffic characteristics and data interaction behaviors of the GPU cluster, and the Tcpreplay sends instructions to the network card at a preset rate, so that the network card sends messages consistent with the virtual GPU card to the switch based on the loaded traffic template. It should be noted that the preset rate can be set as needed, and the present application does not make specific limitations.

[0051] In the embodiment of the present application, by mapping the general network card as a simulated GPU card, the high-cost emu special hardware is not needed, and the hardware procurement and debugging cost is reduced. In addition, the mapping process only needs to configure network parameters and load traffic templates, which is simple to operate and can increase or decrease the number of network cards as needed, and flexibly adapts to different scale cluster verification requirements. At the same time, the messages sent by the network card match the real GPU traffic characteristics, which guarantees the authenticity and verification effectiveness of the simulation scene.

[0052] S2, at least one virtual GPU card generated by all network cards and hardware emulators is respectively connected with the switch.

[0053] Exemplarily, all network cards mapped as simulated GPU cards and virtual GPU cards generated by emus are respectively connected to the corresponding ports of the switch through network cables, so that the network parameters of all connected devices are in the same communication network segment and there is no address conflict, so that the switch can identify and forward the messages of each device, and the communication link between each device and the switch is confirmed to be smooth by sending test data packets, which lays a hardware interconnection foundation for subsequent simulation traffic transmission and message mirroring.

[0054] S3, each network card sends simulated traffic to the switch according to a preset message sending rule.

[0055] Exemplarily, the message sending rule indicates the characteristics and transmission logic of the simulated traffic. The simulated traffic refers to the simulated traffic generated based on the data capture of the virtual GPU card communication, which can accurately reproduce the actual data interaction scene of large-scale GPU cluster interconnection.

[0056] Further, the message sending rule is a preset rule matching the traffic characteristics and data interaction behaviors of the virtual GPU card; wherein the traffic characteristics include at least one of message sending rate, message length and traffic load type, and the data interaction behaviors include at least one of message sending timing, retransmission mechanism and congestion response strategy.

[0057] For example, the real communication data of the virtual GPU card is captured by the emu, the traffic characteristics thereof are extracted, such as a packet sending rate of 5000 pps (packets per second), an average packet length of 1500 bytes, and a traffic load type mainly of GPU instruction data and tensor data, and the interaction logic of the virtual GPU card is analyzed to determine the data interaction behavior thereof, such as a packet sending timing of continuous 3 frames of data with a periodic mode of 10 ms interval, a retransmission mechanism of automatic retransmission after a timeout of 20 ms, and a congestion response strategy of reducing the rate by 30% after detecting a packet loss. It should be noted that the specific values of the above traffic characteristics and data interaction behavior are only examples, and other values can be used in actual applications. The above characteristics and behaviors are solidified into configurable packet sending rules and stored in the control module of the CPU. When silicon verification is needed, the CPU calls the rules to drive the network cards, so that the simulation traffic sent by the network cards is completely consistent with the virtual GPU card in terms of rate, length, timing, and the like, and accurate simulation of the real cluster interaction scene is achieved.

[0058] In the embodiment of the application, by establishing the communication connection of the network card and the virtual GPU card and the switch, a hybrid cluster interconnection topology compatible with the virtual node and the simulation node is constructed, and the high-speed data interaction channel between multiple nodes is ensured.

[0059] S4, based on the target port in the switch, mirroring all the packets sent by the virtual GPU card to at least one target network card.

[0060] For example, mirroring refers to the function of mirroring the port of the switch, which copies a copy of all the original packets sent by the virtual GPU card to the switch in real time and forwards them to the preset target port without interrupting the normal communication of the virtual GPU card with other nodes, and then the target network card connected to the target port receives the mirrored packets. The target network card refers to a network card used to receive the mirrored packets of the switch, so as to Figure 2 For example, the target network card is the No. 2 network port of the network card in Figure 2 which is both a network card mapped as a simulation GPU card and a free interface connected to the mirror destination port of the switch, can participate in the sending of cluster simulation traffic (as a simulation GPU card), and can also receive the mirrored packets forwarded by the switch to realize the reuse of hardware resources, and can further transmit the received mirrored packets to the PC end for packet analysis.

[0061] It should be noted that the mirror is a process of realizing non-interference capture and replication of the message sent by the virtual GPU card. Through the native mirroring function of the switch, the message data of the virtual GPU card, including the header information, the payload content, the sending timing, etc. can be completely obtained without affecting the core communication link of the cluster, and the message is transmitted to the analysis software associated with the CPU through the target network card, so that potential problems such as protocol compatibility, path integrity, message loss / reordering can be checked, and data support for verifying the effectiveness of the results is provided.

[0062] Further, the step S4 specifically comprises: setting the access ports corresponding to all virtual GPU cards in the switch as mirror source ports, and setting the target port in the switch as a mirror destination port; establishing a communication connection between the target port and at least one target network card, so that the switch replicates the original message sent by the virtual GPU card received by all mirror source ports, generates a mirror message, and forwards the mirror message to the corresponding target network card through the target port.

[0063] As shown in Figure 2 As shown in FIG. 1, the access ports corresponding to GPU0 and GPU1 in the switch, i.e. the 0th port and the 1st port of the switch, are set as mirror source ports, and the 5th port of the switch is set as a mirror destination port. The 2nd network port of the target network card is connected to the 5th port of the switch through a network cable, the physical connection between the target port and the target network card is completed, and it is ensured that the network parameters of the two match. When GPU0 sends an original message to the 0th port of the switch and GPU1 sends an original message to the 1st port of the switch, the switch automatically replicates these messages to generate mirror messages, and then forwards the mirror messages to the target network card through the 5th port. After receiving the mirror messages, the target network card transmits the mirror messages to the PC end.

[0064] In the embodiment of the present application, the non-interference capture of the virtual GPU card message is realized through the port mirroring of the switch, which does not interrupt the core communication link of the cluster, and can completely obtain the original message data. In addition, the mirror port configuration is flexible, can adapt to the multi-virtual card scene, and the target network card can reuse the idle interface of the simulation network card in the cluster, without the need for additional hardware, which simplifies the debugging process, guarantees the authenticity and integrity of the verification data, and improves the efficiency and reliability of the pre-silicon verification.

[0065] S5, capturing and analyzing the mirror message received by the target network card by using analysis software.

[0066] Exemplarily, the analysis software refers to a tool software with network message capturing, analyzing and counting functions, which is used to read the mirror message data received by the target network card, analyze and count the format, content and transmission time sequence of the message, so as to locate the potential problems in the GPU cluster communication. For example, the analysis software can be Wireshark, which is an open source network protocol analysis tool, supports cross-platform (Windows, Linux, macOS, etc.) running, has the functions of real-time message capturing, multi-protocol analysis, traffic counting and visual analysis, and is widely used in network fault diagnosis, protocol compatibility verification, data interaction behavior analysis and other scenes.

[0067] In the embodiment of the application, a large-scale GPU interconnection scene can be quickly built by a common PC network card, the deployment cost is reduced and the resource utilization rate is improved. In addition, the application maps the network card as a simulation GPU card, expands the cluster verification scale, does not need to completely rely on a hardware simulator to generate a large-scale virtual GPU card, reduces the verification cost and the hardware deployment threshold, and through the construction of the interconnection environment of the virtual GPU card and the simulation GPU card, combined with the message sending rules matching the real device characteristics, the actual communication scene of the large-scale GPU cluster can be accurately simulated, the authenticity and effectiveness of the verification are guaranteed. Moreover, with the help of the mirroring function of the target port of the switch, the complete capture of the virtual GPU card message can be realized without interfering with the transmission of the core business traffic, multi-dimensional message analysis can be completed by cooperating with the analysis software and the analysis capability, potential problems such as protocol compatibility, transmission time sequence and path integrity in the cluster interconnection can be found in advance, the silicon verification period is effectively shortened, the rework risk after chip tape-out is reduced, and reliable data support is provided for the optimization of the GPU cluster interconnection design.

[0068] Further, the step S5 specifically comprises: starting the analysis software, and selecting a target network card receiving a mirror message as a capture port to obtain the mirror message; loading a preset analysis plug-in through the analysis software; wherein the analysis plug-in is defined according to a preset private protocol field; using the analysis plug-in to analyze the mirror message to obtain field information; and analyzing the mirror message based on the field information.

[0069] Exemplarily, after starting the analysis software, a target network card receiving a mirror message is selected, such as Figure 2The No. 2 network port of the middle network card captures the mirror message forwarded by the switch in real time, at this time, the analysis software without loading the plug-in can only display the original form of the message, such as 0x78, 0x04, 0x32, 0x91, 0xad, 0x7d and other difficult-to-read raw data, the analysis software loads the pre-developed GPU private protocol analysis plug-in, the plug-in has completed the coding according to the field definition of the private protocol, such as the first byte is the destination address, the second byte is the data length, and the subsequent bytes are the data payload. The plug-in automatically disassembles the original data, for example, 0x78 is parsed as the destination address field, 0x04 is parsed as the data length field, 0x32, 0x91, 0xad and 0x7d are parsed as the data payload field, and these parsed field information is displayed in the analysis software, and then the analysis software further analyzes the compliance of the message based on the parsed destination address whether it is a valid node of the cluster, the data length whether it matches the payload and other information.

[0070] In the embodiment of the application, the analysis software blind area of the GPU private protocol is solved by customizing the analysis plug-in, the original message data which is difficult to read is converted into intuitive field information, the technical threshold of message analysis is reduced, in addition, the plug-in adapts to the existing functions of the analysis software, without the need to develop independent analysis tools, saving the software development cost.

[0071] Further, the analysis of the mirror message based on the field information comprises: verifying the mirror message; wherein the verification operation comprises at least one of format integrity verification, field legality verification and transmission timing rationality verification; and generating an analysis report based on the verification result.

[0072] For example, based on the field parsed by the plug-in, when performing format integrity verification, it is checked whether the message contains all fields required by the private protocol, such as destination address, data length, data payload, etc. If a message only parses the destination address field and lacks the data length field, it is determined that the format is incomplete. When performing field legality verification, it is checked whether the field value (such as 0x78) of the destination address belongs to the node address configured in the cluster. If the cluster node address only contains 0x00-0x0F, it is determined that 0x78 is an illegal value. When performing transmission timing rationality verification, the sending interval of consecutive messages is counted to determine whether it conforms to the timing rules preset by the GPU cluster. If a message with an interval exceeding 50ms appears, it is determined that the timing is abnormal. Finally, all verification results are summarized to generate an analysis report, which clearly shows the actual situation of cluster communication.

[0073] In the embodiment of the present application, the verification process relies on the accurate field information parsed by the plug-in to realize multi-dimensional compliance verification of the GPU private protocol packet, covering the verification dimensions of format, content, timing, etc., and the generated analysis report intuitively presents the problem type and quantitative data, facilitating rapid positioning of potential risks of cluster interconnection.

[0074] Further, the present application also provides a simulation platform, which comprises at least one network card, a hardware emulator, a switch and a processor; wherein the processor is configured to: map the at least one network card as a simulation GPU card; generate at least one virtual GPU card by all network cards and the hardware emulator, and respectively establish a communication connection with the switch; control each network card to send simulation traffic to the switch according to a preset packet sending rule; mirror the packets sent by all virtual GPU cards to at least one target network card based on a target port in the switch; and capture and analyze the mirrored packets received by the target network card by using analysis software.

[0075] Further, the present application realizes all or part of the processes in the GPU cluster silicon pre-verification method described in the above embodiment, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer readable storage medium, and when the computer program is executed by a processor, the steps of each method embodiment described above can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0076] Further, the present application also provides a computer program product, which comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the GPU cluster silicon pre-verification method described in any of the above embodiments is realized.

[0077] The above is the preferred embodiment of the present application, and it should be noted that for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, and these improvements and refinements are also considered to be within the protection scope of the present application.

Claims

1. A method for pre-silicon verification of GPU clusters, characterized in that, include: Map at least one network card to a simulated GPU card using a TCP traffic replay tool; Establish communication connections between all network cards and at least one virtual GPU card generated by the hardware emulator and the switch; Control each network interface card to send simulated traffic to the switch according to preset message sending rules; Based on the target port in the switch, mirror all packets sent by the virtual GPU cards to at least one target network card; The mirrored packets received by the target network card are captured and analyzed using analysis software.

2. The GPU cluster pre-silicon verification method as described in claim 1, characterized in that, The message sending rules are preset rules that match the traffic characteristics and data interaction behavior of the virtual GPU card.

3. The GPU cluster pre-silicon verification method as described in claim 1, characterized in that, The step of mirroring all packets sent by the virtual GPU cards to at least one target network card based on the target port in the switch includes: Set the access ports corresponding to all virtual GPU cards in the switch as mirror source ports, and set the target ports in the switch as mirror destination ports; Establish a communication connection between the target port and at least one target network card, so that the switch can copy the original packets sent by the virtual GPU card received by all mirror source ports, generate mirror packets, and forward the mirror packets to the corresponding target network card through the target port.

4. The GPU cluster pre-silicon verification method as described in claim 1, characterized in that, The process of capturing and analyzing the mirrored packets received by the target network card using analysis software includes: Start the analysis software and select the target network card that receives the mirrored packets as the capture port to obtain the mirrored packets; The analysis software loads a preset parsing plugin; wherein the parsing plugin is defined according to a preset private protocol field. The parsing plugin is used to parse the mirrored message to obtain field information; The mirror message is analyzed based on the field information.

5. The GPU cluster pre-silicon verification method as described in claim 4, characterized in that, The analysis of the mirrored message based on the field information includes: The mirrored message is verified; wherein the verification operation includes at least one of format integrity verification, field validity verification, and transmission timing rationality verification. An analysis report is generated based on the verification results.

6. The GPU cluster pre-silicon verification method as described in claim 2, characterized in that, The traffic characteristics include at least one of message sending rate, message length and traffic load type, and the data interaction behavior includes at least one of message sending timing, retransmission mechanism and congestion response strategy.

7. A simulation platform, characterized in that, It includes at least one network interface card (NIC), a hardware emulator, a switch, and a processor, wherein the processor is configured to: Map at least one network card to a simulated GPU card using a TCP traffic replay tool; Establish communication connections between all network cards and at least one virtual GPU card generated by the hardware emulator and the switch; Control each network interface card to send simulated traffic to the switch according to preset message sending rules; Based on the target port in the switch, mirror all packets sent by the virtual GPU cards to at least one target network card; The mirrored packets received by the target network card are captured and analyzed using analysis software.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium resides to perform the GPU cluster pre-silicon verification method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the GPU cluster pre-silicon verification method as described in any one of claims 1 to 6.

Images