Communication bandwidth calibration methods, equipment, storage media and computer program products

Abstract

The application provides a communication bandwidth calibration method, device, storage medium and computer program product. The method comprises the following steps: obtaining first creation parameter information of completing matrix creation on a first processor or a second processor, first copy parameter information of completing matrix copying, and data information corresponding to the matrix; determining the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information and the data information, so as to calibrate the preconfigured communication bandwidth of the second processor based on the communication bandwidth.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a communication bandwidth calibration method, device, storage medium, and computer program product. Background Technology

[0002] With the rapid advancement and increasing prevalence of artificial intelligence (AI) technology, the demand for AI computing power is growing rapidly. Various AI accelerator cards (also known as AI accelerators) are emerging, including PCIe (Peripheral Component Interconnect Express) and OAM (Operation Administration and Maintenance) cards, making it a fiercely competitive area for both traditional IT giants and numerous startups. Therefore, comprehensive, objective, and unified evaluation of AI accelerator cards has become an urgent need in the industry.

[0003] In related technologies, the effective bandwidth of the graphics card in an AI accelerator card is provided by the manufacturer. There is a lack of third-party objective evaluation, making it difficult to calibrate the pre-configured communication bandwidth of the graphics card in various scenarios and obtain the actual communication bandwidth. Summary of the Invention

[0004] In view of this, this application aims to provide a communication bandwidth calibration method, device, storage medium, and computer program product, which can calibrate the actual communication bandwidth of a graphics card in various scenarios to obtain the actual communication bandwidth.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, this application provides a communication bandwidth calibration method, the method comprising:

[0007] Obtain the first creation parameter information, the first copy parameter information, and the corresponding data information of the matrix after the matrix is ​​created on the first processor or the second processor.

[0008] Based on the first creation parameter information, the first copy parameter information, and the data information, the communication bandwidth of the second processor is determined, so as to calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth.

[0009] Secondly, this application provides a communication bandwidth calibration device, the device comprising:

[0010] The acquisition unit is used to acquire the first creation parameter information, the first copy parameter information, and the data information corresponding to the matrix, which are obtained on the first processor or the second processor after matrix creation is completed.

[0011] The determining unit is used to determine the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information, and the data information, so as to calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth.

[0012] Thirdly, this application provides a communication bandwidth calibration device, the device comprising: a processor and a memory; the processor implements the above-described communication bandwidth calibration method when executing a running program stored in the memory.

[0013] Fourthly, this application provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described communication bandwidth calibration method.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described communication bandwidth calibration method.

[0015] This application provides a communication bandwidth calibration method, device, storage medium, and computer program product. The method includes: acquiring first creation parameter information (for matrix creation completed on a first processor or a second processor), first copy parameter information (for matrix copying completed on a first processor), and data information corresponding to the matrix; determining the communication bandwidth of the second processor based on the first creation parameter information, first copy parameter information, and data information, so as to calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth. By adopting the above implementation scheme, by creating a matrix on the first processor side or the second processor side, performing a corresponding copy operation on the created matrix, and simultaneously acquiring the first creation parameter information and the first copy parameter information (for matrix copying completed on a first processor), the communication bandwidth of the second processor can be determined using only the first creation parameter information, first copy parameter information, and data information corresponding to the matrix. This enables the determination of the actual communication bandwidth of the graphics card in various scenarios. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of a communication bandwidth calibration method provided in an embodiment of this application;

[0017] Figure 2 This application provides an overall interaction flowchart for obtaining the first and second times among multiple server nodes.

[0018] Figure 3 A schematic diagram of the composition structure of a communication bandwidth calibration device provided in this application embodiment. Figure 1;

[0019] Figure 4 A schematic diagram of the composition structure of a communication bandwidth calibration device provided in this application embodiment. Figure 2 . Detailed Implementation

[0020] To gain a more detailed understanding of the features and technical content of the embodiments of this application, the technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The accompanying drawings are for reference only and are not intended to limit the embodiments of this application.

[0021] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0022] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first / second / third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0023] In related technologies, with the rapid development and increasing application of artificial intelligence (AI) technology, the demand for AI computing power is also growing rapidly. Various AI accelerator cards, including PCIe and OAM card types, are emerging to provide AI computing power, making it a fiercely competitive area for both traditional IT giants and numerous startups. Therefore, comprehensive, objective, and unified evaluation of AI accelerator cards has become an urgent need in the industry.

[0024] As is well known, the computing power, memory bandwidth, and inter-card interconnect network bandwidth of an AI accelerator card collectively determine its performance. These three factors act like the planks of a "barrel" in an AI accelerator card's performance, with the capacity of the barrel depending on its shortest plank. In fact, the memory (commonly known as "video memory") configured inside AI accelerator cards is typically Graphics Double Data Rate (GDDR), and the more advanced High Bandwidth Memory (HBM) is also widely used. The theoretical memory bandwidth values ​​of GDDR and HBM are very high, currently reaching hundreds of gigabits per second (GBPS), and are provided by their respective manufacturers. However, there is currently a lack of unified and objective third-party evaluation standards for the effective memory bandwidth of GDDR and HBM in the actual application scenarios of AI accelerator cards.

[0025] Among related technologies, third-party testing tools for testing memory bandwidth include GPU Stream (GPU-STREAM), The Scalable Heterogeneous Computing (SHOC) benchmark suite, CLPeak, and SPEC ACCEL. GPU-STREAM is suitable for AI accelerator cards that support the Open Computing Language (OPENCL) and Compute Unified Device Architecture (CUDA) programming frameworks. The SHOC benchmark suite is suitable for heterogeneous computing systems and scientific applications, and is also only applicable to AI accelerator cards that support OPENCL and CUDA programming frameworks. CLPeak is used to test the peak performance of OPENCL-compatible devices and is more commonly used for peak testing of devices such as mobile phones and desktop laptops. SPEC ACCEL is a proprietary application test suite developed by the Standard Performance Evaluation Corporation (SPEC). Based on numerous scientific computing applications, it employs SPEC's unique testing methodologies and logic and supports the OPENCL and Open Accelerator (OPENACC) programming frameworks.

[0026] However, most products from startups in the AI ​​accelerator chip industry do not support the OpenCL interface and are incompatible with CUDA, rendering the aforementioned testing tools unusable. Therefore, this application aims to address the issue of objectively evaluating the actual achievable memory bandwidth of AI accelerator card products from startups that do not support the OpenCL interface or are incompatible with CUDA in real-world application scenarios, thereby calibrating the communication bandwidth of AI accelerator cards provided by manufacturers at the time of shipment.

[0027] In this application embodiment, the PyTorch and TensorFlow deep learning frameworks have become industry standards, sweeping across the AI ​​field. Statistics show that 70% of the most popular AI models (from small to large language models) support only the PyTorch framework, while 30% support both PyTorch and TensorFlow. Therefore, for any AI accelerator card product to be promoted and applied commercially, it must support the mainstream deep learning framework, namely PyTorch. Thus, this application embodiment uses the PyTorch deep learning framework as a standard unified evaluation method for memory bandwidth in practical application scenarios of AI accelerator cards.

[0028] In view of this, embodiments of this application provide a communication bandwidth calibration method, such as... Figure 1 As shown, the method may include:

[0029] S101. Obtain the first creation parameter information, the first copy parameter information, and the data information corresponding to the matrix after the matrix is ​​created on the first processor or the second processor.

[0030] In this embodiment of the application, the first processor is a central processing unit (CPU), and the second processor is an artificial intelligence accelerator card.

[0031] In this embodiment of the application, the first creation parameter information may include the creation time of the matrix, etc.

[0032] In this embodiment of the application, the first copy parameter information may include the copying time of the matrix, etc.

[0033] In this embodiment of the application, the data information may be the amount of data corresponding to the matrix.

[0034] It should be noted that the first creation parameter information and the first copy parameter information mentioned above are not limited to the time parameters in the embodiments of this application, and the data information is not limited to the data volume of the matrix in the embodiments of this application. The specific data can be selected according to the actual situation.

[0035] In this embodiment, to evaluate the effective bandwidth of the AI ​​accelerator card's memory, the AI ​​accelerator card under test is deployed in a matching server dock. To minimize the influence of the matching server equipment on the effective bandwidth test results of the AI ​​accelerator card, it is first ensured that the hardware and software configurations of the matching server docks are completely identical. Server docks of the same model and series from the same server manufacturer with the same hardware configuration can be selected, and the AI ​​accelerator card under test is installed in the same physical location (e.g., for PCIe-type AI accelerator cards, select the same PCIe slot; for snap-on module-type AI accelerator cards, select AI chip modules with the same location number). The BIOS, BMC, or LINUX OS versions and parameter settings of the server docks are completely identical; the same versions of PyTorch and Python dependencies are deployed in the server docks; in addition, the corresponding AI accelerator card's driver, software stack, SDK, PyTorch adapter package, and Command Control Language (CCL) communication package also need to be installed.

[0036] In this application embodiment, the calibration of the effective bandwidth of AI accelerator card memory is mainly aimed at the following types: the effective bandwidth of AI accelerator card memory includes bandwidth in multiple scenarios, including the effective bandwidth of host to memory (i.e., Host->Device, H2D) and memory to host (i.e., Device->Host, D2H) within a single server node; the effective bandwidth of memory to memory (i.e., Device->Device, D2D) within a single AI accelerator card; and the effective bandwidth of memory between multiple AI accelerator cards (i.e., D2D') (wherein, multiple AI accelerator cards include multiple cards within a single server node and multiple cards between multiple server nodes).

[0037] In this embodiment of the application, for the first scenario: effective bandwidth from host to video memory within a single server node, in the specific implementation process, the first creation parameter information and the first copy parameter information for completing matrix creation on the first processor or the second processor are obtained. This can be achieved in the following ways:

[0038] A matrix is ​​created on a first processor and copied to a second processor; the first time when the matrix creation is completed on the first processor and the second time when the copying of the matrix to the second processor is completed are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0039] In this embodiment of the application, a matrix, such as Tensor A, is first created on the CPU side of the host through the PyTorch API. Tensor A can be a 2D M*N matrix (i.e., M rows and N columns).

[0040] In this embodiment of the application, the data element in Tensor A is of data type float (i.e., single-precision floating-point number, float 16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0041] In this embodiment of the application, the data size corresponding to Tensor A is 2*M*N bytes.

[0042] In this embodiment, PyTorch's high-precision timing interface is invoked to record the current timestamp (i.e., the first time) after the CPU completes the creation of Tensor A. This current timestamp can be represented by T1. Simultaneously, Tensor A is copied to the video memory of the AI ​​accelerator card under test.

[0043] Next, call PyTorch's sync interface to ensure that the copy operation is completed and record the corresponding current timestamp (i.e., the second time), which can be represented by T2.

[0044] In this embodiment of the application, for the second scenario: the effective bandwidth from the video memory to the host within a single server node, in the specific implementation process, the first creation parameter information and the first copy parameter information for completing matrix creation on the first processor or the second processor are obtained. This can be achieved in the following way:

[0045] A matrix is ​​created on the second processor and copied to the first processor; the first time when the matrix creation is completed on the second processor and the second time when the matrix is ​​copied to the first processor are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0046] In this embodiment, a Tensor A: a 2D M*N matrix (i.e., M rows and N columns) is first created in the video memory of the AI ​​accelerator card using the PyTorch API.

[0047] In this embodiment of the application, the data element in Tensor A is of data type float (i.e., single-precision floating-point number, float16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0048] In this embodiment of the application, the data size of Tensor A is 2*M*N Bytes.

[0049] In this embodiment, PyTorch's high-precision timing interface is invoked to record the current timestamp after Tensor A is created in the video memory of the AI ​​accelerator card. This current timestamp can be represented as T1. Simultaneously, Tensor A is copied to the host memory side of the server dock where the AI ​​accelerator card under test is located.

[0050] Next, call PyTorch's sync interface to ensure that the copy operation is completed and record the current timestamp (i.e., the second time), which can be represented as T2.

[0051] In this embodiment of the application, for the third scenario: effective bandwidth between video memory and video memory within a single AI accelerator card, in the specific implementation process, the first creation parameter information for completing matrix creation and the first copy parameter information for completing matrix copying on the first processor or the second processor are obtained. Specifically, this can be achieved in the following way:

[0052] Create a matrix on the second processor, wherein the matrix includes a first type matrix and a second type matrix; copy the first type matrix to the second type matrix; obtain the first time when the matrix creation is completed on the second processor, and the second time when the copying of the first type matrix to the second type matrix on the second processor is completed, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0053] In this embodiment of the application, for the third scenario, the process of creating the matrix on the second processor can be implemented in the following way:

[0054] First, three Tensors, Tensor A, Tensor B, and Tensor C, are created on the host CPU side through the PyTorch interface. Tensor A, Tensor B, and Tensor C are all 1-dimensional vectors (with a length of M).

[0055] In this embodiment of the application, the data elements of the three created Tensors are of data type float (i.e., single-precision floating-point number, float16). The data elements in Tensor A and Tensor B are randomly generated to avoid being hit by the cache in the video memory, while the data elements in Tensor C are empty.

[0056] In this embodiment, the data size of Tensor A and Tensor B is 2 * MBytes. Simultaneously, a scalar is defined, which is a vector of length 1, with a data type of float (i.e., single-precision floating-point number, float16), and its corresponding value is randomly generated.

[0057] In this embodiment of the application, the above-mentioned Tensor A, Tensor B, Tensor C and Scalar are copied to the video memory of the AI ​​accelerator card under test by calling the PyTorch API interface. This can be understood as creating the above-mentioned Tensor A, Tensor B, Tensor C and Scalar on the second processor.

[0058] In this embodiment, PyTorch's Sync interface is called to ensure that the above copying action is completed; secondly, PyTorch's high-precision timing interface is called to record the corresponding current timestamp (i.e., the first time), which can be represented as T1.

[0059] In this embodiment of the application, the first type matrix may include the created Tensor A, Tensor B and Scalar; the second type matrix may include the created Tensor C.

[0060] In this embodiment of the application, copying a first-type matrix to a second-type matrix can be achieved in the following ways:

[0061] The PyTorch operation is invoked to calculate Scalar*Tensor A + Tensor B, and the result is assigned to TensorC. Then, the PyTorch high-precision timing interface is invoked again to record the corresponding current timestamp (i.e., the second time), which can be represented as T2.

[0062] In this embodiment of the application, for the fourth scenario: the effective bandwidth of video memory between multiple AI accelerator cards within a single server node, in the specific implementation process, the first creation parameter information and the first copy parameter information for completing matrix creation on the first processor or the second processor can be obtained in the following ways:

[0063] A matrix is ​​created on a second processor and copied to a third processor, wherein the second and third processors are deployed on the same server; the first time when the matrix creation is completed on the second processor and the second time when the matrix is ​​copied to the third processor are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0064] In this application embodiment, the third processor can be an artificial intelligence accelerator card. In the application embodiment, the third processor can be referred to as a cooperating artificial intelligence accelerator card, used to cooperate with the artificial intelligence accelerator card under test.

[0065] In this embodiment of the application, a Tensor A_Card1: 2D M*N matrix (i.e., M rows and N columns) is first created on the video memory of the AI ​​accelerator card under test using the PyTorch API.

[0066] In this embodiment of the application, the data element in Tensor A_Card1 is of data type float (i.e., single-precision floating-point number, float16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0067] In this embodiment of the application, the data size of Tensor A_Card1 is 2*M*N Bytes.

[0068] In this embodiment of the application, the high-precision timing interface of PyTorch is called to record the current timestamp after Tensor A_Card1 is created on the video memory of the AI ​​accelerator card under test. This current timestamp can be represented as T1.

[0069] In this embodiment, an empty Tensor B_Card2 is created on the video memory of the third processor (i.e., the cooperating AI accelerator card): a 2M*N matrix (i.e., M rows and N columns), with data type float (single-precision floating-point number, float16). The data size of Tensor B_Card2 is 2*M*N bytes.

[0070] In this embodiment of the application, the copy to method of Tensor A_Card1 is called to copy Tensor A_Card1 to Tensor A_Card2.

[0071] Next, call PyTorch's sync interface to ensure that the copy operation is completed and record the current timestamp (i.e., the second time), which can be represented as T2.

[0072] In this embodiment of the application, for the fifth scenario: the effective bandwidth of video memory between multiple AI accelerator cards among multiple server nodes, in the specific implementation process, the first creation parameter information and the first copy parameter information of the matrix creation completed on the first processor or the second processor are obtained. Specifically, this can be achieved in the following way:

[0073] A matrix is ​​created on a second processor and copied to a fourth processor; wherein the second and fourth processors are deployed on different servers; the first time when the matrix creation is completed on the second processor and the second time when the matrix is ​​copied to the fourth processor are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0074] In this embodiment of the application, the fourth processor can be an artificial intelligence accelerator card.

[0075] In this embodiment, any artificial intelligence server cluster comprises P artificial intelligence server nodes (such as bare metal servers), each node containing Q AI accelerator cards (typically Q is 2, 4, or 8). The P AI server nodes are connected via Infiniband or Ethernet, and the hardware and software configurations of the P AI server nodes are identical. One of the P AI server nodes contains the AI ​​accelerator card under test and serves as the Master node, while the remaining nodes serve as cooperating nodes.

[0076] First, the PyTorch distributed API is called to create a process (i.e., worker) for each AI accelerator card in each server node of the AI ​​server cluster. This process can be identified by WORLD_RANK. All processes form a process group, which selects a backend communication library (usually XCCL). This communication library usually supports Message Passing Interface (MPI) to implement fast communication algorithms between AI server cluster nodes.

[0077] In this embodiment, the worker process within the Master node (identified by LOCAL_RANK, corresponding to the AI ​​accelerator card under test, typically with a value of 0) creates a Tensor A_Card1: 2D M*N matrix (i.e., M rows and N columns) in the video memory of the AI ​​accelerator card under test via the PyTorch API.

[0078] In this embodiment of the application, the data element in Tensor A_Card1 is of data type float (i.e., single-precision floating-point number, float16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0079] In this embodiment of the application, the data size of Tensor A_Card1 is 2*M*N Bytes.

[0080] In this embodiment of the application, the high-precision timing interface of PyTorch is called to record the current timestamp after the creation of Tensor A_Card1 in the video memory of the AI ​​accelerator card under test. This current timestamp can be represented as T1.

[0081] In this embodiment, the LOCAL_RANK worker process on the Master node calls the PyTorch distributed API to send TensorA_Card1 to the AI ​​accelerator card (identified by WORLD_RANK, with a value not equal to 0, since 0 refers to the AI ​​accelerator card under test on the Master node) located on the cooperating node (i.e., a node other than the Master node). Simultaneously, the WORLD_RANK worker within the cooperating node begins receiving TensorA_Card1 via the PyTorch distributed API.

[0082] When the WORLD_RANK worker in the cooperating node receives Tensor A_Card1 sent by the AI ​​accelerator card under test (i.e., LOCAL_RANK) on the Master node, it sends a reception confirmation message to the LOCAL_RANK worker on the Master node through the cooperating node.

[0083] Subsequently, after the LOCAL_RANK worker process on the Master node receives the above confirmation message, it records the current timestamp (i.e., the second time), which can be represented as T2.

[0084] Based on the implementation of the fifth scenario described above, the overall interaction process can be referred to... Figure 2 , Figure 2 The specific process shown has been described in detail in the foregoing embodiments and will not be repeated here.

[0085] S102. Based on the first creation parameter information, the first copy parameter information, and the data information, determine the communication bandwidth of the second processor, and calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth.

[0086] In this embodiment of the application, for the first scenario, the first creation parameter information is the time T1 corresponding to the completion of Tensor A creation on the CPU side, the first copy parameter information is the time T2 corresponding to the completion of copying Tensor A to the video memory of the AI ​​accelerator card under test; the data information is the data size of Tensor A, which is 2*M*N bytes.

[0087] In this embodiment of the application, the data copy time T = T2 - T1 for this H2D is calculated using T1 and T2.

[0088] Therefore, the bandwidth of the AI ​​accelerator card under test for this operation is (2*M*N) / (T2-T1).

[0089] In this embodiment of the application, for the second scenario, the first creation parameter information is the time T1 corresponding to the completion of Tensor A in the video memory of the artificial intelligence accelerator card, the first copy parameter information is the time T2 corresponding to the completion of copying Tensor A to the CPU side, and the data information is the data size of Tensor A, which is 2*M*N bytes.

[0090] In this embodiment of the application, the data copy time T = T2 - T1 for this D2H is calculated using T1 and T2.

[0091] Therefore, the bandwidth of the AI ​​accelerator card under test for this operation is (2*M*N) / (T2-T1).

[0092] In this embodiment of the application, for the third scenario, the first creation parameter information is the creation of Tensor A, Tensor B, Tensor C and Scalar in the video memory of the AI ​​accelerator card under test, and the corresponding completion time T1. The first copy parameter information is the assignment of the calculation result of Scalar*Tensor A+Tensor B to Tensor C, and the corresponding completion time T2. The data information includes a data size of 3*M (i.e., reading Tensor A, reading Tensor B, and writing Tensor C).

[0093] In this embodiment of the application, the data copy time T = T2 - T1 is calculated using T1 and T2.

[0094] Therefore, the bandwidth of the AI ​​accelerator card under test for this operation is (3*M) / (T2-T1).

[0095] In this application embodiment, for the fourth scenario, the first creation parameter information is the time T1 corresponding to the creation of Tensor A_Card1 in the video memory of the AI ​​accelerator card under test, the first copy parameter information is the time T2 corresponding to the copying of Tensor A_Card1 to the third processor, and the data information is the data volume 2*M*N Bytes corresponding to Tensor A_Card1.

[0096] In this embodiment of the application, the data copy time T = T2 - T1 is calculated using T1 and T2.

[0097] Therefore, the bandwidth of the AI ​​accelerator card under test for this operation is (2*M*N) / (T2-T1).

[0098] In this application embodiment, for the fifth scenario, the first creation parameter information is the time T1 corresponding to the completion of creating Tensor A_Card1 in the video memory of the AI ​​accelerator card under test, the first copy parameter information is the time T2 corresponding to the confirmation message received from the AI ​​accelerator card of the cooperating node after copying Tensor A_Card1 to the cooperating node, and the data information is the data volume corresponding to Tensor A_Card1 as 2*M*N Bytes.

[0099] In this embodiment of the application, the data copy time T = T2 - T1 is calculated using T1 and T2.

[0100] Therefore, the bandwidth of the AI ​​accelerator card under test for this operation is (2*M*N) / (T2-T1).

[0101] In the embodiments of this application, the actual memory bandwidth of the AI ​​accelerator card can be obtained by testing in the above manner, which can correct the effective bandwidth of the pre-configured AI accelerator card provided by the manufacturer.

[0102] It is understood that the positioning method provided in this application provides a method that creates a matrix on the first processor side or the second processor side, performs a corresponding copy operation on the created matrix, and obtains the first creation parameter information and the first copy parameter information after the matrix is ​​created and copied. Thus, the communication bandwidth of the second processor can be determined by only the first creation parameter information, the first copy parameter information, and the data information corresponding to the matrix, which can realize the determination of the actual communication bandwidth of the graphics card in various scenarios.

[0103] In one embodiment of this application, in a first scenario, the following process may also be performed:

[0104] The process of calculating the communication bandwidth of the AI ​​accelerator card under test in the first scenario above is repeated n times (where n times > 10), and Bandwidth[i] (i = 0, 1, ..., n times - 1) is obtained. The first Bandwidth[0] is removed, and the average of the remaining test values ​​is used to obtain the effective bandwidth H2D-Bandwidth of the AI ​​accelerator card under test in this scenario, as shown in the following formula (1):

[0105]

[0106] In the embodiments of this application, the calculation method of Bandwidth in formula (1) can refer to the calculation method in the first scenario, and will not be repeated here.

[0107] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to H2D that the AI ​​accelerator card under test can achieve in a real-world scenario can be measured, and this value is stable. The possible value ranges of M and N are shown in Table 1 below. Different values ​​of M and N can be selected to determine different sizes of Tensors to determine the effective bandwidth of the AI ​​accelerator card under test.

[0108] Table 1

[0109]

[0110] It should be noted that the above-mentioned tensor dimensions cover the tensor size of the currently popular large language model GPT. In related technologies, the size of the Q / K / V matrix of GPT-3 is 128*12288 / 128*12288 / 12288*12288.

[0111] In this application embodiment, in the second scenario, the following process can also be performed:

[0112] The process of calculating the communication bandwidth of the AI ​​accelerator card under the second scenario above is repeated n times (where n times > 10), and Bandwidth[i] (i = 0, 1, ..., n times - 1) is obtained. The first Bandwidth[0] is removed, and the average of the remaining test values ​​is used to obtain the effective bandwidth D2H-Bandwidth of the AI ​​accelerator card under the scenario, as shown in the following formula (2):

[0113]

[0114] In this embodiment of the application, the calculation method of Bandwidth in formula (2) can refer to the calculation method in the second scenario, and will not be repeated here.

[0115] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to D2H that the AI ​​accelerator card under test can achieve in a real-world scenario can be measured, and this value is stable. The possible value ranges of M and N are shown in Table 2 below. Different values ​​of M and N can be selected to determine different sizes of Tensors to determine the effective bandwidth of the AI ​​accelerator card under test.

[0116] Table 2

[0117]

[0118] It should be noted that the above-mentioned tensor dimensions cover the tensor size of the currently popular large language model GPT. In related technologies, the size of the Q / K / V matrix of GPT-3 is 128*12288 / 128*12288 / 12288*12288.

[0119] In this application embodiment, in the third scenario, the following process can also be performed:

[0120] The process of calculating the communication bandwidth of the AI ​​accelerator card under test in the third scenario above is repeated n times (where n times > 10), and the Bandwidth[i] (i = 0, 1, ..., n times - 1) is obtained. The first Bandwidth[0] is removed, and the average of the remaining test values ​​is used to obtain the effective bandwidth D2D-V2V-Bandwidth of the AI ​​accelerator card under test in this scenario, as shown in the following formula (3):

[0121]

[0122] In the embodiments of this application, the calculation method of Bandwidth in formula (3) can refer to the calculation method in the third scenario, and will not be repeated here.

[0123] It should be noted that in the above calculation process, by gradually increasing the value of M, the effective bandwidth value corresponding to D2D that the AI ​​accelerator card under test can be measured in the actual scenario is a stable value. The possible value range of M is shown in Table 3 below. Different values ​​of M can be selected to determine the effective bandwidth of the AI ​​accelerator card under test.

[0124] Table 3

[0125]

[0126] It should be noted that the above-mentioned tensor dimensions cover the tensor size of the currently popular large language model GPT. Among related technologies, the word vector size of GPT-3 is 12288 dimensions.

[0127] In this application embodiment, in the fourth scenario, the following process can also be performed:

[0128] The process of calculating the communication bandwidth of the AI ​​accelerator card under the fourth scenario above is repeated n times (where n times > 10), and Bandwidth[i] (i = 0, 1, ..., n times - 1) is obtained. The first Bandwidth[0] is removed, and the average of the remaining test values ​​is used to obtain the effective bandwidth SS-D2D'-Bandwidth of the AI ​​accelerator card under the D2D scenario as shown in the following formula (4):

[0129]

[0130] In this embodiment of the application, the calculation method of Bandwidth in formula (4) can refer to the calculation method in the fourth scenario, and will not be repeated here.

[0131] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to D2D' that the AI ​​accelerator card under test can be measured in the actual scenario, and it is a stable value. The possible value ranges of M and N are shown in Table 4 below. Different values ​​of M and N can be selected to determine different sizes of Tensors to determine the effective bandwidth of the AI ​​accelerator card under test.

[0132] Table 4

[0133]

[0134] It should be noted that for data copying between different AI accelerator cards within the same server node, some AI accelerator cards support direct copying from AI accelerator card 1 to AI accelerator card 2, while others require CPU assistance to achieve the copying process from AI accelerator card 1 to CPU and then to AI accelerator card 2.

[0135] In this application embodiment, under the fifth scenario, the following process can also be performed:

[0136] The process of calculating the communication bandwidth of the AI ​​accelerator card under the fifth scenario above is repeated n times (where n times > 10), and Bandwidth[i] (i = 0, 1, ..., n times - 1) is obtained. The first Bandwidth[0] is removed, and the average of the remaining test values ​​is used to obtain the effective bandwidth MS-D2D'-Bandwidth of the AI ​​accelerator card under the D2D' scenario as shown in the following formula (5):

[0137]

[0138] In this embodiment of the application, the calculation method of Bandwidth in formula (5) can refer to the calculation method in the fifth scenario, and will not be repeated here.

[0139] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to D2D' that the AI ​​accelerator card under test can be measured in the actual scenario, and it is a stable value. The possible value ranges of M and N are shown in Table 5 below. Different values ​​of M and N can be selected to determine different sizes of Tensors to determine the effective bandwidth of the AI ​​accelerator card under test.

[0140] Table 5

[0141]

[0142] It should be noted that for data copying between different AI accelerator cards' memory located on different server nodes within the same cluster, some AI accelerator cards support direct copying from AI accelerator card 1 to network card 1 to network card 2 to AI accelerator card 2, while others require CPU assistance to achieve the copying from AI accelerator card 1 to CPU1 to network card 1 to network card 2 to CPU2 to AI accelerator card 2.

[0143] Based on the above embodiments, the memory bandwidth testing method for AI accelerator cards in this application is a standard third-party testing method. Based on the widely used PyTorch deep learning framework, which all accelerator card manufacturers must adapt to, it achieves a manufacturer-independent, cross-platform comparable testing objective. It tests the effective bandwidth of AI accelerator cards in various real-world application scenarios, including: host-to-memory (H2D) effective bandwidth of AI accelerator cards within a single server node; memory-to-host (D2H) effective bandwidth of AI accelerator cards within a single server node; memory-to-memory (D2D) effective bandwidth within the memory of AI accelerator cards; memory-to-memory (D2D') effective bandwidth between different AI accelerator cards within a single server node; and memory-to-memory (D2D') effective bandwidth between AI accelerator cards across different server nodes. Compared to data obtained through theoretical calculations and manufacturers' own testing tools, this method is more accurate and has more practical reference and guiding significance, meeting the objective evaluation needs of the industry for the effective memory bandwidth of all AI accelerator cards.

[0144] Based on the above embodiments, another embodiment of this application provides a communication bandwidth calibration device 1, such as... Figure 3 As shown, the communication bandwidth calibration device 1 includes:

[0145] The acquisition unit 10 is used to acquire the first creation parameter information, the first copy parameter information, and the data information corresponding to the matrix after the matrix is ​​created on the first processor or the second processor.

[0146] The determining unit 11 is used to determine the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information, and the data information, so as to calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth.

[0147] In one embodiment, the communication bandwidth calibration device 1 may further include a creation unit.

[0148] A creation unit is used to create a matrix on the first processor and copy the matrix to the second processor.

[0149] The acquisition unit 10 is also used to acquire the first time when matrix creation is completed on the first processor and the second time when copying the matrix to the second processor is completed, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0150] In one embodiment, the creation unit is further configured to create a matrix on a second processor and copy the matrix to a first processor.

[0151] The acquisition unit 10 is also used to acquire the first time when matrix creation is completed on the second processor and the second time when copying the matrix to the first processor is completed, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0152] In one embodiment, the creation unit is further configured to create a matrix on a second processor, wherein the matrix includes a first type matrix and a second type matrix; and to copy the first type matrix to the second type matrix.

[0153] The acquisition unit 10 is also used to acquire the first time when matrix creation is completed on the second processor, and the second time when copying the first type matrix on the second processor to the second type matrix, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0154] In one embodiment, the creation unit is further configured to create a matrix on a second processor and copy the matrix to a third processor, wherein the second and third processors are deployed on the same server.

[0155] The acquisition unit 10 is also used to acquire the first time when the matrix creation is completed on the second processor and the second time when the matrix is ​​copied to the third processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0156] In one embodiment, the creation unit is further configured to create a matrix on a second processor and copy the matrix to a fourth processor, wherein the second and fourth processors are deployed on different servers.

[0157] The acquisition unit 10 is also used to acquire the first time when matrix creation is completed on the second processor, and the second time when the second processor receives a confirmation message returned by the fourth processor in the case of copying the matrix to the fourth processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0158] In one embodiment, the first processor is a CPU, and the second processor is an artificial intelligence accelerator card.

[0159] This application provides a communication bandwidth calibration device that acquires first creation parameter information, first copy parameter information, and corresponding data information of a matrix created on a first processor or a second processor. Based on the first creation parameter information, first copy parameter information, and data information, the communication bandwidth of the second processor is determined, and the pre-configured communication bandwidth of the second processor is calibrated based on the communication bandwidth. Therefore, the communication bandwidth calibration device proposed in this application, by creating a matrix on the first processor side or the second processor side, performing a corresponding copy operation on the created matrix, and simultaneously acquiring the first creation parameter information and the first copy parameter information of the completed matrix creation and copying, can determine the communication bandwidth of the second processor using only the first creation parameter information, first copy parameter information, and corresponding data information of the matrix. This enables the determination of the actual communication bandwidth of the graphics card in various scenarios.

[0160] Figure 4 This is a schematic diagram of the composition structure of a communication bandwidth calibration device 1 provided in an embodiment of this application. In practical applications, based on the same disclosed concept of the above embodiments, such as... Figure 4 As shown, the communication bandwidth calibration device 1 of this application embodiment includes a processor 12, a memory 13 and a communication bus 14.

[0161] In specific embodiments, the acquisition unit 10, determination unit 11, and creation unit described above can be implemented by a processor 12 located on the communication bandwidth calibration device 1. The processor 12 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor functions can also be other types, and this application embodiment does not impose specific limitations.

[0162] In this embodiment, the communication bus 14 is used to realize the connection communication between the processor 12 and the memory 13; when the processor 12 executes the running program stored in the memory 13, it implements the following communication bandwidth calibration method:

[0163] The system acquires the first creation parameter information, the first copy parameter information, and the corresponding data information of the matrix, which are obtained on the first processor or the second processor after matrix creation is completed. Based on the first creation parameter information, the first copy parameter information, and the data information, the system determines the communication bandwidth of the second processor and calibrates the pre-configured communication bandwidth of the second processor based on the communication bandwidth.

[0164] In one embodiment, the processor 12 is further configured to create a matrix on the first processor and copy the matrix to the second processor; obtain a first time when the matrix creation is completed on the first processor and a second time when the copying of the matrix to the second processor is completed, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0165] In one embodiment, the processor 12 is further configured to create a matrix on the second processor and copy the matrix to the first processor; obtain a first time when the matrix creation is completed on the second processor and a second time when the matrix is ​​copied to the first processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0166] In one embodiment, the processor 12 is further configured to create a matrix on a second processor, wherein the matrix includes a first type matrix and a second type matrix; copy the first type matrix to the second type matrix; obtain a first time when matrix creation is completed on the second processor, and a second time when copying the first type matrix on the second processor to the second type matrix is ​​completed, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0167] In one embodiment, the processor 12 is further configured to create a matrix on the second processor and copy the matrix to the third processor, wherein the second processor and the third processor are deployed on the same server; and to obtain a first time when the matrix creation is completed on the second processor and a second time when the matrix is ​​copied to the third processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0168] In one embodiment, the processor 12 is further configured to create a matrix on the second processor and copy the matrix to the fourth processor, wherein the second processor and the fourth processor are deployed on different servers; obtain a first time when the matrix creation is completed on the second processor, and a second time when the matrix is ​​copied to the fourth processor, the second processor receives a confirmation message returned by the fourth processor, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0169] In one embodiment, the first processor is a CPU, and the second processor is an artificial intelligence accelerator card.

[0170] Based on the above embodiments, this application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a communication bandwidth calibration device. The computer program implements the communication bandwidth calibration method as described above.

[0171] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors and applied in a communication bandwidth calibration device. The computer program implements the communication bandwidth calibration method described above.

[0172] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the embodiments of this application.

[0174] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method of communication bandwidth calibration, the method comprising: The method comprises: obtaining first creation parameter information of completing matrix creation on the first processor or the second processor, first copy parameter information of completing matrix copying, and data information corresponding to the matrix; based on the first creation parameter information, the first copy parameter information and the data information, determining the communication bandwidth of the second processor, so as to calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth.

2. The method of claim 1, wherein, The obtaining of the first creation parameter information of completing matrix creation on the first processor or the second processor, and the first copy parameter information of completing matrix copying comprises: creating a matrix on the first processor and copying the matrix to the second processor; obtaining a first time of completing the matrix creation on the first processor and a second time of completing the copying of the matrix to the second processor, wherein the first creation parameter information comprises the first time and the first copy parameter information comprises the second time.

3. The method of claim 1, wherein, The obtaining of the first creation parameter information of completing matrix creation on the first processor or the second processor, and the first copy parameter information of completing matrix copying comprises: creating a matrix on the second processor and copying the matrix to the first processor; obtaining a first time of completing the matrix creation on the second processor and a second time of completing the copying of the matrix to the first processor, wherein the first creation parameter information comprises the first time and the first copy parameter information comprises the second time.

4. The method of claim 1, wherein, The obtaining of the first creation parameter information of completing matrix creation on the first processor or the second processor, and the first copy parameter information of completing matrix copying comprises: creating a matrix on the second processor, wherein the matrix comprises a first type matrix and a second type matrix; copying the first type matrix to the second type matrix; obtaining a first time of completing the matrix creation on the second processor and a second time of completing the copying of the first type matrix on the second processor to the second type matrix, wherein the first creation parameter information comprises the first time and the first copy parameter information comprises the second time.

5. The method of claim 1, wherein, The obtaining of the first creation parameter information of completing matrix creation on the first processor or the second processor, and the first copy parameter information of completing matrix copying comprises: creating a matrix on the second processor and copying the matrix to a third processor, wherein the second processor and the third processor are deployed in the same server; obtaining a first time of completing the matrix creation on the second processor and a second time of completing the copying of the matrix to the third processor, wherein the first creation parameter information comprises the first time and the first copy parameter information comprises the second time.

6. The method of claim 1, wherein, The obtaining of the first creation parameter information of completing matrix creation on the first processor or the second processor, and the first copy parameter information of completing matrix copying comprises: creating a matrix on the second processor and copying the matrix to a fourth processor, wherein the second processor and the fourth processor are deployed in different servers; obtain a first time when the matrix creation is completed on the second processor, and a second time when the second processor receives a return acknowledgement message of the fourth processor in case of copying the matrix to the fourth processor, wherein the first creation parameter comprises the first time, and the first copy parameter information comprises the second time.

7. The method of claim 1, wherein, The first processor is a central processing unit (CPU), and the second processor is an artificial intelligence acceleration card.

8. A communication bandwidth calibration device, characterized by, The device comprises: an obtaining unit, configured to obtain first creation parameter information of completing matrix creation on a first processor or a second processor, first copy parameter information of completing matrix copying, and data information corresponding to the matrix; a determining unit, configured to determine a communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information, and the data information, so as to calibrate a preconfigured communication bandwidth of the second processor based on the communication bandwidth.

9. A communication bandwidth calibration device, characterized by The device comprises a processor and a memory; the processor implements the method in any one of claims 1 to 7 when executing a running program stored in the memory.

10. A storage medium having stored thereon a computer program, characterized in that The computer program implements the method in any one of claims 1 to 7 when executed by a processor.

11. A computer program product comprising a computer program, characterized in that, The computer program implements the method in any one of claims 1 to 7 when executed by a processor.

Images