Communication bandwidth calibration method and device, storage medium and computer program product

Abstract

The invention provides a communication bandwidth calibration method and device, a storage medium and a computer program product. The method comprises the steps of obtaining first creation parameter information completing matrix creation on a first processor or a second processor, first copy parameter information completing matrix copy, and data information corresponding to a matrix; and determining a communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information and the data information, so as to calibrate a pre-configured communication bandwidth of the second processor based on the communication bandwidth.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a communication bandwidth calibration method, device, storage medium and computer program product. Background Art

[0002] With the rapid advancement of artificial intelligence technology and the increasing popularity of its applications, the demand for artificial intelligence computing power has grown rapidly. Various types of artificial intelligence acceleration cards (AI accelerators in English) that provide artificial intelligence computing power, including peripheral component interconnect express (PCIe) form, operation administration and maintenance (OAM) card form, and other products have emerged in an endless stream, becoming a field where traditional giants in the IT industry and many start-ups are competing. Comprehensive, objective and unified evaluation of artificial intelligence acceleration cards has also become an urgent need in the industry.

[0003] In the related art, the effective bandwidth of the graphics card of the artificial intelligence accelerator card is provided by the manufacturer. Due to the lack of objective evaluation by a third party, it is difficult to calibrate the pre-configured communication bandwidth of the graphics card of the artificial intelligence accelerator card in various scenarios to obtain the actual communication bandwidth. Summary of the invention

[0004] In view of this, the present application hopes to provide a communication bandwidth calibration method, device, storage medium and computer program product, which can calibrate the actual communication bandwidth of the graphics card in various scenarios to obtain the actual communication bandwidth.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, the present application provides a communication bandwidth calibration method, the method comprising:

[0007] Acquire first creation parameter information for completing matrix creation on the first processor or the second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix;

[0008] Based on the first creation parameter information, the first copy parameter information, and the data information, a communication bandwidth of the second processor is determined to calibrate a preconfigured communication bandwidth of the second processor based on the communication bandwidth.

[0009] In a second aspect, the present application provides a communication bandwidth calibration device, the device comprising:

[0010] An acquisition unit, used to acquire first creation parameter information for completing matrix creation on the first processor or the second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix;

[0011] The determining unit is used to determine the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information and the data information, so as to calibrate the preconfigured communication bandwidth of the second processor based on the communication bandwidth.

[0012] In a third aspect, the present application provides a communication bandwidth calibration device, the device comprising: a processor and a memory; the processor implements the above-mentioned communication bandwidth calibration method when executing an operating program stored in the memory.

[0013] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, which implements the above-mentioned communication bandwidth calibration method when executed by a processor.

[0014] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned communication bandwidth calibration method when executed by a processor.

[0015] The present application provides a communication bandwidth calibration method, device, storage medium and computer program product, the method comprising: obtaining first creation parameter information for completing matrix creation on a first processor or a second processor, first copy parameter information for completing matrix copying, and data information corresponding to the matrix; determining the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information and the data information, so as to calibrate the communication bandwidth preconfigured by the second processor based on the communication bandwidth. With the above implementation scheme, by creating a matrix on the first processor side or the second processor side, performing a corresponding copy operation on the created matrix, and obtaining the first creation parameter information for completing matrix creation and the first copy parameter information for completing matrix copying, the communication bandwidth of the second processor can be determined only through the first creation parameter information, the first copy parameter information and the data information corresponding to the matrix, and the actual communication bandwidth of the graphics card in various scenarios can be determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of a communication bandwidth calibration method provided in an embodiment of the present application;

[0017] Figure 2 An overall interaction flow chart for obtaining a first time and a second time between multiple server nodes provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of a communication bandwidth calibration device provided in an embodiment of the present application Figure 1;

[0019] Figure 4 A schematic diagram of the structure of a communication bandwidth calibration device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0020] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the technical solution of the present application is further elaborated in detail below in combination with the drawings and specific embodiments of the specification. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0021] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0022] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first / second / third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0023] In related technologies, with the rapid development and increasing popularity of artificial intelligence technology, the demand for artificial intelligence computing power is also growing rapidly. Various artificial intelligence acceleration cards that provide artificial intelligence computing power, including PCIe form factors, OAM card form factors, and other products are emerging in an endless stream, becoming a field that traditional IT industry giants and many start-ups are competing for. Comprehensive, objective, and unified evaluation of artificial intelligence acceleration cards has also become an urgent need in the industry.

[0024] As we all know, the computing power, memory bandwidth and inter-card interconnection bandwidth of artificial intelligence accelerator cards jointly determine the performance of artificial intelligence accelerator cards. The three indicators are like the baffles of the "barrel" of artificial intelligence accelerator card performance. The capacity of the "barrel" depends on the shortest board. In fact, the memory (commonly known as "video memory") configured inside the artificial intelligence accelerator card is usually graphics double data rate (Graphics Double Data Rate, GDDR), and the more advanced high bandwidth memory (High Bandwidth Memory, HBM) has also been widely adopted. The theoretical video memory bandwidth values ​​of the above GDDR and HBM are very high, currently up to several hundred gigabits per second (GigaBit Per Second, GBPS), and are all provided by the corresponding manufacturers. However, the effective video memory bandwidth of the above GDDR and HBM in the actual application scenarios of artificial intelligence accelerator cards currently lacks a unified and objective third-party evaluation standard.

[0025] In the related technologies, the third-party testing tools used by the industry to test video memory bandwidth include GPU stream (Graphics Processing Unit, GPU-STREAM), Scalable Heterogeneous Computing Benchmark Suite (SHOC), CLPeak, and SPEC ACCEL. Among them, GPU-STREAM is suitable for artificial intelligence accelerator cards that support the Open Computing Language (OPENCL) and the Compute Unified Device Architecture (CUDA) programming framework; the SHOC benchmark test set is suitable for heterogeneous computing systems and is suitable for scientific applications. It is also only suitable for artificial intelligence accelerator cards that support the OPENCL and CUDA programming frameworks; CLPeak is used to test the peak performance of OPENCL-compatible devices, and is more used for peak testing of mobile phones, desktop notebooks and other devices; SPEC ACCEL is a proprietary application test set developed by the Standard Performance Evaluation Corporation (SPEC), based on many scientific computing applications, using SPEC's unique testing methodology and logic, and supporting OPENCL and Open Accelerator (OPENACC) programming frameworks.

[0026] However, most of the products of many AI acceleration chip startups in the industry do not support the OPENCL interface and are not compatible with CUDA, so the above test tools cannot be used. Therefore, the embodiments of this application are intended to solve the problem of objectively evaluating the actual achievable video memory bandwidth of the products of many AI acceleration card startups that do not support the OPENCL interface and are not compatible with CUDA in actual application scenarios, so as to calibrate the communication bandwidth of the AI ​​acceleration card provided by the manufacturer when the AI ​​acceleration card leaves the factory.

[0027] In the embodiments of the present application, the deep learning framework based on Pytorch and Tensorflow has become the industry standard and has swept the AI ​​field. According to statistics, 70% of the most popular AI models in the industry (from small and medium models to large language models) only support the Pytorch framework, and 30% support both the Pytorch framework and the Tensorflow framework. Therefore, any artificial intelligence acceleration card product that is to be promoted and applied in the business world must support the mainstream deep learning framework, namely the Pytorch framework. Therefore, the embodiments of the present application are based on the Pytorch framework deep learning framework as a standard unified evaluation method for video memory bandwidth in actual application scenarios of artificial intelligence acceleration cards.

[0028] In view of this, an embodiment of the present application provides a communication bandwidth calibration method, such as Figure 1 As shown, the method may include:

[0029] S101, obtaining first creation parameter information for completing matrix creation on a first processor or a second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix.

[0030] In an embodiment of the present application, the first processor is a central processing unit (CPU), and the second processor is an artificial intelligence accelerator card.

[0031] In the embodiment of the present application, the first creation parameter information may include the creation time of the matrix, etc.

[0032] In the embodiment of the present application, the first copy parameter information may include the copy time of the matrix, etc.

[0033] In the embodiment of the present application, the data information may be the amount of data corresponding to the matrix.

[0034] It should be noted that the above-mentioned first creation parameter information and first copy parameter information are not limited to the time parameters in the embodiment of the present application, and the data information is not limited to the data volume of the matrix in the embodiment of the present application, and the specific ones can be selected according to actual conditions.

[0035] In an embodiment of the present application, in order to evaluate the effective bandwidth of the video memory of the artificial intelligence accelerator card, the artificial intelligence accelerator card being evaluated is deployed in a matching server base. In order to eliminate the influence of the matching server equipment on the test results of the effective bandwidth of the graphics card of the artificial intelligence accelerator card being tested as much as possible, first ensure that the hardware and software configurations of the matching server base are exactly the same. You can choose a server base of the same model and the same hardware configuration of the same series from the same server manufacturer, and install the artificial intelligence accelerator card being tested in the same physical position (such as for artificial intelligence accelerator cards in the form of PCIe, select the same PCIe slot; for artificial intelligence accelerator cards in the form of buckle card modules, select AI chip modules with the same position number). The BIOS or BMC or LINUX OS version and parameter settings of the server base are exactly the same; the same version of Pytorch and python and other dependent packages are deployed in the server base; in addition, it is also necessary to install the driver, software stack, SDK, Pytorch adaptation package and command control language (Command Control Language, CCL) communication package of the corresponding artificial intelligence accelerator card.

[0036] In the embodiments of the present application, the calibration is mainly aimed at the effective bandwidth of the video memory of the following artificial intelligence acceleration cards: the effective bandwidth of the video memory of the artificial intelligence acceleration card includes the bandwidth in multiple scenarios, including the effective bandwidth from the host to the video memory (i.e. Host->Device, H2D) and the effective bandwidth from the video memory to the host (i.e. Device->Host, D2H) within a single server node; the effective bandwidth from the video memory to the video memory (i.e. Device->Device, D2D) within a single artificial intelligence accelerator card; the effective bandwidth of the video memory between multiple artificial intelligence acceleration cards (i.e. D2D') (wherein, multiple artificial intelligence acceleration cards include between multiple cards within a single server node and between multiple cards between multiple server nodes).

[0037] In the embodiment of the present application, for the first scenario: effective bandwidth from the host to the video memory in a single server node, in the specific implementation process, the first creation parameter information for completing the matrix creation on the first processor or the second processor and the first copy parameter information for completing the matrix copy are obtained, which can be specifically implemented in the following manner:

[0038] A matrix is ​​created on a first processor, and the matrix is ​​copied to a second processor; a first time when the matrix creation is completed on the first processor, and a second time when the matrix is ​​copied to the second processor are obtained, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0039] In an embodiment of the present application, a matrix is ​​first created on the CPU side of the host through the Pytorch API, such as Tensor A, where Tensor A can be a 2-dimensional M*N matrix (ie, M rows and N columns).

[0040] In the embodiment of the present application, the data type of the data element in Tensor A is float (ie, single-precision floating point number, float 16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0041] In this embodiment of the present application, the data size corresponding to Tensor A is 2*M*N bytes.

[0042] In the embodiment of the present application, the high-precision timing interface of Pytorch is called to record the current timestamp (i.e., the first time) after the CPU completes the creation of Tensor A. The current timestamp can be represented by T1. At the same time, the Tensor A is copied to the video memory of the artificial intelligence accelerator card under test.

[0043] Again, call the sync synchronization interface of Pytorch to ensure that the copy operation is completed and record the corresponding current timestamp (ie, the second time), which can be represented by T2.

[0044] In the embodiment of the present application, for the second scenario: the effective bandwidth from the video memory in a single server node to the host, in the specific implementation process, the first creation parameter information for completing the matrix creation on the first processor or the second processor and the first copy parameter information for completing the matrix copy are obtained, which can be implemented in the following manner:

[0045] A matrix is ​​created on the second processor, and the matrix is ​​copied to the first processor; a first time of completing the creation of the matrix on the second processor and a second time of completing the copying of the matrix to the first processor are obtained, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0046] In the embodiment of the present application, a Tensor A is first created in the video memory of the artificial intelligence accelerator card through the Pytorch API: a 2-dimensional M*N matrix (i.e., M rows and N columns).

[0047] In an embodiment of the present application, the data type of the data elements in Tensor A is float (ie, single-precision floating point number, float16), and the data elements can be randomly generated to avoid being hit by the cache in the video memory.

[0048] In this embodiment of the present application, the data size of Tensor A is 2*M*N Bytes.

[0049] In the embodiment of the present application, the high-precision timing interface of Pytorch is called to record the current timestamp after the creation of Tensor A in the video memory of the artificial intelligence accelerator card, which can be represented as T1. At the same time, the Tensor A is copied to the host memory side of the server base where the artificial intelligence accelerator card under test is located.

[0050] Again, call the sync synchronization interface of Pytorch to ensure that the copy action is completed and record the current timestamp (ie, the second time), which can be expressed as T2.

[0051] In the embodiment of the present application, for the third scenario: effective bandwidth from video memory to video memory in a single artificial intelligence acceleration card, in the specific implementation process, the first creation parameter information for completing matrix creation on the first processor or the second processor and the first copy parameter information for completing matrix copy are obtained, which can be implemented in the following manner:

[0052] A matrix is ​​created on a second processor, wherein the matrix includes a first type matrix and a second type matrix; the first type matrix is ​​copied to the second type matrix; a first time for completing the creation of the matrix on the second processor is obtained, and a second time corresponding to completing the copying of the first type matrix on the second processor to the second type matrix is ​​obtained, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0053] In the embodiment of the present application, for the third scenario, the process of creating a matrix on the second processor can be implemented in the following manner:

[0054] First, three Tensors are created on the host CPU side through the Pytorch interface, namely Tensor A, Tensor B, and Tensor C. Tensor A, Tensor B, and Tensor C are all 1-dimensional vectors (length M).

[0055] In an embodiment of the present application, the data type of the data elements in the three created Tensors is float (i.e., single-precision floating point numbers, float16), where the data elements in Tensor A and Tensor B are randomly generated to avoid being hit by the cache in the video memory, and the data elements in Tensor C are empty.

[0056] In the embodiment of the present application, the data size of Tensor A and Tensor B is 2*MBytes respectively. At the same time, a scalar is defined, Scalar is a vector with a length of 1, the data type is float (i.e. single-precision floating point number, float16), and the corresponding value is randomly generated.

[0057] In an embodiment of the present application, the Pytorch API interface is called to copy the above-mentioned Tensor A, Tensor B, Tensor C, and Scalar to the video memory of the artificial intelligence accelerator card under test, which can be understood as creating the above-mentioned Tensor A, Tensor B, Tensor C, and Scalar on the second processor.

[0058] In an embodiment of the present application, the Sync synchronization interface of Pytorch is called to ensure that the above-mentioned copying action is completed; secondly, the high-precision timing interface of Pytorch is called to record the corresponding current timestamp (ie, the first time), which can be expressed as T1.

[0059] In an embodiment of the present application, the first type of matrix may include the created Tensor A, Tensor B, and Scalar; the second type of matrix may include the created Tensor C.

[0060] In the embodiment of the present application, copying the first type matrix to the second type matrix can be achieved by the following method:

[0061] Call the Pytorch operation to calculate Scalar*Tensor A+Tensor B, and assign the calculation result to Tensor C. Call the Pytorch high-precision timing interface again to record the corresponding current timestamp (ie, the second time), which can be expressed as T2.

[0062] In the embodiment of the present application, for the fourth scenario: the effective bandwidth of the video memory between multiple artificial intelligence acceleration cards in a single server node, in the specific implementation process, the first creation parameter information of completing the matrix creation on the first processor or the second processor and the first copy parameter information of completing the matrix copy are obtained, which can be specifically implemented in the following manner:

[0063] A matrix is ​​created on a second processor, and the matrix is ​​copied to a third processor, wherein the second processor and the third processor are deployed in the same server; a first time of completing the creation of the matrix on the second processor and a second time of completing the copying of the matrix to the third processor are obtained, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0064] In an embodiment of the present application, the third processor may be an artificial intelligence acceleration card. In an embodiment of the application, the third processor may be referred to as a cooperating artificial intelligence acceleration card, which is used to cooperate with the artificial intelligence acceleration card under test.

[0065] In the embodiment of the present application, a Tensor A_Card1 is first created on the video memory of the artificial intelligence accelerator card under test through the Pytorch API: a 2-dimensional M*N matrix (i.e., M rows and N columns).

[0066] In the embodiment of the present application, the data type of the data element in Tensor A_Card1 is float (ie, single-precision floating point number, float16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0067] In the embodiment of the present application, the data size of Tensor A_Card1 is 2*M*N Bytes.

[0068] In an embodiment of the present application, the high-precision timing interface of Pytorch is called to record the current timestamp after the creation of Tensor A_Card1 on the video memory of the artificial intelligence accelerator card under test is completed. The current timestamp can be expressed as T1.

[0069] In the embodiment of the present application, an empty Tensor B_Card2 is created on the video memory of the third processor (i.e., the matched artificial intelligence acceleration card): 2 is an M*N matrix (i.e., M rows and N columns), and the data type is float (single-precision floating point number, float16). The data size of Tensor B_Card2 is 2*M*N Bytes.

[0070] In the embodiment of the present application, the copy to method of Tensor A_Card1 is called to copy Tensor A_Card1 to Tensor A_Card2.

[0071] Again, call the sync synchronization interface of Pytorch to ensure that the copy action is completed and record the current timestamp (ie, the second time), which can be expressed as T2.

[0072] In the embodiment of the present application, for the fifth scenario: the effective bandwidth of the video memory between multiple artificial intelligence acceleration cards between multiple server nodes, in the specific implementation process, the first creation parameter information of completing the matrix creation on the first processor or the second processor and the first copy parameter information of completing the matrix copy are obtained, which can be specifically implemented in the following manner:

[0073] A matrix is ​​created on a second processor, and the matrix is ​​copied to a fourth processor; wherein the second processor and the fourth processor are deployed in different servers; a first time when the matrix creation is completed on the second processor, and a second time when a confirmation message is returned by the fourth processor and received by the second processor when the matrix is ​​copied to the fourth processor are obtained, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0074] In this embodiment of the present application, the fourth processor may be an artificial intelligence acceleration card.

[0075] In the embodiment of the present application, for any artificial intelligence server cluster, it includes P artificial intelligence server nodes (such as bare metal servers), each node includes Q artificial intelligence acceleration cards (usually Q is 2, 4, or 8); the P artificial intelligence server nodes are connected via infinite bandwidth (which can be expressed as Infiniband in English) or Ethernet, and the software and hardware configurations of the P artificial intelligence server nodes are exactly the same. One of the P artificial intelligence server nodes includes the tested artificial intelligence acceleration card as the Master node, and the remaining nodes are used as matching nodes.

[0076] First, call the Pytorch distributed API interface to create a process (i.e., worker) for each artificial intelligence accelerator card in each server node in the artificial intelligence server cluster. The process can be identified by WORLD_RANK. All processes constitute a process group, which selects a backend communication library (usually: XCCL). The communication library usually supports the Message Passing Interface (MPI) to implement a fast communication algorithm between nodes in the artificial intelligence server cluster.

[0077] In an embodiment of the present application, a worker process in the Master node (labeled by LOCAL_RANK, corresponding to the artificial intelligence accelerator card under test, usually with a value of 0) creates a Tensor A_Card1 in the video memory of the artificial intelligence accelerator card under test through the Pytorch API: a 2-dimensional M*N matrix (i.e., M rows and N columns).

[0078] In the embodiment of the present application, the data type of the data element in Tensor A_Card1 is float (ie, single-precision floating point number, float16), and the data element can be randomly generated to avoid being hit by the cache in the video memory.

[0079] In the embodiment of the present application, the data size of Tensor A_Card1 is 2*M*N Bytes.

[0080] In an embodiment of the present application, the high-precision timing interface of Pytorch is called to record the current timestamp after the creation of Tensor A_Card1 in the video memory of the artificial intelligence accelerator card under test is completed. The current timestamp can be expressed as T1.

[0081] In the embodiment of the present application, the LOCAL_RANK worker process on the Master node calls the Pytorchdistributed API to send TensorA_Card1 to the artificial intelligence accelerator card (identified by WORLD_RANK, the value is not equal to 0, because 0 refers to the tested artificial intelligence accelerator card of the Master node) located in the cooperating node (i.e., the node other than the Master node). At the same time, the WORLD_RANK worker in the cooperating node starts to receive Tensor A_Card1 through the Pytorch distributed API.

[0082] When the WORLD_RANK worker in the cooperating node receives Tensor A_Card1 sent by the AI ​​accelerator card under test (i.e., LOCAL_RANK) on the Master node, it sends a reception confirmation message to the LOCAL_RANKworker on the Master node through the cooperating node.

[0083] Subsequently, after receiving the above confirmation message, the LOCAL_RANK worker process on the Master node records the current timestamp (ie, the second time), which can be expressed as T2.

[0084] Based on the above fifth scenario, the overall interaction process can be referred to Figure 2 , Figure 2 The specific process shown in has been specifically described in the above embodiments and will not be repeated here.

[0085] S102: Determine a communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information, and the data information, so as to calibrate a preconfigured communication bandwidth of the second processor based on the communication bandwidth.

[0086] In the embodiment of the present application, for the first scenario, the first creation parameter information is the time T1 corresponding to the completion of creating Tensor A on the CPU side, and the first copy parameter information is the time T2 corresponding to the completion of copying Tensor A to the video memory of the artificial intelligence accelerator card under test; the data information is the data volume size corresponding to Tensor A, 2*M*N bytes.

[0087] In the embodiment of the present application, the data copy time T of this H2D is calculated by T1 and T2 to obtain T=T2-T1.

[0088] Then the memory bandwidth of the tested artificial intelligence acceleration card corresponding to this operation is Bandwidth = (2*M*N) / (T2-T1).

[0089] In the embodiment of the present application, for the second scenario, the first creation parameter information is the time T1 corresponding to the completion of creating Tensor A in the video memory of the artificial intelligence accelerator card, and the first copy parameter information is the time T2 corresponding to the completion of copying Tensor A to the CPU side; the data information is the data volume size corresponding to Tensor A, 2*M*N bytes.

[0090] In the embodiment of the present application, the data copy time T of this D2H is calculated by T1 and T2 to obtain T=T2-T1.

[0091] Then the memory bandwidth of the tested artificial intelligence acceleration card corresponding to this operation is Bandwidth = (2*M*N) / (T2-T1).

[0092] In an embodiment of the present application, for the third scenario, the first creation parameter information is to create Tensor A, Tensor B, Tensor C and Scalar in the video memory of the artificial intelligence accelerator card under test, and the corresponding time T1 is completed. The first copy parameter information is to assign the calculation result of Scalar*Tensor A+Tensor B to Tensor C to complete the corresponding time T2. The data information includes a data amount of 3*M (i.e., read Tensor A, read Tensor B, and write Tensor C).

[0093] In the embodiment of the present application, the current data copy time T=T2-T1 is calculated by T1 and T2.

[0094] Then the memory bandwidth of the tested artificial intelligence accelerator card corresponding to this operation is Bandwidth = (3*M) / (T2-T1).

[0095] In an embodiment of the present application, for the fourth scenario, the first creation parameter information is the time T1 corresponding to the creation of Tensor A_Card1 in the video memory of the artificial intelligence accelerator card under test, the first copy parameter information is the time T2 corresponding to the copying of Tensor A_Card1 to the third processor, and the data information is the data volume 2*M*N Bytes corresponding to Tensor A_Card1.

[0096] In the embodiment of the present application, the current data copy time T=T2-T1 is calculated by T1 and T2.

[0097] Then the memory bandwidth of the tested artificial intelligence acceleration card corresponding to this operation is Bandwidth = (2*M*N) / (T2-T1).

[0098] In an embodiment of the present application, for the fifth scenario, the first creation parameter information is the time T1 corresponding to the completion of creating Tensor A_Card1 in the video memory of the artificial intelligence accelerator card under test, the first copy parameter information is the time T2 corresponding to the receipt of the confirmation message replied by the cooperating artificial intelligence accelerator card after copying Tensor A_Card1 to the artificial intelligence accelerator card of the cooperating node, and the data information is the data volume 2*M*N Bytes corresponding to Tensor A_Card1.

[0099] In the embodiment of the present application, the current data copy time T=T2-T1 is calculated by T1 and T2.

[0100] Then the memory bandwidth of the tested artificial intelligence acceleration card corresponding to this operation is Bandwidth = (2*M*N) / (T2-T1).

[0101] In the embodiment of the present application, the actual memory bandwidth of the artificial intelligence acceleration card can be tested in the above manner, and the effective bandwidth of the pre-configured artificial intelligence acceleration card provided by the manufacturer can be corrected.

[0102] It can be understood that a positioning method provided by an embodiment of the present application creates a matrix on the first processor side or the second processor side, performs a corresponding copy operation on the created matrix, and simultaneously obtains the first creation parameter information of the matrix creation and the first copy parameter information of the matrix copy. Therefore, the communication bandwidth of the second processor can be determined only through the first creation parameter information, the first copy parameter information and the data information corresponding to the matrix, and the actual communication bandwidth of the graphics card in various scenarios can be determined.

[0103] In an embodiment of the present application, in the first scenario, the following process may also be performed:

[0104] The process of calculating the communication bandwidth of the tested artificial intelligence accelerator card in the first scenario is repeated ntimes times (where ntimes>10), and Bandwidth[i] (i=0, 1, ..., ntimes-1) is obtained. The first Bandwidth[0] is removed, and the effective bandwidth H2D-Bandwidth of the H2D tested artificial intelligence accelerator card in this scenario is obtained by using the average value of the remaining test values ​​as shown in the following formula (1):

[0105]

[0106] In the embodiment of the present application, the calculation method of Bandwidth in formula (1) can refer to the calculation method in the first scenario, which will not be repeated here.

[0107] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to H2D that the tested AI accelerator card can achieve in the actual scenario can be measured, which is a stable value. The possible value ranges of M and N are shown in the following Table 1. Different M and N values ​​can be selected to determine the Tensor of different data sizes to determine the effective bandwidth of the tested AI accelerator card.

[0108] Table 1

[0109]

[0110] It should be noted that the above-mentioned tensor dimensions cover the tensor scale of the currently popular large language model GPT. In related technologies, the scale of the Q / K / V matrices of GPT-3 is 128*12288 / 128*12288 / 12288*12288.

[0111] In the embodiment of the present application, in the second scenario, the following process may also be performed:

[0112] The process of calculating the communication bandwidth of the artificial intelligence accelerator card under test in the second scenario is repeated ntimes times (where ntimes>10), and Bandwidth[i] (i=0, 1, ..., ntimes-1) is obtained. The first Bandwidth[0] is removed, and the effective bandwidth D2H-Bandwidth of the D2H artificial intelligence accelerator card under test in this scenario is obtained by using the average value of the remaining test values ​​as shown in the following formula (2):

[0113]

[0114] In the embodiment of the present application, the calculation method of Bandwidth in formula (2) can refer to the calculation method in the second scenario, which will not be repeated here.

[0115] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to D2H that the tested artificial intelligence acceleration card can achieve in the actual scenario can be measured, which is a stable value. The possible value ranges of M and N are shown in the following Table 2. Different M and N values ​​can be selected to determine the Tensor of different data sizes to determine the effective bandwidth of the tested artificial intelligence acceleration card.

[0116] Table 2

[0117]

[0118] It should be noted that the above-mentioned tensor dimensions cover the tensor scale of the currently popular large language model GPT. In related technologies, the scale of the Q / K / V matrices of GPT-3 is 128*12288 / 128*12288 / 12288*12288.

[0119] In the embodiment of the present application, in the third scenario, the following process may also be performed:

[0120] The process of calculating the communication bandwidth of the tested artificial intelligence accelerator card in the third scenario is repeated ntimes times (where ntimes>10), and Bandwidth[i] (i=0, 1, ..., ntimes-1) is obtained. The first Bandwidth[0] is removed, and the effective bandwidth D2D-V2V-Bandwidth of the tested artificial intelligence accelerator card in the D2D scenario is obtained by using the average value of the remaining test values ​​as shown in the following formula (3):

[0121]

[0122] In the embodiment of the present application, the calculation method of Bandwidth in formula (3) can refer to the calculation method in the third scenario, which will not be repeated here.

[0123] It should be noted that in the above calculation process, by gradually increasing the M value, the effective bandwidth value corresponding to D2D that the tested artificial intelligence acceleration card can achieve in the actual scenario can be measured, which is a stable value. The possible value range of M is shown in the following Table 3. Different M values ​​can be selected to determine the effective bandwidth of the tested artificial intelligence acceleration card.

[0124] Table 3

[0125]

[0126] It should be noted that the above tensor dimensions cover the tensor scale of the currently popular large language model GPT. In related technologies, the word vector size of GPT-3 is 12288 dimensions.

[0127] In the embodiment of the present application, in the fourth scenario, the following process may also be performed:

[0128] The process of calculating the communication bandwidth of the artificial intelligence accelerator card under test in the fourth scenario is repeated ntimes times (where ntimes>10), and Bandwidth[i] (i=0, 1, ..., ntimes-1) is obtained. The first Bandwidth[0] is removed, and the average value of the remaining test values ​​is used to obtain the effective bandwidth SS-D2D'-Bandwidth of the artificial intelligence accelerator card under test in the D2D' scenario as shown in the following formula (4):

[0129]

[0130] In the embodiment of the present application, the calculation method of Bandwidth in formula (4) can refer to the calculation method in the fourth scenario, which will not be repeated here.

[0131] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to D2D' that the tested artificial intelligence acceleration card can achieve in the actual scenario can be measured, which is a stable value. The possible value ranges of M and N are shown in the following Table 4. Different M and N values ​​can be selected to determine the Tensor of different data sizes to determine the effective bandwidth of the tested artificial intelligence acceleration card.

[0132] Table 4

[0133]

[0134] It should be noted that for data copying between the video memories of different AI accelerator cards in the same server node, some AI accelerator cards support direct copying from AI accelerator card 1 -> AI accelerator card 2, while some cards must implement AI accelerator card 1 -> CPU -> AI accelerator card 2 with the assistance of the CPU.

[0135] In the embodiment of the present application, in the fifth scenario, the following process may also be performed:

[0136] The process of calculating the communication bandwidth of the artificial intelligence accelerator card under test in the fifth scenario is repeated ntimes times (where ntimes>10), and Bandwidth[i] (i=0, 1, ..., ntimes-1) is obtained. The first Bandwidth[0] is removed, and the effective bandwidth MS-D2D'-Bandwidth of the artificial intelligence accelerator card under test in this scenario is obtained by using the average value of the remaining test values ​​as shown in the following formula (5):

[0137]

[0138] In the embodiment of the present application, the calculation method of Bandwidth in formula (5) can refer to the calculation method in the fifth scenario, which will not be repeated here.

[0139] It should be noted that in the above calculation process, by gradually increasing the values ​​of M and N, the effective bandwidth value corresponding to D2D' that the tested artificial intelligence acceleration card can achieve in the actual scenario can be measured, which is a stable value. The possible value ranges of M and N are shown in the following Table 5. Different M and N values ​​can be selected to determine the Tensor of different data sizes to determine the effective bandwidth of the tested artificial intelligence acceleration card.

[0140] Table 5

[0141]

[0142] It should be noted that for data copying between the video memories of different AI accelerator cards located in different server nodes in the same cluster, some AI accelerator cards support direct copying from AI accelerator card 1->network card 1->network card 2->AI accelerator card 2, while some cards must implement AI accelerator card 1->CPU1->network card 1->network card 2->CPU2->AI accelerator card 2 with the assistance of the CPU.

[0143] Based on the above embodiments, the memory bandwidth test method of the artificial intelligence accelerator card in the embodiment of the present application is a standard third-party test method, based on the widely popular Pytorch deep learning framework that each accelerator card manufacturer must adapt, and realizes the test goal of independent manufacturers and comparable results across platforms. It tests the effective bandwidth of artificial intelligence accelerator cards in multiple application scenarios in reality, including: the effective bandwidth of the host to memory (H2D) memory of the artificial intelligence accelerator card in a single server node, the effective bandwidth of the memory to the host (D2H) memory of the artificial intelligence accelerator card in a single server node, the effective bandwidth of the memory to the memory (D2D) memory in the memory of the artificial intelligence accelerator card, the effective bandwidth of the memory to the memory (D2D') memory between different artificial intelligence accelerator cards in a single server node, and the effective bandwidth of the memory to the memory (D2D') memory between artificial intelligence accelerator cards between different server nodes. Compared with the data obtained by theoretical calculations and the manufacturer's own test tools, it is more accurate and has more realistic reference and guiding significance, and can meet the objective evaluation needs of the effective memory bandwidth of all artificial intelligence accelerator cards in the industry.

[0144] Based on the above embodiment, in another embodiment of the present application, a communication bandwidth calibration device 1 is provided. Figure 3 As shown, the communication bandwidth calibration device 1 includes:

[0145] The acquisition unit 10 is used to acquire first creation parameter information for completing matrix creation on the first processor or the second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix.

[0146] The determining unit 11 is configured to determine the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information and the data information, so as to calibrate the preconfigured communication bandwidth of the second processor based on the communication bandwidth.

[0147] In one embodiment, the communication bandwidth calibration device 1 may further include: a creating unit.

[0148] The creation unit is used to create a matrix on the first processor and copy the matrix to the second processor.

[0149] The acquisition unit 10 is further used to acquire a first time for completing matrix creation on the first processor and a second time for completing copying the matrix to the second processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0150] In one embodiment, the creating unit is further configured to create a matrix on the second processor and copy the matrix to the first processor.

[0151] The acquisition unit 10 is further used to acquire a first time for completing matrix creation on the second processor and a second time for completing copying the matrix to the first processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0152] In one embodiment, the creating unit is further configured to create a matrix on the second processor, wherein the matrix includes a first type matrix and a second type matrix; and copy the first type matrix to the second type matrix.

[0153] The acquisition unit 10 is also used to acquire a first time for completing matrix creation on the second processor, and a second time corresponding to completing copying the first type matrix on the second processor to the second type matrix, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0154] In one embodiment, the creating unit is further configured to create a matrix on the second processor and copy the matrix to the third processor, wherein the second processor and the third processor are deployed in the same server.

[0155] The acquisition unit 10 is further used to acquire a first time for completing matrix creation on the second processor and a second time for completing copying the matrix to the third processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0156] In one embodiment, the creating unit is further configured to create a matrix on the second processor and copy the matrix to the fourth processor, wherein the second processor and the fourth processor are deployed in different servers.

[0157] The acquisition unit 10 is further used to acquire a first time when the matrix creation is completed on the second processor, and a second time when the fourth processor returns a confirmation message received by the second processor when the matrix is ​​copied to the fourth processor, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0158] In one embodiment, the first processor is a CPU and the second processor is an artificial intelligence accelerator card.

[0159] An embodiment of the present application provides a communication bandwidth calibration device, which obtains first creation parameter information for completing matrix creation on a first processor or a second processor, first copy parameter information for completing matrix copying, and data information corresponding to the matrix; based on the first creation parameter information, the first copy parameter information and the data information, the communication bandwidth of the second processor is determined to calibrate the pre-configured communication bandwidth of the second processor based on the communication bandwidth. It can be seen that a communication bandwidth calibration device proposed in an embodiment of the present application creates a matrix on the first processor side or the second processor side, performs a corresponding copy operation on the created matrix, and simultaneously obtains the first creation parameter information for completing the matrix creation and the first copy parameter information for completing the matrix copying, so that the communication bandwidth of the second processor can be determined only through the first creation parameter information, the first copy parameter information and the data information corresponding to the matrix, and the actual communication bandwidth of the graphics card in various scenarios can be determined.

[0160] Figure 4 The schematic diagram of the structure of a communication bandwidth calibration device 1 provided in an embodiment of the present application is shown in FIG. 1 . In practical applications, based on the same disclosure concept of the above embodiments, Figure 4 As shown, the communication bandwidth calibration device 1 of the embodiment of the present application includes a processor 12 , a memory 13 and a communication bus 14 .

[0161] In the specific implementation process, the acquisition unit 10, the determination unit 11, and the creation unit can be implemented by a processor 12 located on the communication bandwidth calibration device 1, and the processor 12 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing image processing device (DSPD), a programmable logic image processing device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiments of the present application are not specifically limited.

[0162] In the embodiment of the present application, the communication bus 14 is used to realize the connection communication between the processor 12 and the memory 13; when the processor 12 executes the running program stored in the memory 13, the following communication bandwidth calibration method is implemented:

[0163] Acquire first creation parameter information for completing matrix creation on the first processor or the second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix; determine the communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information, and the data information, so as to calibrate the preconfigured communication bandwidth of the second processor based on the communication bandwidth.

[0164] In one embodiment, the above-mentioned processor 12 is also used to create a matrix on the first processor and copy the matrix to the second processor; obtain a first time for completing the matrix creation on the first processor and a second time for completing the copying of the matrix to the second processor, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0165] In one embodiment, the above-mentioned processor 12 is also used to create a matrix on the second processor and copy the matrix to the first processor; obtain a first time for completing the matrix creation on the second processor and a second time for completing the copying of the matrix to the first processor, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

[0166] In one embodiment, the above-mentioned processor 12 is also used to create a matrix on a second processor, wherein the matrix includes a first type matrix and a second type matrix; copy the first type matrix to the second type matrix; obtain a first time to complete the matrix creation on the second processor, and a second time corresponding to completing the copying of the first type matrix on the second processor to the second type matrix, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0167] In one embodiment, the processor 12 is further used to create a matrix on a second processor and copy the matrix to a third processor, wherein the second processor and the third processor are deployed in the same server; obtain a first time for completing the creation of the matrix on the second processor and a second time for completing the copying of the matrix to the third processor, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0168] In one embodiment, the processor 12 is further used to create a matrix on a second processor and copy the matrix to a fourth processor, wherein the second processor and the fourth processor are deployed in different servers; obtain a first time when the matrix creation is completed on the second processor, and a second time when the second processor receives a confirmation message returned by the fourth processor when the matrix is ​​copied to the fourth processor, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

[0169] In one embodiment, the first processor is a CPU and the second processor is an artificial intelligence accelerator card.

[0170] Based on the above embodiments, an embodiment of the present application provides a storage medium on which a computer program is stored. The above computer-readable storage medium stores one or more programs. The above one or more programs can be executed by one or more processors and applied to a communication bandwidth calibration device. The computer program implements the communication bandwidth calibration method as described above.

[0171] Based on the above embodiments, an embodiment of the present application provides a computer program product, including a computer program. The computer program can be executed by one or more processors and is applied to a communication bandwidth calibration device. The computer program implements the communication bandwidth calibration method as described above.

[0172] It should be noted that in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for an image display device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the method described in each embodiment of the embodiment of the present application.

[0174] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A communication bandwidth calibration method, characterized in that: The method comprises: Acquire first creation parameter information for completing matrix creation on the first processor or the second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix; Based on the first creation parameter information, the first copy parameter information, and the data information, a communication bandwidth of the second processor is determined to calibrate a preconfigured communication bandwidth of the second processor based on the communication bandwidth.

2. The method according to claim 1, characterized in that The obtaining of first creation parameter information for completing matrix creation on the first processor or the second processor and first copy parameter information for completing matrix copying includes: Creating a matrix on the first processor and copying the matrix to the second processor; A first time for completing creation of the matrix on the first processor and a second time for completing copying the matrix to the second processor are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

3. The method according to claim 1, characterized in that The obtaining of first creation parameter information for completing matrix creation on the first processor or the second processor and first copy parameter information for completing matrix copying includes: Creating a matrix on the second processor and copying the matrix to the first processor; A first time for completing creation of the matrix on the second processor and a second time for completing copying the matrix to the first processor are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

4. The method according to claim 1, characterized in that: The obtaining of first creation parameter information for completing matrix creation on the first processor or the second processor and first copy parameter information for completing matrix copying includes: Creating a matrix on the second processor, wherein the matrix includes a first type matrix and a second type matrix; Copying the first type matrix to the second type matrix; Obtain a first time for completing creation of the matrix on the second processor, and a second time for completing copying the first type matrix on the second processor to the second type matrix corresponding to the second type matrix, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

5. The method according to claim 1, characterized in that The obtaining of first creation parameter information for completing matrix creation on the first processor or the second processor and first copy parameter information for completing matrix copying includes: Creating a matrix on the second processor, and copying the matrix to a third processor, wherein the second processor and the third processor are deployed in the same server; A first time for completing creation of the matrix on the second processor and a second time for completing copying the matrix to the third processor are obtained, wherein the first creation parameter includes the first time and the first copy parameter information includes the second time.

6. The method according to claim 1, characterized in that The obtaining of first creation parameter information for completing matrix creation on the first processor or the second processor and first copy parameter information for completing matrix copying includes: Creating a matrix on the second processor, and copying the matrix to a fourth processor, wherein the second processor and the fourth processor are deployed in different servers; Obtain a first time when the matrix creation is completed on the second processor, and a second time when the fourth processor returns a confirmation message received by the second processor when the matrix is ​​copied to the fourth processor, wherein the first creation parameter includes the first time, and the first copy parameter information includes the second time.

7. The method according to claim 1, characterized in that The first processor is a central processing unit (CPU), and the second processor is an artificial intelligence accelerator card.

8. A communication bandwidth calibration device, characterized in that: The device comprises: An acquisition unit, used to acquire first creation parameter information for completing matrix creation on the first processor or the second processor, first copy parameter information for completing matrix copy, and data information corresponding to the matrix; A determining unit is used to determine a communication bandwidth of the second processor based on the first creation parameter information, the first copy parameter information and the data information, so as to calibrate a preconfigured communication bandwidth of the second processor based on the communication bandwidth.

9. A communication bandwidth calibration device, characterized in that: The device comprises: a processor and a memory; when the processor executes the running program stored in the memory, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images