Communication data compression method and system based on quantization technology
By using quantization techniques to compress and transmit communication data in the distributed GMRES algorithm, the problem of high inter-process communication overhead is solved, the amount of data transmitted is reduced and the bit width is effectively utilized, thereby improving the efficiency of the iterative algorithm.
Patent Information
- Application Number
- CN202411125553.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-16
AI Technical Summary
In the distributed GMRES algorithm, inter-process data communication becomes the main performance bottleneck, and existing technologies struggle to effectively reduce communication overhead.
A communication data compression method based on quantization technology is adopted. The local calculation results are compressed and transmitted through quantization and dequantization strategies. The quantization technology maps the floating-point number set to the fixed-point number range. Data compression is performed only during communication and does not affect the calculation accuracy.
It significantly reduces data transmission volume, improves bit width utilization efficiency, reduces the impact of iterative convergence, and optimizes the overall efficiency of distributed GMRES.
Smart Images

Figure CN118944678A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication, and in particular relates to a communication data compression method and system based on quantization technology. Background Art
[0002] The generalized minimum residual method (GMRES) is an iterative solution algorithm for solving the sparse linear system Ax=b, where A is a sparse matrix, and x and b are dense vectors. There are usually two methods to solve sparse linear systems, namely the direct method and the iterative method. The direct method has the advantages of good versatility and high accuracy of calculation results, but it faces problems such as high storage requirements and large amount of calculation when solving large-scale sparse linear systems. The iterative method generates a series of approximate solutions that are close to the exact solution through multiple iterations. It has the advantages of small amount of calculation, low memory requirements, and can make full use of the sparsity of the matrix. It is the mainstream method for solving large-scale sparse linear systems and is widely used in scientific computing and simulation fields such as computational electromagnetics and computational fluid dynamics.
[0003] Since a single computing node cannot meet the storage and computing resource requirements for solving large-scale sparse linear systems, distributed iterative solvers are more commonly used in scientific and engineering computing. Although distributed systems have more computing and storage resources available for problem solving, the communication delay between nodes will be significantly increased compared to the communication delay within a single node.
[0004] Distributed computing of GMRES is an important means to solve large-scale sparse linear systems. Its distributed implementation distributes the main computing kernels such as sparse matrix-vector multiplication (SpMV), vector addition, and vector inner product to each node for calculation. To ensure the normal operation of iteration, each process needs to merge and reduce the local calculation results of typical vector operators such as vector addition, vector inner product, and bi-norm. These calculation results are usually scalars with small communication overhead. For the SpMV operator y=Ap involving sparse matrix operations, each process needs to obtain the complete multiplication vector p through inter-process communication before calling the SpMV kernel in each iteration. The data for inter-process communication is a vector. The experimental results of distributed GMRES on 29 sparse linear systems show that the average ratio of vector communication time before the execution of the SpMV operator to the total running time of the iterative algorithm exceeds 80%. This means that data communication between distributed nodes is the main performance bottleneck of distributed GMRES. How to reduce its communication overhead is an important direction for optimizing the overall efficiency of distributed GMRES. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a communication data compression method and system based on quantization technology. Among them, a communication data compression method based on quantization technology includes:
[0006] Distributed iteration is performed based on quantization and dequantization strategies. When each process obtains its own local calculation result, the transmission vector is quantized and compressed based on the numerical characteristics of the local calculation result.
[0007] Each process sends the quantized result to other processes through the network, and each process obtains the complete quantized result;
[0008] Based on the complete quantization result, each process dequantizes the data it receives to obtain a vector represented by the original precision, and then performs local SpMV calculations and subsequent operations on the vector represented by the original precision.
[0009] Preferably, the process of performing distributed iteration based on quantization and dequantization strategies includes:
[0010] In each iteration, each process first transmits the high-precision representation of the data to be transmitted to other processes. Quantize it into a fixed-point representation of w bits through the quantization function FP2INTw, then send it to other processes, and receive the quantized data from other processes to obtain the complete quantization result v w ; Then call INTw2FP to convert the complete quantization result v w Dequantize to high-precision floating point representation v out , and then continue with subsequent SpMV and other operations.
[0011] Preferably, the process of quantizing and compressing the transmission vector based on the numerical characteristics of the local calculation result includes:
[0012] The communication compression method based on uniform quantization maps all floating-point numbers in the interval [a, b] to an integer interval with a fixed step size. Given a floating-point vector x in And an unsigned integer grid, quantization operation is performed to map the real vector to a low-bit-width integer grid to achieve data compression.
[0013] Preferably, the quantification process is expressed as:
[0014]
[0015] in, Indicates that the input data is rounded to the nearest integer. The definition of the clamp function is:
[0016]
[0017] Preferably, the communication compression method based on uniform quantization has different selections of zero points, and the uniform quantization includes symmetric quantization and asymmetric quantization;
[0018] The asymmetric quantization parameter composition includes a scaling factor S, a zero point z, and a bit width w; wherein the scaling factor S is a floating point number and is used to determine the step size of the quantizer.
[0019] Preferably, the process of quantizing and compressing the transmission vector based on the numerical characteristics of the local calculation result includes:
[0020] The communication compression method based on non-uniform quantization introduces a segmentation point in the quantization range by using the quantization technology based on piecewise linear function, splits the quantization interval into two non-overlapping areas, and obtains the central dense area and the edge discrete area;
[0021] The quantization technology based on the piecewise linear function uniformly quantizes the data distributed in the two quantization intervals respectively, and the quantization loss of the edge discrete area and the central dense area is regulated by controlling the segmentation point.
[0022] Preferably, a quantization technique based on a piecewise linear function is used to introduce a segmentation point in the quantization range, and a process of splitting the quantization interval into two non-overlapping areas includes:
[0023] The parameter p is introduced to represent the proportion of elements contained in the central dense area. By adjusting the parameter p, the central dense area and the edge discrete area are segmented, and the segmentation point setting for data distribution adaptation is completed.
[0024] Preferably, the process of uniformly quantizing the data distributed in two quantization intervals respectively based on the piecewise linear function quantization technology includes:
[0025] Through the hybrid compression of quantization mapping and floating point representation, the elements in the edge discrete area are not quantized and compressed, and the original FP64 or FP32 representation is still used;
[0026] Define the proportion of elements contained in the central dense area as α, then the proportion of elements contained in the edge discrete area is 1-α, assuming that a 64-bit integer is used to represent the edge discrete area, and the central dense area is compressed by uniform quantization, and the quantization bit width is w, then the communication compression ratio η achieved by quantization is:
[0027]
[0028] Preferably, based on the complete quantization result, the process of each process dequantizing the data received by itself includes:
[0029] After the compressed data is communicated between processes, each process dequantizes the received data, and the floating-point vector obtained by dequantization is approximately equal to the initial floating-point vector;
[0030] The dequantization process is expressed as follows:
[0031] x out =s·(x w -z)
[0032] The quantization factor s is determined by the quantization range [a, b] and the quantization bit width w as follows:
[0033]
[0034] The present invention also provides a communication data compression system based on quantization technology, comprising:
[0035] A quantization module is used to quantize and compress the transmission vector based on the numerical characteristics of the local calculation results after each process obtains its own local calculation results; each process sends the quantized results to other processes through the network, and each process obtains a complete quantization result;
[0036] The dequantization module is connected to the quantization module and is used for each process to dequantize the data it receives based on the complete quantization result to obtain a vector represented by the original precision, and then perform local SpMV calculations and subsequent operations on the vector represented by the original precision.
[0037] Compared with the prior art, the present invention has the following advantages and technical effects:
[0038] The present invention quantizes floating-point numbers before transmission, which can significantly reduce the amount of data transmission on the one hand, and can achieve effective utilization of the bit width on the other hand, thereby reducing the impact on iterative convergence.
[0039] The communication compression method based on quantization technology proposed by the present invention maps a given floating-point number set to a fixed-point number interval within a specific range according to the numerical characteristics of limited floating-point data. Compared with the floating-point format representation divided into exponent and mantissa, the quantization representation can improve the utilization efficiency of limited bit width, thereby improving the representation accuracy of floating-point numbers. More importantly, the compressed data based on quantization technology is only used for communication, not for calculation, so it supports compression of arbitrary bit width. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0041] Figure 1 : is a numerical distribution diagram of the communication vector in the iterative algorithm of the embodiment of the present invention; wherein, (a) is a middle exponential distribution diagram; (b) is a mantissa distribution diagram; (c) is a distribution diagram of the change of the information entropy of the value;
[0042] Figure 2 : are the element proportion distribution diagrams of the same index in the embodiments of the present invention; wherein (a) is the nd12k proportion distribution diagram; (b) is the nd24k proportion distribution diagram; (c) is the bmwcra_1 proportion distribution diagram;
[0043] Figure 3 A schematic diagram of a distributed iterative process based on quantization technology according to an embodiment of the present invention;
[0044] Figure 4 1 is a data distribution diagram of a communication floating-point vector according to an embodiment of the present invention; wherein (a) is a bmwcra_1 data distribution diagram; and (b) is a nd12k data distribution diagram. DETAILED DESCRIPTION
[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0047] Embodiment 1
[0048] This embodiment provides a communication data compression method based on quantization technology, including:
[0049] Distributed iteration is performed based on quantization and dequantization strategies. When each process obtains its own local calculation results, the transmission vector is quantized and compressed based on the numerical characteristics of the local calculation results.
[0050] Each process sends the quantized result to other processes through the network, and each process obtains the complete quantized result;
[0051] Based on the complete quantization result, each process dequantizes the data it receives to obtain a vector represented using the original precision, and then performs local SpMV calculations and subsequent operations on the vector represented using the original precision.
[0052] Furthermore, the process of distributed iteration based on quantization and anti-quantization strategies includes:
[0053] In each iteration, each process first transmits the high-precision representation of the data to be transmitted to other processes. Quantize it into a fixed-point representation of w bits through the quantization function FP2INTw, then send it to other processes, and receive the quantized data from other processes to obtain the complete quantization result v w; Then call INTw2FP to complete the quantization result v w Dequantize to high-precision floating point representation v out , and then continue with subsequent SpMV and other operations.
[0054] Furthermore, the process of quantizing and compressing the transmission vector based on the numerical characteristics of the local calculation result includes:
[0055] The communication compression method based on uniform quantization maps all floating-point numbers in the interval [a, b] to an integer interval with a fixed step size. Given a floating-point vector x in And an unsigned integer grid, quantization operation is performed to map the real vector to a low-bit-width integer grid to achieve data compression.
[0056] Furthermore, the quantization process is expressed as:
[0057]
[0058] in, Indicates that the input data is rounded to the nearest integer. The definition of the clamp function is:
[0059]
[0060] Furthermore, the communication compression method based on uniform quantization has different zero point selections, and uniform quantization includes symmetric quantization and asymmetric quantization;
[0061] The parameters of asymmetric quantization include a scaling factor S, a zero point z, and a bit width w; wherein the scaling factor S is a floating point number used to determine the step size of the quantizer.
[0062] Furthermore, the process of quantizing and compressing the transmission vector based on the numerical characteristics of the local calculation result includes:
[0063] The communication compression method based on non-uniform quantization introduces a segmentation point in the quantization range by using the quantization technology based on piecewise linear function, splits the quantization interval into two non-overlapping areas, and obtains the central dense area and the edge discrete area;
[0064] The quantization technology based on piecewise linear function uniformly quantizes the data distributed in two quantization intervals respectively, and regulates the quantization loss of the edge discrete area and the central dense area by controlling the segmentation point.
[0065] Furthermore, a segmentation point is introduced into the quantization range by using a quantization technique based on a piecewise linear function, and the process of splitting the quantization interval into two non-overlapping regions includes:
[0066] The parameter p is introduced to represent the proportion of elements contained in the central dense area. By adjusting the parameter p, the central dense area and the edge discrete area are segmented, and the segmentation point setting for data distribution adaptation is completed.
[0067] Furthermore, the process of uniformly quantizing the data distributed in two quantization intervals based on the piecewise linear function quantization technology includes:
[0068] Through the hybrid compression of quantization mapping and floating-point representation, the elements in the edge discrete area are not quantized and compressed, and the original FP64 or FP32 representation is still used;
[0069] Define the proportion of elements contained in the central dense area as α, then the proportion of elements contained in the edge discrete area is 1-α. Assuming that a 64-bit integer is used to represent the edge discrete area, uniform quantization is used to compress the central dense area, and the quantization bit width is w, the communication compression ratio η achieved by quantization is:
[0070]
[0071] Furthermore, based on the complete quantization result, each process performs dequantization on the data it receives, including:
[0072] After the compressed data is communicated between processes, each process dequantizes the received data, and the floating-point vector obtained by dequantization is approximately equal to the initial floating-point vector;
[0073] Among them, the dequantization process is expressed as:
[0074] x out =s·(x w -z)
[0075] The quantization factor s is determined by the quantization range [a, b] and the quantization bit width w as follows:
[0076]
[0077] Embodiment 2
[0078] This embodiment also provides a communication data compression system based on quantization technology, including:
[0079] The quantization module is used to quantize and compress the transmission vector based on the numerical characteristics of the local calculation results after each process obtains its own local calculation results. Each process sends the quantized results to other processes through the network, and each process obtains a complete quantization result.
[0080] The dequantization module is connected to the quantization module and is used to dequantize the data received by each process based on the complete quantization result to obtain a vector represented by the original precision, and then perform local SpMV calculations and subsequent operations on the vector represented by the original precision.
[0081] Embodiment 3
[0082] The existing technology hides data communication or reduces communication overhead through methods such as sparse matrix rearrangement, sparse matrix partitioning based on graph partitioning algorithm, overlapping calculation and communication, and selective transmission of communication vectors. Among them, sparse matrix rearrangement and sparse matrix partitioning based on graph partitioning are to reduce communication requirements as much as possible from the source, and overlapping calculation and communication hides part of the communication process through asynchronous execution of tasks. Selective transmission means only communicating the data that each process really needs. These methods have significantly improved the computational efficiency of the distributed GMRES algorithm. However, statistical analysis of the optimized algorithm found that communication overhead still accounts for a large proportion.
[0083] Table 1
[0084]
[0085] At present, the communication data in the distributed iterative algorithm adopts IEEE754-like floating point formats, such as FP64, FP32, BF16 and FP16. Such floating point formats have significant storage redundancy for the communication data representation of the iterative algorithm. As shown in Table 1, taking the GMRES algorithm as an example, Algorithm 1 shows its iterative process. Through the inner iteration of Algorithm 1 (lines 3-14), it can be found that the SpMV operation Av j The vector v that needs to be communicated in (line 4) j After standardization, its elements are all floating-point numbers with absolute values less than 1. The exponent bits of the four traditional mainstream floating-point formats are 11, 7, 7 and 5 respectively, and their representation range far exceeds the value range of vector elements in the iterative process. This means that when the transmission vector elements are represented by IEEE754, the exponent part has very limited values, and the limited floating-point exponent bits are not fully utilized in the iterative solution algorithm scenario. Taking the sparse linear equations with coefficient matrices of bmwcra_1, nd12k and nd24k from the SuiteSparse matrix set as an example, the inventors further analyzed its iterative solution process, as shown in the following figure. Figure 1 As shown in the figure, the information entropy of the exponent, mantissa, and value of the transmitted vector v changes during 155 iterations. It can be observed that during the solution of the same linear system, the information entropy of the mantissa and value of the transmitted floating-point vector elements is the same, which is much larger than the information entropy of the exponent. In the three linear systems, the information entropy of the exponent is basically distributed between 2 and 3. Further statistics on the proportion of elements with the same exponent are as follows: Figure 2As shown, it can be observed that in most iterations, 1, 2, 4 and 8 shared indices can cover about 30%, 50%, 80% and 96% of the non-zero elements, respectively.
[0086] The above analysis shows that the floating point representation based on the IEEE74 standard or similar representation methods using exponent + mantissa will result in the problem that bits cannot be effectively utilized in actual data representation.
[0087] To solve the above technical problems, this embodiment provides a communication data compression method based on quantization technology, such as Figure 3 The figure shows the distributed iterative process using quantization and dequantization strategies. After each process obtains its local calculation results, it quantizes and compresses the transmission vector based on the numerical characteristics of the calculation results. Then each process sends the quantized results to other processes through the network, so that each process has a complete quantization result. Finally, each process dequantizes the data it receives to obtain a vector represented by the original precision, and then performs local SpMV operations and subsequent operations.
[0088] As shown in Table 2, Algorithm 2 shows the core steps of the distributed iterative algorithm using quantization optimization. In each iteration, each process first transmits the high-precision representation of the data to be transmitted to other processes. Quantize it into a fixed-point representation of w bits through the quantization function FP2INTw, then send it to other processes, and receive the quantized data from other processes to get the complete v w . Then call INTw2FP to dequantize it into a high-precision floating point representation v out , and then continue with the subsequent SpMV and other operations. Although the quantization before transmission and the dequantization after transmission introduce additional data conversion overhead, the quantization and dequantization processes are performed element by element, the elements are independent of each other, and the data operated on are all in the GPU memory, so they can be completed quickly using the tens of thousands of GPU computing cores. Compared with the benefits in communication, the overhead of quantization and dequantization is almost negligible.
[0089] Table 2
[0090]
[0091]
[0092] The floating-point format defined by the existing IEEE754 standard adopts the same representation rules for all floating-point numbers, which can meet general computing needs. In practical applications, the data of each application usually has unique numerical distribution characteristics, such as the number of data and the range of data values. However, the existing floating-point representation of the IEEE754-like format is difficult to match the numerical distribution characteristics of the application, resulting in significant waste of bit width. Quantization technology maps a given floating-point number set to a fixed-point number interval within a specific range according to the numerical characteristics of limited floating-point data. Compared with the floating-point format representation that divides the exponent and the mantissa, the quantized representation can improve the utilization efficiency of the limited bit width, thereby improving the representation accuracy of the floating-point number. However, the existing quantization technology is designed for artificial intelligence applications, and model training and reasoning are not as sensitive to the representation accuracy of data as scientific and engineering calculations, so the quantization technology cannot be directly applied to the communication optimization of distributed iterative algorithms. This embodiment considers how to quantize the communication vector of the iterative algorithm from two situations: uniform quantization and non-uniform quantization.
[0093] (1) Communication compression method based on uniform quantization;
[0094] Uniform quantization maps all floating-point numbers in the interval [a, b] to an integer interval with a fixed step size. Depending on the choice of the zero point, uniform quantization can be divided into symmetric and asymmetric quantization. Symmetric quantization can be regarded as a special asymmetric quantization. Asymmetric quantization consists of three parameters: scaling factor S, zero point z, and bit width w. The scaling factor S is usually a floating-point number that determines the step size of the quantizer. Given a floating-point vector x in and an unsigned integer grid, the quantization process can be expressed as:
[0095]
[0096] in, Indicates that the input data is rounded to the nearest integer. The definition of the clamp function is:
[0097]
[0098] The above quantization operation can map the real vector to a lower bit width integer grid, thereby achieving data compression. After the compressed data is communicated between processes, each process also needs to dequantize the received data in the following way. The dequantized floating-point vector is approximately equal to the original floating-point vector.
[0099] x out =s·(x w -z)(3)
[0100] The quantization factor s is determined by the quantization range [a, b] and the quantization bit width w as follows:
[0101]
[0102] In some quantization methods in the field of artificial intelligence, the quantization range is not necessarily an interval consisting of the maximum and minimum values of the current tensor. In this case, if the quantized result exceeds the interval [min, max], it will be directly clamped to the min or max value. This is applicable to the compression of certain data in artificial intelligence applications, because errors in a small part of the data do not necessarily affect the global performance of the model. However, for the scientific computing problems discussed in the present invention, in the iterative solution process, large errors on some components may have a greater impact on the convergence of the iterative algorithm. Therefore, the value of the quantization range of this embodiment is statistically derived from the current vector to be compressed. Specifically, before quantization, the communication vector to be compressed is statistically analyzed to obtain its maximum and minimum values, which is very efficient using multi-threaded parallel technology on mainstream accelerator GPUs.
[0103] In order to perform efficient calculations directly based on existing instruction sets on quantized data, the quantization bit width used in artificial intelligence applications is generally 4 or 8. However, in the problem considered in the present invention, it is not necessary to perform iterative calculations directly based on quantized data, but only for data synchronization and transmission between different computing nodes, so there is no need to be limited to 4 or 8 bit widths, but can be quantized to any bit width as needed. Of course, for iterative algorithms, the bit width below 8 bits indicates that the precision is too low, and the data compression effect is very limited when it exceeds 24 bits, so it is more desirable to adjust as needed near 16 bits. When the bit width is an integer multiple of 8 (i.e., byte alignment), the quantization from floating point numbers to integers is more intuitive, but when the bit width is not an integer multiple of 8 (e.g., 12 and 20), it is necessary to further fill it with bit operations during data encoding and decoding. Bit operations have a lower execution overhead in computers, so that arbitrary bit width quantization can be supported.
[0104] (2) Hybrid compression method based on non-uniform quantization;
[0105] From the uniform quantization formula, we can see that the effect of quantization is very dependent on the maximum and minimum values in the input vector. When the bit width is fixed at w, the larger the interval between the maximum and minimum values, the lower the precision loss caused by quantization. Figure 4 The data distribution of the communication floating-point vector in a certain iteration of the iterative solution of two linear equations is shown. It can be observed that the values of the elements in the vector are very unevenly distributed, with only a few elements far from 0, while the vast majority of elements are near 0. It is precisely because of the very few elements far from 0 that the maximum and minimum values are determined.
[0106] The present invention adopts a quantization method based on piecewise linear function to introduce a segmentation point in the quantization range, and divides the quantization interval into two non-overlapping areas: a dense central area and a sparse edge area. The quantization technology based on piecewise linear function uniformly quantizes the data distributed in the two quantization intervals respectively. By controlling the segmentation point, the quantization loss of the discrete edge area and the central dense area can be regulated. In artificial intelligence applications, segmentation points that increase the representation accuracy of the central dense area are usually selected. In the problem discussed in the present invention, the values of the central dense area and the discrete edge area are equally important, and a higher precision loss on either side will affect the final convergence of the iteration. To this end, the present invention proposes two improvements:
[0107] 1) Data distribution adaptive segmentation point setting. In the linear equation system with different sparse matrices as coefficient matrices, the numerical characteristics of the communication vector are different, so it is necessary to use different segmentation point settings. The present invention introduces a parameter p to represent the proportion of elements contained in the central dense area, and the segmentation of the central dense area and the edge discrete area is achieved by adjusting p.
[0108] 2) Hybrid compression of quantization mapping and floating-point representation. The edge discrete area contains fewer elements, so this part of the data can be unquantized and compressed, and still represented by the original FP64 or FP32. If the proportion of elements contained in the central dense area is α, then the proportion of elements contained in the edge discrete area is 1-α. Assuming that a 64-bit integer is used to represent the edge discrete area, uniform quantization is used to compress the central dense area, and the quantization bit width is w, the communication compression ratio η achieved by quantization is:
[0109]
[0110] When α=0.9 and w=16, hybrid storage can still achieve a compression ratio of about 3.08 times. However, it should be noted that as α decreases, the precision loss caused by quantization becomes smaller, but the compression ratio will also decrease. Therefore, a trade-off needs to be made according to the needs.
[0111] (3) Internal and external iterative mixed compression;
[0112] Quantization optimization is essentially a lossy compression technology for data, so it will have a certain impact on the convergence of the distributed iterative algorithm. By observing Algorithm 2, it can be found that the GMRES algorithm contains two layers of iteration, inner and outer, and both involve SpMV operations (lines 1 and 4), so both need to obtain the latest multiplication vector through inter-process communication. The communication of the v vector in the inner iteration is mainly used to construct the Krylov subspace, while the SpMV calculation of the outer iteration is used to obtain the residual caused by the current numerical solution, and its multiplication vector is the approximate solution obtained by the current iteration. Using a quantization method to compress the approximate solution will cause a significant loss of precision. Therefore, this embodiment does not use quantization optimization for the communication vector in the outer iteration, and only uses quantization compression for the communication vector of the inner iteration. In fact, the number of executions of the inner iteration is much higher than that of the outer iteration, that is, the communication proportion of the inner iteration is significantly higher than that of the outer iteration. Therefore, this internal and external mixed optimization method can significantly reduce communication overhead without causing a high loss of precision.
[0113] The floating-point format defined by the existing IEEE754 standard adopts the same representation rules for all floating-point numbers, which can meet general computing needs. In practical applications, the data of each application usually has unique numerical distribution characteristics. However, the existing floating-point representation of the IEEE754-like format is difficult to match the application numerical distribution characteristics, resulting in significant waste of bit width. The communication compression method based on quantization technology proposed in the present invention maps a given floating-point number set to a fixed-point number interval in a specific range according to the numerical characteristics of limited floating-point data. Compared with the floating-point format representation divided into exponents and mantissas, the quantized representation can improve the utilization efficiency of limited bit width, thereby improving the representation accuracy of floating-point numbers. More importantly, the compressed data based on quantization technology is only used for communication, not for calculation, and therefore supports compression of arbitrary bit widths.
[0114] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A communication data compression method based on quantization technology, It is characterized in that include: Distributed iteration is performed based on quantization and dequantization strategies. When each process obtains its own local calculation result, the transmission vector is quantized and compressed based on the numerical characteristics of the local calculation result. Each process sends the quantized result to other processes through the network, and each process obtains the complete quantized result; Based on the complete quantization result, each process dequantizes the data it receives to obtain a vector represented by the original precision, and then performs local SpMV calculations and subsequent operations on the vector represented by the original precision.
2. The communication data compression method based on quantization technology according to claim 1, It is characterized in that The process of distributed iteration based on quantization and anti-quantization strategies includes: In each iteration, each process first transmits the high-precision representation of the data to be transmitted to other processes. Quantize it into a fixed-point representation of w bits through the quantization function FP2INTw, then send it to other processes, and receive the quantized data from other processes to obtain the complete quantization result v w ; Then call INTw2FP to convert the complete quantization result v w Dequantize to high-precision floating point representation v out , and then continue with subsequent SpMV and other operations.
3. The communication data compression method based on quantization technology according to claim 1, It is characterized in that The process of quantizing and compressing the transmission vector based on the numerical characteristics of the local calculation result includes: The communication compression method based on uniform quantization maps all floating-point numbers in the interval [a, b] to an integer interval with a fixed step size. Given a floating-point vector x in And an unsigned integer grid, quantization operation is performed to map the real vector to a low-bit-width integer grid to achieve data compression.
4. The communication data compression method based on quantization technology according to claim 3, It is characterized in that The quantization process is expressed as: in, Indicates that the input data is rounded to the nearest integer. The definition of the clamp function is:
5. The communication data compression method based on quantization technology according to claim 3, It is characterized in that The communication compression method based on uniform quantization has different zero point selections. Uniform quantization includes symmetric quantization and asymmetric quantization. The asymmetric quantization parameter composition includes a scaling factor S, a zero point z, and a bit width w; wherein the scaling factor S is a floating point number and is used to determine the step size of the quantizer.
6. The communication data compression method based on quantization technology according to claim 1, It is characterized in that The process of quantizing and compressing the transmission vector based on the numerical characteristics of the local calculation result includes: The communication compression method based on non-uniform quantization introduces a segmentation point in the quantization range by using the quantization technology based on piecewise linear function, splits the quantization interval into two non-overlapping areas, and obtains the central dense area and the edge discrete area; The quantization technology based on the piecewise linear function uniformly quantizes the data distributed in the two quantization intervals respectively, and the quantization loss of the edge discrete area and the central dense area is regulated by controlling the segmentation point.
7. The communication data compression method based on quantization technology according to claim 6, It is characterized in that The process of introducing a segmentation point in the quantization range by using the quantization technology based on piecewise linear function and splitting the quantization interval into two non-overlapping areas includes: The parameter p is introduced to represent the proportion of elements contained in the central dense area. By adjusting the parameter p, the central dense area and the edge discrete area are segmented, and the segmentation point setting for data distribution adaptation is completed.
8. The communication data compression method based on quantization technology according to claim 6, It is characterized in that The process of uniformly quantizing the data distributed in two quantization intervals based on the piecewise linear function quantization technology includes: Through the hybrid compression of quantization mapping and floating point representation, the elements in the edge discrete area are not quantized and compressed, and the original FP64 or FP32 representation is still used; Define the proportion of elements contained in the central dense area as α, then the proportion of elements contained in the edge discrete area is 1-α, assuming that a 64-bit integer is used to represent the edge discrete area, and the central dense area is compressed by uniform quantization, and the quantization bit width is w, then the communication compression ratio η achieved by quantization is:
9. The communication data compression method based on quantization technology according to claim 1, It is characterized in that Based on the complete quantization result, the process of each process dequantizing the data it receives includes: After the compressed data is communicated between processes, each process dequantizes the received data, and the floating-point vector obtained by dequantization is approximately equal to the initial floating-point vector; The dequantization process is expressed as follows: x out =s·(x w -z) The quantization factor s is determined by the quantization range [a, b] and the quantization bit width w as follows:
10. A communication data compression system based on quantization technology, It is characterized in that include: A quantization module is used to quantize and compress the transmission vector based on the numerical characteristics of the local calculation results after each process obtains its own local calculation results; each process sends the quantized results to other processes through the network, and each process obtains a complete quantization result; The dequantization module is connected to the quantization module and is used for each process to dequantize the data it receives based on the complete quantization result to obtain a vector represented by the original precision, and then perform local SpMV calculations and subsequent operations on the vector represented by the original precision.
Citation Information
Patent Citations
Communication optimization method and system for distributed deep learning
CN109951438A
Adaptive sparse parameter model design and quantitative transmission method for distributed machine learning
CN113660113A
Convolutional neural network operation method and related equipment
CN114418057A
Model training method
CN114418088A
Method, system and device for optimizing federated learning compression communication
CN114861790A
Cited By
Large model training video memory optimization method and system based on low-precision integer storage
CN121351902A
Memory optimization method and system for large model training based on low-precision integer storage
CN121351902B
