PC quantitative reasoning convolution avx acceleration method
By optimizing convolutional calculations on the PC -end AVX instruction set, the problem of slow convolution calculation at ordinary PC configuration is solved, and a faster model reasoning effect is achieved.
Patent Information
- Application Number
- CN202310892509.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-20
- Publication Date
- 2025-01-21
AI Technical Summary
In ordinary PC configuration, convolutional calculations in deep learning are slower due to the large amount of calculation, and cannot effectively utilize the advantages of high-configuration hardware and software.
By analyzing the convolution calculation method, combined with the AVX instruction set, optimize the implementation of convolution calculations, improve the calculation speed, and perform model reasoning calculation on the PC side to achieve convolution acceleration.
It significantly improves the speed of convolution calculation, reduces the time of the overall network model reasoning, and makes model reasoning on PCs faster and efficient.
Smart Images
Figure CN119337939A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of data processing, and in particular relates to a method for accelerating PC quantization inference convolution AVX. Background Art
[0002] In the prior art, with the development of deep learning, the demand for hardware and software is getting higher and higher. In order to improve the efficiency and speed of calculation, better hardware and software configuration is often required. In a large part of cases, it may be necessary to verify the rationality and correctness of the network result reasoning first. Under this premise, good hardware and software configuration is not required, and only estimated calculations are allowed under the ordinary configuration of personal computers. However, the convolution calculations in deep learning often have a large amount of calculations, which are time-consuming and slow to calculate in ordinary CPUs.
[0003] The reasoning of deep network models uses a large number of operators, especially a large number of convolutions to obtain data features. Under PC reasoning, ordinary convolution is just a step-by-step accumulation of feature maps and weights. Due to the large amount of calculation, the speed is relatively slow on ordinary computers without GPU. The avx instruction is used in the implementation of convolution calculation, which can speed up the calculation of convolution and perform convolution calculations on multiple data streams in parallel.
[0004] Therefore, the main defect of the existing technology is that the quantized model inference convolution on the PC does not really use high-configuration hardware and software. A large number of multiplication and addition calculations are required during the inference process. A large number of convolution multiplication and addition calculations consume a lot of time and the calculation speed is very slow.
[0005] In addition, common terms in the prior art include:
[0006] PC: A personal computer is a general-purpose computer of a size, price, and performance suitable for personal use. Examples include desktops, laptops, subnotebooks, tablets, and ultrabooks.
[0007] Quantization: To speed up the inference of the algorithm model, the features in the middle layer of the network are fixed to 8, 4, or 2 bits, which can significantly reduce the memory usage of the network.
[0008] Convolution: A method of computing features in deep learning.
[0009] AVX: A set of instruction sets for the CPU that can be fully utilized in high-performance workstations involving large-scale mathematically complex and heavy applications; improves the processing speed of the CPU.
[0010] Protobuf serialization: is the process of converting a data structure or object into a binary byte stream. Protobuf uses different encoding methods and data storage methods for different field types to serialize message fields to ensure efficient and compact data compression;
[0011] The Protobuf serialization process is as follows:
[0012] (1) Determine whether each field has a set value, and encode it only if it has a value;
[0013] (2) Encode the field value using different encoding methods according to the field identification number and data type;
[0014] (3) Encapsulate the encoded data blocks into binary data streams using different data storage methods according to the field types.
[0015] Protobuf deserialization: It is the process of converting the binary byte stream generated during the serialization process into a data structure or object.
[0016] The Protobuf deserialization process is as follows:
[0017] (1) Call the parse From(input) method of the message class to parse the binary byte data stream read from the input stream;
[0018] (2) Read the parsed data into the corresponding structure type of C++, Java, and Phyton according to the specified format. Summary of the invention
[0019] In order to solve the above problems, the purpose of this application is: this method analyzes the calculation method of convolution, based on the existing ordinary calculation of convolution, the convolution implementation is completed using the avx instruction set, thereby improving the calculation speed of convolution and the reasoning speed of the model.
[0020] Specifically, the present invention provides a method for accelerating PC quantization inference convolution avx, the method comprising the following steps:
[0021] S1, parsing the model, obtaining the network name for convolution; that is, obtaining the network convolution node for convolution calculation, the convolution calculation is to extract input data and key features according to the node parameter information; the parsing model obtains the convolution node, that is, the convolution layer, from the model, identifies the layer for convolution calculation; the convolution calculation is to perform specific arithmetic operations on the input data according to the weight, step size, and pad parameters in the convolution layer;
[0022] S2, Protobuf serialization, deserialization and parsing of network parameters, deserialization of convolution nodes to obtain the parameters of the node, save the convolution parameters, and save the parsed original data in the set parameters respectively without changing the data content;
[0023] S3, convolution output shape reasoning, is to reason about the shape size of the output of the convolution layer, i.e., batch, width, height, and output channel;
[0024] 1). If padding is same, use the input shape size divided by the strides to get the output shape: the output width is equal to the input width divided by the strides, and the output height is equal to the input height divided by strides; the output shape is: input first dimension, height, width, input last dimension;
[0025] 2). If padding is value, use the obtained parameter information, input shape, convolution kernel size, pad step length strides, and dilation value dilations to infer the shape and calculate the shape characteristics of the convolution output; specific calculation: input width and height plus pad, minus convolution kernel size, multiplied by dilations, divided by strides;
[0026] S4, convolution interface, passes in convolution parameters: input data, convolution kernel data, bias data, malloc to open up space, set the number of threads, pthread_t to open up thread space, use for loop to assign the same convolution calculation data to each thread, that is, a loop thread; use loop batch to output width, height and output channel, and obtain input and weight values;
[0027] S5, use For loop to loop all data and obtain data for calculation;
[0028] Store the input data into the vector of avx's _mm256_set1_ps.
[0029] Use _mm256_loadu_ps memory address to load floating point output vector and weight, use _mm256_mul_ps to multiply input and weight,
[0030] Use _mm256_add_ps to add the weight and input two floating-point vectors to the output vector, and use _mm256_storeu_ps to store the calculated value in the output.
[0031] Among them, the instructions used for avx calculations include:
[0032] _mm256_set1_ps fills the vector with float type values,
[0033] _mm256_loadu_ps loads the output vector and convolution kernel vector from the memory address,
[0034] _mm256_mul_ps multiplies the input value by the value of the convolution kernel.
[0035] _mm256_add_ps performs data accumulation operations.
[0036] _mm256_storeu_ps stores the overall data of the addition;
[0037] S6, end the convolution.
[0038] The training quantization models include yolov5 series models, resnet series models, and mobilinet series models containing convolution.
[0039] The parameters in step S2 include: input shape, convolution kernel shape, padding supplement width and height, padding filling mode, quantization parameter scale, THRESHOD, shift parameter mutify_factor, activation parameter activate_type, input bit, weight bit, output bit, clip maximum value, clip minimum value, strides, supplement value pad_value, group convolution group value, and dilations expansion size parameter.
[0040] The step S4 further comprises:
[0041] S4.1, for convolution calculation, set the number of multi-threaded parallel calculations according to the shape of the output. The specific number of threads is set according to the output width multiplied by the height minus one divided by the number of convolution threads. The number of convolution threads is set to 8. If it exceeds 16, the overall thread setting is 16; get pthread_t from the pthread.h file to open up thread space, use the pthread_create function to create a thread, run the thread function after creating the thread, and use the pthread_join() function interface to loop and control the execution of the thread;
[0042] S4.2, copy the parameters required for convolution to the same number of threads, copy the same number of parameter data according to the number of threads, each thread can obtain complete data, use the malloc function to open up the space of the corresponding thread, and each thread stores the same data parameters, including input, output, pad_value, mean variance, quantization parameter scale, activation information, step size information, pads information, and weight information;
[0043] S4.3, since the weight is uint8 type data, the input weight value needs to be preprocessed to meet the quantization calculation, and the weight data is converted from uint8 data to float data and saved in the opened space for convolution calculation. The code saves the uint8 data type data into the float space pointer.
[0044] The step S5 further comprises:
[0045] Each thread intercepts a specific input value and weight value for corresponding multiplication and addition, that is, according to the width, height and number of threads, a for loop is used to obtain the corresponding amount of input data and parameters. The "corresponding amount" means obtaining the size of the input data according to the shape and size of the weight. By setting the environment variable AVX_OPTION_OPEN, it is expressed as: option(AVX_OPTION_OPEN"option forAVX"ON) is not set to perform ordinary multiplication and addition calculations. After setting, avx's multiplication and addition calculations are enabled.
[0046] The method is suitable for model inference calculation on the PC side, and convolution is implemented in combination with the avx operation instruction set.
[0047] Therefore, the advantage of this application is that adding the avx instruction set to the convolution implementation can accelerate the calculation speed of the convolution and reduce the time of the overall network model reasoning. That is, using the avx instruction set to implement convolution, improve the calculation speed of the model reasoning convolution, effectively and quickly verify the reasoning model implementation on the PC, and reduce the reasoning time. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0049] Figure 1 It is a flow chart of the present application method.
[0050] Figure 2 It is a code diagram of the for loop in the present application method. DETAILED DESCRIPTION
[0051] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0052] In order to improve the calculation speed of convolution calculation, this application uses the avx instruction set for calculation acceleration. The avx instruction set is added to the network reasoning convolution implementation on the PC to increase the reasoning and running speed of the network.
[0053] Specifically, an embodiment of the present application relates to a method for accelerating PC quantized reasoning convolution AVX. This method performs model reasoning calculations on the PC side, and the convolution is implemented in combination with the AVX operation instruction set to improve the convolution calculation speed and the reasoning speed of the overall model.
[0054] First, the convolution calculation is as follows:
[0055] For an input feature map, the value of the feature map and the value of the convolution kernel are multiplied and added to obtain the inferred convolution calculation result value; according to the width and height of the convolution kernel corresponding to the width and height of the feature map, from left to right and from top to bottom, the stride of the mobile convolution kernel is controlled according to the parameter strides, and the value in the convolution kernel is used to correspond to the value in the feature map to perform data multiplication and addition, and the calculation result data is obtained as output;
[0056] In the training model, such as the commonly used yolov5 series models, resnet series models, mobilenet series and other models containing convolution, some input parameters are obtained, including the input shape, the shape of the convolution kernel, the padding size and the strides step size. If padding is the same mode, the input shape size is divided by the strides to obtain the output shape. If padding is the value mode, the input shape plus the pad value minus the convolution kernel size is divided by the strides to perform shape reasoning and calculate the shape characteristics of the convolution output, obtain the input shape and data value, weight shape and data value, and the ordinary convolution directly obtains the required parameters. After a cyclic calculation, the corresponding input value and weight value are obtained, and the multiplication and addition operations are directly performed to obtain the corresponding reasoning result value;
[0057] Convolution calculation:
[0058] First, obtain the parameter information set in the model structure, such as the data value and shape size of the input feature map, the data value and shape size of the convolution kernel, the value of padding, and the value of pad to add corresponding rows and columns to the feature map up, down, left, and right. The value of padvalue determines the size of the pad value, and the value of strides controls the length of the convolution kernel's movement at one time.
[0059] Secondly, we perform inference on the output shape.
[0060] Finally, the data is calculated. The calculation is in the form of cumulative multiplication and addition. The value of the convolution kernel is multiplied by the value of the corresponding feature map and then added. The convolution kernel is moved from left to right and from top to bottom according to the strides value and then multiplied and added with the value of the corresponding feature map. The calculation ends when it moves to the last position of the feature map, and the value calculated at each step is saved as the output data.
[0061] In this method, in order to improve the calculation speed of convolution, the convolution operation is implemented using the avx instruction set. The quantized avx convolution calculation is as follows:
[0062] Among them, the instructions used are _mm256_set1_ps to fill the vector with float type values, _mm256_loadu_ps to load the output vector and convolution kernel vector from the memory address, _mm256_mul_ps to multiply the input value by the value of the convolution kernel, _mm256_add_ps to perform data accumulation operations, and _mm256_storeu_ps to store the overall data of the addition.
[0063] For convolution calculation, the number of multi-threaded parallel calculations will be set according to the shape of the output. The total number of threads is based on the output width multiplied by the height minus one divided by the number of convolution threads. The number of convolution threads is set to 8. If it exceeds 16, the total number of threads is set to 16. Get pthread_t from the pthread.h file to open up thread space, use the pthread_create function to create a thread, and run the thread function after creating the thread. pthread_ join() Function, which waits for the thread specified by thread to end in a blocking manner.
[0064] Copy the parameters required for convolution to the same number of threads, malloc to open up space for the corresponding threads, and store the same data parameters in each thread, including input, output, pad_value, mean variance, quantization parameter scale, activation information, step size information, pads information, and weight information. Since the weight is uint8 data, the input weight value needs to be preprocessed to meet the quantization calculation, and the weight data is converted from uint8 data to float data and saved in the opened space for convolution calculation.
[0065] Each thread intercepts a specific input value and weight value, multiplying and adding them accordingly. According to the width, height and number of threads, the for loop obtains the corresponding amount of input data and parameters. By setting the environment variable AVX_OPTION_OPEN, ordinary multiplication and addition calculations are not set. After setting, avx multiplication and addition calculations are enabled. The following instructions are used for avx calculations
[0066] Store the input data into the vector of avx's _mm256_set1_ps.
[0067] _mm256_loadu_ps loads floating point output vectors and weights from memory addresses,
[0068] _mm256_mul_ps multiplies the input and weight,
[0069] _mm256_add_ps adds the weight and input two floating-point vectors to the output vector, and _mm256_storeu_ps stores the calculated value in the output.
[0070] In summary, if Figure 1 As shown, the specific process of this method is as follows:
[0071] S1, parsing the model, obtaining the network name for convolution; that is, obtaining the network convolution node for convolution calculation, the convolution calculation is to extract input data and key features according to the node parameter information; the parsing model obtains the convolution node, that is, the convolution layer, from the model, identifies the layer for convolution calculation; the convolution calculation is to perform specific arithmetic operations on the input data according to the weight, step size, and pad parameters in the convolution layer;
[0072] S2, Protobuf serialization, deserialization and parsing of network parameters, deserialization of convolution nodes to obtain the parameters of the node, save the convolution parameters, and save the parsed original data in the set parameters respectively without changing the data content;
[0073] The parameters in step S2 include: input shape, convolution kernel shape, padding supplement width and height, padding filling mode, quantization parameter scale, THRESHOD, shift parameter mutify_factor, activation parameter activate_type, input bit, weight bit, output bit, clip maximum value, clip minimum value, strides, supplement value pad_value, group convolution group value, and dilations expansion size parameter.
[0074] S3, convolution output shape reasoning, is to reason about the shape size of the output of the convolution layer, i.e., batch, width, height, and output channel;
[0075] 1). If padding is same, use the input shape size divided by the strides to get the output shape: the output width is equal to the input width divided by the strides, and the output height is equal to the input height divided by strides; the output shape is: input first dimension, height, width, input last dimension;
[0076] 2). If padding is value, use the obtained parameter information, input shape, convolution kernel size, pad step length strides, and dilation value dilations to infer the shape and calculate the shape characteristics of the convolution output; specific calculation: input width and height plus pad, minus convolution kernel size, multiplied by dilations, divided by strides;
[0077] S4, convolution interface, passes in convolution parameters: input data, convolution kernel data, bias data, malloc to open space, set the number of threads, pthread_t to open thread space, use for loop to assign the same convolution calculation data to each thread, that is, one is a loop thread; use loop batch to output width, height and output channel, and obtain input and weight values;
[0078] The step S4 further comprises:
[0079] S4.1, for convolution calculation, set the number of multi-threaded parallel calculations according to the shape of the output. The specific number of threads is set according to the output width multiplied by the height minus one divided by the number of convolution threads. The number of convolution threads is set to 8. If it exceeds 16, the overall thread setting is 16; get pthread_t from the pthread.h file to open up thread space, use the pthread_create function to create a thread, run the thread function after creating the thread, and use the pthread_join() function interface to loop and control the execution of the thread;
[0080] S4.2, copy the parameters required for convolution to the same number of threads, copy the same number of parameter data according to the number of threads, each thread can obtain complete data, use the malloc function to open up the space of the corresponding thread, and each thread stores the same data parameters, including input, output, pad_value, mean variance, quantization parameter scale, activation information, step information, pads information, and weight information. These parameters are the data analyzed above;
[0081] S4.3, since the weight is uint8 type data, the input weight value needs to be preprocessed to meet the quantization calculation, and the weight data is converted from uint8 data to float data and saved in the opened space for convolution calculation. The code saves the uint8 data type data into the float space pointer.
[0082] S5, use For loop to loop all data and get data calculation, such as Figure 2 As shown, the input data is stored in the vector of avx's _mm256_set1_ps.
[0083] Use _mm256_loadu_ps memory address to load floating point output vector and weight, use _mm256_mul_ps to multiply input and weight,
[0084] Use _mm256_add_ps to add the weight and input two floating-point vectors to the output vector, and use _mm256_storeu_ps to store the calculated value in the output.
[0085] Among them, the instructions used for avx calculations include:
[0086] _mm256_set1_ps fills the vector with float type values,
[0087] _mm256_loadu_ps loads the output vector and convolution kernel vector from the memory address,
[0088] _mm256_mul_ps multiplies the input value by the value of the convolution kernel.
[0089] _mm256_add_ps performs data accumulation operations.
[0090] _mm256_storeu_ps stores the overall data of the addition;
[0091] The step S5 further comprises:
[0092] Each thread intercepts a specific input value and weight value, and multiplies and adds them accordingly. That is, according to the width, height and number of threads, a for loop is used to obtain the corresponding amount of input data and parameters. The "corresponding amount" means obtaining the size of the input data according to the shape and size of the weight. By setting the environment variable AVX_OPTION_OPEN, it is expressed as: option(AVX_OPTION_OPEN"option forAVX"ON) is not set to perform ordinary multiplication and addition calculations. After setting, avx multiplication and addition calculations are enabled. S6, end the convolution.
[0093] The method is suitable for model inference calculation on the PC side, and convolution is implemented in combination with the avx operation instruction set.
[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for accelerating PC quantized inference convolution avx, characterized in that: The method comprises the following steps: S1, parse the model, obtain the network name for convolution; that is, obtain the network convolution node for convolution calculation. The convolution calculation is to extract input data and key features based on the node parameter information; The analytical model obtains a convolution node, i.e., a convolution layer, from the model, and identifies the layer for convolution calculation; the convolution calculation is to perform a specific arithmetic operation on the input data according to the weight, step size, and pad parameters in the convolution layer; S2, Protobuf serialization, deserialization and parsing of network parameters, deserialization of convolution nodes to obtain the parameters of the node, save the convolution parameters, and save the parsed original data in the set parameters respectively without changing the data content; S3, convolution output shape reasoning, is to reason about the shape size of the output of the convolution layer, i.e., batch, width, height, and output channel; 1). If padding is same, use the input shape size divided by the strides to get the output shape: the output width is equal to the input width divided by the strides, and the output height is equal to the input height divided by strides; the output shape is: input first dimension, height, width, input last dimension; 2). If padding is value, use the obtained parameter information, input shape, convolution kernel size, pad step length strides, and dilation value dilations to infer the shape and calculate the shape characteristics of the convolution output; specific calculation: input width and height plus pad, minus convolution kernel size, multiplied by dilations, divided by strides; S4, convolution interface, passes in convolution parameters: input data, convolution kernel data, bias data, malloc to open up space, set the number of threads, pthread_t to open up thread space, use for loop to assign the same convolution calculation data to each thread, that is, a loop thread; use loop batch to output width, height and output channel, and obtain input and weight values; S5, use For loop to loop all data and obtain data for calculation; Store the input data into the vector of avx's _mm256_set1_ps. Load the floating point output vector and weights using the _mm256_loadu_ps memory address, Use _mm256_mul_ps to multiply the input and weight, Use _mm256_add_ps to add the weight and input two floating-point vectors to the output vector, and use _mm256_storeu_ps to store the calculated value in the output. Among them, the instructions used for avx calculations include: _mm256_set1_ps fills the vector with float type values, _mm256_loadu_ps loads the output vector and convolution kernel vector from the memory address, _mm256_mul_ps multiplies the input value by the value of the convolution kernel. _mm256_add_ps performs data accumulation operations. _mm256_storeu_ps stores the overall data of the addition; S6, end the convolution.
2. According to claim 1, a method for accelerating PC quantization inference convolution avx, characterized in that: The training quantization models include yolov5 series models, resnet series models, and mobilinet series models containing convolution.
3. According to claim 1, a method for accelerating PC quantization inference convolution avx, characterized in that: The parameters in step S2 include: input shape, convolution kernel shape, padding supplement width and height, padding filling mode, quantization parameter scale, THRESHOD, shift parameter mutify_factor, activation parameter activate_type, input bit, weight bit, output bit, clip maximum value, clip minimum value, strides, supplement value pad_value, group convolution group value, and dilations expansion size parameter.
4. A method for accelerating PC quantization inference convolution avx according to claim 1, characterized in that: The step S4 further comprises: S4.1, for convolution calculation, set the number of multi-threaded parallel calculations according to the shape of the output. The specific number of threads is set according to the output width multiplied by the height minus one divided by the number of convolution threads. The number of convolution threads is set to 8. If it exceeds 16, the overall thread setting is 16; get pthread_t from the pthread.h file to open up thread space, use the pthread_create function to create a thread, run the thread function after creating the thread, and use the pthread_join() function interface to loop and control the execution of the thread; S4.2, copy the parameters required for convolution to the same number of threads, copy the same number of parameter data according to the number of threads, each thread can obtain complete data, use the malloc function to open up the space of the corresponding thread, and each thread stores the same data parameters, including input, output, pad_value, mean variance, quantization parameter scale, activation information, step size information, pads information, and weight information; S4.3, since the weight is uint8 type data, the input weight value needs to be preprocessed to meet the quantization calculation, and the weight data is converted from uint8 data to float data and saved in the opened space for convolution calculation. The code saves the uint8 data type data into the float space pointer.
5. A method for accelerating PC quantization inference convolution avx according to claim 1, characterized in that: The step S5 further comprises: Each thread intercepts a specific input value and weight value for corresponding multiplication and addition, that is, according to the width, height and number of threads, a for loop is used to obtain the corresponding amount of input data and parameters. The "corresponding amount" means obtaining the size of the input data according to the shape and size of the weight. By setting the environment variable AVX_OPTION_OPEN, it is expressed as: option(AVX_OPTION_OPEN"option forAVX"ON) is not set to perform ordinary multiplication and addition calculations. After setting, avx's multiplication and addition calculations are enabled.
6. A method for accelerating PC quantization inference convolution avx according to claim 1, characterized in that: The method is suitable for model inference calculation on the PC side, and convolution is implemented in combination with the avx operation instruction set.