Implementation method of deep learning preprocessing and deep learning network
By adjusting the parameters of the first convolutional layer of a deep learning network, a preprocessing function is implemented, which solves the problems of latency and bandwidth impact in deep learning preprocessing, simplifies the model inference process, and improves efficiency and performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- RONG MING MICROELECTRONICS (JINAN) CO LTD
- Filing Date
- 2022-07-20
- Publication Date
- 2026-06-09
AI Technical Summary
In the preprocessing of deep learning, traditional methods cause latency and bandwidth issues, especially in high-throughput, low-latency, and low-power application scenarios. The CPU performance of hardware accelerators is insufficient to meet the requirements of large-scale data preprocessing, and multiple data exchanges increase latency and DDR performance bottlenecks.
By adjusting the parameters of the first convolutional layer in a deep learning network to make its output consistent with the result after preprocessing, the preprocessing function is achieved, simplifying the model inference process and improving inference efficiency.
Without altering the model's accuracy and structure, the preprocessing steps are reduced, multiple data transfers between the host and hardware accelerators are avoided, DDR bandwidth is saved, and inference performance and efficiency are improved.
Smart Images
Figure CN115294354B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a method for implementing deep learning preprocessing and a deep learning network. Background Technology
[0002] With the continuous development of deep learning in the fields of sound, image, and video, more and more application scenarios are being discovered.
[0003] For such applications, using deep learning for intelligent analysis and optimization is becoming the industry standard. At the same time, a large number of GPU-based or other customized hardware accelerators are emerging, providing low-power, high-performance hardware acceleration for these applications.
[0004] Modern deep learning is often based on deep convolutional neural networks. To train accurate models, the industry commonly uses model training architectures such as TensorFlow and PyTorch. Depending on the architecture, some image models support RGB input format, while others support BGR format. To ensure convergence during training, the input images need to be preprocessed before training, usually through some form of normalization. Because of these preprocessing steps during model training, the same preprocessing must be used during model inference to ensure model accuracy.
[0005] In traditional GPU+CPU-based inference architectures, the input image is first loaded into the host's memory and then preprocessed, including normalization and RGB color channel swapping. It is then loaded into the GPU's video memory, where the GPU performs inference on the preprocessed data based on the model's parameters, generates results, and then transfers them to the host's memory. This data preprocessing, performed on the host's CPU, has a certain impact on the host's workload.
[0006] For applications requiring high throughput, low latency, and low power consumption, customized hardware accelerators are often limited by chip size and power consumption, resulting in CPUs with poor performance that cannot meet the requirements of large-scale data preprocessing. Conversely, using a host-side CPU for preprocessing requires multiple data exchanges between the accelerator and the host, increasing latency and limiting performance due to DDR performance bottlenecks. Summary of the Invention
[0007] The technical problem to be solved by this invention is how to solve the problems of latency and bandwidth in deep learning preprocessing. This invention proposes an implementation method for deep learning preprocessing and a deep learning network.
[0008] The deep learning preprocessing implementation method according to embodiments of the present invention includes:
[0009] Based on deep learning preprocessing operations, the original parameters of the first convolutional layer in the deep learning network are adjusted so that the output of the first convolutional layer after parameter adjustment is the same as the output of the first convolutional layer after preprocessing and then passing through the original parameters.
[0010] The adjusted parameters are used as the running parameters for the first convolutional layer.
[0011] The deep learning preprocessing method according to embodiments of the present invention achieves the function of model inference preprocessing by changing the parameters of the first convolutional layer of deep learning, thereby simplifying model inference and improving inference efficiency. The process of modifying the model does not require retraining, modification of the model structure, or increase in computational load; its accuracy remains consistent with the original model, and it solves the latency and bandwidth performance issues in existing preprocessing techniques.
[0012] According to some embodiments of the present invention, the original parameters for adjusting the first convolutional layer include weights and offsets.
[0013] In some embodiments of the present invention, the preprocessing operations include: color channel conversion and / or normalization processing.
[0014] According to some embodiments of the present invention, when the preprocessing operation includes color channel switching, the adjustment of the original parameters includes swapping the weighted channels corresponding to the colors.
[0015] According to some embodiments of the present invention, when the preprocessing operation includes normalization, the adjustment of the original parameters includes modifying the weights of the first convolutional layer.
[0016] According to some embodiments of the present invention, when there is a padding operation before the first convolutional layer, the padding parameters are adjusted based on the average value.
[0017] According to an embodiment of the present invention, the deep learning network implements the preprocessing operation in the deep learning network using the deep learning preprocessing implementation method described above.
[0018] According to embodiments of the present invention, the deep learning network achieves model inference preprocessing by changing the parameters of the first convolutional layer, thereby simplifying model inference and improving inference efficiency. The process of modifying the model does not require retraining, modification of the model structure, or increased computation; its accuracy remains consistent with the original model, and it solves the latency and bandwidth performance issues inherent in existing preprocessing techniques.
[0019] According to some embodiments of the present invention, the deep learning network is a sound processing deep learning network, an image processing deep learning network, or a video processing deep learning network. Attached Figure Description
[0020] Figure 1 This is a flowchart illustrating the implementation method of deep learning preprocessing according to an embodiment of the present invention.
[0021] Figure 2 This is a flowchart illustrating the parameter adjustment process of the first convolutional layer in the deep learning preprocessing implementation method according to an embodiment of the present invention. Detailed Implementation
[0022] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.
[0023] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily to be strictly followed according to the step numbers; the execution order of the steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.
[0024] In existing technologies, using a host-side CPU requires transferring image data from the on-chip decoder to host memory, and then sending the preprocessed result back to the on-chip AI accelerator. This inevitably increases latency and significantly increases DDR bandwidth, severely impacting performance.
[0025] Using on-chip resources for preprocessing increases the load on the on-chip CPU (Vector processor), and due to power consumption and size considerations, its on-chip computing resources usually do not have much redundancy. It will also consume other on-chip bandwidth and other resources.
[0026] Of course, preprocessing can also be achieved by adding corresponding preprocessing nodes to the computation graph of an existing model. This changes the model architecture, increases the number of layers in the deep learning model, and also consumes more computational resources, increasing the computational load and impacting performance.
[0027] This invention discloses a data preprocessing method that modifies the parameters of the trained model, thereby omitting common preprocessing steps without affecting the model's accuracy. This avoids multiple data transfers between the host and the hardware accelerator, saves DDR bandwidth, and improves inference performance and efficiency.
[0028] To facilitate understanding of the present invention, the technical aspects involved are explained below:
[0029] Deep learning models generally require some degree of preprocessing during the inference process. This mainly includes:
[0030] Scale to the required size for the model. Most video / image hardware accelerators support image scaling.
[0031] Format conversion: Most video / image hardware accelerators support image format conversion.
[0032] Color: Converts the decompressed video image from YUV format to RGB format.
[0033] Data layout: There are usually two methods: [NCHW], corresponding to the RGB Planer format, and [NHWC], corresponding to the RGB packed format.
[0034] Color channel switching: The usual color channels are RGB or BGR.
[0035] Image normalization: Usually, the original RGB UINT8 image is normalized by subtracting the average value and then multiplying it by the normalization value (for example, if the average value is [0, 0, 0] and the normalization value is [1 / 255, 1 / 255, 1 / 255], the image is normalized to [0, 1]).
[0036] Scaling and format conversion typically require hardware support. Most chips involved in image and video encoding / decoding generally support these operations in hardware.
[0037] Color channel swapping and image normalization preprocessing are typically implemented in modern deep learning accelerators in the following ways:
[0038] This can be done using the host computer or an on-chip CPU.
[0039] This is done using a special on-chip accelerator, such as a vector calculator.
[0040] For the traditional CPU+GPU approach, the host-side CPU is used for preprocessing.
[0041] A convolutional layer transforms an input image to extract features. This process involves convolving the image with a kernel. A two-dimensional convolutional layer, as the name suggests, takes a two-dimensional input image and a two-dimensional kernel array, and convolves them through a cross-correlation operation to output a two-dimensional array.
[0042] The kernel is a small matrix whose height and width are smaller than the image to be convolved. The kernel slides along the height and width of the input image, and the kernel's dot product and the image are computed at each spatial location. The most important parameters are the weights and the offsets.
[0043] When defining convolutional layers, commonly used parameters also include stride, which is the size of the sliding displacement during each convolution process. Since the output size may also need to be controlled during convolution, certain values need to be padded at the edges of the input to ensure the output meets the target requirements. Common padding modes include: none, constant, symmetric, and mirror.
[0044] Based on the above explanation of the technology involved in this invention, the solution of this invention is described below:
[0045] like Figure 1 As shown, the deep learning preprocessing implementation method according to an embodiment of the present invention includes:
[0046] S100, based on deep learning preprocessing operations, adjusts the original parameters of the first convolutional layer in the deep learning network so that the output of the first convolutional layer after parameter adjustment is the same as the output of the first convolutional layer after preprocessing and then passing through the original parameters.
[0047] S200 uses the adjusted parameters as the running parameters for the first convolutional layer.
[0048] The deep learning preprocessing method according to embodiments of the present invention achieves the function of model inference preprocessing by changing the parameters of the first convolutional layer of deep learning, thereby simplifying model inference and improving inference efficiency. The process of modifying the model does not require retraining, modification of the model structure, or increase in computational load; its accuracy remains consistent with the original model.
[0049] According to some embodiments of the present invention, the original parameters for adjusting the first convolutional layer include weights and offsets. That is, the present invention can adjust the weights and offsets of the first convolutional layer to ensure that the output calculated by the first convolutional layer meets the objectives of the preprocessing operation.
[0050] In some embodiments of the present invention, the preprocessing operation includes color channel conversion and / or normalization. That is, the preprocessing operation can be color channel conversion or normalization, and of course, the preprocessing operation can also include both color channel conversion and normalization.
[0051] According to some embodiments of the present invention, such as Figure 2As shown, when the preprocessing operation includes color channel switching, the adjustment of the original parameters includes swapping the weight channels corresponding to the colors. In other words, this invention can achieve the preprocessing objective of color channel swapping in the preprocessing operation by adjusting the weights of the first convolutional layer.
[0052] According to some embodiments of the present invention, such as Figure 2 As shown, when the preprocessing operation includes normalization, the adjustment of the original parameters includes modifying the weights of the first convolutional layer. In other words, this invention can achieve the normalization purpose in the preprocessing operation by modifying the weights.
[0053] According to some embodiments of the present invention, when there is a padding operation before the first convolutional layer, the padding parameters are adjusted based on the average value.
[0054] According to the embodiments of the present invention, the deep learning network implements the preprocessing operations in the deep learning network using the above-described deep learning preprocessing implementation method.
[0055] According to embodiments of the present invention, the deep learning network achieves model inference preprocessing by changing the parameters of the first convolutional layer, thereby simplifying model inference and improving inference efficiency. The process of modifying the model does not require retraining, modification of the model structure, or increased computation; its accuracy remains consistent with the original model.
[0056] According to some embodiments of the present invention, the deep learning network is a sound processing deep learning network, an image processing deep learning network, or a video processing deep learning network.
[0057] The implementation method of deep learning preprocessing and the deep learning network according to the present invention are described in detail below with reference to the accompanying drawings. It is to be understood that the following description is merely exemplary and should not be construed as a specific limitation of the present invention.
[0058] like Figure 1 and Figure 2 As shown, this invention achieves preprocessing by changing the parameters (W1, B1) of the first convolutional layer (L1) of a deep learning model. Deep learning models typically contain multiple convolutional layers. Their parameters include weights (W) and offsets (B), and they are among the most important layers in a deep learning model.
[0059] Let A be the original model input before preprocessing. A' = P(A, c, m, s) is the input after preprocessing, where c is the color channel swap, m is the normalized mean vector, and s is the normalized parameter vector.
[0060] Typically, the first layer of a model is a convolutional layer (L1). The presence of other linear layers or transition layers will not affect the results.
[0061] R = L1(A'), meaning R is the result of the L1 layer for the normalized input.
[0062] This invention generates new parameters L1' = (W1', B1') by modifying the parameters (W1, B1) of L1. This satisfies:
[0063] L1'(A)=R.
[0064] Since the output of the first convolutional layer after modification is the same as the original output, it has no effect on the other layers. Therefore, the entire model is completely consistent with the original model after modification, without any loss of accuracy.
[0065] Since only the L1 parameters were modified and the model structure was not changed, there was no impact on the model's performance.
[0066] The resulting new model can be quantized and operated on in the same way as the original model, thus enabling it to work normally on different deep learning accelerators.
[0067] When working in conjunction with a hardware decoding module, the results of its hardware decoding can be directly used as input to a deep learning accelerator after standard color and size conversion, without the need for additional preprocessing, thus reducing the consumption of computing and bandwidth resources.
[0068] It is worth emphasizing that:
[0069] This invention achieves the function of model inference preprocessing by changing the parameters of the first convolutional layer in deep learning, thereby simplifying model inference and improving inference efficiency.
[0070] This invention achieves the purpose of exchanging color channels during the preprocessing process by swapping the weight channels corresponding to the colors.
[0071] This paper modifies the model weights based on the normalization parameters, enabling the use of normalization parameters during the preprocessing stage.
[0072] This invention achieves the function of subtracting the average value during the preprocessing process by modifying the offset.
[0073] The method disclosed in this invention is applicable to operations on various convolutional layers, including padding. When using a constant C as padding, the defined constant C needs to be modified to C plus the average value (C+M).
[0074] The process of modifying the model does not require retraining or changing the model structure, and its accuracy remains consistent with the original model.
[0075] The solution of the present invention is described below with reference to a specific example:
[0076] Suppose we need to perform inference on an image classification model K, the required preprocessing is as follows:
[0077] To switch the color channel to BGR, a color channel swap is required.
[0078] The average value of its color channels is M = [m1, m2, m3].
[0079] Its normalized parameter S = [s1, s2, s3]
[0080] The original input image format is RGB, and its size is [H,W,3], where 3 corresponds to the three channels of RGB.
[0081] 1. First, find the first convolutional layer C1 of model K, and set its parameters as follows:
[0082] kernel size: 4x4;
[0083] Input channel number: 3 (corresponding to BGR color channels);
[0084] Number of output channels: 2;
[0085] Step size: 1;
[0086] Fill mode: None;
[0087] The dimensions of the weights (W) of the first convolutional layer C1 are [4,4,3,2].
[0088] The dimension of the extracted offset (B) corresponds to the output dimension, which is [2].
[0089] 2. Because color channel switching is required, RGB needs to be converted to BGR during the preprocessing process.
[0090] The specific steps are as follows:
[0091] W_b = W[:,:,0,:] represents the weight of the corresponding color channel B;
[0092] W_g = W[:,:,1,:] represents the weight of the corresponding color channel G;
[0093] W_r = W[:,:,2,:] represents the weight of the corresponding color channel R;
[0094] W = [W_b, W_g, W_r] represents the weights of the first convolutional layer C, adapted to the input BGR format.
[0095] Since the original input is in RGB format, the positions of W_b and W_r need to be swapped, i.e., W[:,:,0,:],W[:,:,2,:]=W[:,:,2,:],W[:,:,0,:].
[0096] The new weights are W = [W_r, W_g, W_b].
[0097] No changes need to be made to offset B.
[0098] 3. Mean and normalization.
[0099] The preprocessing process requires subtracting the corresponding average value *m* from the value of each color channel in the image, and then multiplying it by the corresponding normalization parameter *s*. Further adjustments are needed to the parameters of the first convolutional layer, C1, to integrate the image preprocessing into the model parameters. The specific steps are as follows:
[0100] Weights multiplied by normalization parameters: W' = [W_r*s1, W_g*s2, W_b*s3];
[0101] Modify the offset B. B' = B – Sum(M*W'), where Sum(M*W') is equivalent to using an image T of [4,4,3], with the values of each color channel being [m1,m2,m3], and then performing an inner product of each output channel of T and W', and finally summing the results.
[0102] Replace the original W and B with the modified weights W' and offsets B', and then output the model.
[0103] It is important to note that if there is a padding operation before or within the first convolutional layer, and the padding pattern is constant, the constant C for that operation needs to be modified: C' = C + M
[0104] It is obvious that if M = [0,0,0], C' = C. No modifications are needed.
[0105] The modified model of this invention retains no structural changes, and therefore its performance remains unchanged. Since all modifications are reversible, its accuracy is also identical. Through these modifications, the preprocessing step can be completely eliminated during model inference, allowing the model to directly obtain the preprocessed result using the original input. This improves inference efficiency and simplifies the business process.
[0106] Through the description of specific embodiments, a more in-depth and specific understanding should be gained of the technical means and effects adopted by the present invention to achieve the intended purpose. However, the accompanying drawings are only provided for reference and illustration and are not intended to limit the present invention.
Claims
1. A method for implementing deep learning preprocessing, characterized in that, The preprocessing involves swapping the RGB color channel order of the input image to the BGR color channel order. The method includes: Input image; Based on the preprocessing operation of exchanging RGB to BGR, the weight parameters of the first convolutional layer in the deep learning network are adjusted. The adjustment method is to perform a data permutation operation on the weight tensor, that is, directly swap the weights corresponding to the red channel R and the blue channel B in the weight tensor of the first convolutional layer, so that the output of the first convolutional layer after parameter adjustment is exactly the same as the output result after the input image is first processed by exchanging the channel order from RGB to BGR and then processed by the first convolutional layer with the original parameters. The adjusted parameters are used as the running parameters for the first convolutional layer; Specifically, when there is a constant padding operation before the first convolutional layer, the constant padding parameter is adjusted to the original padding parameter plus the average value, so that the output result of the first convolutional layer after parameter adjustment is exactly the same as the output result of the input image after constant padding and after processing by the first convolutional layer with the original parameters.
2. The implementation method of deep learning preprocessing according to claim 1, characterized in that, The original parameters for adjusting the first convolutional layer include weights and offsets.
3. The implementation method of deep learning preprocessing according to claim 1, characterized in that, The preprocessing operation also includes: normalization.
4. The implementation method of deep learning preprocessing according to claim 3, characterized in that, When the preprocessing operation includes normalization, the adjustment of the original parameters includes modifying the weights of the first convolutional layer.
5. A deep learning network, characterized in that, The deep learning network implements the preprocessing operation in the deep learning network using the deep learning preprocessing implementation method as described in any one of claims 1-4.
6. The deep learning network according to claim 5, characterized in that, The deep learning network is a sound processing deep learning network, an image processing deep learning network, or a video processing deep learning network.