Linear filtering algorithm optimization method based on depth separable convolution thought

Abstract

The invention discloses a linear filtering algorithm optimization method based on a depth separable convolution idea. The method comprises the following steps: S1, generating the weight of a filtering kernel according to a selected linear filtering algorithm and a set filtering kernel size; s2, filling the image boundary according to the set filtering kernel size; s3, processing the input image according to the channel number of the input image; s4, combining the plurality of channels after the linear filtering operation, and outputting a filtered image; s5, using a mean square error, a peak signal-to-noise ratio and structural similarity as indexes to carry out image quality analysis on the output filtered image; and S6, comparing with a naive linear filtering algorithm, a linear filtering algorithm realized based on a row-column separation thought and a linear filtering algorithm of an OpenCV library realized based on a buffer area and the row-column separation thought. The method can effectively adapt to various hardware environments, and a specific chip does not need to be modified greatly.

Description

Technical Field

[0001] The present invention relates to the field of parallel computing and deep learning technology, and more specifically, to a linear filtering algorithm optimization method based on the idea of ​​deep separable convolution. Background Art

[0002] In the process of image acquisition, transmission and processing, there is usually a certain amount of noise interference. This noise can come from a variety of sources, such as thermal noise of sensors, electromagnetic interference of electronic devices, errors in data transmission, etc. The presence of noise will make the image blurry, cover up some important features of the image, and bring inconvenience to subsequent image processing. For example, in medical images, noise may cover up lesions or diseased areas, affecting doctors' diagnosis. Therefore, effective noise removal is an important step in image processing.

[0003] Filtering is an effective means to eliminate noise components in images. Through filtering operations, images can be smoothed, the impact of noise can be reduced, and images can be made clearer. Most of the energy of a signal or image is concentrated in the low and medium frequency bands of the amplitude spectrum, while in the higher frequency bands, the information of interest is often drowned out by noise. Therefore, a filter that can reduce the amplitude of high-frequency components can reduce the impact of noise and improve the quality of the image. The effect of filtering directly affects the effectiveness and reliability of subsequent image processing and analysis.

[0004] For images with high resolution, the speed requirement of filtering is particularly critical. High-resolution images contain a large amount of pixel data, which means that the amount of data that the filtering algorithm needs to process increases exponentially. In order to complete the filtering process in a reasonable time, the algorithm must be efficient. At the same time, for embedded systems and mobile devices with limited hardware resources, filtering technology needs to pay special attention to the effective use of computing resources to ensure that it can still run efficiently under lower hardware configurations. In addition, when filtering is used as part of image preprocessing, its speed directly affects the efficiency of the entire image processing process. Especially in real-time applications such as autonomous driving, filtering must be completed in a very short time to ensure the response speed and stability of the system. Summary of the invention

[0005] The purpose of the present invention is to provide a linear filtering algorithm optimization method based on the concept of depthwise separable convolution to overcome the defects of the prior art.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A linear filtering algorithm optimization method based on the idea of ​​depthwise separable convolution includes the following steps:

[0008] S1. Generate the weight of the filter kernel according to the selected linear filtering algorithm and the set filter kernel size;

[0009] S2. Filling the image boundary according to the set filter kernel size;

[0010] S3. Process the input image according to the number of channels of the input image. If the input image is a grayscale image, the optimized linear filtering algorithm is executed to process the input image. If the input image is an RGB image, the multi-threading idea is used to separate the channels of the input image, and the optimized linear filtering operation is executed on each channel in parallel.

[0011] S4, merging the multiple channels after the linear filtering operation, and outputting the filtered image;

[0012] S5, using mean square error, peak signal-to-noise ratio and structural similarity as indicators to perform image quality analysis on the image output after filtering in step S4, so as to measure the denoising ability of the linear filtering algorithm based on the idea of ​​depthwise separable convolution on the noisy image;

[0013] S6. Compare with a simple linear filtering algorithm, a linear filtering algorithm based on the idea of ​​row-column separation, and a linear filtering algorithm of the OpenCV library based on the idea of ​​buffer and row-column separation to measure the processing time acceleration of the linear filtering algorithm based on the idea of ​​depthwise separable convolution.

[0014] Furthermore, the linear filtering algorithm selected in step S1 is box filtering, mean filtering or Gaussian filtering, wherein the filter kernel weights of box filtering and mean filtering are both 1, and the filter kernel weight of Gaussian filtering is calculated by a Gaussian function and normalized so that the sum of the filter weights of Gaussian filtering is 1.

[0015] Furthermore, the filter kernel of the Gaussian filter is calculated by a two-dimensional Gaussian function, and the standard deviations σ in the x-direction and the y-direction are regarded as the same standard deviation, and the calculation formula is:

[0016]

[0017] By using the given filter kernel size N*N, the center offset is calculated using the following formula:

[0018]

[0019] After calculating the center offset, by traversing each position (i, j) in the filter kernel, using the Gaussian function, calculate the corresponding Gaussian function value. The calculation formula is:

[0020]

[0021] The standard deviation represented by σ is used to adjust the degree of blurring at different distances from the center. It is obtained by inputting a value or calculating it using the formula:

[0022]

[0023] After the weights in the filter kernel are calculated, normalization is performed, and the calculation formula is:

[0024]

[0025] The denominator represents the sum of all weights of the filter kernel.

[0026] Furthermore, the formula for filling the image boundary in step S2 is:

[0027]

[0028] In the formula, p is the padding value and the filter kernel size is N*N.

[0029] Furthermore, the filtering algorithm optimized based on the idea of ​​deep separable convolution executed on the input image in the step S3 is found to have general rules by splitting the filtering formula, and the filtering process performs continuous data access and has the conditions for vectorization. It is optimized using the Neon vector instruction set, and a single cycle simultaneously processes the filtering operations of eight elements of a single channel. The number of instructions is reduced to one eighth of the original compared to the naive algorithm, and the delay problem caused by cache misses is reduced. At the same time, the time used for memory access is reduced through data prefetching operations.

[0030] Furthermore, the step S4 also includes saving the output filtered image.

[0031] Furthermore, the mean square error calculates the sum of squared differences using pixel information of the denoised image and the original image, and takes the average according to the number of channels and the image size; the peak signal-to-noise ratio is further calculated through image depth and mean square error to obtain an image quality score; the structural similarity calculates the structural similarity score between images using the mean and standard deviation of two images.

[0032] Compared with the prior art, the advantages of the present invention are:

[0033] 1. The linear filtering algorithm designed by the present invention combines the idea of ​​depthwise separable convolution to make full use of the vector registers of the ARM architecture for filtering operations, greatly reducing the number of instruction entries, which is reduced to one eighth of that of the simple linear filtering algorithm. At the same time, the multi-threading idea of ​​OpenMP is adopted to perform filtering operations channel by channel, further accelerating the processing of linear filtering, making the image preprocessing time shorter, and processing high-resolution images more quickly, and suitable for systems with real-time requirements.

[0034] 2. The linear filtering algorithm designed by the present invention based on the idea of ​​depthwise separable convolution is general and can be extended to any chip with SIMD instructions, such as chips from Intel, Apple and other manufacturers. By making full use of the SIMD instruction set on different chip platforms, the algorithm can effectively adapt to various hardware environments without requiring extensive modifications for specific chips.

[0035] 3. The linear filtering algorithm designed by the present invention based on the idea of ​​deep separable convolution has the same denoising ability as the original linear filtering algorithm. It can directly replace the originally designed function without losing image quality, without modifying the internal code details, and has plug-and-play characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a flow chart of the linear filtering algorithm optimization method based on the concept of deep separable convolution of the present invention.

[0038] Figure 2 It is a schematic diagram of the algorithm execution flow based on the depth-separable convolution algorithm in the present invention.

[0039] Figure 3 This is a comparison chart of the execution time of the box filter in the linear filtering algorithm based on the row-column separation idea, the OpenCV library, and the depth-separable convolution idea. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0041] Embodiment 1

[0042] See also Figure 1 As shown, this embodiment discloses a linear filtering algorithm optimization method based on the idea of ​​deep separable convolution, comprising the following steps:

[0043] Step S1, generate the weight of the filter kernel according to the selected linear filtering algorithm and the set filter kernel size, wherein three linear filtering algorithms, box filtering, mean filtering and Gaussian filtering, are supported. At the same time, some common filter kernel sizes, such as 3x3, 5x5, and 7x7 filter kernels, have been pre-generated and can be directly read from the file.

[0044] Specifically, this embodiment supports three linear filtering algorithms: box filtering, mean filtering and Gaussian filtering. The filter kernel weights of box filtering and mean filtering are both 1, and the filter kernel weights of Gaussian filtering need to be calculated by Gaussian function and normalized so that the weight sum is 1. At the same time, some common filter kernel sizes, such as 3x3, 5x5, and 7x7 filter kernels, have been pre-generated, and the selected standard deviation is the corresponding default value. Such filter kernel weights can be directly read from the file.

[0045] Specifically, the filter kernel of the Gaussian filter is calculated by a two-dimensional Gaussian function. In this embodiment, the standard deviation σ in the x-direction and the y-direction is regarded as the same standard deviation, and the calculation formula is as follows:

[0046]

[0047] First, the center offset is calculated using the given filter kernel size N*N. The calculation formula is as follows:

[0048]

[0049] After calculating the center offset, by traversing each position (i, j) in the filter kernel, the corresponding Gaussian function value can be calculated using the Gaussian function. The calculation formula is as follows:

[0050]

[0051] The standard deviation represented by σ is used to adjust the blur degree of the distance from the center. It can be obtained by inputting a value or calculated by the formula. The calculation formula is as follows:

[0052]

[0053] After the weights in the filter kernel are calculated, normalization is required to make the sum of the weights in the filter kernel equal to 1. The calculation formula is as follows:

[0054]

[0055] The denominator represents the sum of all weights of the filter kernel. Since the filter kernel generated by the Gaussian function needs to be normalized, the coefficients are not calculated in the actual calculation.

[0056] Step S2: Fill the image border according to the set filter kernel size to ensure that the image size after filtering is consistent with the original image.

[0057] In this embodiment, the calculation formula of the boundary filling value is as follows:

[0058]

[0059] The calculated filling value p needs to be used for filling in four directions of the image.

[0060] Step S3, processing the input image according to the number of channels of the input image. If the input image is a grayscale image, executing the optimized linear filtering algorithm to process the input image; if the input image is an RGB image, using the multi-threading idea to separate the channels of the input image, and executing the optimized linear filtering operation on each channel in parallel, that is, realizing channel-by-channel processing of the image based on the idea of ​​deep separable convolution.

[0061] In this embodiment, the linear filtering algorithm optimized based on the idea of ​​deep separable convolution executed by the input image is derived from the split filtering formula. Figure 2 (a) shows a 3x3 filter kernel, which is a single-line linear filter calculation. It can be found that the element d obtained after the filter calculation is 00 You need to use the three consecutive elements s in the row where the current channel is located 00 to 02 , calculate the element d 01 Need to use the three consecutive elements s of the current row 01 to 03 , they also need to be the first row of k in the filter kernel 00 to k 02 Multiply the corresponding bits. For ARMv8 vector registers, the bit width is 128 bits, which means that it can store four elements represented by 32-bit floating point numbers. 00 to d 03 The elements obtained after these three filtering calculations need to be combined with k 00 The calculation is s 00 to 03 , needs to be related to k 01 The calculation is s 01 to 04 , needs to be related to k 02 The calculation is s 02 to05 These four elements, therefore s 00 to 03 、s 01 to 04 and 02 to 05 Three vector registers can be used for storage, and the weights of the filter kernel can also be stored in the vector registers. By using the vector and vector corresponding channels of the Neon instruction set (that is, the corresponding element k of the vector register where the filter kernel weight is stored) 00 , k 01 and k 02 ) performs a multiplication and addition operation, and the result of the calculation is stored in another vector register. After accumulation, d 00 to d 03 The values ​​of the four elements; similarly, this algorithm is extended to Figure 2 (b) The 5x5 filter kernel requires two vector registers to store k 00 to k 04 The weight value of this single-row filter kernel is realized by the above method. 00 to d 03 Element calculation. Since data access has the principle of locality, d 10 to d 13 The elements, i.e., the elements from the 0th to the 3rd columns of the 1st row, are also calculated in the same loop, i.e., the filtering calculation of eight elements is implemented in one loop. The calculation details are calculated and stored using the Neon instruction set and vector registers, which can reduce the number of cache misses and accelerate linear filtering processing.

[0062] Step S4: merge the multiple channels after the linear filtering operation, output the filtered image and save it.

[0063] Step S5, using mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as indicators to perform image quality analysis on the image output after filtering in step S4, so as to measure the denoising ability of the linear filtering algorithm based on the idea of ​​deep separable convolution for noisy images.

[0064] In this embodiment, the mean square error (MSE) uses the pixel information of the denoised image and the original image to calculate the sum of squared differences, and takes the average according to the number of channels and the image size; the peak signal-to-noise ratio (PSNR) is further calculated through the image depth and the mean square error to obtain the image quality score; the structural similarity (SSIM) uses the mean and standard deviation of the two images to calculate the structural similarity score between the images.

[0065] In this embodiment, the mean square error is calculated for the pixel information of the two images, and the calculation formula is as follows:

[0066]

[0067] In the formula, I(i, j) represents the pixel information of the original image, k(i, j) represents the pixel information of the image after denoising by the linear filtering algorithm proposed in the present invention, m and n represent the width and height of the image, and for RGB images, it is also necessary to divide by the number of channels 3 to obtain the corresponding mean square error.

[0068] After obtaining the mean square error, it is necessary to calculate the peak signal-to-noise ratio score based on the image depth and the mean square error. The calculation formula is as follows:

[0069]

[0070] Where MAX is the maximum value of the image depth. When the image depth is 8 bits, the MAX value is 255.

[0071] The structural similarity calculation is performed on the denoised image and the original image. The calculation formula is as follows:

[0072]

[0073] In the formula, μ x and μ y Represent the mean of the two images, σ x and σ y Represents the standard deviation of the two images, c 1 and c 2 is a constant used to maintain stability. When the image depth is 8 bits, c 1 and c 2 The values ​​are 6.5025 and 58.5225 respectively.

[0074] Step S6, compare with a simple linear filtering algorithm, a linear filtering algorithm based on the row-column separation idea, and a linear filtering algorithm of the OpenCV library based on the buffer and row-column separation idea, so as to measure the processing time acceleration of the linear filtering algorithm based on the depthwise separable convolution idea.

[0075] In this embodiment, the simple linear filtering algorithm is similar to convolution to process pixel by pixel, and is implemented internally through four loops; the filtering algorithm based on the row-column separation idea is optimized through a buffer, splits the row and column calculations to achieve, and is implemented internally through three loops; the linear filtering algorithm of the OpenCV library implemented based on the buffer and row-column separation ideas is optimized by splitting the filter kernel size N to reuse the intermediate calculation results, reducing a large number of redundant calculations, and using the buffer to perform addition and subtraction operations at both ends, and using the vectorization operation of the Neon instruction set locally. They are used to compare with the linear filtering algorithm based on the depth-separable convolution idea proposed in the present invention.

[0076] Embodiment 2

[0077] In order to better illustrate the filtering effect of this embodiment on some noise, this embodiment implements a strict procedure and uses MSE, PSNR and SSIM as indicators to measure the filtering effect. Table 1 lists the denoising effects of the Gaussian filtering algorithm in the linear filtering algorithm used in this embodiment on pictures with different resolutions, and compares it with the Gaussian filtering algorithm provided by the OpenCV library:

[0078] Table 1 Multi-index test of Gaussian noise denoising effect of multi-resolution images using Gaussian filtering

[0079]

[0080] It can be seen from Table 1 that this embodiment has achieved high evaluation scores in multiple indicators, and is similar to the Gaussian filter algorithm provided by the OpenCV library in multiple indicators under multiple resolution images. This result shows that the present invention has the same denoising ability as the linear filter algorithm provided by the OpenCV library.

[0081] Embodiment 3

[0082] In order to test the execution time consumption of the present invention and the simple linear filtering algorithm, the linear filtering algorithm based on row-column separation, and the linear filtering algorithm of the OpenCV library implemented based on the buffer and row-column separation ideas, the RGB image and the box filtering in the linear filtering algorithm are uniformly used for experiments. The experimental results are shown in Table 2 below:

[0083] Table 2 Comparison of execution time of the method proposed in the present invention and the above-mentioned methods

[0084]

[0085] Figure 3 The execution time histogram results of the method proposed in this invention and other comparison methods are shown. The height of the histogram represents the millisecond execution time of the method. Since the execution time of the naive algorithm is too different, the algorithm is not reflected in the image. Figure 3 It can be seen from the comparison that the present invention has a faster execution speed than the simple linear filtering algorithm and even the linear filtering algorithm of the existing OpenCV library, and its comprehensive performance is better than other methods.

[0086] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.

Claims

1. A linear filtering algorithm optimization method based on the idea of ​​deep separable convolution, characterized in that: The following steps are involved: S1. Generate the weight of the filter kernel according to the selected linear filtering algorithm and the set filter kernel size; S2. Filling the image boundary according to the set filter kernel size; S3, processing the input image according to the number of channels of the input image. If the input image is a grayscale image, the optimized linear filtering algorithm is executed to process the input image; if the input image is an RGB image, the multi-threading idea is adopted to separate the channels of the input image, and the optimized linear filtering operation is executed on each channel in parallel; S4, merging the multiple channels after the linear filtering operation, and outputting the filtered image; S5, using mean square error, peak signal-to-noise ratio and structural similarity as indicators to perform image quality analysis on the image output after filtering in step S4, so as to measure the denoising ability of the linear filtering algorithm based on the idea of ​​depthwise separable convolution on the noisy image; S6. Compare with a simple linear filtering algorithm, a linear filtering algorithm based on the idea of ​​row-column separation, and a linear filtering algorithm of the OpenCV library based on the idea of ​​buffer and row-column separation to measure the processing time acceleration of the linear filtering algorithm based on the idea of ​​depthwise separable convolution.

2. According to claim 1, the linear filtering algorithm optimization method based on the concept of depthwise separable convolution is characterized in that: The linear filtering algorithm selected in step S1 is box filtering, mean filtering or Gaussian filtering, wherein the filter kernel weights of box filtering and mean filtering are both 1, and the filter kernel weight of Gaussian filtering is calculated by a Gaussian function and normalized so that the sum of the filter weights of Gaussian filtering is 1.

3. According to claim 2, the linear filtering algorithm optimization method based on the concept of depth-separable convolution is characterized in that: The filter kernel of the Gaussian filter is calculated by a two-dimensional Gaussian function, and the standard deviations σ in the x-direction and the y-direction are regarded as the same standard deviation. The calculation formula is: By using the given filter kernel size N*N, the center offset is calculated using the following formula: After calculating the center offset, by traversing each position (i, j) in the filter kernel, using the Gaussian function, calculate the corresponding Gaussian function value. The calculation formula is: Among them, the standard deviation represented by σ is used to adjust the degree of blurring at a distance from the center, which is obtained by inputting a value or calculating it through a formula. The calculation formula is: After the weights in the filter kernel are calculated, normalization is performed, and the calculation formula is: The denominator represents the sum of all weights of the filter kernel.

4. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The formula for filling the image boundary in step S2 is: In the formula, p is the padding value and the filter kernel size is N*N.

5. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The filtering algorithm optimized based on the idea of ​​deep separable convolution executed on the input image in the step S3 is found to have a general rule by splitting the filtering formula, and the filtering process performs continuous data access and has the conditions for vectorization. The Neon vector instruction set is used for optimization, and a single cycle simultaneously processes the filtering operation of eight elements of a single channel. The number of instructions is reduced to one eighth of the original compared to the naive algorithm, and the delay problem caused by cache misses is reduced. At the same time, the time used for memory access is reduced through data prefetching operations.

6. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The step S4 also includes saving the output filtered image.

7. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The mean square error uses the pixel information of the denoised image and the original image to calculate the sum of squared differences, and takes the average according to the number of channels and the image size; the peak signal-to-noise ratio is further calculated through the image depth and the mean square error to obtain the image quality score; the structural similarity uses the mean and standard deviation of the two images to calculate the structural similarity score between the images.

Images