A linear filtering algorithm optimization method based on the idea of depth separable convolution

Abstract

The application discloses a linear filtering algorithm optimization method based on a depth separable convolution idea, and comprises the following steps: S1, generating the weight of a filter kernel according to a selected linear filtering algorithm and a set filter kernel size; S2, filling the image boundary according to the set filter kernel size; S3, processing an input image according to the channel number of the input image; S4, merging the multi-channels after the linear filtering operation, and outputting the filtered image; S5, using mean square error, peak signal-to-noise ratio and structural similarity as indexes to analyze the image quality of the output filtered image; and S6, comparing the linear filtering algorithm with a naive linear filtering algorithm, a linear filtering algorithm based on a row-column separation idea and a linear filtering algorithm of an OpenCV library based on a buffer and the row-column separation idea. The application can effectively adapt to various hardware environments without the need of making a large number of modifications for specific chips.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of parallel computing and deep learning, and more particularly, to a linear filtering algorithm optimization method based on the idea of deep separable convolution. BACKGROUND

[0002] In the process of image acquisition, transmission and processing, there will be certain noise interference. These noises can come from various sources, such as thermal noise of sensors, electromagnetic interference of electronic devices, errors in data transmission process, etc. The existence of noise will make the image blurred, which covers some important features of the image, and brings inconvenience to the subsequent image processing. For example, in medical images, noise may cover the lesion or lesion area, affecting the doctor's diagnosis. Therefore, effectively removing noise is an important step in image processing.

[0003] Filtering is an effective means for eliminating noise components in images. Through filtering operation, the image can be smoothed, the influence of noise can be reduced, and the image can be made clearer. The energy of a signal or image is mostly concentrated in the low and medium frequency bands of the amplitude spectrum, while in the higher frequency bands, the information of interest is often overwhelmed by noise. Therefore, a filter that can reduce the amplitude of high frequency components can reduce the influence of noise and improve the quality of the image. The effect of filtering processing directly affects the effectiveness and reliability of subsequent image processing and analysis.

[0004] For images with high resolution, the speed requirement of filtering processing is particularly critical. High-resolution images contain a large amount of pixel data, which means that the amount of data that filtering algorithms need to process increases exponentially. In order to complete the filtering processing within a reasonable time, it is necessary to ensure the efficiency of the algorithm. At the same time, for the case of limited hardware resources such as embedded systems and mobile devices, filtering processing technology needs to pay special attention to the effective use of computing resources to ensure that it can still run efficiently under lower hardware configurations. In addition, when filtering is part of image preprocessing, its speed directly affects the efficiency of the entire image processing flow. Especially in real-time applications such as autonomous driving, filtering processing must be completed within a very short time to ensure the response speed and stability of the system. SUMMARY

[0005] The purpose of the present application is to provide a linear filtering algorithm optimization method based on the idea of deep separable convolution to overcome the defects of the prior art.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A linear filtering algorithm optimization method based on the idea of deep separable convolution, comprising the following steps:

[0008] S1. Generate filter kernel weights according to the selected linear filtering algorithm and the set filter kernel size;

[0009] S2. Filling the image boundary according to the set filter kernel size;

[0010] S3. Process the input image according to the number of channels of the input image. If the input image is a grayscale image, the optimized linear filtering algorithm is executed to process the input image. If the input image is an RGB image, the multi-threaded idea is used to separate the channels of the input image and the optimized linear filtering operation is executed on each channel in parallel.

[0011] S4, merging the multiple channels after the linear filtering operation and outputting the filtered image;

[0012] S5. Performing image quality analysis on the filtered image output in step S4 using mean square error, peak signal-to-noise ratio, and structural similarity as indicators to measure the denoising ability of the linear filtering algorithm based on the depthwise separable convolution idea on the noisy image;

[0013] S6. Compare with a simple linear filtering algorithm, a linear filtering algorithm based on row-column separation, and a linear filtering algorithm of the OpenCV library based on a buffer and row-column separation idea to measure the processing time acceleration of the linear filtering algorithm based on the depthwise separable convolution idea.

[0014] Furthermore, the linear filtering algorithm selected in step S1 is box filtering, mean filtering or Gaussian filtering, wherein the filter kernel weights of box filtering and mean filtering are both 1, and the filter kernel weight of Gaussian filtering is calculated by a Gaussian function and normalized so that the sum of the filter weights of Gaussian filtering is 1.

[0015] Furthermore, the filter kernel of the Gaussian filter is calculated by a two-dimensional Gaussian function, and the standard deviations σ in the x-direction and the y-direction are regarded as the same standard deviation. The calculation formula is:

[0016]

[0017] By using the given filter kernel size N*N, the center offset is calculated using the following formula:

[0018]

[0019] After calculating the center offset, by traversing each position (i, j) in the filter kernel, the corresponding Gaussian function value is calculated using the Gaussian function. The calculation formula is:

[0020]

[0021] The standard deviation σ represents is used to adjust the degree of blurring based on the distance from the center. It is obtained by inputting a value or calculating it using the formula:

[0022]

[0023] After the weights in the filter kernel are calculated, normalization is performed. The calculation formula is:

[0024]

[0025] The denominator represents the sum of all the weights of the filter kernels.

[0026] Furthermore, the formula for filling the image boundary in step S2 is:

[0027]

[0028] Where p is the padding value and the filter kernel size is N*N.

[0029] Furthermore, the filtering algorithm optimized based on the idea of ​​depthwise separable convolution executed on the input image in step S3 is found to have general rules by splitting the filtering formula, and the filtering process performs continuous data access and has the conditions for vectorization. It is optimized using the Neon vector instruction set, and a single loop simultaneously processes the filtering operations of eight elements of a single channel. The number of instructions is reduced to one-eighth of the original compared to the naive algorithm, and the delay problem caused by cache misses is reduced. At the same time, the time used for memory access is reduced through data prefetching operations.

[0030] Furthermore, the step S4 also includes saving the output filtered image.

[0031] Furthermore, the mean square error calculates the sum of squared differences using pixel information of the denoised image and the original image, and takes the average based on the number of channels and image size; the peak signal-to-noise ratio is further calculated using image depth and mean square error to obtain an image quality score; and the structural similarity calculates the structural similarity score between images using the mean and standard deviation of the two images.

[0032] Compared with the prior art, the advantages of the present invention are:

[0033] 1. This invention incorporates the concept of depthwise separable convolution to design a linear filtering algorithm that fully utilizes the ARM architecture's vector registers for filtering operations, significantly reducing the number of instruction entries by one-eighth compared to a naive linear filtering algorithm. Furthermore, by adopting the multi-threading concept of OpenMP, filtering operations are performed channel by channel, further accelerating the linear filtering process. This reduces image preprocessing time, enables faster processing of high-resolution images, and is suitable for systems with real-time requirements.

[0034] 2. The linear filtering algorithm designed in this paper, based on the concept of depthwise separable convolution, is general and can be extended to any chip equipped with SIMD instructions, such as those from Intel, Apple, and other manufacturers. By fully utilizing the SIMD instruction sets on different chip platforms, the algorithm can effectively adapt to various hardware environments without requiring extensive modification for specific chips.

[0035] 3. The linear filtering algorithm designed by the present invention based on the idea of ​​depthwise separable convolution has the same denoising ability as the original linear filtering algorithm. It can directly replace the originally designed function without losing image quality, without modifying the internal code details, and has plug-and-play characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is a flow chart of the linear filtering algorithm optimization method based on the depth-separable convolution idea of ​​the present invention.

[0038] Figure 2 It is a schematic diagram of the algorithm execution flow based on the depthwise separable convolution algorithm in the present invention.

[0039] Figure 3 This is a comparison chart of the execution time of the box filter in the linear filtering algorithm based on the row-column separation idea, the OpenCV library, and the depth-separable convolution idea. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0041] Example 1

[0042] Referring to Figure 1 As shown in the embodiment, the embodiment discloses a linear filtering algorithm optimization method based on the idea of depth separable convolution, including the following steps:

[0043] Step S1, according to the selected linear filtering algorithm and the set filter kernel size, the weight of the filter kernel is generated, wherein the box filter, mean filter and Gaussian filter three linear filtering algorithms are supported, and part of the common filter kernel size, such as 3x3, 5x5, 7x7 size filter kernel has been pre-generated, which can be directly read from the file.

[0044] Specifically, the embodiment supports three linear filtering algorithms of box filter, mean filter and Gaussian filter. The filter kernel weight of box filter and mean filter is 1, and the filter kernel weight of Gaussian filter needs to be calculated by Gaussian function, and is normalized to make the weight sum 1. At the same time, part of the common filter kernel size, such as 3x3, 5x5, 7x7 size filter kernel has been pre-generated, and the standard deviation used is the corresponding default value, and such filter kernel weight can be directly read from the file.

[0045] Specifically, the filter kernel of Gaussian filter is calculated by two-dimensional Gaussian function, and the standard deviation σ in x direction and y direction is regarded as the same standard deviation in the embodiment, and the calculation formula is as follows:

[0046]

[0047] Firstly, the center offset is calculated by the given filter kernel size N*N, and the calculation formula is as follows:

[0048]

[0049] After calculating the center offset, the corresponding Gaussian function value can be calculated by traversing each position (i, j) in the filter kernel using Gaussian function, and the calculation formula is as follows:

[0050]

[0051] Wherein σ represents the standard deviation for adjusting the blur degree of the distance from the center, which can be obtained by passing in the value, or can be calculated by formula, and the calculation formula is as follows:

[0052]

[0053] After the weight in the filter kernel is calculated, in order to make the total weight in the filter kernel 1, normalization operation is also needed, and the calculation formula is as follows:

[0054]

[0055] The denominator represents the sum of all weights of the filter kernel. Since the filter kernel generated by the Gaussian function needs to be normalized, the coefficients are not calculated in the actual calculation.

[0056] Step S2: Fill the image boundary according to the set filter kernel size to ensure that the image size after filtering is consistent with the original image.

[0057] In this embodiment, the calculation formula of the boundary filling value is as follows:

[0058]

[0059] The calculated filling value p needs to be used for filling in four directions of the image.

[0060] Step S3: Process the input image according to the number of channels of the input image. If the input image is a grayscale image, the optimized linear filtering algorithm is executed to process the input image. If the input image is an RGB image, the multi-threaded idea is used to separate the channels of the input image, and the optimized linear filtering operation is performed on each channel in parallel, thereby realizing channel-by-channel processing of the image based on the depthwise separable convolution idea.

[0061] In this embodiment, the linear filtering algorithm optimized based on the depth-separable convolution idea executed on the input image is derived by the split filtering formula. Figure 2 (a) The 3x3 filter kernel here reflects the linear filtering calculation of a single row. It can be found that the element d obtained after the filtering calculation 00 It is necessary to use the three consecutive elements s in the row where the current channel is located 00 to s 02 , calculate the element d 01 Need to use the three consecutive elements s of the current row 01 to s 03 , they also need to be the first row of k in the filter kernel 00 to k 02 Perform corresponding bit multiplication. For ARMv8 vector registers, the bit width is 128 bits, which means it can store 4 elements represented by 32-bit floating point numbers. 00 to d 03 The elements obtained after the three filtering calculations need to be combined with k 00 The calculation is s 00 to s 03 , needs to be consistent with k 01 The calculation is s 01 to s 04 , needs to be consistent with k 02 The calculation is s 02 to s05 These four elements, therefore s 00 to s 03 、s 01 to s 04 and s 02 to s 05 Three vector registers can be used for storage, and the weights of the filter kernel can also be stored in the vector register. By using the vector and the corresponding channel of the vector of the Neon instruction set (that is, the corresponding element k of the vector register where the filter kernel weight is stored) 00 , k 01 and k 02 ) performs a multiplication and addition operation, and the result of the calculation is stored in another vector register. After accumulation, d 00 to d 03 The values ​​of the four elements; similarly, this algorithm is extended to Figure 2 (b) The 5x5 filter kernel requires two vector registers to store k 00 to k 04 The weight value of this single-line filter kernel is realized by the above method. 00 to d 03 Element calculation. Since data access has the principle of locality, d 10 to d 13 The elements from row 1, column 0 to column 3 are also calculated within the same loop, that is, filtering calculations for eight elements are implemented within one loop. The calculation details are calculated and stored using the Neon instruction set and vector registers, which can reduce the number of cache misses and accelerate linear filtering processing.

[0062] Step S4: Merge the multiple channels after the linear filtering operation, output the filtered image and save it.

[0063] Step S5: Use mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) as indicators to perform image quality analysis on the filtered image output in step S4 to measure the denoising ability of the linear filtering algorithm based on the idea of ​​depthwise separable convolution for the noisy image.

[0064] In this embodiment, the mean square error (MSE) uses the pixel information of the denoised image and the original image to calculate the sum of squared differences, and takes the average based on the number of channels and image size; the peak signal-to-noise ratio (PSNR) is further calculated using the image depth and mean square error to obtain an image quality score; and the structural similarity (SSIM) uses the mean and standard deviation of the two images to calculate the structural similarity score between the images.

[0065] In this embodiment, the mean square error (MSE) of the pixel information of the two images is calculated using the following formula:

[0066]

[0067] Wherein, I(i, j) represents the pixel information of the original image, k(i, j) represents the pixel information of the image after denoising using the linear filtering algorithm proposed in this invention, m and n represent the width and height of the image. For RGB images, it is also necessary to divide by the number of channels 3 to obtain the corresponding mean square error.

[0068] After obtaining the mean square error, it is necessary to calculate the peak signal-to-noise ratio score based on the image depth and the mean square error. The calculation formula is as follows:

[0069]

[0070] Where MAX is the maximum value of the image depth. For an image depth of 8 bits, MAX is 255.

[0071] The structural similarity calculation is performed on the denoised image and the original image. The calculation formula is as follows:

[0072]

[0073] Where μ x and μ y Represent the mean of the two images, σ x and σ y They represent the standard deviations of the two images respectively. c1 and c2 are constants used to maintain stability. When the image depth is 8 bits, the values ​​of c1 and c2 are 6.5025 and 58.5225 respectively.

[0074] Step S6: Compare the linear filtering algorithm based on the naive linear filtering algorithm, the linear filtering algorithm based on the row-column separation idea, and the linear filtering algorithm of the OpenCV library based on the buffer and row-column separation idea to measure the processing time acceleration of the linear filtering algorithm based on the depthwise separable convolution idea.

[0075] In this embodiment, a simple linear filtering algorithm processes pixels one by one similar to convolution, and is implemented internally through four loops. A filtering algorithm based on the row-column separation concept is optimized through a buffer, splitting the row and column calculations, and implemented internally through three loops. The linear filtering algorithm of the OpenCV library, implemented based on the buffer and row-column separation concept, is optimized by splitting the filter kernel size N and reusing the intermediate calculation results, reducing a large amount of redundant calculations. At the same time, the buffer is used for addition and subtraction operations at both ends, and the vectorized operations of the Neon instruction set are used locally. These are compared with the linear filtering algorithm based on the depthwise separable convolution concept proposed in this invention.

[0076] Example 2

[0077] In order to better illustrate the filtering effect of this embodiment on some noise, this embodiment implements a strict procedure and uses MSE, PSNR and SSIM as indicators to measure the filtering effect. Table 1 lists the denoising effects of the Gaussian filtering algorithm in the linear filtering algorithm used in this embodiment on pictures with different resolutions, and compares them with the Gaussian filtering algorithm provided by the OpenCV library:

[0078] Table 1 Multi-index test of Gaussian noise denoising effect of multi-resolution images using Gaussian filtering

[0079]

[0080] As can be seen from Table 1, this embodiment achieves high evaluation scores across multiple metrics and is similar to the Gaussian filter algorithm provided by the OpenCV library across multiple image resolutions. This result demonstrates that the present invention has comparable denoising capabilities to the linear filter algorithm provided by the OpenCV library.

[0081] Example 3

[0082] In order to test the execution time of the present invention and the simple linear filtering algorithm, the linear filtering algorithm based on row-column separation, and the linear filtering algorithm of the OpenCV library implemented based on the buffer and row-column separation ideas, the RGB image and the box filtering in the linear filtering algorithm are uniformly used for experiments. The experimental results are shown in Table 2 below:

[0083] Table 2 Comparison of execution time of the method proposed in this invention and the above-mentioned methods

[0084]

[0085] Figure 3 The execution time histogram of the method proposed in this invention and other comparison methods is shown. The height of the histogram represents the execution time of the method in milliseconds. Since the execution time of the naive algorithm is too different, the algorithm is not reflected in the image. Figure 3 From the comparison, it can be seen that the present invention has a faster execution speed than the simple linear filtering algorithm and even the linear filtering algorithm of the existing OpenCV library, and its overall performance is better than other methods.

[0086] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various changes or modifications within the scope of the appended claims. As long as they do not exceed the scope of protection described in the claims of the present invention, they should be within the scope of protection of the present invention.

Claims

1. A linear filtering algorithm optimization method based on the idea of ​​depthwise separable convolution, characterized in that: The following steps are involved: S1. Generate filter kernel weights according to the selected linear filtering algorithm and the set filter kernel size; S2. Filling the image boundary according to the set filter kernel size; S3. Process the input image according to the number of channels of the input image. If the input image is a grayscale image, execute the optimized linear filtering algorithm to process the input image. If the input image is an RGB image, use a multi-threading approach to separate the channels of the input image and execute the optimized linear filtering operation on each channel in parallel. S4, merging the multiple channels after the linear filtering operation and outputting the filtered image; S5. Performing image quality analysis on the filtered image output in step S4 using mean square error, peak signal-to-noise ratio, and structural similarity as indicators to measure the denoising ability of the linear filtering algorithm based on the depthwise separable convolution idea on the noisy image; S6. Compare the linear filtering algorithm based on the naive linear filtering algorithm, the linear filtering algorithm based on the row-column separation idea, and the linear filtering algorithm of the OpenCV library based on the buffer and row-column separation idea to measure the processing time acceleration of the linear filtering algorithm based on the depthwise separable convolution idea; The filtering algorithm optimized based on the idea of ​​depthwise separable convolution executed on the input image in step S3 is found to have general rules by splitting the filtering formula, and the filtering process performs continuous data access and has the conditions for vectorization. It is optimized using the Neon vector instruction set, and a single loop simultaneously processes the filtering operations of eight elements of a single channel. The number of instructions is reduced to one-eighth of the original compared to the naive algorithm, and the delay problem caused by cache misses is reduced. At the same time, the time used for memory access is reduced through data prefetching operations.

2. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The linear filtering algorithm selected in step S1 is box filtering, mean filtering or Gaussian filtering, wherein the filter kernel weights of box filtering and mean filtering are both 1, and the filter kernel weight of Gaussian filtering is calculated by a Gaussian function and normalized so that the sum of the filter weights of Gaussian filtering is 1.

3. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 2 is characterized in that: The filter kernel of the Gaussian filter is calculated by a two-dimensional Gaussian function, and the standard deviation σ in the x-direction and the y-direction is regarded as the same standard deviation. The calculation formula is: By using the given filter kernel size N*N, the center offset is calculated using the following formula: After calculating the center offset, by traversing each position (i, j) in the filter kernel, the corresponding Gaussian function value is calculated using the Gaussian function. The calculation formula is: The standard deviation σ represents is used to adjust the degree of blurring based on the distance from the center. It is obtained by inputting a value or calculating it using the formula: After the weights in the filter kernel are calculated, normalization is performed. The calculation formula is: The denominator represents the sum of all the weights of the filter kernels.

4. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The formula for filling the image boundary in step S2 is: Where p is the padding value and the filter kernel size is N*N.

5. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The step S4 also includes saving the output filtered image.

6. The linear filtering algorithm optimization method based on the depthwise separable convolution concept according to claim 1 is characterized in that: The mean square error calculates the sum of squared differences between the denoised image and the original image using pixel information, and takes the average based on the number of channels and image size. The peak signal-to-noise ratio is further calculated using image depth and mean square error to obtain an image quality score. The structural similarity calculates the structural similarity score between the two images using the mean and standard deviation.

Images