Image bilateral filtering method based on kernel parameter lookup table

Abstract

The invention discloses an image bilateral filtering method based on a kernel parameter lookup table. The method comprises the following steps: S1, training a kernel parameter prediction model; s2, according to the trained kernel parameter prediction model, obtaining a kernel parameter lookup table of bilateral filtering; and S3, according to the kernel parameter lookup table, selecting an optimal kernel parameter for each input pixel to carry out bilateral filtering processing. According to the method, the neural network is adopted to learn the kernel parameters capable of effectively counteracting noise interference, and the kernel parameter table is generated. During use, appropriate kernel parameters are determined for each pixel based on a kernel parameter table and a four-side interpolation method. In this way, the image can be effectively smoothed in the presence of noise disturbance.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to an image bilateral filtering method based on a kernel parameter lookup table. Background Art

[0002] In the field of computer vision, edge-preserving smoothing is considered an important step in many tasks, including removing image texture, extracting image structure, and image enhancement. Recent studies have pointed out that the computational burden of bilateral filters can be reduced through Taylor approximation and trigonometric (Fourier) approximation techniques. At the same time, researchers have found that bilateral filters may produce abnormal results in the face of malicious interference. In recent years, the research progress of bilateral filtering is mainly reflected in the adaptive mechanism of constructing range kernels, but this adaptive mechanism only tries to find an adaptive function to convert pixels into another representation according to the characteristics of the image itself. In the face of slight noise disturbances, this adaptive mechanism may fail, resulting in poor edge-preserving smoothing performance. When using bilateral filters to perform global smoothing on images with slight noise, inappropriate kernel parameter selection may cause edge protection to fail. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides an image bilateral filtering method based on a kernel parameter lookup table, which can find the most suitable kernel parameter for each pixel when performing filtering processing, thereby protecting the image edge.

[0004] The technical solution adopted by the present invention to solve the technical problem is: an image bilateral filtering method based on a kernel parameter lookup table, comprising:

[0005] S1, training kernel parameter prediction model;

[0006] S2. Obtain a bilateral filtering kernel parameter lookup table based on the trained kernel parameter prediction model;

[0007] S3. According to the kernel parameter lookup table, select the optimal kernel parameter for each input pixel to perform bilateral filtering.

[0008] Furthermore, the kernel parameter prediction model includes: an input network layer, an intermediate network layer, and an output layer;

[0009] The input network layer includes three input channels, and the convolution kernel forms of the three input channels are respectively form S, form C and form X, the form S is 2*2 arrangement, the form C is 1*4 and arranged in a horizontal straight line, and the form X is 1*4 and arranged in an inclined straight line;

[0010] The convolution kernel size of the intermediate network layer is 1*1, and the ReLU activation function is used;

[0011] The output layer contains four output channels, corresponding to four kernel parameters σ r , σ sx , σ sy and ρ, σ r represents the standard deviation of the range-domain Gaussian distribution, σ sx and σ sy They represent the x-direction standard deviation and y-direction standard deviation of the Gaussian distribution in the spatial domain, and ρ represents the correlation of two-dimensional variables.

[0012] Furthermore, in the input network, rotation operations are performed simultaneously on the three types of convolution kernels, with rotation angles of 0°, 90°, 180°, and 270°.

[0013] Furthermore, the bilateral filtering process for the input image I can be expressed as:

[0014]

[0015] Where p=(p x ,p y ) represents the position of the central pixel, q=(q x ,q y ) represents the position of the surrounding pixels, I p and I q Indicates the pixel value of the pixel at position p and the pixel at position q, G Σ represents the Gaussian distribution in the spatial domain, represents the range domain Gaussian distribution, W p represents the normalization factor, and U represents the pixel neighborhood set.

[0016] Furthermore, in,

[0017] Σ| represents the determinant of covariance Σ;

[0018] Furthermore, the output result of the output layer can be expressed as:

[0019]

[0020] Among them, x i represents the i-th image block of the input containing noise interference, f p represents the network kernel parameter, R j represents the rotation operation, j=0,1,2,3, represents the derotation operation, and BF represents the bilateral filtering operation.

[0021] Furthermore, the network kernel parameters are transferred via the loss function loss=MSE(y',y).

[0022] Furthermore, a bilateral filtering kernel parameter lookup table is obtained, including:

[0023] Obtain the kernel parameter tables LUT-S, LUT-C, and LUT-X corresponding to the form S, form C, and form X;

[0024] Integrate the core parameter tables LUT-S, LUT-C, and LUT-X to obtain a complete core parameter table LUT;

[0025] The pixels in the kernel parameter table LUT are pixels sampled at equal intervals.

[0026] Furthermore, in step S3, the optimal kernel parameters are selected for each input pixel to perform bilateral filtering, including:

[0027] Using the four-sided interpolation method to interpolate and fill in the missing pixels in the kernel parameter table LUT;

[0028] The pixel K corresponding to the input pixel is searched in the padded kernel parameter table LUT, and the kernel parameter corresponding to the pixel K is obtained, and the input pixel is subjected to bilateral filtering using the kernel parameter.

[0029] The beneficial effect of the present invention is that the present invention adopts a neural network to learn the kernel parameters that can effectively offset the noise interference and generate a kernel parameter table. When used, the appropriate kernel parameters are determined for each pixel based on the kernel parameter table and the four-sided interpolation method. In this way, it is possible to effectively smooth the image in the presence of noise disturbance. The present invention is superior to the existing bilateral filtering method in terms of retaining image structure, removing image texture and resisting small noise disturbance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0031] Figure 1 It is a flow chart of the image bilateral filtering method based on the kernel parameter lookup table of the present invention.

[0032] Figure 2 It is a schematic diagram of the anisotropic bilateral filtering kernel of the present invention.

[0033] Figure 3 This is a comparison chart of the filtering and smoothing effects of Gaussian noise on the BSDS100 test data set using different filtering methods.

[0034] Figure 4 This is a comparison diagram of the effects of different parameter combinations on filtering and smoothing results in the present invention.

[0035] Figure 5 It is a comparison chart of the smoothing effect of the present invention and the conventional bilateral filter on the BSDS100 data set for different noise levels. DETAILED DESCRIPTION

[0036] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0037] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0038] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Bilateral filtering is a widely used image smoothing technique. Recent studies have shown that even small noise interferences can hinder the ability of bilateral filters to effectively preserve image edges. To solve this problem, the present invention uses a neural network to learn kernel parameters that can effectively offset noise interference. In addition, in order to improve the adaptability of the learned kernel parameters to the local edge features of the image, the present invention uses an edge-sensitive indexing method to generate a lookup table (LUT) for the kernel parameters. When used, the lookup table and interpolation method are used to determine the appropriate kernel parameters for each pixel. In this way, the image can be effectively smoothed in the presence of noise disturbances.

[0040] like Figure 1 to Figure 2As shown, the image bilateral filtering method based on the kernel parameter lookup table of the present invention includes: S1, training the kernel parameter prediction model; S2, obtaining the kernel parameter lookup table of bilateral filtering according to the trained kernel parameter prediction model; S3, selecting the optimal kernel parameter for each input pixel for bilateral filtering processing according to the kernel parameter lookup table.

[0041] In this embodiment, the kernel parameter prediction model adopts a neural network model. The kernel parameter prediction model includes: an input network layer, an intermediate network layer and an output layer. Among them, the input network layer contains three input channels, and the convolution kernel forms of the three input channels are form S, form C and form X respectively. Form S is arranged in 2*2, form C is 1*4 and arranged in a horizontal straight line, and form X is 1*4 and arranged in an inclined straight line. The convolution kernel size of the intermediate network layer is 1*1, and the ReLU activation function is used. The output layer contains four output channels, corresponding to the four kernel parameters σ r , σ sx , σ sy and ρ, σ r represents the standard deviation of the range-domain Gaussian distribution, σ sx and σ sy They represent the x-direction standard deviation and y-direction standard deviation of the Gaussian distribution in the spatial domain, and ρ represents the correlation of two-dimensional variables.

[0042] It should be noted that the input pixels in the training of the present invention use a small receptive field. The purpose of using a small receptive field is to simplify the input pixel combination. Assuming that a 2*2 pixel block is used as input, the possibility of pixel values ​​is 255. 4 If four kernel parameters are output, the size of the complete kernel parameter table under 8-bit storage may reach 16G, which takes up too much memory. 4 For equal-interval sampling, when the receptive field is 1 pixel, the size of the sampled kernel parameter table is only 68B, and when the receptive field is 4 pixels, the size of the sampled kernel parameter table is 326.254KB. In other words, for equal-interval sampling, the size of the parameter table is (2 4 +1) n ×4B, when n=4, the size of the parameter table is (2 4 +1) 4 ×4B=326.254KB. In this way, a good balance can be found between the receptive field and the storage size.

[0043] In this embodiment, the receptive field is 4 pixels. Since the image edge has various shapes, the present invention sets three different convolution kernel shapes, shape S is 2*2 arrangement, shape C is 1*4 and arranged in a horizontal straight line, and shape X is 1*4 and arranged in an inclined straight line. In the input network layer, the three different shapes of convolution kernels are rotated at the same time, and the rotation angles are 0°, 90°, 180°, and 270° to obtain a larger receptive field.

[0044] It should be noted that the bilateral filtering process for the input image I can be expressed as: Where p=(p x ,p y ) represents the position of the central pixel, q=(q x ,q y ) represents the position of the surrounding pixels, I p and I q Indicates the pixel value of the pixel at position p and the pixel at position q, G Σ represents the Gaussian distribution in the spatial domain, represents the range domain Gaussian distribution, W p Represents the normalization factor, and U represents the pixel domain set (i.e., the set of surrounding pixels). It should be explained that the surrounding pixels here refer to the pixels other than the central pixel in the input image. Considering that the Gaussian distribution with fixed parameters cannot adapt well to the local features of the input image, this embodiment learns the range domain anisotropic Gaussian distribution parameters Σ through a neural network with a small receptive field. For each pixel p, a neural network is required to predict a set of kernel parameters to achieve adaptive bilateral filtering.

[0045] At different positions in the image, slight noise will destroy the spatial domain weights of bilateral filtering. Therefore, when encountering irregular noise, using isotropic Gaussian distribution in the spatial domain will cause bilateral filtering to lose its smoothing effect. In order to improve this problem, this embodiment adopts a bilateral filtering method combined with feature correlation, and uses anisotropic Gaussian distribution spatial kernel parameters to effectively alleviate slight noise interference, which is different from the commonly used isotropic Gaussian distribution that is independent of the features. When considering the adaptation to different local image features, the spatial domain blur kernel can be expressed as: Where |Σ| represents the determinant of covariance Σ; For example, Figure 2 Part (a) of shows a set of anisotropic Gaussian distributions, Figure 2 Part (b) shows a set of spatial kernel examples that are adjusted according to the details of the image itself. It can be seen that the neural network successfully learns adaptive kernel parameters for different image details and smoothes the image while protecting the edges.

[0046] During the model training process, the output of the neural network is rotated 4 times and then integrated to obtain the final result. The output result of the output layer can be expressed as: Among them, x i represents the i-th image block of the input containing noise interference, f p represents the network kernel parameter, R j represents the rotation operation, j=0,1,2,3, Denotes the derotation operation, and BF denotes the bilateral filtering operation. The network kernel parameters are passed through the loss function loss = MSE (y', y). MSE denotes mean square error.

[0047] It should be noted that, in order to improve the ability of the neural network to adapt to image features, the present invention adopts edge-sensitive convolution kernel forms (i.e. forms X, C, S). In this way, the kernel parameters within the bilateral filtering range can dynamically adapt to the local edges of the image. Figure 1 Step S1 shows the adaptability of various convolution kernel forms for image localization. Compared with edge-insensitive kernel forms, the kernel form used in the present invention can better keep consistent with the image edge, thereby helping the neural network to more accurately predict the parameters of the anisotropic Gaussian distribution. In addition, during the neural network training process, the present invention rotates and combines three different convolution kernel forms to obtain a larger receptive field.

[0048] It should be noted that obtaining the nuclear parameter lookup table for bilateral filtering includes: obtaining the nuclear parameter tables LUT-S, LUT-C, and LUT-X corresponding to the morphology S, morphology C, and morphology X; integrating the nuclear parameter tables LUT-S, LUT-C, and LUT-X to obtain a complete nuclear parameter table LUT; wherein the pixels in the nuclear parameter table LUT are pixels sampled at equal intervals. In the nuclear parameter table LUT, one pixel corresponds to a set of nuclear parameters, and this nuclear parameter table LUT is stored in the model. In actual use, for the input pixel, a corresponding set of nuclear parameters is found in the nuclear parameter table LUT for bilateral filtering. It should be explained that, in order to balance the storage size of the receptive field and the kernel parameter table during model training, this embodiment adopts an equidistant sampling method, for example, within the range of 0 to 255 pixels, the sampled pixels are 0, 17, 33, 49, etc. Therefore, when in use, this embodiment uses the four-sided interpolation method to interpolate and fill in the missing pixels in the kernel parameter table LUT; in the filled kernel parameter table LUT, the pixel K corresponding to the input pixel is searched, and the kernel parameter corresponding to the pixel K is obtained, and the input pixel is bilaterally filtered using the kernel parameter. The four-sided interpolation method does not involve resource-intensive value-added quantification, and maximizes the interpolation efficiency by comparing the size.

[0049] The technical effect of this embodiment is illustrated below through test cases.

[0050] During testing, the Waterloo Exploration database was used for training, and Gaussian noise with a standard deviation of 0.1 and a mean of 0.0 was added as noise interference in the quantitative test. The test data set uses five data sets: Set5, BSDS100, Urban100, Adobe FiveK, and Manga-109 to evaluate the smoothing effect of this embodiment and other bilateral filters in the face of slight noise disturbances. During training, PyTorch is used to implement the training part of the neural network. At the same time, training is performed on a single NVIDIA GTX 3090TI GPU and an Inter(R) Core(TM) i7-12700F CPU, with a batch size of 32 and a learning rate of 10 -3 Initially, each training round is multiplied by epoch represents the current round number (ranging from 1 to 60,000). All networks use the Adam optimizer for parameter optimization, the maximum number of training rounds is 60,000, and the Kaiming initialization method is used to initialize the parameters.

[0051] The present invention compares the quantitative (as shown in Table 2) and qualitative (as shown in Table 3) comparison results of this embodiment and the recently proposed bilateral filtering methods (BF, OFABF, GABF, TPBF, FABF, A2CBF) on multiple data sets to evaluate their effects in smoothing noise interference.

[0052] Please refer to Table 2. From the perspective of the PSNR (peak signal-to-noise ratio) evaluation index, this embodiment is superior to most existing bilateral filtering methods. Compared with the more complex A2CBF, this embodiment only requires table lookup operations during the test process, making the entire process simpler, while A2CBF requires more parameters to train the neural network to implement the adaptive width strategy. In addition, the multi-step decision method used in the test phase causes the demand for computing resources of A2CBF to increase linearly with the increase in decision steps. In contrast, this embodiment can successfully smooth the image even in the presence of slight noise influence, and does not require additional computing modules. Figure 3 The comparison results of smoothing processing of this embodiment and other filtering methods are shown. It can be seen that under the influence of noise, this embodiment effectively retains the edge of the image, while other filtering methods do not have good effects.

[0053] Please refer to Table 3. The first column shows different parameter combinations. W / o is the abbreviation of without, which means that there is no such parameter. Fixed range kernel parameters or fixed spatial kernel parameters will lead to performance degradation. In addition, removing the edge-sensitive index obviously leads to the most obvious performance degradation. Figure 4The filter visual effects under different parameter combinations are shown. It can be seen that although the fixed range kernel parameters retain details, it also causes smoothing failure. The fixed spatial kernel parameters cannot capture enough details. Removing the edge-sensitive index module will cause the edges of the smoothed image to be blurred. Figure 5 In FIG. 1 , the difference in the effects of this embodiment and the conventional bilateral filter in image smoothing and edge preservation under different degrees of Gaussian noise disturbance is also shown. It can be seen that as the noise increases, the basic bilateral filtering algorithm with fixed global parameters gradually loses the ability to resist disturbance. In contrast, this embodiment can successfully predict different kernel parameters based on the texture differences of the image itself. In this way, the image can be smoothed to the maximum extent while retaining the edges of the image.

[0054] Table 2

[0055]

[0056] Table 3

[0057]

[0058] In summary, the image bilateral filtering method based on the kernel parameter lookup table of the present invention can improve the problem of smoothing failure caused by noise interference in the image smoothing task by searching the most suitable kernel parameter in the kernel parameter table to filter the current pixel. Compared with the existing filtering method, the present invention has better filtering and smoothing effect and high processing efficiency.

[0059] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An image bilateral filtering method based on a kernel parameter lookup table, characterized in that: include: S1, training kernel parameter prediction model; S2. Obtain a bilateral filtering kernel parameter lookup table based on the trained kernel parameter prediction model; S3. According to the kernel parameter lookup table, select the optimal kernel parameter for each input pixel to perform bilateral filtering.

2. The image bilateral filtering method based on a kernel parameter lookup table according to claim 1, characterized in that: The kernel parameter prediction model includes: input network layer, intermediate network layer and output layer; The input network layer includes three input channels, and the convolution kernel forms of the three input channels are respectively form S, form C and form X, the form S is 2*2 arrangement, the form C is 1*4 and arranged in a horizontal straight line, and the form X is 1*4 and arranged in an inclined straight line; The convolution kernel size of the intermediate network layer is 1*1, and the ReLU activation function is used; The output layer contains four output channels, corresponding to four kernel parameters σ r , σ sx , σ sy and ρ, σ r represents the standard deviation of the range-domain Gaussian distribution, σ sx and σ sy They represent the x-direction standard deviation and y-direction standard deviation of the Gaussian distribution in the spatial domain, and ρ represents the correlation of two-dimensional variables.

3. The image bilateral filtering method based on the kernel parameter lookup table according to claim 2, characterized in that: In the input network, rotation operations are performed simultaneously on the three types of convolution kernels, with rotation angles of 0°, 90°, 180°, and 270°.

4. The image bilateral filtering method based on a kernel parameter lookup table as claimed in claim 2, characterized in that: The bilateral filtering process for the input image I can be expressed as: Where p=(p x ,p y ) represents the position of the central pixel, q=(q x ,q y ) represents the position of the surrounding pixels, I p and I q Indicates the pixel value of the pixel at position p and the pixel at position q, G Σ represents the Gaussian distribution in the spatial domain, represents the range domain Gaussian distribution, W p represents the normalization factor, and U represents the pixel neighborhood set.

5. The image bilateral filtering method based on kernel parameter lookup table according to claim 4, characterized in that: in, |Σ| represents the determinant of covariance Σ; 6. The image bilateral filtering method based on a kernel parameter lookup table according to claim 5, characterized in that: The output result of the output layer can be expressed as: Among them, x i represents the i-th image block of the input containing noise interference, f p represents the network kernel parameter, R j represents the rotation operation, j=0,1,2,3, represents the derotation operation, and BF represents the bilateral filtering operation.

7. The image bilateral filtering method based on kernel parameter lookup table according to claim 6, characterized in that: The network kernel parameters are transferred via the loss function loss=MSE(y',y).

8. The image bilateral filtering method based on kernel parameter lookup table according to claim 2, characterized in that: Get the kernel parameter lookup table for bilateral filtering, including: Obtain the kernel parameter tables LUT-S, LUT-C, and LUT-X corresponding to the form S, form C, and form X; Integrate the core parameter tables LUT-S, LUT-C, and LUT-X to obtain a complete core parameter table LUT; The pixels in the kernel parameter table LUT are pixels sampled at equal intervals.

9. The image bilateral filtering method based on a kernel parameter lookup table as claimed in claim 8, characterized in that: In step S3, the optimal kernel parameters are selected for each input pixel to perform bilateral filtering, including: Using the four-sided interpolation method to interpolate and fill in the missing pixels in the kernel parameter table LUT; The pixel K corresponding to the input pixel is searched in the padded kernel parameter table LUT, and the kernel parameter corresponding to the pixel K is obtained, and the input pixel is subjected to bilateral filtering using the kernel parameter.

Images