An image smoothing method based on region-fragment smoothing and edge consistency constraints

Abstract

The present invention provides an image smoothing method with regional fragmentation smoothing and edge consistency constraints, comprising the following steps: using a feature extraction module to extract features from an input image by expanding the sensory field; transmitting the extracted features to an edge extraction module to mine more significant image edge information; combining the mined edge information and the extracted features at the feature level, and transmitting the combined feature information to an image smoothing module to obtain a preliminary smoothing result; combining the extracted features, mined edge information, and preliminary smoothing result at the same level, and then transmitting the combined feature information to a weak structure enhancement module to obtain weak structure information; and combining the weak structure information with the preliminary smoothing result to obtain a final smoothing result. This method can more fully mine and utilize significant edge information and weak structure details, suppress the generation of edge artifacts, and simultaneously achieve a better smoothing effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image smoothing, and in particular to an image smoothing method with regional fragmentation smoothing and edge consistency constraints. Background Art

[0002] Image smoothing is a key task in computer vision. In the era of big data, images often carry vast amounts of information. However, during image formation, transmission, reception, and processing, external and internal interference is unavoidable. For example, quantization noise during digitization, errors during image transmission, and various human factors can all cause image blur and loss of image feature detail. Therefore, preserving significant edge structure information while filtering out texture and noise is crucial. To address this, researchers have proposed image smoothing methods based on deep learning.

[0003] Deep network models can fully leverage large amounts of data and hierarchical structures to extract image features, thus enabling image smoothing techniques to achieve better results and generalization capabilities. In 2015, Xu et al. proposed a CNN-based filter that uses the gradient domain to preserve edge information. In 2017, Chen et al. used a fully convolutional network to learn various edge-preserving image smoothing methods, leveraging the fully convolutional network to learn global image features. In 2018, Zhu et al. used a deep network to extract salient structural features of an image and then iteratively solved an optimization problem to achieve smoothing results. In 2019, Li et al. designed a learning-based convolutional neural network framework for constructing joint filters. In 2021, Feng et al. constructed a network model using residual mixed dilated convolutions to effectively extract image feature information, allowing the subsequent network to better learn and preserve detailed features during the smoothing process. In 2022, Pan et al. proposed a dual convolutional network to address the problem of a single network architecture's inadequate processing of image smoothing tasks. The dual network branches are used to process image structure and detail information, respectively. These methods provide new insights into deep learning-based image smoothing methods and achieve promising results.

[0004] However, the inventors of this application have discovered several deficiencies in existing methods: a) They fail to fully utilize significant edge information in the image and provide insufficient protection for weak structural details, resulting in the loss of edge structure information during the smoothing process. b) They retain edges using only residual connections or a combination of edge extraction and smoothing, which introduces noise or pseudo-edges that interfere with model learning. c) They impose a single constraint on the smoothing result, ignoring the diversity of image content and edge changes during the convolution process, which can cause edge artifacts in the smoothed result or unsatisfactory smoothing effects.

[0005] In view of some problems existing in the above methods, it is particularly important to design a deep learning image smoothing method with better edge preservation and smoothing performance. Summary of the Invention

[0006] In order to address the technical problems that existing deep learning-based image smoothing methods lose edge structure information during the smoothing process, introduce noise or pseudo edges that interfere with model learning, and cause edge artifacts in the smoothing results or unsatisfactory smoothing effects, the present invention provides an image smoothing method with regional fragmentation smoothing and edge consistency constraints. This method can more fully mine and utilize significant edge information and weak structural details, and can suppress the generation of edge artifacts, while achieving better smoothing effects.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] An image smoothing method with region-fragment smoothing and edge consistency constraint comprises the following steps:

[0009] The feature extraction module is used to extract features from the input image by expanding the sensory field;

[0010] The features extracted by the feature extraction module are transferred to the edge extraction module, and the edge extraction module is used to detect the edge of the image and mine more significant image edge information;

[0011] The edge information mined by the edge extraction module and the features extracted by the feature extraction module are combined at the feature level, and the combined feature information is passed to the image smoothing module to filter out image texture details to obtain a preliminary smoothing result;

[0012] The features extracted by the feature extraction module, the edge information mined by the edge extraction module and the preliminary smoothing result obtained by the image smoothing module are cascaded and combined, and then transmitted to the weak structure enhancement module to obtain weak structure information;

[0013] The final smoothing result is obtained by combining the weak structure information obtained by the weak structure enhancement module with the preliminary smoothing result obtained by the image smoothing module.

[0014] Furthermore, the method further comprises: using an edge consistency constraint module to constrain the input image and the final smoothing result using an edge consistency method to retain significant edge structure information of the image.

[0015] Furthermore, the method also includes: utilizing the difference between the final smoothing result and the true value edge response feature through the regional fragmentation smoothing module, dividing different regions and selecting different norms to constrain the final smoothing result to improve the image smoothing quality.

[0016] Furthermore, the feature extraction module is composed of a combination of sequentially connected convolutional layers and multiple residual mixed dilated convolutional layers, and the weak structure enhancement module is composed of a combination of sequentially connected convolutional layers, multiple residual mixed dilated convolutional layers and a convolutional layer.

[0017] Furthermore, the edge extraction module is used to detect the image edge and mine more significant image edge information, and the high-precision and low-noise-sensitivity Canny edge detection algorithm is used to calculate the result E of the algorithm. GT , as the true value of the edge prediction information E output by the edge extraction module, the loss function of the edge extraction module is as follows:

[0018]

[0019] Among them, L E is the loss function of the edge extraction module.

[0020] Furthermore, the image smoothing module filters out the image texture details to obtain the preliminary smoothing result, and uses four representative traditional methods, namely L0 smoothing, L1 smoothing, weighted median filtering, and relative total variation filtering, to generate the true value, which is recorded as S GT , used to constrain the preliminary smoothing result obtained by the image smoothing module. The loss function of the image smoothing module is as follows:

[0021]

[0022] Among them, L pre is the loss function of the image smoothing module, S pre is the preliminary smoothing result, S GT is the true value corresponding to the preliminary smoothing result.

[0023] Furthermore, the weak structure information obtained by the weak structure enhancement module and the preliminary smoothing result obtained by the image smoothing module are combined to obtain the final smoothing result. Four representative traditional methods, namely L0 smoothing, L1 smoothing, weighted median filtering, and relative total variation filtering, are used to generate the true value, which is recorded as S GT , which is used to constrain the final smoothing result obtained by combining the preliminary smoothing result with the weak structural information, where the loss function L final As shown below:

[0024]

[0025] Among them, S final It is the final smoothing result obtained by combining the preliminary smoothing result with the weak structural information.

[0026] Furthermore, in the edge consistency constraint module, the input image and the final smoothed result are constrained by an edge consistency method. The edge consistency constraint module uses the local gradient amplitudes of the input image and the output image as the edge response map, and makes the edge response map of the output image, i.e., the final smoothed result, as similar as possible to the edge response map of the input image by analyzing the similarity relationship between the edge response maps of the two. The local gradient amplitude of the image is calculated as follows:

[0027]

[0028]

[0029] Among them, T i (I) is the edge response map of the input image I, T i (S) is the edge response map of the output image S, N(i) is the neighborhood of pixel i, c represents the number of channels of the image, I i,k is the pixel i of the kth channel in the input image I, I j,k is the pixel j in the neighborhood of pixel i in the kth channel of the input image I, S i,k is the pixel i of the kth channel in the output image S, S j,k is the pixel j in the neighborhood of pixel i in the kth channel of the output image S;

[0030] After calculating the edge response maps of the input image and the output image, the similarity relationship between the edge response maps of the input image and the output image is expressed as the L2 norm of the two in the loss function. The following correlation judgment loss function is used for constraint:

[0031]

[0032] Among them, L ECM is the correlation determination loss function, N is the total number of pixels of the input image I or the output image S, is the total number of pixels contained in the important edge, that is, the significant edge, M i is the decision weight of important edges.

[0033] Furthermore, the regional fragmentation smoothing module processes different regions in an adaptive constraint manner, and uses the difference between the output image, i.e., the final smoothed result, and the true edge response feature to divide different regions and select different norms to solve the image smoothing problem. The following variable norm loss function is used for constraint:

[0034]

[0035] Among them, L RFSM is a variable paradigm loss function, N is the total number of pixels of the input image I or the output image S, Nr (i) represents the pixel set within the r×r neighborhood of the pixel point, Z i,j Represents the weight of the pixel pair, S i is the pixel i in the output image S, S i is the pixel j in the neighborhood of pixel i in the output image S, |·| p For L p norm.

[0036] Furthermore, the Z representing the weight of the pixel point i,j Divided into color space calculation weights and position space calculation weights

[0037]

[0038]

[0039] Where exp(·) represents the exponential function with the natural constant e as the base, S i,c is the pixel i in the cth channel of the output image S, S j,c is the pixel j in the neighborhood of pixel i in the c-th channel of the output image S, Represents the Gaussian standard deviation calculated in the color space, x, y are the spatial coordinates of the final smoothed pixel point, Represents the Gaussian standard deviation computed in position space.

[0040] Compared with the prior art, the image smoothing method with regional fragmentation smoothing and edge consistency constraint provided by the present invention has the following advantages:

[0041] (1) This paper proposes an Edge Consistency Region Fragment Smooth Network (ECRFS-Net), which is the first model to use the consistency relationship of edge features in supervised learning, combine the idea of ​​regional fragment smoothing, and make full use of significant edge information and weak structural details to achieve edge protection.

[0042] (2) An edge consistency constraint module (ECM) is proposed, which can effectively mine and utilize the important edge information of the image itself, map the consistency relationship between edge features to the image smoothing result, and thus effectively retain the significant edge structure information.

[0043] (3) A Region Fragment Smooth Model (RFSM) is proposed, which can be effectively combined with the current supervision model and adopts a flexible approach of regional feature constraints to effectively improve the image smoothing quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of the edge consistency region sharding smoothing network model provided by the present invention.

[0045] Figure 2 It is a schematic diagram of the specific structure of the rHDC layer provided by the present invention.

[0046] Figure 3 This is a comparison chart of the smoothing results of various algorithms for the first image provided by the present invention.

[0047] Figure 4 This is a comparison chart of the smoothing results of various algorithms for the second image provided by the present invention.

[0048] Figure 5 This is a comparison chart of the smoothing results of various algorithms for the third image provided by the present invention.

[0049] Figure 6 This is a comparison chart of the smoothing results of various algorithms for the fourth image provided by the present invention.

[0050] Figure 7 This is a comparison chart of the smoothing results of various algorithms for the fifth image provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below with reference to specific illustrations.

[0052] The overall structure of the edge consistency region fragmentation smoothing network (ECRFS-Net) proposed in this invention is as follows: Figure 1 As shown. The overall network consists of six modules, including a feature extraction module (FeatureExtractor) that extracts high-level semantic information of the image, an edge extraction module (Edge Extractor) that detects and mines image edges, an image smoothing module (Smoother) that filters out image texture details, a weak structure enhancement module (WSR) that protects and strengthens the weak structural information of the image, an edge consistency constraint module (ECM) that retains the edge structural information of the image, and a regional fragment smoothing module (RFSM) that improves smoothing quality and suppresses artifacts. According to the overall network model, the present invention provides an image smoothing method with regional fragment smoothing and edge consistency constraints, comprising the following steps:

[0053] A feature extraction module is used to extract features from the input image by expanding the receptive field to learn more high-level semantic information of the image. Specifically, the feature extraction module is composed of sequentially connected convolutional layers (ConV) and multiple residual hybrid dilated convolutional (rHDC) layers, aiming to extract deeper high-level semantic information of the image.

[0054] The feature F extracted by the feature extraction module is passed to the edge extraction module, and the edge extraction module is used to detect the edge of the image and mine more significant image edge information;

[0055] In order to make full use of the significant edge information in the image and retain as much important edge information as possible during the smoothing process, the edge information mined by the edge extraction module and the feature F extracted by the feature extraction module are combined at the feature level, and the combined feature information is passed to the image smoothing module to filter out the image texture details, and the preliminary smoothing result S is obtained. pre ;

[0056] Image features often contain structural detail information. In order to make full use of the feature information of each feature layer and extract the weak structural information in the image, the features extracted by the feature extraction module, the edge information mined by the edge extraction module, and the preliminary smoothing result obtained by the image smoothing module are cascaded and combined, and then passed to the weak structure enhancement module to obtain the weak structural information W;

[0057] In order to reduce the structural detail information lost in the image smoothing process and enhance it, the weak structure information obtained by the weak structure enhancement module is combined with the preliminary smoothing result obtained by the image smoothing module to obtain the final smoothing result S final , in order to achieve the purpose of protecting weak structure information and enhancing edge preservation effect. The weak structure enhancement module is composed of sequentially connected convolutional layers, multiple residual hybrid dilated convolutional (rHDC) layers and convolutional layers. The specific structure of the rHDC layer is as follows: Figure 2 shown.

[0058] As a specific embodiment, the method also includes: in order to further improve the image smoothing quality and retain the image's significant edges and weak structural information, the edge consistency constraint module (ECM) makes full use of the image's own significant edge information, and constrains the input image and the final smoothing result using an edge consistency method to retain more image significant edge structure information.

[0059] As a specific embodiment, the method also includes: in order to solve the problem that the single paradigm constraint of the ground truth (GT) and the smoothing result causes edge artifacts in the output image, the regional fragmentation smoothing module (RFSM) utilizes the difference between the final smoothing result and the true value edge response characteristics, divides different regions and selects different norms to constrain the final smoothing result to improve the image smoothing quality.

[0060] As a specific embodiment, in order to extract more significant edge information, namely important edge information, the edge extraction module is used to detect the edge of the image and mine more significant image edge information, and the high-precision and low-noise-sensitivity Canny edge detection algorithm is used to calculate the result E of the algorithm. GT , as the true value of the edge prediction information E output by the edge extraction module to achieve better edge extraction effect, the loss function of the edge extraction module is as follows:

[0061]

[0062] Among them, L E is the loss function of the edge extraction module.

[0063] As a specific embodiment, considering the problem that it is difficult to determine the true value in image smoothing, the image smoothing module filters out the image texture details to obtain the preliminary smoothing result, and uses four representative traditional methods: L0 smoothing, L1 smoothing, weighted median filtering (WMF), and relative total variation filtering (RTV) to generate the ground truth (Ground Truth), denoted as S GT , used to constrain the preliminary smoothing result obtained by the image smoothing module. The loss function of the image smoothing module is as follows:

[0064]

[0065] Among them, L pre is the loss function of the image smoothing module, S pre is the preliminary smoothing result, S GT is the true value corresponding to the preliminary smoothing result.

[0066] As a specific embodiment, from the perspective of protecting weak structure information and enhancing edge preservation effect, the weak structure information obtained by the weak structure enhancement module and the preliminary smoothing result obtained by the image smoothing module are combined to obtain the final smoothing result. Four representative traditional methods, namely L0 smoothing, L1 smoothing, weighted median filtering (WMF), and relative total variation filtering (RTV), are used to generate the ground truth (Ground Truth), denoted as S GT , which is used to constrain the final smoothing result obtained by combining the preliminary smoothing result with the weak structural information, where the loss function Lfinal As shown below:

[0067]

[0068] Among them, S final It is the final smoothing result obtained by combining the preliminary smoothing result with the weak structural information.

[0069] As a specific embodiment, the present invention takes the edge similarity between input image and output image as the starting point and proposes a method of edge consistency constraint, such as Figure 1 The ECM module in the embodiment uses the edge consistency constraint module to constrain the input image and the final smoothed result using the edge consistency method. The edge consistency constraint module uses the local gradient amplitude of the input image and the output image as the edge response map. By analyzing the similarity relationship between the edge response maps of the two, the edge response map of the output image, i.e., the final smoothed result, is made as similar as possible to the edge response map of the input image, thereby retaining important edge structure information. The local gradient amplitude of the image is calculated as follows:

[0070]

[0071]

[0072] Among them, T i (I) is the edge response map of the input image I, T i (S) is the edge response map of the output image S, N(i) is the neighborhood of pixel i, c represents the number of channels of the image, such as c = 3, I i,k is the pixel i of the kth channel in the input image I, I j,k is the pixel j in the neighborhood of pixel i in the kth channel of the input image I, S i,k is the pixel i of the kth channel in the output image S, S j,k is the pixel j in the neighborhood of pixel i in the kth channel of the output image S;

[0073] After calculating the edge response maps of the input image and the output image, the similarity relationship between the edge response maps of the input image and the output image is expressed as the L2 norm of the two in the loss function. The following correlation judgment loss function is used for constraint:

[0074]

[0075] Among them, L ECM is the correlation determination loss function, N is the total number of pixels of the input image I or the output image S, is the total number of pixels contained in the important edge, that is, the significant edge, M i is the decision weight of important edges.

[0076] The edge-preserving image smoothing method needs to retain some significant edge information. Retaining too many pseudo edges will affect the network performance and smoothing results. Therefore, it is necessary to filter out significant / insignificant edge information, that is, to make the edge response of the final smoothed image as consistent as possible with the significant edge response in the input image. In formula (6), the judgment weight of the important edge is set to M i , when the edge is identified as an important edge, M i =1, otherwise M i =0.

[0077] As a specific embodiment, the regional fragmentation smoothing module processes different regions in an adaptive constraint manner, divides different regions into different areas based on the difference between the output image, i.e., the final smoothed result, and the true edge response feature, and selects different norms to solve the image smoothing problem. The following variable norm loss function is used for constraint:

[0078]

[0079] Among them, L RFSM is a variable paradigm loss function, N is the total number of pixels of the input image I or the output image S, N r (i) represents the pixel set within the r×r neighborhood of the pixel point, Z i,j Represents the weight of the pixel pair, S i is the pixel i in the output image S, S i is the pixel j in the neighborhood of pixel i in the output image S, |·| p For L p norm.

[0080] As a specific embodiment, the Z representing the weight of the pixel point i,j That is, Z in formula (7) i,j , which can be divided into color space calculation weights and position space calculation weights

[0081]

[0082]

[0083] Where exp(·) represents the exponential function with the natural constant e as the base, S i,c is the pixel i in the cth channel of the output image S, S j,c is the pixel j in the neighborhood of pixel i in the c-th channel of the output image S, Represents the Gaussian standard deviation calculated in the color space, x, y are the spatial coordinates of the final smoothed pixel point, Represents the Gaussian standard deviation computed in position space.

[0084] As a specific embodiment, to determine different areas in an image, the edge response map in the edge consistency constraint module is used to help distinguish different image areas, distinguish the value of p and the weight value of each pixel pair:

[0085]

[0086] Among them, T i (GT) represents the edge response map of the final smoothed result corresponding to the true value (GT), α1 is the first edge response strength threshold, which can be set to 20 according to experience; α2 is the second edge response strength threshold, which can be set to 10 according to experience.

[0087] By comparing the difference between the edge response features of the input image and the final smoothed result, the image is divided into different regions. The first discriminant in Equation (10) represents the region where the pixel gradient in the ground truth (GT) is relatively flat and the edge response features of the final smoothed result are similar to those of the GT. For these relatively flat image regions with similar color values ​​of the surrounding pixels, the color space information of the pixels needs to be considered and a smaller L is assigned. 0.8 Norm penalty weight. Therefore, the present invention adopts L 0.8 It is constrained by a combination of the norm and the pixel color space weight.

[0088] However, all regions in the image use L 0.8 The penalty weight of the norm can lead to a piecewise constant effect, causing the image to appear oversharpened or have edge artifacts. This is the situation described by the second discriminant in Equation (10): for areas with a small edge response in the ground truth but a significantly enhanced edge response in the output, for such areas of the image that should be flat but have strong edge responses in the surrounding areas, it is necessary to consider the spatial position information of the pixels in this area and assign a larger L2 norm penalty weight. Therefore, a combination of the L2 norm and the pixel position spatial weight is used for constraint.

[0089] Therefore, the overall loss function of the ECRFS-Net network model is:

[0090] L total =L E +L pre +L final +L ECM +L RFSM Formula (11)

[0091] In order to verify the effectiveness of the ECRFS-Net network model and each module, the present invention conducts an ablation experiment on the effectiveness of the ECM, RFSM, and WSR modules on the BSDS500 image dataset. The results are shown in Table 1 below.

[0092] Table 1

[0093]

[0094] It can be seen from Table 1 that each module proposed in the present invention can achieve different degrees of performance improvement of the network model, verifying the rationality and effectiveness of each module. Among them, the edge consistency constraint module and the regional fragmentation smoothing module can effectively improve the network performance and complement each other, so that the PSNR of the network model is improved by 0.88 and the SSIM is improved by 0.04. This shows that the edge consistency constraint module can effectively retain significant edge information, and the use of edge consistency constraints is effective; and the regional fragmentation smoothing module can further constrain the smoothing results. After adding the feature extraction module and the weak structure enhancement module, the PSNR of the network model is improved by 0.2 and the SSIM is improved by 0.01, indicating that the feature extraction module can better utilize the information at the image feature level, and the weak structure enhancement module can protect the fine structure of the image on the basis of the edge consistency constraint module, further improving the performance of the model.

[0095] The proposed method is compared with 12 representative existing image smoothing methods, including two classic local filtering methods (GF and RGF), six global filtering methods (L0, RTV, SDF, PNLS, ILS, and G-smooth), and four representative deep learning-based methods (DEAF, DRF, ResNet, and DualCNN). It should be noted that the DEAF and DualCNN network models were retrained on the BSDS300 dataset. To quantitatively evaluate the performance of the smoothing algorithms, the proposed ECRFS-Net model and other comparison methods were tested on the BSDS500, NKS, and SPS datasets using the PSNR and SSIM metrics. Both the NKS and SPS datasets contain paired input and ground-truth images, allowing for quantitative comparison. Although the BSDS500 dataset lacks ground-truth images, the smoothed images are typically evaluated using subjective human judgment. Therefore, visual comparisons of several images and magnified regions are presented to facilitate intuitive visual comparison.

[0096] Table 2 below shows the comparative experimental results of various methods on the BSDS500, NKS, and SPS datasets using PSNR and SSIM metrics, with the maximum values ​​in bold. As can be seen from Table 2, ECRFS-Net demonstrates significant advantages across all metrics across the different datasets tested. This is attributed to ECRFS-Net's ability to leverage significant image edge information and further improve performance through the synergistic effects of feature extraction, ECM, RFSM, and WSR. The ECM module further preserves image edge information, the RFSM more finely delineates smooth regions, and the feature extraction module and WSR fully utilize deep image features. ECRFS-Net performs well on natural images derived from texture synthesis (such as the SPS dataset), and achieves significant performance on hand-drawn cartoon textures (such as the NKS dataset) and natural images without ground truth (such as the BSDS500 dataset), demonstrating its excellent generalization performance.

[0097] Table 2

[0098]

[0099] In order to better verify the effectiveness of the method proposed in this paper, three sets of visual comparison experiments are conducted to specifically demonstrate the performance of the model in image smoothing and its ability to retain weak edge structures. The comparison results are as follows: Figure 3 、 Figure 4 and Figure 5 Specifically, from the three sets of comparison results, it can be seen that the present invention can achieve better results than other methods, not only effectively preserving the edge structure information of the image, but also achieving better smoothing effect.

[0100] Image smoothing, as a technology for image preprocessing, has a wide range of applications. This paper will apply it to image enhancement and edge detection to further illustrate the effectiveness and superiority of the proposed method. Figure 6 The image enhancement effects of ECRFS-Net and various methods are compared. Through the results of local magnification experiments, it can be found that other methods have halo and over-sharpening phenomena in some edge areas, while the present invention can better suppress this phenomenon, mainly due to the ECM's role in sensing and preserving edges. Figure 7 The edge detection results based on various methods are displayed. It can be clearly seen from the results that the smoothing result of the present invention not only retains the main edge structure of the image, but also improves the accuracy and reliability of edge detection.

[0101] In summary, compared with the prior art, the image smoothing method with regional segmentation smoothing and edge consistency constraint provided by the present invention has the following advantages:

[0102] (1) This paper proposes an Edge Consistency Region Fragment Smooth Network (ECRFS-Net), which is the first model to use the consistency relationship of edge features in supervised learning, combine the idea of ​​regional fragment smoothing, and make full use of significant edge information and weak structural details to achieve edge protection.

[0103] (2) An edge consistency constraint module (ECM) is proposed, which can effectively mine and utilize the important edge information of the image itself, map the consistency relationship between edge features to the image smoothing result, and thus effectively retain the significant edge structure information.

[0104] (3) A Region Fragment Smooth Model (RFSM) is proposed, which can be effectively combined with the current supervision model and adopts a flexible approach of regional feature constraints to effectively improve the image smoothing quality.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An image smoothing method with region-slice smoothing and edge consistency constraint, characterized in that: The following steps are involved: The feature extraction module is used to extract features from the input image by expanding the sensory field; The features extracted by the feature extraction module are transferred to the edge extraction module, and the edge extraction module is used to detect the edge of the image and mine more significant image edge information; The edge information mined by the edge extraction module and the features extracted by the feature extraction module are combined at the feature level, and the combined feature information is passed to the image smoothing module to filter out image texture details to obtain a preliminary smoothing result; The features extracted by the feature extraction module, the edge information mined by the edge extraction module and the preliminary smoothing result obtained by the image smoothing module are cascaded and combined, and then transmitted to the weak structure enhancement module to obtain weak structure information; The final smoothing result is obtained by combining the weak structure information obtained by the weak structure enhancement module with the preliminary smoothing result obtained by the image smoothing module; The edge consistency constraint module is used to constrain the input image and the final smoothed result using the edge consistency method to preserve the significant edge structure information of the image; The regional segmentation smoothing module utilizes the difference between the final smoothing result and the true edge response characteristics, divides different regions and selects different norms to constrain the final smoothing result to improve the image smoothing quality. The feature extraction module is composed of a sequentially connected convolutional layer and a plurality of residual mixed dilated convolutional layers, and the weak structure enhancement module is composed of a sequentially connected convolutional layer, a plurality of residual mixed dilated convolutional layers and a convolutional layer.

2. The image smoothing method with regional fragmentation smoothing and edge consistency constraint according to claim 1, characterized in that: In the process of detecting image edges and mining more significant image edge information using the edge extraction module, a high-precision and low-noise-sensitivity Canny edge detection algorithm is used to calculate the result E of the algorithm. GT , as the true value of the edge prediction information E output by the edge extraction module, the loss function of the edge extraction module is as follows: Among them, L E is the loss function of the edge extraction module.

3. The image smoothing method with regional fragmentation smoothing and edge consistency constraint according to claim 1, characterized in that: The image smoothing module filters out the image texture details to obtain the preliminary smoothing result, and uses four representative traditional methods: L0 smoothing, L1 smoothing, weighted median filtering, and relative total variation filtering to generate the true value, which is recorded as S GT , used to constrain the preliminary smoothing result obtained by the image smoothing module. The loss function of the image smoothing module is as follows: Among them, L pre is the loss function of the image smoothing module, S pre is the preliminary smoothing result, S GT is the true value corresponding to the preliminary smoothing result.

4. The image smoothing method with regional fragmentation smoothing and edge consistency constraint according to claim 1, characterized in that: The final smoothing result is obtained by combining the weak structure information obtained by the weak structure enhancement module with the preliminary smoothing result obtained by the image smoothing module. The four representative traditional methods of L0 smoothing, L1 smoothing, weighted median filtering, and relative total variation filtering are used to generate the true value, which is recorded as S GT , which is used to constrain the final smoothing result obtained by combining the preliminary smoothing result with the weak structural information, where the loss function L final As shown below: Among them, S final It is the final smoothing result obtained by combining the preliminary smoothing result with the weak structural information.

5. The image smoothing method with regional fragmentation smoothing and edge consistency constraint according to claim 1, characterized in that: In the edge consistency constraint module, the input image and the final smoothed result are constrained by the edge consistency method. The edge consistency constraint module uses the local gradient amplitude of the input image and the output image as the edge response map, and makes the edge response map of the output image, i.e., the final smoothed result, as similar as possible to the edge response map of the input image by analyzing the similarity relationship between the edge response maps of the two. The local gradient amplitude of the image is calculated as follows: Among them, T i (I) is the edge response map of the input image I, T i (S) is the edge response map of the output image S, N(i) is the neighborhood of pixel i, c represents the number of channels of the image, I i,k is the pixel i of the kth channel in the input image I, I j,k is the pixel j in the neighborhood of pixel i in the kth channel of the input image I, S i,k is the pixel i of the kth channel in the output image S, S j,k is the pixel j in the neighborhood of pixel i in the kth channel of the output image S; After calculating the edge response maps of the input image and the output image, the similarity relationship between the edge response maps of the input image and the output image is expressed as the L2 norm of the two in the loss function. The following correlation judgment loss function is used for constraint: Among them, L ECM is the correlation determination loss function, N is the total number of pixels of the input image I or the output image S, is the total number of pixels contained in the important edge, that is, the significant edge, M i is the decision weight of important edges.

6. The image smoothing method with regional fragmentation smoothing and edge consistency constraint according to claim 1, characterized in that: The regional fragmentation smoothing module processes different regions in an adaptive constraint manner. It divides different regions into different areas and selects different norms to solve the image smoothing problem by using the difference between the output image, i.e., the final smoothed result, and the true edge response feature. The following variable norm loss function is used for constraint: Among them, L RFSM is a variable paradigm loss function, N is the total number of pixels of the input image I or the output image S, N r (i) represents the pixel set within the r×r neighborhood of the pixel point, Z i,j Represents the weight of the pixel pair, S i is the pixel i in the output image S, S i is the pixel j in the neighborhood of pixel i in the output image S, |·| p For L p norm.

7. The image smoothing method with regional fragmentation smoothing and edge consistency constraint according to claim 6, characterized in that: The Z that represents the weight of the pixel point i,j Divided into color space calculation weights and position space calculation weights Where exp(·) represents the exponential function with the natural constant e as the base, S i,c is the pixel i in the cth channel of the output image S, S j,c is the pixel j in the neighborhood of pixel i in the c-th channel of the output image S, Represents the Gaussian standard deviation calculated in the color space, x, y are the spatial coordinates of the final smoothed pixel point, Represents the Gaussian standard deviation computed in position space.

Images