Image processing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an image processing method, and the method comprises the steps: carrying out the noise detection and noise reduction of a to-be-processed image, and obtaining a denoised image; performing multi-scale texture analysis on the denoised image, and extracting texture feature maps under different spatial frequencies; performing local enhancement processing on a weak texture region of the texture feature map through a directional convolution kernel group according to gradient amplitudes and direction information of the texture feature map under different spatial frequencies to generate a detail enhanced image; and inputting the detail enhanced image into a pre-trained deep super-resolution network, performing iterative optimization on high-frequency details of the detail enhanced image through a multi-layer feature fusion module of the deep super-resolution network, and generating an image with a target resolution. According to the technical scheme, noise and texture can be distinguished in a self-adaptive mode, and details of the image are kept complete in the super-resolution process.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Existing image super-resolution techniques face significant challenges when processing low-quality input images. Traditional methods typically employ a single-stage processing flow, performing noise suppression concurrently with super-resolution upscaling. However, this approach suffers from three key drawbacks: first, the noise reduction process is prone to misidentifying and eliminating valid texture information; second, high-frequency details in the upscaled image are not fully reconstructed; and third, it is difficult to effectively distinguish texture components at different spatial frequencies, resulting in residual noise particles in flat areas of the upscaled image. In short, existing techniques struggle to maintain the integrity of details in the upscaled image during the super-resolution process. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, and storage medium that can adaptively distinguish between noise and texture, and maintain the integrity of image details during super-resolution.

[0004] On one hand, this application provides an image processing method, the method comprising:

[0005] Noise detection and denoising are performed on the image to be processed to obtain the denoised image;

[0006] Multi-scale texture analysis is performed on the denoised image to extract texture feature maps at different spatial frequencies;

[0007] Based on the gradient magnitude and direction information of the texture feature map, a group of directional convolution kernels is used to perform local enhancement processing on the weak texture regions of the texture feature map to generate a detail-enhanced image.

[0008] The enhanced detail image is input into a pre-trained deep super-resolution network. The high-frequency details of the enhanced detail image are iteratively optimized through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

[0009] On the other hand, this application provides an image processing apparatus, the apparatus comprising:

[0010] The noise reduction module is used to detect and reduce noise in the image to be processed, and obtain the noise-reduced image.

[0011] The extraction module is used to perform multi-scale texture analysis on the denoised image and extract texture feature maps at different spatial frequencies;

[0012] The enhancement module is used to perform local enhancement processing on the weak texture regions of the texture feature map by using a group of directional convolution kernels based on the gradient magnitude and direction information of the texture feature map, thereby generating a detail-enhanced image;

[0013] An optimization module is used to input the detail-enhanced image into a pre-trained deep super-resolution network, and to iteratively optimize the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

[0014] Thirdly, this application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the technical solution of the image processing method described above.

[0015] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the technical solution of the image processing method described above.

[0016] As can be seen from the technical solution provided in this application, on the one hand, after noise reduction processing of the image to be processed, multi-scale texture analysis is performed on the denoised image. This effectively eliminates noise patches while extracting texture feature maps at different spatial frequencies, comprehensively capturing the feature distribution from macroscopic texture structure to microscopic details, ensuring that texture components at different scales are appropriately enhanced in subsequent stages. On the other hand, by using directional convolutional kernel groups to locally enhance weak texture regions, the enhancement direction can be adaptively adjusted according to the texture direction, significantly improving the reconstruction quality of textures with directional features. Thirdly, compared to traditional methods of single feature extraction, high-frequency detail iterative optimization can more accurately reconstruct detail information that conforms to the statistical characteristics of natural images, reducing artifact generation. In summary, the technical solution of this application can adaptively distinguish between noise and texture, and maintain the integrity of image details during the super-resolution process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the image processing method provided in the embodiments of this application;

[0019] Figure 2This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element, component, or step (etc.) should not be construed as limited to only one element, component, or step, but may include one or more of the elements, components, or steps, etc.

[0023] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0024] Existing image super-resolution techniques face significant challenges when processing low-quality input images. Traditional methods typically employ a single-stage processing flow, performing noise suppression concurrently with super-resolution upscaling. However, this approach suffers from two key drawbacks: first, noise and real texture overlap in the frequency domain, leading to misjudgment and the elimination of effective texture information during denoising; second, while the upscaled image achieves increased resolution, high-frequency details are not fully reconstructed, resulting in artifacts such as texture structure breaks and blurred edges. Furthermore, when processing multi-scale texture features, traditional methods struggle to effectively distinguish texture components with different spatial frequencies due to the use of single-scale convolutional kernels for feature extraction. This approach results in noise particles remaining in flat areas of the upscaled image while losing detail in high-frequency texture regions. In short, existing techniques struggle to maintain the integrity of details in upscaled images during super-resolution.

[0025] To address the aforementioned problems in the prior art, this application proposes an image processing method, the flowchart of which is attached. Figure 1 As shown, the main steps include S101 to S104, which are detailed below:

[0026] Step S101: Perform noise detection and denoising on the image to be processed to obtain the denoised image.

[0027] Traditional denoising algorithms (such as Gaussian filtering and median filtering) typically perform uniform processing on the entire image, which can easily lead to the loss of details in non-noise areas (such as blurred edges and weakened textures), while residual noise in noisy areas may affect the subsequent super-resolution effect. For example, subsequent deep super-resolution networks are sensitive to noise in the input image. If denoising is insufficient, the noise will be amplified during network iterations; if denoising is excessive, high-frequency details will be lost, limiting the network's ability to recover textures. Therefore, noise detection and denoising processing can be performed on the image to be processed, and after obtaining the denoised image, multi-scale texture analysis can be further performed on the denoised image. Specifically, as an embodiment of this application, noise detection and denoising processing of the image to be processed to obtain the denoised image can be achieved through steps S1011 and S1012, which are described in detail below:

[0028] Step S1011: Perform adaptive noise detection on the image to be processed, and generate a dynamic noise mask based on the detected noise distribution characteristics.

[0029] It should be noted that real-world scene noise typically exhibits non-uniformity (e.g., differences in noise intensity in areas of varying illumination) and time-varying characteristics (e.g., changes in noise distribution under dynamic shooting conditions). Dynamic noise masks, generated based on the noise distribution characteristics of the current image, can adaptively adjust detection parameters (e.g., local variance threshold, frequency domain energy threshold), avoiding the limitations of preset parameters for different scenes. Furthermore, encoding noise distribution characteristics, such as noise intensity and spatial frequency, into the mask provides prior noise information for subsequent steps such as multi-scale texture analysis and directional convolution enhancement, assisting the algorithm in distinguishing between real textures and noise interference. Specifically, as an embodiment of this application, adaptive noise detection of the image to be processed, and generating a dynamic noise mask based on the detected noise distribution characteristics, can be achieved by: performing wavelet transform decomposition on the image to be processed to extract high-frequency subband coefficients; calculating a dynamic noise threshold based on the statistical distribution of the high-frequency subband coefficients; clustering pixels in the image to be processed that exceed the dynamic noise threshold into noise regions using a region growing algorithm; and generating a binary image identifying the location of noise pixels based on the noise regions as the dynamic noise mask. Specifically, based on the statistical distribution of high-frequency subband coefficients, the dynamic noise threshold can be calculated using the formula T = μ + k * σ, where μ is the mean of the high-frequency subband coefficients, σ ​​is the standard deviation of the high-frequency subband coefficients, and k is the gain coefficient dynamically adjusted based on the image signal-to-noise ratio. It should be noted that in an image, the high-frequency subbands carry the rapidly changing parts, such as edges, textures, and noise, etc., where pixel values ​​can change drastically over short distances. High-frequency subband coefficients refer to the mathematical representations of high-frequency details in an image after wavelet transform. To extract high-frequency subband coefficients from the image to be processed by wavelet transform decomposition, the image can be subjected to a first-level wavelet decomposition to filter out low-frequency information and retain high-frequency subband coefficients; a second-level wavelet decomposition can then be performed on the low-frequency components filtered out in the first-level wavelet decomposition to obtain more refined low-frequency information and another set of high-frequency subband coefficients; this process can be repeated to achieve multi-scale analysis. It should also be noted that the dynamic noise mask is a two-dimensional matrix (i.e., an image) with the same size as the original image. In this matrix, the pixel positions identified as noise regions are assigned a value (e.g., 1 or 255), and the pixels in non-noise regions are assigned another value (e.g., 0). Therefore, after identifying the noise regions in the image to be processed using a region growing algorithm, a binary image identifying the locations of noise pixels can be generated based on the noise regions as a dynamic noise mask.

[0030] Step S1012: Based on the dynamic noise mask, a nonlocal mean filtering algorithm is used to selectively denoise the noise region of the image to be processed, while retaining the original pixel information of the non-noise region of the image to be processed, thus obtaining the denoised image.

[0031] Simply using the Nonlocal Mean Filter (NLM) algorithm to denoise noisy regions results in high computational complexity and sensitivity to similar block matching in uniform texture regions. Direct global application leads to inefficiency and the risk of mismatches. If non-noise regions (e.g., sharp edges, regular textures) are mistakenly included in NLM processing, artifacts may be introduced due to incorrect similar block matching. However, by limiting the computational scope of NLM through a dynamic noise mask, block matching and weight calculation are performed only on noisy regions. This significantly reduces algorithm complexity while skipping redundant calculations in non-noise regions, improving real-time performance. Therefore, based on a dynamic noise mask, the NLM algorithm can be used to selectively denoise noisy regions of the image to be processed, preserving the original pixel information of non-noise regions to obtain the denoised image. It should be noted that the granularity of selective noise reduction (e.g., mask resolution, NLM window size) can be dynamically adjusted according to the noise distribution. For example, a finer block matching strategy can be used for high noise density areas, while low noise areas can be processed quickly to improve overall robustness. Specifically, the NLM window size can be adaptively adjusted according to the noise intensity. When the noise intensity is higher than a set threshold, a 5×5 window is used, and when it is lower than the threshold, a 3×3 window is used. In addition, the limitation of the NLM calculation range by the dynamic noise mask mentioned in the above embodiments can be specifically: the dynamic noise mask is morphologically dilated (e.g., dilated by 3 pixels) to form a searchable region mask as a search window; for the target pixel to be processed, when performing similar block matching within the search window, only candidate pixels located within the searchable region mask are considered; candidate pixels falling outside the searchable region mask are directly ignored, and their similarity with the reference block is not calculated. This scheme actively avoids searching for similar blocks outside the noise region, reduces the probability of misjudging irrelevant textures (e.g., a clear edge) as similar blocks, makes the weight allocation more accurate, and the noise reduction effect better.

[0032] As can be seen from the above embodiments, the dynamic noise mask generation mechanism can perform differentiated processing based on the noise distribution characteristics of local regions of the image to be processed. Combined with selective noise reduction using non-local mean filtering, noise patches are effectively eliminated while avoiding the destruction of texture details in non-noise areas by traditional global filtering, thus preserving the details of the original image to the maximum extent while accurately reducing noise.

[0033] Step S102: Perform multi-scale texture analysis on the denoised image to extract texture feature maps at different spatial frequencies.

[0034] As mentioned earlier, existing super-resolution techniques are generally not refined enough in terms of noise reduction and texture enhancement. For example, when using global noise reduction or single-scale texture analysis, if there is overlap between noise and real texture in the frequency domain features, it means that such global noise reduction or single-scale texture analysis cannot adaptively handle different noise and texture structures. To overcome the above-mentioned shortcomings of existing technologies, after obtaining the denoised image, multi-scale texture analysis can be performed on the denoised image to extract texture feature maps at different spatial frequencies. Specifically, step S102 can be implemented through steps S1021 to S1024, as detailed below:

[0035] Step S1021: Use Laplacian pyramid decomposition to generate feature maps of different scales from the denoised image.

[0036] It should be noted that each layer of the Laplacian pyramid is actually the difference between a certain layer of the Gaussian pyramid and the layer above it after upsampling and interpolation (e.g., using bilinear interpolation). It captures detailed information at different scales, i.e., high-frequency components. Therefore, using Laplacian pyramid decomposition to generate feature maps of different scales from the denoised image can be done by first constructing a Gaussian pyramid and then constructing a Laplacian pyramid based on the Gaussian pyramid. The construction of the Gaussian pyramid mainly involves taking the denoised image as the 0th layer (G0) of the pyramid, downsampling G0 (e.g., using Gaussian blur and then sampling every other row and column) to obtain the 1st layer image G1, repeating the above process for G1 to obtain the 2nd layer image G2, and repeating this process to generate a set of images with gradually decreasing resolution {G0, G1, G2, ...}, i.e., the Gaussian pyramid; the construction of the Laplacian pyramid based on the Gaussian pyramid mainly involves: taking the higher layers (lower resolution) of the Gaussian pyramid (e.g., the lower resolution images) of the Gaussian pyramid (e.g., the higher resolution images)... For example, G1 is upsampled (e.g., using bilinear or bicubic interpolation) to make its size the same as the next layer (higher resolution) image (e.g., G0). The upsampled G1 is denoted as Expand(G1). The difference is calculated as L0 = G0 - Expand(G1). L0 is the 0th layer of the Laplacian pyramid, which contains high-frequency details in the G0 image that exist at the G0 scale but not at the G1 scale. Similarly, L1 = G1 - Expand(G2) is calculated to obtain the 1st layer Laplacian image. This process is repeated to generate a set of detail images {L0, L1, L2, ...}, which is the Laplacian pyramid. These L0, L1, L2, ... are feature maps at different scales.

[0037] Step S1022: Perform local binary pattern encoding on feature maps of different scales to generate texture pattern feature maps of different scales.

[0038] First, local binary pattern encoding (LBP) is performed on feature maps at different scales. Taking three sets of feature maps at different scales as examples, the original scale Laplacian feature map is encoded using LBP to generate an original scale texture pattern map, highlighting high-frequency texture details (e.g., skin pores, small scratches). The first downsampled scale Laplacian feature map is encoded using LBP to generate a medium-scale texture pattern map, capturing medium-sized texture variations (e.g., brick wall seams, leaf veins). The second downsampled scale feature map is encoded using LBP to generate a large-scale texture pattern map, describing a wide range of texture distributions (e.g., cloud gradients, large-area shadow transitions). During LBP encoding, an adaptive threshold is used to suppress low-frequency noise. That is, if the adaptive threshold of a certain region detects low-frequency noise (i.e., the standard deviation is too low), the LBP encoding value of that region is forced to be set to zero to avoid random noise in smooth regions being misencoded as texture. Finally, histogram equalization is performed on the LBP encoding results corresponding to each scale feature map to enhance texture contrast and ensure consistency of texture patterns under different lighting conditions.

[0039] Step S1023: Generate a deep texture feature map by performing cross-scale feature fusion on texture pattern feature maps of different scales.

[0040] Cross-scale feature fusion is performed on texture pattern feature maps at different scales to generate deep texture feature maps. Specifically, texture pattern feature maps at different scales can be input into a convolutional neural network, and the cross-scale feature fusion module can extract the associated texture features between each scale to generate deep texture feature maps. Considering that gradients may decay layer by layer during the training of deep neural networks, resulting in the inability to effectively update the lower-level parameters, the residual module creates a "shortcut" through skip connections (directly superimposing the input to the output), allowing the gradient to be directly propagated back to the shallow layers, alleviating the gradient vanishing problem. In addition, skip connections can preserve the original input information and prevent the deep network from losing important low-level features (such as texture edges) due to multiple nonlinear transformations. Therefore, the above convolutional neural network can contain several residual modules, each of which can consist of two convolutional layers with l×l kernels and one skip connection.

[0041] It should be noted that stacking multiple small convolutional kernels is equivalent to the receptive field of a single large convolutional kernel. For example, stacking two 3×3 convolutional layers is equivalent to the receptive field of a 5×5 convolutional layer, but the former has fewer parameters (3×3×2 = 18 parameters vs. 5×5 = 25 parameters). Furthermore, multiple small convolutional layers can increase network depth and introduce more non-linear activations (e.g., ReLU). Therefore, from the perspective of reducing computational complexity and improving feature representation, the two l×l convolutional layers mentioned above could be two 3×3 convolutional layers. In addition, ablation experiments revealed that if the above convolutional neural network has fewer than four residual modules (e.g., two), it cannot fully extract cross-scale correlated features, while having more than four residual modules (e.g., six) leads to a dramatic increase in computational cost and diminishing returns. Therefore, a convolutional neural network can contain four residual modules, which form a progressive structure of "shallow → medium → deep → deeper", gradually fusing multi-scale information and capturing the texture correlation from local details to global structure.

[0042] Step S1024: Perform channel attention weighting on the deep texture feature map to obtain texture feature maps at different spatial frequencies.

[0043] Specifically, step S1024 can be implemented through steps S10241 to S10244, as detailed below:

[0044] Step S10241: Perform global feature compression on each feature channel of the deep texture feature map to generate a channel-level global description vector.

[0045] That is, step S10241 can be implemented as follows: for each feature channel of the deep texture feature map, all feature response values ​​of the channel in the entire spatial dimension (i.e., the height and width of the image) are aggregated into a scalar value with global representativeness. The scalar values ​​corresponding to all feature channels constitute the channel-level global description vector. In other words, each element of the channel-level global description vector corresponds to the global aggregation result of an original feature channel.

[0046] Step S10242: Based on the channel-level global description vector, calculate the weight coefficients of each feature channel through a feedforward neural network containing a bottleneck structure.

[0047] Specifically, step S10242 can be implemented as follows: inputting the channel-level global description vector into a feedforward neural network containing a bottleneck structure. The network first projects the channel-level global description vector into a lower-dimensional space for nonlinear transformation, then projects it back to the original channel dimension, and finally maps the output value to a preset numerical range (e.g., between 0 and 1). The mapped value is the weight coefficient of the feature channel.

[0048] Step S10243: Multiply each weight coefficient in the channel weight vector with the corresponding feature channel in the deep texture feature map channel by channel to generate a weighted feature map.

[0049] Specifically, step S10243 can be implemented as follows: multiply the first weight coefficient in the channel weight vector with all the data of the first feature channel in the deep texture feature map; multiply the second weight coefficient with the second feature channel, and so on, traversing all feature channels to obtain a weighted feature map. This map has the same shape as the deep texture feature map, but the feature values ​​in each channel are scaled according to their importance coefficients; the features of important channels are enhanced, while the features of secondary channels are suppressed.

[0050] Step S10244: Output the weighted feature map as a texture feature map at different spatial frequencies.

[0051] As can be seen from steps S1021 to S1024 of the above embodiment, by employing multi-scale texture analysis technology to extract texture feature maps at different spatial frequencies, it is possible to comprehensively capture the feature distribution from macroscopic texture structure to microscopic details, providing accurate feature localization for subsequent local enhancement, and ensuring that texture components at different scales can be appropriately enhanced.

[0052] In the above embodiments, between the noise detection and denoising of the image to be processed to obtain the denoised image and the multi-scale texture analysis of the denoised image to extract texture feature maps at different spatial frequencies, the noise suppression verification can also be performed in the following way: perform local variance analysis on the denoised image; when an overly smoothed region is detected, restore the texture details of the corresponding region of the image to be processed.

[0053] Step S103: Based on the gradient magnitude and direction information of the texture feature map at different spatial frequencies, perform local enhancement processing on the weak texture region of the texture feature map through a group of directional convolution kernels to generate a detail-enhanced image.

[0054] To effectively address the texture orientation confusion caused by traditional isotropic enhancement and significantly improve the reconstruction quality of textures with directional features in images, this application embodiment can utilize directional convolution kernel groups to locally enhance weak texture regions based on the gradient magnitude and direction information of texture feature maps at different spatial frequencies, generating a detail-enhanced image. Specifically, as one embodiment of this application, the generation of a detail-enhanced image by locally enhancing weak texture regions of texture feature maps using directional convolution kernel groups based on the gradient magnitude and direction information of texture feature maps at different spatial frequencies can be achieved through steps S1031 to S1033, as detailed below:

[0055] Step S1031: Determine the weak texture region of the texture feature map based on the gradient magnitude of the texture feature map.

[0056] Specifically, step S1031 can be implemented as follows: First, an edge detection operator (such as the Sobel operator or the Prewitt operator) is used to perform a convolution operation on the texture feature map to obtain the horizontal gradient component and the vertical gradient component. Based on the horizontal gradient component and the vertical gradient component, the square root of the sum of squares of each pixel is calculated to generate a gradient magnitude map of the texture feature map. These gradient magnitude maps represent the texture intensity at each pixel position of the texture feature map. Then, adaptive threshold segmentation is performed on the gradient magnitude map to generate an initial weak texture region label map. Finally, morphological opening operation is performed on the initial weak texture region label map to eliminate isolated noise points. Connectivity analysis is performed on the label map after morphological opening operation to filter out connected regions with an area greater than a preset area threshold and merge them to generate the final weak texture region.

[0057] Step S1032: Dynamically generate directional convolution kernel groups based on the gradient direction information of the texture feature map.

[0058] Specifically, step S1032 can be implemented as follows: based on the horizontal and vertical gradient components obtained when calculating the gradient magnitude map, calculate the arctangent value pixel by pixel to generate a gradient direction map of the texture feature map. These gradient direction maps represent the texture orientation of each pixel position in the texture feature map. Extract all pixel positions covered by a connected region in the weak texture region, obtain the gradient direction values ​​corresponding to these pixel positions in the gradient direction map, and calculate the statistical histogram of these gradient direction values. Determine the direction corresponding to the peak in the statistical histogram as the dominant gradient direction of the connected region. Based on the dominant gradient direction of the connected region, perform a rotation transformation on a preset direction-sensitive basis kernel (e.g., an anisotropic Gaussian derivative kernel or Gabor kernel) to generate a directional convolution kernel aligned with the dominant gradient direction of the connected region. Traverse all connected regions in the weak texture region and repeat the above calculation and generation steps to generate a directional convolution kernel group containing multiple directional convolution kernels.

[0059] Step S1033: Perform directional selective enhancement processing on the weak texture regions of the texture feature map using dynamically generated directional convolution kernel groups to generate a detail-enhanced image.

[0060] Specifically, step S1033 can be implemented as follows: create a blank image of the same size as the texture feature map as a detail enhancement image buffer, copy the pixel data of the texture feature map to the detail enhancement image buffer as the initial state; extract a connected region from the weak texture region, select a directional convolution kernel from the directional convolution kernel group corresponding to the dominant gradient direction of the connected region, and perform convolution operation on the texture feature map only within the spatial range corresponding to the current connected region to obtain the enhanced feature response value, update the corresponding position in the detail enhancement image buffer with the calculated enhanced feature response value; traverse all connected regions in the weak texture region and repeat the above directional convolution enhancement operation, retain the original texture feature value in the detail enhancement image buffer unchanged for the pixel regions not covered in the weak texture region, and output the image in the detail enhancement image buffer as the final detail enhancement image after all regions have been processed.

[0061] Step S104: Input the detail-enhanced image into the pre-trained deep super-resolution network, and iteratively optimize the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

[0062] In this embodiment, the pre-trained deep super-resolution network is a trained ESRGAN architecture, which may include a feature extraction module consisting of 10 residual blocks, a feature fusion module with channel attention mechanism, and an adversarial discriminator using spectral normalization constraints, etc. The residual dense block may contain four cascaded 3×3 convolutional layers, each convolutional layer is followed by a LeakyReLU activation function, and cross-layer features can be aggregated through dense connections. As one embodiment of this application, the high-frequency details of the enhanced image are input into a pre-trained deep super-resolution network. The high-frequency details of the enhanced image are iteratively optimized through a multi-layer feature fusion module of the deep super-resolution network to generate an image at the target resolution. This can be achieved by: extracting high-frequency components from the feature map output at the current stage during each iteration of the multi-layer feature fusion module, where the input for the first iteration is the feature map generated from the enhanced image after initial convolution processing; performing a preset number of iterative corrections on the extracted high-frequency components through a recursive network, generating a residual map containing the enhancement weights each time; after completing the preset number of iterative corrections, fusing the final residual map with the previous stage feature map input at the start of the current iteration using an adaptive weighting method; and generating the image at the target resolution after a preset number of iterative optimizations by the multi-layer feature fusion module. Compared to traditional methods of single-step feature extraction, in the above embodiment, the multi-layer feature fusion module gradually corrects high-frequency details through an iterative optimization mechanism, more accurately reconstructing detail information that conforms to the statistical characteristics of natural images, thereby reducing artifact generation.

[0063] To enhance important contour features while avoiding over-sharpening noise in flat areas, thus making the final generated image more consistent with the perceptual characteristics of the human visual system, the method in the above embodiments may further include edge sharpening post-processing of the image at the target resolution. Specifically, edge sharpening post-processing of the image at the target resolution may involve: employing an unsharpened mask algorithm, whose enhancement coefficient is dynamically adjusted based on edge gradient values ​​and a visual saliency map. The visual saliency map is generated by: converting the image to Lab color space; calculating the color difference and brightness difference between each pixel and its surrounding area; and generating multi-scale saliency features through Gaussian pyramid fusion.

[0064] Since super-resolution networks (such as SRGAN and ESRGAN) primarily focus on spatial resolution enhancement, they may still fall short in optimizing local contrast. This can lead to issues such as low contrast in flat areas (e.g., sky, walls), blurred details, local overexposure or underexposure in textured areas (e.g., fabric, leaves), texture adhesion, and halo artifacts or jagged edges in sharpened edge areas (e.g., building outlines, text). Therefore, after edge sharpening of the target resolution image, dynamic quantization enhancement can be performed on the resulting image. This involves dividing the edge-sharpened image into flat, textured, and edge regions; applying adaptive histogram equalization to each region; and using guided filtering to achieve a smooth transition between these regions. By performing dynamic quantization enhancement on the image obtained after edge sharpening, the dark details of the image (such as objects in shadows) are significantly improved without introducing graininess. It also enhances texture separation (such as clarifying leaf veins) and avoids overexposure. At the same time, it eliminates harsh transitions at area boundaries (such as banded artifacts at the junction of the sky and buildings), achieving natural blending.

[0065] From the above appendix Figure 1 As can be seen from the example image processing method, on the one hand, after noise reduction processing of the image to be processed, multi-scale texture analysis is performed on the denoised image. While effectively eliminating noise patches, texture feature maps at different spatial frequencies can be extracted to comprehensively capture the feature distribution from macroscopic texture structure to microscopic details, ensuring that texture components at different scales can be appropriately enhanced in subsequent stages. On the other hand, by using directional convolution kernel groups to locally enhance weak texture regions, the enhancement direction can be adaptively adjusted according to the texture direction, significantly improving the reconstruction quality of textures with directional features. Thirdly, compared with traditional methods of single feature extraction, high-frequency detail iterative optimization can more accurately reconstruct detail information that conforms to the statistical characteristics of natural images and reduce artifact generation.

[0066] Please see the appendix Figure 2This application provides an image processing apparatus, which may include a noise reduction module 201, an extraction module 202, an enhancement module 203, and an optimization module 204, as detailed below:

[0067] The noise reduction module 201 is used to perform noise detection and noise reduction processing on the image to be processed, so as to obtain the noise-reduced image;

[0068] Extraction module 202 is used to perform multi-scale texture analysis on the denoised image and extract texture feature maps at different spatial frequencies;

[0069] The enhancement module 203 is used to perform local enhancement processing on the weak texture region of the texture feature map through a group of directional convolution kernels based on the gradient magnitude and direction information of the texture feature map at different spatial frequencies, so as to generate a detail-enhanced image.

[0070] The optimization module 204 is used to input the detail-enhanced image into a pre-trained deep super-resolution network, and to iteratively optimize the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

[0071] From the above appendix Figure 2 As can be seen from the example image processing apparatus, on the one hand, after noise reduction processing of the image to be processed, multi-scale texture analysis is performed on the denoised image. While effectively eliminating noise patches, texture feature maps at different spatial frequencies can be extracted to comprehensively capture the feature distribution from macroscopic texture structure to microscopic details, ensuring that texture components at different scales can be appropriately enhanced in subsequent stages. On the other hand, by using directional convolution kernel groups to locally enhance weak texture regions, the enhancement direction can be adaptively adjusted according to the texture direction, significantly improving the reconstruction quality of textures with directional features. Thirdly, compared with traditional methods of single feature extraction, high-frequency detail iterative optimization can more accurately reconstruct detail information that conforms to the statistical characteristics of natural images and reduce artifact generation.

[0072] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for an image processing method. When the processor 30 executes the computer program 32, it implements the steps described in the above-described image processing method embodiment, for example... Figure 1 The steps S101 to S104 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2The functions of the noise reduction module 201, extraction module 202, enhancement module 203, and optimization module 204 shown are illustrated.

[0073] For example, the computer program 32 of the image processing method mainly includes: performing noise detection and denoising on the image to be processed to obtain a denoised image; performing multi-scale texture analysis on the denoised image to extract texture feature maps at different spatial frequencies; performing local enhancement processing on the weak texture regions of the texture feature maps through directional convolution kernel groups based on the gradient magnitude and direction information of the texture feature maps at different spatial frequencies to generate a detail-enhanced image; inputting the detail-enhanced image into a pre-trained deep super-resolution network, and iteratively optimizing the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 32 in the electronic device 3. For example, computer program 32 can be divided into the functions of a noise reduction module 201, an extraction module 202, an enhancement module 203, and an optimization module 204 (modules in a virtual device). The specific functions of each module are as follows: the noise reduction module 201 is used to perform noise detection and noise reduction processing on the image to be processed, and obtain a noise-reduced image; the extraction module 202 is used to perform multi-scale texture analysis on the noise-reduced image and extract texture feature maps at different spatial frequencies; the enhancement module 203 is used to perform local enhancement processing on the weak texture regions of the texture feature maps through directional convolution kernel groups based on the gradient magnitude and direction information of the texture feature maps at different spatial frequencies, and generate a detail-enhanced image; the optimization module 204 is used to input the detail-enhanced image into a pre-trained deep super-resolution network, and perform iterative optimization on the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

[0074] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0075] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0076] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0078] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the image processing method can be stored in a storage medium. When the computer program is executed by a processor, it can implement the steps of the various method embodiments described above, namely, performing noise detection and denoising on the image to be processed to obtain a denoised image; performing multi-scale texture analysis on the denoised image to extract texture feature maps at different spatial frequencies; based on the gradient magnitude and direction information of the texture feature maps at different spatial frequencies, performing local enhancement processing on the weak texture regions of the texture feature maps through directional convolution kernel groups to generate a detail-enhanced image; inputting the detail-enhanced image into a pre-trained deep super-resolution network, and iteratively optimizing the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added to or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media may not include electrical carrier signals and telecommunication signals.

[0084] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this invention.

Claims

1. An image processing method, characterized in that, The method includes: Noise detection and denoising are performed on the image to be processed to obtain the denoised image; Multi-scale texture analysis is performed on the denoised image to extract texture feature maps at different spatial frequencies; Based on the gradient magnitude and direction information of the texture feature map, a group of directional convolution kernels is used to perform local enhancement processing on the weak texture regions of the texture feature map to generate a detail-enhanced image. The enhanced detail image is input into a pre-trained deep super-resolution network. The high-frequency details of the enhanced detail image are iteratively optimized through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

2. The method according to claim 1, characterized in that, The process of performing noise detection and denoising on the image to be processed to obtain a denoised image includes: Adaptive noise detection is performed on the image to be processed, and a dynamic noise mask is generated based on the detected noise distribution characteristics; Based on the dynamic noise mask, a nonlocal mean filtering algorithm is used to selectively denoise the noise region of the image to be processed, while retaining the original pixel information of the non-noise region of the image to be processed, thus obtaining the denoised image.

3. The method according to claim 2, characterized in that, The step of adaptively detecting noise in the image to be processed and generating a dynamic noise mask based on the detected noise distribution features includes: The image to be processed is decomposed by wavelet transform to extract high-frequency subband coefficients; The dynamic noise threshold is calculated based on the statistical distribution of the high-frequency subband coefficients. The region growing algorithm is used to cluster pixels in the image to be processed that exceed the dynamic noise threshold into noise regions. A binary image identifying the location of noise pixels is generated based on the noise region as the dynamic noise mask.

4. The method according to claim 1, characterized in that, The step of performing multi-scale texture analysis on the denoised image to extract texture feature maps at different spatial frequencies includes: The denoised image is used to generate feature maps of different scales by employing Laplacian pyramid decomposition. Local binary pattern encoding is performed on the feature maps at different scales to generate texture pattern feature maps at different scales; A deep texture feature map is generated by performing cross-scale feature fusion on the texture pattern feature maps of different scales. The deep texture feature map is subjected to channel attention weighting to obtain texture feature maps at different spatial frequencies.

5. The method according to claim 1, characterized in that, The step of performing local enhancement processing on the weak texture regions of the texture feature map using directional convolution kernel groups based on the gradient magnitude and direction information of the texture feature map to generate a detail-enhanced image includes: The weak texture regions of the texture feature map are determined based on the gradient magnitude of the texture feature map. Directional convolution kernel groups are dynamically generated based on the gradient direction information of the texture feature map; The dynamically generated directional convolution kernel group is used to perform directional selective enhancement processing on the weak texture regions of the texture feature map to generate the detail-enhanced image.

6. The method according to claim 1, characterized in that, The step of iteratively optimizing high-frequency details through a multi-layer feature fusion module to generate an image at the target resolution includes: In each iteration of the multi-layer feature fusion module, high-frequency components are extracted from the feature map output at the current stage; The extracted high-frequency components are iteratively corrected a preset number of times using a recursive network, and each correction generates a residual map containing detail enhancement weights. After completing the preset number of iterations for correction, an adaptive weighting method is used to fuse the final residual map with the feature map of the previous stage input at the beginning of this iteration process; After a preset number of iterations and optimizations by the multi-layer feature fusion module, an image at the target resolution is generated.

7. The method according to claim 1, characterized in that, After generating the image at the target resolution, the method further includes: The image at the target resolution is then subjected to edge sharpening post-processing.

8. An image processing apparatus, characterized in that, The device includes: The noise reduction module is used to detect and reduce noise in the image to be processed, and obtain the noise-reduced image. The extraction module is used to perform multi-scale texture analysis on the denoised image and extract texture feature maps at different spatial frequencies; The enhancement module is used to perform local enhancement processing on the weak texture regions of the texture feature map by using a group of directional convolution kernels based on the gradient magnitude and direction information of the texture feature map, thereby generating a detail-enhanced image; An optimization module is used to input the detail-enhanced image into a pre-trained deep super-resolution network, and to iteratively optimize the high-frequency details of the detail-enhanced image through the multi-layer feature fusion module of the deep super-resolution network to generate an image of the target resolution.

9. An electronic device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images