Image denoising method and device based on double U-shaped network, equipment and medium

Abstract

This application provides an image denoising method, apparatus, device, and medium based on a dual U-shaped network, relating to the field of image processing technology and applicable to the financial technology field. The method generates a simulated noisy image using a generative adversarial network (GAN), and superimposes the simulated noisy image with a real image to generate a noisy image. The noisy image is then input into a dual U-shaped network. The main U-shaped module of the dual U-shaped network extracts global multi-scale features from the noisy image, and the branch U-shaped modules split the noisy image into positive and negative value parts for multi-scale feature extraction, obtaining local multi-scale features. Subsequently, the local multi-scale features are fused with the global multi-scale features along the channel dimension, outputting a denoised image. This achieves complementarity between local detail features and global features, preserving the global structure and local polarity details of the effective signal while improving the denoising effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image denoising method, apparatus, device and medium based on a dual U-shaped network. Background Technology

[0002] In the field of digital image processing, image noise suppression is one of the core technologies for improving image quality. It reduces the impact on the original image information while suppressing noise in noisy images and repairs damaged features as much as possible, thereby improving image quality.

[0003] In existing technologies, deep learning-based image denoising methods mainly include Multi-Scale Residual Dense Networks (MSRDN). MSRDN enhances noise suppression capabilities through multi-scale convolutional fusion and residual dense connections. However, MSRDN has a complex network structure, requires significant GPU memory for training, and has a long training time, making it difficult to adapt to the computational resource constraints of practical applications. Furthermore, it tends to over-smooth image details when suppressing noise, leading to the loss of effective signals such as edges and textures.

[0004] Therefore, while existing technologies improve noise reduction performance, they not only consume a large amount of video memory, but also suffer from the problem of losing effective image signals. Summary of the Invention

[0005] This application provides an image denoising method, apparatus, device, and medium based on a dual U-shaped network, which aims to improve noise suppression capabilities, optimize the protection of effective signals, and reduce model resource consumption.

[0006] In a first aspect, this application provides an image denoising method based on a dual U-shaped network, wherein the dual U-shaped network includes a main U-shaped module and a branch U-shaped module, and the method includes:

[0007] A simulated noisy image is generated by a generative adversarial network, and then the simulated noisy image is superimposed on a real image to generate a noisy image.

[0008] The noisy image is input into a dual U-shaped network. The main U-shaped module of the dual U-shaped network extracts global multi-scale features from the noisy image. The branch U-shaped module then splits the noisy image into positive and negative parts for multi-scale feature extraction, thus obtaining local multi-scale features.

[0009] The local multi-scale features are fused with the global multi-scale features in the channel dimension to output the denoised image.

[0010] In one possible implementation, the aforementioned generative adversarial network includes: a noise generator and a noise discriminator, which generate simulated noisy images by means of the generative adversarial network, including:

[0011] A random array is input into a noise generator, which generates a simulated noise image.

[0012] The simulated noise image and the real noise sample are input into the noise discriminator to determine whether they are real or fake.

[0013] The parameters of the noise generator and noise discriminator are optimized through adversarial training based on the judgment results.

[0014] In one possible implementation, a branched U-shaped module is used to split the noisy image into positive and negative parts for multi-scale feature extraction, resulting in local multi-scale features, including:

[0015] The noisy image is split into positive and negative parts using a feature splitting function;

[0016] The absolute value of the negative part is taken and then concatenated with the positive part to obtain the first feature data. The first feature data includes the intensity distribution features of the positive and negative effective signals in the noisy image, as well as the superposition features of noise under the corresponding polarity.

[0017] Multi-scale feature extraction is performed on the first feature data to obtain positive features;

[0018] By splitting and reconstructing the positive features using feature splitting and feature reconstruction functions, local multi-scale features are obtained. These local multi-scale features are used to characterize the effective signal strength features after noise suppression.

[0019] In one possible implementation, global multi-scale features in the noisy image are extracted via the main U-shaped module of a dual U-shaped network, including:

[0020] Feature extraction is performed on noisy images using convolutional layers and average pooling layers in the main U-shaped module to obtain global multi-scale features with multiple resolutions.

[0021] In one possible implementation, local multi-scale features are fused with global multi-scale features along the channel dimension to output a denoised image, including:

[0022] In the channel dimension, global multi-scale features and local multi-scale features at the corresponding levels of global multi-scale features are fused to obtain fused features;

[0023] The second feature data of the fused feature and the noisy image are stitched together, and the obtained first global multi-scale feature is convolved multiple times to obtain the denoised image. The second feature data is the feature data obtained by convolving the noisy image.

[0024] In one possible implementation, after obtaining the local multi-scale features, the method further includes:

[0025] The first local multi-scale feature is obtained by performing dimensionality reduction on the local multi-scale feature through multiple convolutional layers.

[0026] The first local multi-scale feature is split using a feature splitting function to obtain the denoised positive value part and the denoised negative value part.

[0027] In one possible implementation, the method further includes:

[0028] After the parameters of the noise discriminator are iterated and updated a preset number of times, the parameters of the noise generator are updated based on the judgment results.

[0029] Secondly, this application provides an image denoising device based on a dual U-shaped network, wherein the dual U-shaped network includes a main U-shaped module and a branch U-shaped module, and the device includes:

[0030] The generation module is used to generate simulated noisy images through a generative adversarial network and to overlay the simulated noisy images with real images to generate noisy images.

[0031] The input module is used to input the noisy image into the dual U-shaped network. The main U-shaped module of the dual U-shaped network extracts global multi-scale features from the noisy image, and the branch U-shaped module splits the noisy image into positive and negative parts for multi-scale feature extraction to obtain local multi-scale features.

[0032] The fusion module is used to fuse local multi-scale features with global multi-scale features in the channel dimension, and output a denoised image.

[0033] In one possible implementation, the generative adversarial network includes a noise generator and a noise discriminator, and the apparatus further includes an optimization module.

[0034] The input module is also used to input a random array into the noise generator, which generates a simulated noise image.

[0035] The input module is also used to input simulated noise images and real noise samples into the noise discriminator for real vs. fake determination;

[0036] The optimization module is used to optimize the parameters of the noise generator and noise discriminator through adversarial training based on the judgment results.

[0037] In one possible implementation, the above-mentioned device further includes: a splitting module, a processing module, and a feature extraction module;

[0038] The splitting module is used to split a noisy image into positive and negative value parts using a feature splitting function;

[0039] The processing module is used to take the absolute value of the negative part and concatenate the obtained absolute value part with the positive part to obtain the first feature data. The first feature data includes the intensity distribution features of the positive effective signal and the negative effective signal in the noisy image, as well as the superposition features of noise under the corresponding polarity.

[0040] The feature extraction module is used to perform multi-scale feature extraction on the first feature data to obtain positive features;

[0041] The processing module is also used to split and polarity-restore the positive features through feature splitting and feature reconstruction functions to obtain local multi-scale features. The aforementioned local multi-scale features are used to characterize the effective signal strength features after noise suppression.

[0042] In one possible implementation, the feature extraction module is also used to extract features from the noisy image through the convolutional layers and average pooling layers in the main U-shaped module to obtain multi-resolution global multi-scale features.

[0043] In one possible implementation, the fusion module is also used to fuse global multi-scale features and local multi-scale features at the corresponding level of the global multi-scale features in the channel dimension to obtain fused features.

[0044] The processing module is also used to stitch together the fused features and the second feature data of the noisy image, and to perform multiple convolutions on the obtained first global multi-scale features to obtain a denoised image. The second feature data is the feature data obtained by convolving the noisy image.

[0045] In one possible implementation, the processing module is also used to perform dimensionality reduction on the local multi-scale features through multiple convolutional layers to obtain the first local multi-scale features.

[0046] The splitting module is also used to split the first local multi-scale features using a feature splitting function to obtain the denoised positive value part and the denoised negative value part.

[0047] In one possible implementation, the above-mentioned device further includes: an update module;

[0048] The update module is used to update the parameters of the noise generator based on the judgment result after the parameters of the noise discriminator have been iterated and updated a preset number of times.

[0049] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0050] The aforementioned memory stores instructions executed by the computer;

[0051] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0053] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0054] The image denoising method based on a dual U-shaped network provided in this application generates a simulated noisy image through a generative adversarial network (GAN), and then superimposes the simulated noisy image with a real image to generate a noisy image. This ensures that the generated simulated noisy image matches the real scene. Using this noisy image to train the dual U-shaped network is beneficial for improving the generalization ability of the denoising network. Furthermore, the noisy image is input into the dual U-shaped network. The main U-shaped module of the dual U-shaped network extracts global multi-scale features from the noisy image, and the branch U-shaped modules split the noisy image into positive and negative value parts for multi-scale feature extraction, obtaining local multi-scale features. This method splits the noisy image into positive and negative value parts for multi-scale feature extraction, enhancing the expression of local features and avoiding the loss of local details. Subsequently, the local multi-scale features are fused with the global multi-scale features along the channel dimension to output the denoised image. This achieves complementarity between local detail features and global features, preserving the global structure and local polarity details of the effective signal while improving the denoising effect. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 A flowchart illustrating an image denoising method based on a dual U-shaped network provided in this application embodiment. Figure 1 ;

[0057] Figure 2 A flowchart illustrating an image denoising method based on a dual U-shaped network provided in this application embodiment. Figure 2 ;

[0058] Figure 3 This is a schematic diagram of a dual U-shaped network provided in an embodiment of this application;

[0059] Figure 4 A schematic diagram of the image denoising device based on a dual U-shaped network provided in this application;

[0060] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0061] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0064] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0065] It should be noted that the image denoising method, apparatus, device and medium based on dual U-shaped networks provided in this application can be used in the field of financial technology, or in any field other than financial technology. The application field of the image denoising method, apparatus, device and medium based on dual U-shaped networks in this application is not limited.

[0066] Efficient and accurate noise suppression of noisy images can not only significantly improve the visual quality of the images, but also provide more reliable input data for subsequent image analysis (such as object detection, feature extraction, and semantic segmentation).

[0067] Currently, with the rapid development of deep learning technology, neural network-based image denoising methods are gradually becoming the mainstream solution. For example, Multi-Scale Residual Dense Network (MSRDN) enhances noise suppression capabilities through multi-scale convolutional fusion and residual dense connections. MSRDN utilizes convolutional kernels of different scales to extract features and retains more intermediate feature information through dense connections.

[0068] However, MSRDN's network structure is relatively complex, requiring a large amount of GPU memory and taking a long time to train, making it difficult to adapt to the constraints of computing resources in practical applications. Furthermore, this method is prone to losing key edge and texture information of the image during noise suppression, resulting in damage to the integrity of the effective signal and affecting the restoration of the image's true features.

[0069] Therefore, while existing technologies improve noise reduction performance, they not only consume a large amount of video memory, but also suffer from the problem of losing effective image signals.

[0070] The image denoising method based on a dual U-shaped network provided in this application extracts global multi-scale features from a noisy image through the main U-shaped module of the dual U-shaped network, and then splits the noisy image into positive and negative parts through a branch U-shaped module to extract local multi-scale features. The local multi-scale features are then fused with the global multi-scale features in the channel dimension to output the denoised image. This method achieves complementarity between local detail features and global features, and preserves the global structure and local polarity details of the effective signal while improving the denoising effect.

[0071] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0072] Figure 1 A flowchart illustrating an image denoising method based on a dual U-shaped network provided in this application embodiment. Figure 1A dual U-shaped network includes a TUN (trunkunet) module and a BUN (branchunet) module. For example... Figure 1 As shown, the method includes:

[0073] S101. Generate a simulated noisy image using a generative adversarial network, and then overlay the simulated noisy image with the real image to generate a noisy image.

[0074] Specifically, a generative adversarial network (GAN) includes a noise generator and a noise discriminator. The noise generator generates simulated noise image data based on an input random array, and inputs this simulated noise image and real noise samples into the noise discriminator for realism judgment. Then, based on the realism judgment results, the parameters of the noise generator and the noise discriminator are optimized through adversarial training, and finally, a simulated noise image consistent with the noise distribution of the real scene is generated.

[0075] S102. Input the noisy image into the dual U-shaped network. Extract global multi-scale features from the noisy image through the main U-shaped module of the dual U-shaped network. Then, split the noisy image into positive and negative parts through the branch U-shaped module to extract multi-scale features and obtain local multi-scale features.

[0076] Specifically, the main U-shaped module extracts global multi-scale features from the noisy image. The branch U-shaped module splits the noisy image into positive and negative parts using a feature splitting function. The positive parts and the negative parts (after taking their absolute values) are then concatenated along the batch dimension to form concatenated feature data. This concatenated feature data is then used for feature extraction through multiple convolutional layers. After each convolutional layer extracts feature data, an average pooling layer progressively downsamples the feature data to obtain multiple sets of low-resolution feature data. These feature data sets characterize the overall noise distribution and effective signal features of the noisy image.

[0077] Furthermore, the above feature data set is progressively upsampled through multiple deconvolution layers to restore its resolution, and polarity is restored through a feature reconstruction function to obtain local multi-scale feature extraction.

[0078] For example, positive values ​​are extracted using a 3×3 convolution kernel to extract local edge features, while negative values ​​are extracted using a 5×5 convolution kernel to extract global structural features. The feature reconstruction function restores the data distribution through batch dimensionality cropping and reconstruction, ensuring the integrity of the denoised image.

[0079] This method enhances local feature representation by independently processing positive and negative values ​​through a feature splitting function; multi-scale feature extraction captures image details and structural information using different convolutional kernel sizes; and a feature reconstruction function restores data distribution through cropping and reconstruction, avoiding signal loss during denoising. This mechanism significantly improves denoising performance and image detail preservation, enabling the dual U-shaped network to perform better in complex noisy scenes.

[0080] S103. Perform feature fusion between local multi-scale features and global multi-scale features in the channel dimension, and output the denoised image.

[0081] This step fuses the local multi-scale features extracted by the branch U-shaped module with the global multi-scale features in the channel dimension. By leveraging the representational ability of the branch U-shaped module for local data features, the global multi-scale features are supplemented. Then, after dimensionality reduction in the channel dimension by the convolutional layer, the denoised image is output.

[0082] The image denoising method based on a dual U-shaped network provided in this embodiment generates a simulated noisy image through a generative adversarial network (GAN), and then superimposes the simulated noisy image with a real image to generate a noisy image. This ensures that the generated simulated noisy image matches the real scene. Using this noisy image to train the dual U-shaped network is beneficial for improving the generalization ability of the denoising network. Furthermore, the noisy image is input into the dual U-shaped network. The main U-shaped module of the dual U-shaped network extracts global multi-scale features from the noisy image, and the branch U-shaped modules split the noisy image into positive and negative value parts for multi-scale feature extraction, obtaining local multi-scale features. This method splits the noisy image into positive and negative value parts for multi-scale feature extraction, enhancing the expression of local features and avoiding the loss of local details. Subsequently, the local multi-scale features are fused with the global multi-scale features in the channel dimension, outputting the denoised image. This achieves complementarity between local detail features and global features, preserving the global structure and local polarity details of the effective signal while improving the denoising effect.

[0083] Figure 2 A flowchart illustrating an image denoising method based on a dual U-shaped network provided in this application embodiment. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, a possible image denoising method based on a dual U-shaped network is described in detail. This method includes:

[0084] S201. Generate a simulated noisy image using a generative adversarial network, and then overlay the simulated noisy image with the real image to generate a noisy image.

[0085] Optionally, a training method for a generative adversarial network is provided herein, the method comprising: inputting a random array into a noise generator to generate a simulated noise image; inputting the simulated noise image and a real noise sample into a noise discriminator for real / fake judgment; and using the judgment result to optimize the parameters of the noise generator and the noise discriminator through adversarial training.

[0086] In generative adversarial networks (GANs), a noise generator progressively expands an input random array through deconvolutional layers to generate simulated noise images. A noise discriminator extracts features from the simulated noise images and real noise samples through convolutional layers and outputs a true / false result. Adversarial training optimizes the parameters of the noise generator and the noise discriminator, making the noise images output by the noise generator approximate real noise in statistical properties (such as distribution patterns and spatial correlation).

[0087] Optionally, during training, after the parameters of the noise discriminator are iteratively updated a preset number of times, the parameters of the noise generator are updated based on the judgment results.

[0088] For example, the random array is a 1×128 one-dimensional array with values ​​between [0, 1). This random array is input into the noise generator module, where four deconvolutional layers reduce the dimensionality of the random array in the channel dimension while expanding the data size. To ensure the output range matches the range of the real noise, the final layer uses a tanh activation function. The simulated noise image output by the noise generator module is then fed into a noise discriminator along with real noise samples for evaluation. In the noise discriminator, the input data includes real noise samples of size 128×128, which are then multiplied in the channel dimension by four convolutional layers. Finally, a linear layer and a sigmoid activation function are used to output labels, thereby determining whether the samples are real or fake.

[0089] During the training of the generative adversarial network, the learning rate was set to 2e−6, the batch size to 8, the Adam optimization algorithm was used as the optimizer, the number of iterations was 20,000, and the mean squared error loss function was used. Notably, to further improve the discrimination accuracy of the noise discriminator module, the noise generator underwent one iteration of training after every two iterations of the noise discriminator. Furthermore, to improve the quality of the noise discriminator, the label of an input sample was determined to be true when its label output value was greater than 0.8.

[0090] S202. Input the noisy image into the dual U-shaped network, and extract features from the noisy image through the convolutional layer and average pooling layer in the main U-shaped module to obtain multi-resolution global multi-scale features.

[0091] Specifically, the noisy image is input into the main U-shaped module, where feature extraction is performed through convolutional layers. Each convolutional layer corresponds to an average pooling layer, which is used to downsample the extracted feature data, reducing the resolution of the feature data and expanding the receptive field. Multiple convolutional layers combined with average pooling layers perform feature extraction and downsampling layer by layer, and each combination outputs global multi-scale features at different resolutions.

[0092] Figure 3 This is a schematic diagram of a dual U-shaped network provided in an embodiment of this application, as shown below. Figure 3 As shown, the main U-shaped module TUN consists of 4 average pooling layers, 4 deconvolutional layers, and 15×2 convolutional layers ending with the ReLU activation function. In addition, the dual U-shaped network also includes a feature reconstruction function (FR) and a feature splitting function (FS). The FS and FR functions are used to perform cropping and reconstruction operations on the processed feature data in the batch dimension.

[0093] like Figure 3 As shown, the TUN module has 4 average pooling layers paired with 4 convolutional layers. Through the combination of 4 convolutional layers and average pooling layers, feature extraction and downsampling are performed progressively on noisy images to obtain global multi-scale features.

[0094] S203. The noisy image is split into positive and negative parts using a feature splitting function.

[0095] The branch U-shaped module includes a feature splitting function. After a noisy image is input into the branch U-shaped module, the feature splitting function splits the noisy image into positive and negative value parts. For example, when the input pixel value is -2.5, the feature splitting function splits it into a positive value of 0 and a negative value of 2.5.

[0096] For example, such as Figure 3 As shown, the Branched U-shaped module (BUN) consists of four average pooling layers, four deconvolutional layers, and 11×2 convolutional layers ending with ReLU activation. The feature splitting function in the BUN module is the FS function, which can be expressed mathematically as: [Vpositive, Vnegative] = FS(G). Here, G is the noisy image, Vpositive is the positive value obtained from the splitting, and Vnegative is the negative value obtained from the splitting.

[0097] S204. Take the absolute value of the negative part and concatenate the obtained absolute value part with the positive part to obtain the first feature data.

[0098] The first feature data includes the intensity distribution features of the positive and negative effective signals in the noisy image, as well as the superposition features of noise under the corresponding polarity.

[0099] To prevent the positive and negative effective signals in noisy images from canceling each other out due to their opposite polarities, which would cause the low-amplitude effective signals extracted for feature extraction to be masked by noise, the absolute value of the negative part is taken. The absolute value part contains the intensity distribution characteristics of the negative effective signal and the amplitude distribution characteristics of the negative noise.

[0100] Furthermore, the absolute value part and the positive value part are concatenated to facilitate the learning of the correlation features between the positive and negative effective signals, as well as the individual features of the positive and negative effective signals, in the subsequent feature extraction process.

[0101] For example, the first feature data may be, for instance, Where abs is the absolute value function and Concatb is the concatenation function.

[0102] S205. Perform multi-scale feature extraction on the first feature data to obtain positive features.

[0103] By using multiple convolutional layers and average pooling layers, the first feature data is sampled at multiple scales to obtain positive features. For example... Figure 3 As shown, in the branched U-shaped module, the ADD module outputs the first feature data, which is then processed through multiple convolutional layers (Covn2) and progressively downsampled using an average pooling layer (AvgPool). The Covn2 convolutional layers extract local features from the first feature data, unify the feature distribution through batch normalization, and enhance effective signal features while suppressing noise using the ReLU activation function. The average pooling layer gradually reduces the feature map resolution through downsampling, outputting four sets of low-resolution deep features.

[0104] Furthermore, the low-resolution deep features are progressively upsampled through four deconvolution layers to restore the resolution. Each time a deconvolution layer passes through a layer, higher-resolution deep feature data is output. These feature data are represented in ascending order of resolution as {Vup1, Vup2, Vup3, Vup4}, thus obtaining the forwarded features.

[0105] S206. The positiveized features are cropped and polarity restored by the feature reconstruction function to obtain local multi-scale features.

[0106] Among them, local multi-scale features are used to characterize the effective signal strength features after noise suppression.

[0107] The positive features are input into the feature reconstruction function, and the batch dimension is cropped according to a preset ratio to obtain positive value feature data and absolute value feature data. Then, the absolute value feature data is restored to negative polarity to obtain negative value feature data. Finally, the positive value feature data and negative value feature data are added together to achieve the purpose of restoring the data in various dimensions and outputting local multi-scale features.

[0108] For example, such as Figure 3 As shown, the positively oriented features {Vup1, Vup2, Vup3, Vup4} are input into the feature reconstruction function FR, and the local multi-scale features are output using the following formula:

[0109] {VFR·cut1, VFR·abs·cut2}=Cut(VFR·input)

[0110] VFR·out=VFR·cut1+(−1)×VFR·cut2

[0111] Where VFR·input={Vup1, Vup2, Vup3, Vup4}, Cut is the proportional clipping function, VFR·cut1 represents the first part of VFR·input clipped, VFR·abs·cut2 represents the second part of clipped, and VFR·out is the aforementioned local multi-scale feature.

[0112] Optionally, after obtaining the local multi-scale features of the branch U-shaped module side, the local multi-scale features are reduced in dimensionality by multiple convolutional layers to obtain the first local multi-scale features; the first local multi-scale features are split by a feature splitting function to obtain the denoised positive part and the denoised negative part.

[0113] Among them, the positive value part after denoising is as follows Figure 3 In B, the negative part after denoising is as follows: Figure 3 In the C parameter, the denoised positive and negative values ​​are used to compare with a clean, real image to iteratively optimize the network parameters.

[0114] S207. In the channel dimension, the global multi-scale features and the local multi-scale features of the corresponding level of the global multi-scale features are fused to obtain the fused features.

[0115] In the main U-shaped module, the global multi-scale features are deconvolutionally processed, and the processed feature data, the global multi-scale features of the corresponding level, and the local multi-scale features output by the convolutional layer of the corresponding level in the main U-shaped module are fused to obtain the fused features.

[0116] like Figure 3As shown, in the TUN module, before each deconvolution layer, there is a deep data feature whose resolution needs to be increased. These data features, arranged from low to high resolution, can be represented as {Fup1, Fup2, Fup3, Fup4}. Specifically, in this group of data features, the higher-resolution data feature is obtained by concatenating and fusing the lower-resolution features after passing through the deconvolution layer and the corresponding lower-resolution data features after passing through the average pooling layer, followed by dimensionality reduction. Subsequently, this group of deep data features and the local multi-scale features {VFR·1, VFR·2, VFR·3, VFR·4} output in step S207 are concatenated and fused along the channel dimension. The deep data features obtained after concatenation and fusion then undergo the next deconvolution operation until the final deconvolution, resulting in the fused feature.

[0117] S208. The fused features and the second feature data of the noisy image are spliced ​​together, and the obtained first global multi-scale features are convolved multiple times to obtain the denoised image.

[0118] The second feature data is obtained by convolving the noisy image. The fused feature and the second feature data are used to supplement the strength benchmark of the original valid data, preventing excessive fusion that deviates from the original image data. Further optimization through multiple convolutions outputs a denoised image with the same size, dimensions, and numerical range as the noisy image.

[0119] For example, the denoised image is as follows Figure 3 A in the middle.

[0120] The image denoising method based on a dual U-shaped network provided in this embodiment generates a simulated noisy image using a generative adversarial network (GAN). This simulated noisy image is then superimposed on a real image to generate a noisy image. The noisy image is then input into the dual U-shaped network, where convolutional layers and average pooling layers in the main U-shaped module extract features to obtain multi-resolution, multi-scale global features. This efficiently captures the distribution patterns of global noise and valid data. The convolutional layers accurately capture edge and texture features of the valid data in the noisy image. Furthermore, using an adversarial network to generate a simulated noisy image that approximates the real scene helps the dual U-shaped network learn noise features from the real scene, thereby improving its denoising capabilities.

[0121] Furthermore, a feature splitting function is used to divide the noisy image into positive and negative parts. The absolute value of the negative part is then taken, and the resulting absolute value is concatenated with the positive part to obtain the first feature data. Multi-scale feature extraction is then performed on this first feature data to obtain positive features. Finally, a feature reconstruction function is used to crop and restore the polarity of the positive features to obtain local multi-scale features. This method eliminates positive and negative polarity interference, which helps to preserve the feature data of effective positive and negative signals. Therefore, while achieving noise suppression, more complete and accurate local multi-scale features are obtained.

[0122] Furthermore, at the channel dimension, the global multi-scale features and the corresponding local multi-scale features at the corresponding levels are fused to obtain fused features. These fused features and the second feature data of the noisy image are then concatenated. Finally, the resulting first global multi-scale features are subjected to multiple convolutional processes to obtain the denoised image. This method supplements the global multi-scale features with local multi-scale features, significantly improving the quality of the denoised image and ensuring the integrity of the effective signal.

[0123] Figure 4 This is a schematic diagram of the image denoising device based on a dual U-shaped network provided in this application. Figure 4 As shown, the image denoising device 40 based on a dual U-shaped network provided in this embodiment includes:

[0124] The generation module 401 is used to generate a simulated noisy image through a generative adversarial network, and to superimpose the simulated noisy image with a real image to generate a noisy image.

[0125] The input module 402 is used to input the noisy image into the dual U-shaped network, extract global multi-scale features from the noisy image through the main U-shaped module of the dual U-shaped network, and split the noisy image into positive and negative parts through the branch U-shaped module for multi-scale feature extraction to obtain local multi-scale features.

[0126] The fusion module 403 is used to fuse local multi-scale features with global multi-scale features in the channel dimension and output a denoised image.

[0127] In one possible implementation, the above-mentioned generative adversarial network includes: a noise generator and a noise discriminator, and the above-mentioned device further includes: an optimization module 404;

[0128] Input module 402 is also used to input a random array into a noise generator, and generate a simulated noise image through the noise generator;

[0129] The input module 402 is also used to input the simulated noise image and the real noise sample into the noise discriminator for real-fake judgment;

[0130] The optimization module 404 is used to optimize the parameters of the noise generator and noise discriminator through adversarial training based on the judgment results.

[0131] In one possible implementation, the above-mentioned device further includes: a splitting module 405, a processing module 406, and a feature extraction module 407;

[0132] The splitting module 405 is used to split a noisy image into positive and negative parts using a feature splitting function;

[0133] The processing module 406 is used to take the absolute value of the negative part and concatenate the obtained absolute value part with the positive part to obtain the first feature data. The first feature data includes the intensity distribution features of the positive effective signal and the negative effective signal in the noisy image, as well as the superposition features of noise under the corresponding polarity.

[0134] Feature extraction module 407 is used to perform multi-scale feature extraction on the first feature data to obtain positive features;

[0135] The processing module 406 is also used to split and polarity restore the positive features through the feature splitting function and the feature reconstruction function to obtain local multi-scale features. The local multi-scale features are used to characterize the effective signal strength features after noise suppression.

[0136] In one possible implementation, the feature extraction module 407 is also used to extract features from the noisy image through the convolutional layer and average pooling layer in the main U-shaped module to obtain multi-resolution global multi-scale features.

[0137] In one possible implementation, the fusion module 403 is further used to fuse the global multi-scale features and the local multi-scale features of the corresponding level of the global multi-scale features in the channel dimension to obtain fused features.

[0138] The processing module 406 is also used to stitch together the fused features and the second feature data of the noisy image, and to perform multiple convolutions on the obtained first global multi-scale features to obtain a denoised image. The second feature data is the feature data obtained by convolving the noisy image.

[0139] In one possible implementation, the processing module 406 is further configured to perform dimensionality reduction processing on the local multi-scale features through multiple convolutional layers to obtain the first local multi-scale features.

[0140] The splitting module 405 is also used to split the first local multi-scale features using a feature splitting function to obtain the denoised positive value part and the denoised negative value part.

[0141] In one possible implementation, the above-mentioned device further includes: an update module 408;

[0142] The update module 408 is used to update the parameters of the noise generator based on the judgment result after the parameters of the noise discriminator have been iterated and updated a preset number of times.

[0143] The image denoising device based on a dual U-shaped network provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0144] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0145] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described image denoising method based on a dual U-shaped network.

[0146] The specific implementation process of processor 501 can be found in the above embodiment of the image denoising method based on dual U-shaped network. The implementation principle and technical effect are similar, and will not be repeated here.

[0147] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0148] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0149] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0150] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image denoising method based on a dual U-shaped network.

[0151] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described image denoising method based on a dual U-shaped network.

[0152] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0153] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0154] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0157] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0159] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0160] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0161] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0162] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0163] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0164] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0165] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0166] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0167] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for image denoising based on double U-shaped network, characterized in that, The double U-shaped network comprises a main U-shaped module and a branch U-shaped module, and the method comprises: An artificial noise image is generated by a generative adversarial network, and the artificial noise image is superimposed with a real image to generate a noisy image; The noisy image is input into the double U-shaped network, global multi-scale features in the noisy image are extracted via the main U-shaped module of the double U-shaped network, and the noisy image is split into a positive value part and a negative value part by the branch U-shaped module for multi-scale feature extraction to obtain local multi-scale features; The local multi-scale features are fused with the global multi-scale features in the channel dimension to output a denoised image.

2. The method of claim 1, wherein, The generative adversarial network comprises a noise generator and a noise discriminator, and the artificial noise image is generated by the generative adversarial network, comprising: A random array is input into the noise generator, and the noise generator generates an artificial noise image; The artificial noise image and a real noise sample are input into the noise discriminator for true or false judgment; The parameters of the noise generator and the noise discriminator are optimized by adversarial training using the judgment result.

3. The method of claim 1, wherein, The noisy image is split into a positive value part and a negative value part by the branch U-shaped module for multi-scale feature extraction to obtain local multi-scale features, comprising: The noisy image is split into a positive value part and a negative value part by a feature splitting function; The absolute value of the negative value part is taken, and the obtained absolute value part is spliced with the positive value part to obtain first feature data, the first feature data comprising intensity distribution features of positive and negative effective signals in the noisy image and superposition features of noise corresponding to the polarity; The first feature data is subjected to multi-scale feature extraction to obtain positive features; The positive features are split and polarity restored by the feature splitting function and a feature reconstruction function to obtain local multi-scale features, the local multi-scale features being used to represent effective signal intensity features after noise suppression.

4. The method of claim 3, wherein, The global multi-scale features in the noisy image are extracted via the main U-shaped module of the double U-shaped network, comprising: The noisy image is subjected to feature extraction by a convolution layer and an average pooling layer in the main U-shaped module to obtain multi-resolution global multi-scale features.

5. The method of claim 4, wherein, The local multi-scale features are fused with the global multi-scale features in the channel dimension to output a denoised image, comprising: The global multi-scale features and local multi-scale features corresponding to the hierarchical level of the global multi-scale features are fused in the channel dimension to obtain fused features; The fused features, second feature data of the noisy image are spliced, and the obtained first global multi-scale features are subjected to multiple convolution processing to obtain a denoised image, the second feature data being feature data obtained by convolution of the noisy image.

6. The method of claim 3, wherein, After the local multi-scale features are obtained, the method further comprises: The local multi-scale features are subjected to dimension reduction processing by multiple convolution layers to obtain first local multi-scale features; The first local multi-scale feature is split by using the characteristic split function to obtain a positive value part after denoising and a negative value part after denoising.

7. The method of claim 2, wherein, The method further comprises: After the parameter of the noise discriminator is iteratively updated for a preset number of times, the parameter of the noise generator is updated based on the judgment result.

8. An image denoising device based on a double U-shaped network, characterized in that, The double U-shaped network comprises a main U-shaped module and a branch U-shaped module, and the device comprises: A generation module is configured to generate a simulated noise image by using a generative adversarial network, and to generate a noisy image by superimposing the simulated noise image and a real image; An input module is configured to input the noisy image into a double U-shaped network, to extract a global multi-scale feature in the noisy image via a main U-shaped module of the double U-shaped network, and to split the noisy image into a positive value part and a negative value part for multi-scale feature extraction by using a branch U-shaped module, so as to obtain a local multi-scale feature; A fusion module is configured to perform feature fusion on the local multi-scale feature and the global multi-scale feature in a channel dimension, and to output an image after denoising.

9. An electronic device, comprising: It comprises: a processor, and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to realize the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize the method of any one of claims 1 to 7.

11. A computer program product, characterised in that, It comprises a computer program, which is executed by the processor to realize the method of any one of claims 1 to 7.

Images