Picture denoising method and device

Abstract

The embodiment of the invention provides a picture denoising method and device. The method comprises the following steps: acquiring a to-be-denoised picture; inputting a to-be-denoised picture into the trained denoising model, and performing zooming extraction processing on the to-be-denoised picture through the denoising model to obtain a noise picture of the to-be-denoised picture; and performing pixel value processing on each pixel in the to-be-denoised picture and the corresponding pixel in the noise picture to obtain a denoised picture. Therefore, the method solves the problem that the image quality and the processing duration cannot be considered at the same time when the image is denoised.

Description

Technical Field

[0001] This application relates to the technical field of computers, and more particularly to a method and apparatus for image denoising. Background Technology

[0002] When taking pictures, especially of documents, document noise often appears in the captured images. For example, there may be shadows or transparent text in the document. This document noise can negatively affect the recognition of the captured document image and also negatively affect the readability of the document image.

[0003] Among the relevant denoising methods, on the one hand, since document noise can contain multiple noise types, such as moiré patterns, shadows, and transparent text in document images, different algorithm frameworks are often set for different noise types. However, this method cannot uniformly denoise different noises in the same document image.

[0004] On the other hand, existing methods often require scaling the image to a fixed size first, obtaining the noise prediction result, and then scaling the image back to the original size. This image scaling results in a loss of image quality. If the image size is too large, processing with a large image takes a long time and affects the user experience. Summary of the Invention

[0005] This application provides an image denoising method, apparatus, and device, which solves the problem that image quality and processing time cannot be simultaneously optimized in related technologies. By predicting noise features in the scaled-down image during the denoising process, more detailed noise feature maps can be predicted in the smaller image. After scaling the feature map back to the original size of the image to be denoised, feature extraction and stitching are performed to obtain the original-size noise image. The denoised image can be directly obtained by subtracting pixel values ​​from the original-size image to be denoised and the original-size noise image, thus avoiding scaling the denoised image and achieving better image quality without scaling.

[0006] In a first aspect, embodiments of this application provide an image denoising method, the method comprising: Obtain the image to be denoised; The image to be denoised is input into the trained denoising model, which then performs scaling and extraction processing on the image to be denoised to obtain the noisy image of the image to be denoised. The pixel values ​​of each pixel in the image to be denoised are compared with the corresponding pixels in the noisy image to obtain the denoised image.

[0007] The denoising model includes an encoder and a decoder; Accordingly, the denoising model is used to scale and extract the image to be denoised to obtain the noisy image of the image to be denoised, including; The encoder performs a first-fold downsampling and noise feature extraction on the image to be denoised to obtain a feature map. The size of the image to be denoised is equal to the second-fold size of the feature map, and the square of the first-fold size equals the second-fold size. After the feature map is enlarged by a second factor using a decoder, noise features are spliced, extracted, and fused to obtain a noise image.

[0008] The encoder includes a downsampling layer and a noise feature extraction layer; Accordingly, the encoder performs a first-fold downsampling and noise feature extraction on the image to be denoised, resulting in a feature map, including: The image to be denoised is downsampled by a first factor through a downsampling layer to obtain a smaller image; A noise feature extraction layer is used to extract global texture features from the scaled-down image to obtain a feature map.

[0009] The decoder includes a transposed convolutional layer, a concatenation layer, a feature extraction layer, and a feature fusion layer. Accordingly, after the feature map is enlarged by a second factor using a decoder, noise feature stitching, noise feature extraction, and noise feature fusion are performed to obtain a noise image, including: The feature map size is enlarged to the same size as the image to be denoised by a second multiple of the transposed convolutional layer. The noise features in the magnified feature map are spliced ​​together by a splicing layer to obtain a noise spliced ​​map. Noise features are extracted from the noise splicing image through a feature extraction layer, and then the extracted noise features are fused through a feature fusion layer to obtain a noise image.

[0010] Furthermore, the pixels in the image to be denoised are compared with the corresponding pixels in the noisy image to obtain the denoised image, including: Determine the pixel difference between the pixel value of each pixel in the image to be denoised and the pixel value of the corresponding pixel in the noisy image; The pixel values ​​at each pixel position are adjusted to the corresponding pixel differences to obtain a denoised image.

[0011] Furthermore, the denoising model is trained in the following way: Obtain the training dataset, which includes multiple sets of metadata. Each set of metadata includes multiple noisy training images and their corresponding actual clear images. Input the metadata of a predetermined number of groups into the denoising model to be trained; Each pixel in the noisy training image is compared with the corresponding pixel in the corresponding noisy prediction image to obtain the corresponding denoised training image. The multimodal loss value corresponding to each denoised training image is determined based on the multimodal loss function. Each multimodal loss value represents the noise of the corresponding denoised training image in multiple dimensions. The denoising model to be trained is trained based on the multimodal loss value to obtain the trained denoising model.

[0012] The multimodal loss values ​​include image loss values ​​and adversarial loss values; Accordingly, the multimodal loss value corresponding to each denoised training image is determined based on the multimodal loss function, including: The adversarial loss value corresponding to each denoised training image is determined by a preset discriminator; The image loss value between each denoised training image and the corresponding actual clear image is determined based on the preset image loss function; For each denoised training image, the corresponding adversarial loss value and the corresponding image loss value are weighted to obtain the corresponding multimodal loss value.

[0013] The discriminator includes a first discriminator and a second discriminator; the adversarial loss value includes a global adversarial loss value and a local adversarial loss value. Accordingly, the adversarial loss value corresponding to each denoised training image is determined by a preset discriminator, including: Perform the following processing on each denoised training image: The first discriminator performs global noise judgment on the denoised training images to obtain the global adversarial loss value. The second discriminator performs local noise detection on the denoised training images to obtain the local adversarial loss value. The adversarial loss value is obtained by weighting the global adversarial loss value and the local adversarial loss value.

[0014] Specifically, the first discriminator performs global noise assessment on the denoised training images to obtain a global adversarial loss value, including: The first discriminator determines the noise level of the training images and uses this noise level as the global adversarial loss value.

[0015] Accordingly, the second discriminator performs local noise assessment on the denoised training images to obtain local adversarial loss values, including: The denoising training images are divided into multiple local regions; The second discriminator determines the regional noise level value in each local area and identifies the regional noise level value as the local adversarial loss value.

[0016] Secondly, embodiments of this application also provide an image denoising apparatus, the apparatus comprising: Acquisition module, noise prediction module, and noise reduction module; The noise prediction module is configured to input the image to be denoised into the trained denoising model, and then use the denoising model to scale and extract the image to be denoised to obtain the noise image of the image to be denoised. The denoising module is configured to perform pixel value processing on each pixel in the image to be denoised and the corresponding pixel in the noise image to obtain the denoised image.

[0017] Thirdly, embodiments of this application also provide an image denoising device, the device comprising: One or more processors; Storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image denoising method described in the embodiments of this application.

[0018] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are configured to perform the image denoising method described in embodiments of this application.

[0019] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the image denoising method described in embodiments of this application.

[0020] As can be seen from the above, the image denoising method and apparatus provided in this application, based on the acquired image to be denoised, effectively improves the efficiency of denoising processing by scaling and extracting the image to be denoised. Since noise such as light, shadow, and transparency usually exhibits a uniform distribution in an image, and the noise features in local areas are basically consistent, scaling and extracting the image to be denoised can maintain the accuracy of noise recognition while significantly reducing computational complexity and improving feature extraction speed. Finally, pixel-level processing is performed on the image to be denoised and the noisy image to obtain a high-quality denoised image, avoiding the computational resource consumption caused by directly processing high-resolution images, and overcoming the detail loss problem caused by scaling the denoised image. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of an image denoising method provided in this application embodiment; Figure 2 Another flowchart of an image denoising method provided in this application embodiment; Figure 3 A flowchart of a noisy image prediction method provided in an embodiment of this application; Figure 4 A structural diagram of an encoder provided in an embodiment of this application; Figure 5 A structural diagram of a decoder provided in an embodiment of this application; Figure 6 A flowchart illustrating a denoising model training method provided in this application embodiment; Figure 7 This is a schematic diagram illustrating the process of training a denoising model, as provided in an embodiment of this application. Figure 8 A flowchart illustrating a method for calculating adversarial loss values ​​provided in this application embodiment; Figure 9 A structural block diagram of an image denoising device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an image denoising device provided in an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. Words such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. "Above," "below," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] As described in the background section, existing image denoising methods are still insufficient to meet practical needs. In developing this application, the applicant discovered that the main problem with existing image denoising methods is that, when denoising noisy images, they often employ deep learning methods to design different algorithm frameworks for different types of noise, thereby processing the various noise levels in the image.

[0026] However, the types of noise in images vary depending on factors such as the subject and the shooting environment. For example, moiré noise in images of computer screens and shadows and text transparency in images of documents constitute different types of document noise. Given the difficulty in unifying existing algorithm frameworks for different types of noise, it is impossible to use the same algorithm framework to denoise different types of noise in the same image.

[0027] Furthermore, since the different learning algorithms are generally quite large, typically around several hundred megabytes in size, they are difficult to deploy on mobile devices.

[0028] On the other hand, existing methods are generally implemented in a graph-to-graph manner. Therefore, when the model is denoising, it is usually necessary to first scale the image to a fixed size, and after denoising is completed, the denoised image is scaled back to the original image size. This operation of directly scaling the denoised image often results in a loss of image quality. In addition, if the image to be denoised is too large, denoising a large image takes a long time, which greatly affects the user experience.

[0029] Based on this, one or more embodiments of this application provide an image denoising method. In the image denoising process, by first scaling the image to be denoised, it can be reduced to a predetermined multiple, thereby accelerating the noise feature extraction process when extracting noise from the reduced image. After the feature map is extracted, the size of the feature map is scaled back to the original size of the image to be denoised, thus avoiding scaling the image to be denoised. By subtracting the image to be denoised from the noise image, a denoised image can be obtained, and the denoised image is the same size as the original image to be denoised, thus avoiding scaling the denoised image and ensuring that the image quality of the denoised image is not degraded by the denoising operation.

[0030] Since this method performs denoising through a single trained denoising model, it eliminates the need to design a separate denoising model for each type of noise. This allows the model to be deployed on the client side, such as in a mobile electronic device, enabling denoising to be performed directly on the client side without sending it to the server. This avoids back-and-forth transmission between the client and the server, thus shortening the time required for denoising.

[0031] The image denoising method provided in this application embodiment can be executed by a computer device. The computer device refers to any electronic device with data computing, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices. This application embodiment does not limit this.

[0032] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating an image correction method provided in an embodiment of this application. Figure 1 As shown, it includes the following steps: Step S101: Obtain the image to be denoised.

[0034] The image to be denoised can be, for example, a screen image taken with the screen as the subject, or a document image taken with a paper document as the subject, etc.

[0035] Due to lighting or the subject itself, such screen photos or document photos often contain a lot of noise. For example, moiré patterns may appear in screen photos, or shadows and / or transparent text may appear in document photos.

[0036] Therefore, the captured screen images and / or document images can be used as images to be denoised.

[0037] Step S102: Input the image to be denoised into the trained denoising model. The denoising model performs scaling and extraction processing on the image to be denoised to obtain the noise image of the image to be denoised.

[0038] Based on the image to be denoised obtained in step S101, the trained denoising model can be used to extract the noisy image from the image to be denoised.

[0039] The trained denoising model can downsample the image to be denoised, thereby extracting noise features from the downsized image and forming a noise feature map. Therefore, the resulting feature map is relatively small. However, since the noise in each small part of the image is basically the same, the smaller feature map does not reduce the noise extraction effect. Furthermore, since the noise features are extracted from the downsized image, the efficiency of noise feature extraction is more efficient than extracting noise features from the original size image to be denoised.

[0040] Specifically, for the original-sized image to be denoised, since its noise is often caused by light, such as moiré patterns, shadows, and / or transparent text, the light, shadows, and transparent text in the image to be denoised are usually uniform and do not change abruptly. Therefore, after dividing the original-sized image to be denoised into multiple small parts, it can be found that the noise in each small part of the image to be denoised is basically the same. For example, it can be considered that the noise at each position in a 2-pixel x 2-pixel area is the same.

[0041] Therefore, the noise feature of one pixel within each small locality can be used to represent the noise feature of that locality. For example, for a 2-pixel x 2-pixel region, the noise feature of one pixel can be used as the noise feature of that region. Thus, for a region with 4 pixels, the noise feature of only 1 pixel can be extracted to represent the noise feature of that region. This improves the efficiency of noise feature extraction by 4 times, while the noise within the small locality is basically the same, so it does not change the noise feature extraction effect.

[0042] Based on this, during the noise feature extraction process, the small local area can be reduced in size, so that noise features can be extracted pixel by pixel from the reduced local area. That is, the 2-pixel x 2-pixel region in the above specific example can be reduced by a factor of 4, and the reduced region is a single pixel. Thus, the noise features of the single pixel can be extracted, and the noise features of the single pixel can be used as the noise features of the 2-pixel x 2-pixel region.

[0043] Figure 2 Another flowchart of an image denoising method provided in this application embodiment is shown below. Figure 2As shown, in the process of noise feature extraction of the original size image to be denoised, the original size image to be denoised can be reduced in size, and then noise feature extraction can be performed on the reduced image to obtain the feature map of the corresponding reduced image. The size of the feature map is the same as the size of the reduced image.

[0044] The feature map can be, for example, a noise feature representing each pixel position in the scaled-down image. This noise feature can be represented, for example, the pixel value at that pixel position. In other words, the feature map represents the pixel value of the noise at each pixel position in the scaled-down image.

[0045] Furthermore, since the feature map at this point is different in size from the original image to be denoised before being scaled down, it is difficult to apply it to the original image to be denoised before being scaled down. Therefore, the feature map at this point can be enlarged.

[0046] In some alternative approaches, the feature map can be enlarged to the same size as the original image to be denoised before being reduced in size, for example by interpolation. This allows for feature extraction from the enlarged feature map, resulting in a noise image suitable for the original image to be denoised before being reduced in size.

[0047] Step S103: Perform pixel value processing on each pixel in the image to be denoised and the corresponding pixel in the noise image to obtain the denoised image.

[0048] Based on the noise image determined in step S103, since the size of the noise image is the same as the size of the image to be denoised obtained in step S101, the magnified noise image can represent the noise features of each pixel position in the image to be denoised. The noise features can be represented, for example, the pixel value of the pixel position. That is, the noise image represents the pixel value of the noise at each pixel position in the obtained image to be denoised.

[0049] Based on this, such as Figure 2 As shown, a denoised image of the same size as the denoised image can be obtained by subtracting the denoised image from the denoised image.

[0050] Specifically, during the subtraction operation, the pixel difference between each pixel in the image to be denoised and the corresponding pixel in the noise image can be calculated.

[0051] Furthermore, for each pixel in the obtained image to be denoised, i.e., the pixel position, its pixel value can be adjusted to the pixel difference of the corresponding pixel, thereby obtaining a denoised image of the same size as the obtained image to be denoised.

[0052] As can be seen, the image denoising method of the above embodiments of this application, based on the acquired image to be denoised, effectively improves the efficiency of denoising processing by reducing the size of the image to be denoised before extracting noise features. Since noise such as light, shadow, and transparency usually exhibits a uniform distribution in an image, and the noise features in local areas are basically consistent, extracting noise features from the reduced image can maintain the accuracy of noise recognition while significantly reducing computational complexity and increasing the feature extraction rate. Specifically, the feature map extracted after reducing the size of the image to be denoised is interpolated and enlarged to the original size of the image to be denoised, and then pixel-level subtraction is performed with the image to be denoised to obtain a high-quality denoised image. This avoids the computational resource consumption caused by directly processing high-resolution images and overcomes the problem of detail loss caused by scaling the denoised image.

[0053] In one alternative implementation, the denoising model described above includes an encoder and a decoder, thereby, as follows: Figure 3 As shown, based on the denoising model consisting of an encoder and a decoder, the process of obtaining a noisy image from the image to be denoised in step S102 may include, but is not limited to, the following steps S1021 and S1022: Step S1021: The encoder performs downsampling and noise feature extraction on the image to be denoised by a first multiple to obtain a feature map. The size of the image to be denoised is equal to the second multiple of the feature map, and the square of the first multiple is equal to the second multiple.

[0054] In the process of generating a noisy image, the encoder is used to reduce the size of the acquired image to be denoised and extract its noise features to obtain a noise feature map, which can be in the form of a feature matrix for representing noise, for example.

[0055] It is evident that this encoder possesses both the ability to reduce the size of the image to be denoised and the ability to extract noise features. Therefore, this encoder can be considered a hybrid encoder with mixed functions.

[0056] In the process of extracting noise features from the image to be denoised using this encoder, the obtained image to be denoised can be downsampled first, so that a smaller image can be obtained after downsampling.

[0057] During the downsampling operation, the downsampling operation can be performed at a preset first multiple, such as 2 times, i.e., 2 times downsampling. 2 times downsampling means that after the downsampling operation is completed, the length and width of the resulting reduced image are both half of the original image to be denoised before reduction, i.e., it is reduced by 2 times.

[0058] As can be seen, the reduced image has its area reduced by a factor of four because both its length and width have been reduced by a factor of two.

[0059] In other words, after downsampling by a first factor, the resulting reduced image is smaller than the image to be denoised by a second factor, and the square of the first factor is the same as the second factor.

[0060] Furthermore, feature extraction can be performed on the reduced image.

[0061] Specifically, local feature extraction can be performed on the reduced image first. Based on this, after extracting noise features from the reduced image, the resulting feature map has the same size as the reduced image. That is, the size of the feature map is also reduced by a factor of two compared to the image to be denoised.

[0062] In some embodiments, the encoder may include a downsampling layer and a noise feature extraction layer.

[0063] Figure 4 This is a structural diagram of an encoder provided in an embodiment of this application.

[0064] like Figure 4 As shown, the downsampling layer and the noise feature extraction layer are connected in series.

[0065] The downsampling layer is used to downsample the input image to be denoised by a first factor.

[0066] Specifically, the downsampling layer can be a convolutional neural network, wherein the downsampling layer can include, for example, Figure 4 The first feature extraction layer, the second feature extraction layer, and the third feature extraction layer are shown.

[0067] Therefore, the image to be denoised can be input into the first feature extraction layer in the downsampling layer for feature extraction to obtain shallow local features.

[0068] The first feature extraction layer can be a convolutional layer used to downsample shallow local features of noise, specifically for extracting sharp shadow edges, etc.

[0069] Among them, shallow local features can refer to the features of noise captured within a small receptive field. Since the noise capture range is small, shallow local features can usually be obvious features within a small receptive field, such as brightness abrupt changes and high-frequency noise points.

[0070] Furthermore, shallow local features can be input into the second feature extraction layer in the downsampling layer for feature extraction to obtain deep local features.

[0071] The second feature extraction layer can be a convolutional layer used to downsample deep local features of noise, specifically for extracting local color shifts in transparent characters, etc.

[0072] In contrast to shallow local features, deep local features can refer to the noise features captured within a larger receptive field. Since the noise capture range is larger, deep local features can usually be more complex noises exhibited within a larger receptive field, such as overall moiré noise and shadow noise within the receptive field, as well as abstract noise features, such as abstract semantic noise like transparent characters.

[0073] Furthermore, the deep local features can be input into the third feature extraction layer in the downsampling layer for feature extraction to obtain a scaled-down image.

[0074] The third feature extraction layer can be a convolutional layer used to further compress and refine noise features.

[0075] Based on this, after downsampling by a factor of one in the downsampling layer, a smaller image can be obtained.

[0076] Furthermore, such as Figure 4 As shown, the reduced image can be input into the noise feature extraction layer in the feature extraction layer for feature extraction, resulting in a feature map with the same size as the reduced image.

[0077] The noise feature extraction layer can extract global features of noise in the downsized input image. For example, it can be a neural network layer that combines a lightweight convolution mechanism with a Transformer self-attention mechanism. It can be used to determine the dependencies between each local noise in the downsized image and all other noises, that is, the global dependencies between each local noise feature and the global noise. For example, it can determine the global dependencies between the stripes of each local moiré pattern and the color distribution of each local transparent character and the global noise, thereby avoiding the need to design multiple models for different noises to determine global dependencies.

[0078] Based on this, the noise feature extraction layer can extract the global features of the noise based on the determined local features of the noise and the global dependencies of the local features, thereby obtaining a feature map with the same size as the image after reduction.

[0079] Step S1022: After the feature map is enlarged by a second multiple through the decoder, noise feature splicing, noise feature extraction and noise feature fusion are performed to obtain a noise image.

[0080] Based on the feature map determined in step S1021, since the size of the feature map is a multiple of the second size, which does not match the size of the image to be denoised, it can be input into the decoder to enlarge its size, and then noise feature splicing, noise feature extraction and noise feature fusion operations can be performed after enlargement.

[0081] It is evident that this decoder possesses both the ability to amplify feature maps and process noise features; therefore, it can be considered a hybrid decoder with mixed functions.

[0082] During the process of magnifying the feature map, the decoder can be used to magnify the feature map by a factor of two, so that the magnified feature map is obtained with the same size as the image to be denoised.

[0083] Furthermore, since the noise feature of a single pixel within each small locality in the image to be denoised can represent the noise feature of that small locality, the magnified feature map obtained after magnifying the feature map by a second factor can accurately represent the noise at each pixel location in the image to be denoised.

[0084] Furthermore, based on the obtained magnified feature map, the decoder can be used to perform noise feature splicing, noise feature extraction, and noise feature fusion on the magnified feature map.

[0085] Specifically, the decoder can first extract deep and shallow features from the magnified feature map, and then splice the extracted deep and shallow noise features to obtain spliced ​​features. After that, noise features are extracted from the spliced ​​features.

[0086] In the process of noise feature extraction from the magnified feature map, local feature extraction can be performed first to obtain the local features of the magnified feature map. Then, the global dependency relationship between each local feature and all other noise can be determined, that is, the global dependency relationship between each local feature of the noise and the global noise. Based on this, the global features of the noise in the magnified feature map can be extracted based on the determined local features of the noise in the magnified feature map and the global dependency relationship of each local feature. This achieves global feature extraction from the local features of the magnified feature map to obtain the global features of the magnified feature map.

[0087] Local features can be noise features extracted from various local areas of the image, i.e., features obtained by extracting noise features within a predetermined receptive field. Global features can be noise features extracted from the entire image. Specifically, they can be noise features obtained from the relationship between various local noises and the global dependency between this noise and all global noises.

[0088] Furthermore, noise features can be fused to the global features of the magnified feature map to obtain a noisy image of the same size as the image to be denoised.

[0089] In some embodiments, the decoder may include a transposed convolutional layer, a splicing layer, a feature extraction layer, and a feature fusion layer.

[0090] Figure 5 This is a structural diagram of a decoder provided in an embodiment of this application.

[0091] like Figure 5 As shown, The transposed convolutional layer, the splicing layer, the feature extraction layer, and the feature fusion layer are connected in series.

[0092] In the process of obtaining noisy images using a decoder, after inputting the aforementioned feature map into the decoder, as follows: Figure 5 As shown, the transposed convolutional layer enlarges the size of the feature map, and during the enlargement process, it is enlarged by a second factor, thus obtaining an enlarged feature map with the same size as the image to be denoised.

[0093] The magnified feature map includes deep local features, shallow local features, and global dependencies of various noises from the original feature map.

[0094] Therefore, the magnified feature map can be further input to... Figure 5 The splicing layer in the image is used to splice together the various noise features to obtain a noise splicing image.

[0095] Furthermore, the noise splicing image can be input into the feature extraction layer to extract noise features from the noise splicing image.

[0096] Specifically, the feature extraction layer includes a local feature extraction layer and a global feature extraction layer connected in series.

[0097] like Figure 5 As shown, the structure of the local feature extraction layer can be similar to that of the downsampling layer in the encoder. The difference is that the first feature extraction layer, the second feature extraction layer and the third feature extraction layer are connected in a different order. That is, in the local feature extraction layer, the third feature extraction layer, the second feature extraction layer and the first feature extraction layer are connected in a different order.

[0098] The noise splicing image can be input into the third feature extraction layer in the local feature extraction layer for feature extraction to obtain preliminary features.

[0099] The third feature extraction layer can be used to perform preliminary compression and refinement of noise in various local regions to obtain preliminary features.

[0100] Furthermore, the preliminary features can be input into the second feature extraction layer in the local feature extraction layer for feature extraction to obtain deep local features.

[0101] The second feature extraction layer can be a convolutional layer used to extract deep local features of noise, specifically for extracting local color shifts in transparent characters, etc.

[0102] Deep local features can refer to the noise features captured within a larger receptive field. Since the noise capture range is large, deep local features can usually be more complex noises that appear within a larger receptive field, such as overall moiré noise and shadow noise within the receptive field, as well as abstract noise features, such as abstract semantic noise like transparent characters.

[0103] Furthermore, deep local features can be input into the first feature extraction layer in the local feature extraction layer for feature extraction to obtain shallow local features.

[0104] The first feature extraction layer can be a convolutional layer used to downsample shallow local features of noise, such as for extracting sharp shadow edges.

[0105] In contrast to deep local features, shallow local features can refer to the noise features captured within a smaller receptive field. Since the noise capture range is small, shallow local features can usually be obvious features within a smaller receptive field, such as brightness abrupt changes and high-frequency noise points.

[0106] Furthermore, after the shallow and deep local features of the magnified feature map are extracted by the local feature extraction layer, the shallow local features, deep local features and noise splicing map can be input into the global feature extraction layer. The global feature extraction layer is used to extract global features from the noise in the original magnified feature map to obtain global features.

[0107] Furthermore, the acquired local and global features can be input into the feature fusion layer, which then performs noise feature fusion on the global and local features to obtain a noise image with the same size as the image to be denoised and the size of the magnified feature map.

[0108] As described above, efficient noise image prediction is achieved through the encoder and decoder architecture. Local noise features are extracted layer by layer through the local feature extraction layer, and the global dependencies of noise are modeled by combining the MobileViT Block structure, thereby comprehensively representing various noise characteristics in the feature map of the reduced image. The decoder enlarges the feature map through the transposed convolutional layer and organically combines the shallow noise features with the deep noise features, finally reconstructing a high-quality noise image that matches the size of the image to be denoised.

[0109] Figure 6 This is a flowchart illustrating a denoising model training method provided in an embodiment of this application. Figure 6 As shown, it includes the following steps: Step S601: Obtain the training dataset. The training dataset includes multiple sets of metadata. Each set of metadata includes multiple noisy training images and their corresponding actual clear images.

[0110] In the process of training a denoising model, it is necessary to first obtain a training dataset for training.

[0111] The training dataset contains multiple sets of metadata. Each set of original data contains noisy training images used as input to the denoising model to be trained. These noisy training images are images containing noise. Each set of metadata also includes the actual clear image corresponding to the noisy training image. The actual clear image is the image after the noise is removed from the noisy training image as expected.

[0112] In this step, the acquisition of each noise training image can be done by taking pictures of the document or screen. During the acquisition, pictures can be taken under different lighting conditions, different shooting angles, and different backlighting conditions to obtain noise training images with different levels of noise.

[0113] When photographing the same paper document, different lighting conditions, backlighting conditions, and shooting angles can be used to obtain different noise training images. In other words, paper documents with the same text, images, or graphics can be combined with the same clear actual image to form multiple sets of metadata under different lighting conditions, backlighting conditions, and shooting angles. Similarly, when photographing the same screen, different lighting conditions, backlighting conditions, and shooting angles can be used to obtain different noise training images. In other words, screen documents with the same text, images, or graphics can be combined with the same clear actual image to form multiple sets of metadata under different lighting conditions, backlighting conditions, and shooting angles.

[0114] Step S602: Input the metadata of a predetermined number of groups into the denoising model to be trained, and output the noise prediction image corresponding to each noise training image through the denoising model to be trained.

[0115] Based on the training dataset determined in S601 above, all metadata can be divided into multiple parts. That is, each part contains multiple sets of metadata, and each part constitutes all the metadata. In each round of training, the metadata of each set in one part is input into the correction model to be trained for prediction.

[0116] In the correction model to be trained, the number of metadata groups processed in each round of training can be set, and in the process of dividing all metadata in the training dataset into multiple parts, the division can be carried out according to the set number of groups, so that the number of metadata groups in each part is the same as the set number of groups.

[0117] Based on this, you can refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the process of training a denoising model, as provided in an embodiment of this application.

[0118] like Figure 7 As shown, after inputting the metadata of any part into the denoising model to be trained, the denoising model to be trained can predict the noise prediction image corresponding to the noise training image for each set of metadata noisy training image.

[0119] Step S603: Perform pixel value processing on each pixel in each noisy training image and the corresponding pixel in the corresponding noisy prediction image to obtain the corresponding denoised training image.

[0120] Based on the noise prediction images determined in S602 above, the noise prediction images can be subtracted from the corresponding noise training images to obtain the denoised training images corresponding to the noise training images.

[0121] Specifically, during the subtraction operation, the pixel difference between each pixel in the noisy training image and the corresponding pixel in the noisy prediction image can be calculated.

[0122] Furthermore, the pixel values ​​of each pixel in each noisy training image, that is, the pixel values ​​at each pixel position, are adjusted to the corresponding pixel difference to obtain the corresponding denoised training image.

[0123] Step S604: Determine the multimodal loss value corresponding to each denoised training image based on the multimodal loss function. Each multimodal loss value represents the noise in multiple dimensions of the corresponding denoised training image.

[0124] The multimodal loss function includes evaluations of the denoised training images from multiple dimensions.

[0125] Specifically, for the denoised training image determined in step S603 above, loss values ​​in multiple dimensions can be determined, and the loss values ​​in each dimension are input into the multimodal loss function for calculation, thereby obtaining the multimodal loss value.

[0126] Among them, such as Figure 7 As shown, the multimodal loss values ​​include the image loss value representing the difference between the denoised training image and the actual clear image, and the adversarial loss value representing the adversarial dimension.

[0127] Specifically, after predicting the denoised training image, a preset image loss function can be used to calculate the image loss value between the denoised training image and the corresponding real clear image.

[0128] The image loss value specifically represents the degree of difference between the denoised training image and the corresponding real clear image, that is, whether the denoising effect of the denoised training image is the same as or close to that of the real clear image.

[0129] Meanwhile, during the training of the denoising model, since the denoising model can be regarded as a generator of noise, a discriminator can be set for it. This discriminator is used to determine whether there is noise in the denoised training image, so that the generator and the discriminator can be used to build an adversarial relationship.

[0130] Based on this, the denoised training images obtained from the training are input into the discriminator for discrimination, and after discrimination, the presence of noise and the significance of the noise can be quantified into adversarial loss values.

[0131] Based on this, the determined image loss value and adversarial loss value can be input into the multimodal loss function. The multimodal loss function can then weight the image loss value and the adversarial loss value to determine the multimodal loss value.

[0132] Step S605: Train the denoising model to be trained based on the multimodal loss value to obtain the trained denoising model.

[0133] Based on the multimodal loss value determined in S604 above, the completion of training can be determined by comparing the relationship between the multimodal loss value and the preset loss threshold, as well as the number of training iterations completed.

[0134] In cases where the multimodal loss value is greater than the preset loss value threshold and / or the training dataset has not completed a predetermined number of training iterations, the parameters in the denoising model to be trained are adjusted, and different metadata of the same number of groups are input into the adjusted denoising model to generate multimodal loss value images corresponding to the noisy training images of each group of metadata.

[0135] Specifically, if the multimodal loss value is greater than the loss threshold, it can be considered that the generated denoised training image still has a large error compared with the expected clear image, and the denoising model to be trained still needs to be continuously trained.

[0136] Based on this, the parameters in the denoising model to be trained can be adjusted according to the difference between the multimodal loss value and the loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the adjusted denoising model for the next round of training. In the next round of training, the noise prediction images corresponding to each noisy training image are predicted again. Then, the corresponding noisy training images are subtracted again based on each noise prediction image to generate each actual clear image. The multimodal loss value is calculated again using the actual clear images and the discriminator. Specifically, during the subtraction operation, the pixel difference between each pixel in the noisy training image and the corresponding pixel in the noisy prediction image can be calculated.

[0137] Furthermore, for each pixel in the noisy training image, i.e., the pixel position, its pixel value can be adjusted to the pixel difference of the corresponding pixel, thereby obtaining an actual clear image with the same size as the noisy training image.

[0138] In other cases, the number of training rounds can be preset. Before the training dataset completes the required number of rounds, the parameters of the denoising model to be trained can be adjusted based on the difference between the multimodal loss value and the loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the adjusted correction model for the next round of training. In the next round of training, the noise prediction images corresponding to each noisy training image are predicted again. Then, the corresponding noisy training images are subtracted again based on each noise prediction image to generate each actual clear image. The multimodal loss value is calculated again using the actual clear images and the discriminator.

[0139] In other cases, the multimodal loss value and the number of training rounds can be combined for judgment. For example, if the multimodal loss value is greater than the loss threshold, it is then determined whether the training dataset has completed the predetermined number of rounds. If the number of rounds has not been completed, the parameters in the denoising model to be trained are adjusted based on the difference between the multimodal loss value and the loss threshold. After adjustment, based on the multiple parts of the training dataset mentioned above, the metadata of another part is input into the noise model to be trained for the next round of training. In the next round of training, the noise prediction images corresponding to each noise training image are predicted again, and then the corresponding noise training images are subtracted again based on each noise prediction image to generate each actual clear image. The multimodal loss value is then calculated again using the actual clear images and the discriminator. Based on the multimodal loss value determined in S604 above, if the multimodal loss value is less than or equal to the loss value threshold and / or the training dataset has completed a predetermined number of training iterations, the current denoising model to be trained is determined as the denoising model that has been trained.

[0140] Specifically, if the multimodal loss value is less than or equal to the loss threshold, the generated denoised training image can be considered to be the same as or close to the expected actual clear image, with a small error between them, and the denoised model can be considered to have been trained.

[0141] In other cases, based on a pre-set number of training rounds, it can be determined that a denoising model has been successfully trained when the training dataset has completed that number of rounds. In other cases, the multimodal loss value and the number of training rounds can be combined for judgment. For example, if the multimodal loss value is less than or equal to the loss threshold, it can be determined whether the training dataset has completed the predetermined number of rounds. If the training dataset has completed the predetermined number of rounds, it can be determined that the denoising model has been successfully trained.

[0142] It can be seen that training based on the multimodal loss function brings significant technical improvements to the denoising model. By constructing an adversarial loss value that includes noisy training images, actual clear images, and the discriminator, the model can comprehensively learn the complex features of noise. Dynamically adjusting the parameters of the denoising model during training, coupled with a predetermined number of iterations, ensures the stability and reliability of the denoising model's convergence. This results in a significant improvement in both accuracy and robustness of the trained denoising model, making it suitable for denoising needs under different noise levels.

[0143] Figure 8 A flowchart illustrating a method for calculating adversarial loss values ​​provided in an embodiment of this application. Figure 7As shown, the discriminator includes a first discriminator and a second discriminator. Based on this, after the denoised training image is input into the discriminator, the first discriminator and the second discriminator can simultaneously discriminate the denoised training image, thereby determining the adversarial loss value of the discriminator.

[0144] like Figure 8 As shown, the method of using the first discriminator and the second discriminator to discriminate the denoised training images and calculating the adversarial loss value specifically includes the following steps: Step S801: The first discriminator performs global noise judgment on the denoised training image to obtain the global adversarial loss value.

[0145] The first discriminator is used to perform global noise judgment on the denoised training images, that is, to judge from a global perspective whether there is noise in the denoised training images, or to judge whether the noise in the denoised training images is significant.

[0146] Furthermore, different judgment results regarding the presence or absence of noise can be assigned to different scores, and these different scores can be used as the corresponding global adversarial loss values.

[0147] Alternatively, the noise saliency of the denoised training images can be quantified into an image noise saliency value, and this value can be used as the corresponding global adversarial loss value.

[0148] Based on this, the denoised training image can be input into the first discriminator and global noise judgment can be performed to obtain the global adversarial loss value of the denoised training image.

[0149] Step S802: The second discriminator performs local noise judgment on the denoised training image to obtain the local adversarial loss value.

[0150] The second discriminator is used to judge local noise in the denoised training image, that is, to judge whether there is noise in the local part of the denoised training image from various local perspectives, and whether the noise is significant.

[0151] Specifically, when the denoised training image is input into the first discriminator, the denoised training image can be divided into multiple local regions, for example, into 16 local regions of the same size. The denoised training image after division is then input into the second discriminator, which uses the second discriminator to judge the local noise of each local region.

[0152] Furthermore, for each local region, different judgment results regarding the presence or absence of noise can correspond to different scores, and these different scores can be used as the local adversarial loss values ​​corresponding to that local region.

[0153] Alternatively, the noise significance of each local region can be quantified into a regional noise significance value, and the regional noise significance value can be used as the local adversarial loss value corresponding to that local region.

[0154] Step S803: Weight the global adversarial loss value and the local adversarial loss value to obtain the adversarial loss value.

[0155] In this step, corresponding weights can be pre-set for the local adversarial loss value and the global adversarial loss value.

[0156] Based on the global adversarial loss value determined in step S801 and the local adversarial loss value determined in step S802, the global adversarial loss value and the local adversarial loss value can be weighted according to the preset weights to obtain the adversarial loss value of the denoised training image.

[0157] As described above, a dual discrimination mechanism using a first discriminator and a second discriminator enables multi-dimensional noise evaluation of the denoised training images. The first discriminator analyzes the overall noise distribution characteristics of the denoised training images from a global perspective and outputs a global adversarial loss value that reflects the overall noise level of the image. The second discriminator divides the image into multiple local regions, detects the noise features of each local region, and generates a local adversarial loss value that characterizes the significance of local noise. By weighted and fused the global and local adversarial loss values, a comprehensive adversarial loss value that reflects the noise characteristics of the image is finally obtained, thereby effectively improving the performance and generalization ability of the denoising model.

[0158] Based on the same inventive concept, and corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an image denoising device.

[0159] Figure 9 This is a structural block diagram of an image denoising device provided in an embodiment of this application. The device is configured to execute the image denoising method provided in the above embodiment, and has corresponding functional modules and beneficial effects for executing the method. For example... Figure 9 As shown, the device includes: an acquisition module 901, a noise prediction module 902, and a noise reduction module 903; Module 901 is configured to acquire the image to be denoised. The noise prediction module 902 is configured to input the image to be denoised into the trained denoising model, and then use the denoising model to scale and extract the image to be denoised to obtain the noise image of the image to be denoised. The denoising module 903 is configured to perform pixel value processing on each pixel in the image to be denoised and the corresponding pixel in the noise image to obtain the denoised image.

[0160] As described above, based on the acquired image to be denoised, noise features are extracted after the image is reduced in size, which effectively improves the efficiency of denoising processing. Since noise such as light, shadow, and transparency typically exhibits a uniform distribution in an image, and the noise features in local areas are basically consistent, extracting noise features from the reduced image can maintain the accuracy of noise recognition while significantly reducing computational complexity and increasing the feature extraction rate. Specifically, the feature map extracted from the reduced image is interpolated and enlarged to the original size of the image to be denoised before pixel-level processing is performed with it, ultimately resulting in a high-quality denoised image. This avoids the computational resource consumption caused by directly processing high-resolution images and overcomes the detail loss problem caused by scaling the denoised image.

[0161] In one possible embodiment, the denoising model includes an encoder and a decoder; correspondingly, the noise prediction module 902 is further configured as follows: The denoising model is used to scale and extract the image to be denoised, resulting in a noisy image of the image to be denoised, including: The encoder performs a first-fold downsampling and noise feature extraction on the image to be denoised to obtain a feature map. The size of the image to be denoised is equal to the second-fold size of the feature map, and the square of the first-fold size equals the second-fold size. After the feature map is enlarged by a second factor using a decoder, noise features are spliced, extracted, and fused to obtain a noise image.

[0162] The encoder includes a downsampling layer and a noise feature extraction layer; Accordingly, the encoder performs a first-fold downsampling and noise feature extraction on the image to be denoised, resulting in a feature map, including: The image to be denoised is downsampled by a first factor through a downsampling layer to obtain a smaller image; A noise feature extraction layer is used to extract global texture features from the scaled-down image to obtain a feature map.

[0163] The decoder includes a transposed convolutional layer, a concatenation layer, a feature extraction layer, and a feature fusion layer. Accordingly, after the feature map is enlarged by a second factor using a decoder, noise feature stitching, noise feature extraction, and noise feature fusion are performed to obtain a noise image, including: The feature map size is enlarged to the same size as the image to be denoised by a second multiple of the transposed convolutional layer. The noise features in the magnified feature map are spliced ​​together by a splicing layer to obtain a noise spliced ​​map. Noise features are extracted from the noise splicing image through a feature extraction layer, and then the extracted noise features are fused through a feature fusion layer to obtain a noise image.

[0164] In one possible embodiment, the noise reduction module 903 is further configured as follows: Determine the pixel difference between the pixel value of each pixel in the image to be denoised and the pixel value of the corresponding pixel in the noisy image; The pixel values ​​at each pixel position are adjusted to the corresponding pixel differences to obtain a denoised image.

[0165] In one possible embodiment, the image denoising apparatus further includes a training module 904, configured as follows: Obtain the training dataset, which includes multiple sets of metadata. Each set of metadata includes multiple noisy training images and their corresponding actual clear images. The metadata of a predetermined number of groups is input into the denoising model to be trained, and the denoising model to be trained outputs the noise prediction image corresponding to each noise training image. Each pixel in the noisy training image is compared with the corresponding pixel in the corresponding noisy prediction image to obtain the corresponding denoised training image. The multimodal loss value corresponding to each denoised training image is determined based on the multimodal loss function. Each multimodal loss value represents the noise of the corresponding denoised training image in multiple dimensions. The denoising model to be trained is trained based on the multimodal loss value to obtain the trained denoising model.

[0166] The multimodal loss values ​​include image loss values ​​and adversarial loss values; Accordingly, the multimodal loss value corresponding to each denoised training image is determined based on the multimodal loss function, including: The adversarial loss value corresponding to each denoised training image is determined by a preset discriminator; The image loss value between each denoised training image and the corresponding actual clear image is determined based on the preset image loss function; For each denoised training image, the corresponding adversarial loss value and the corresponding image loss value are weighted to obtain the corresponding multimodal loss value.

[0167] The discriminator includes a first discriminator and a second discriminator; the adversarial loss value includes a global adversarial loss value and a local adversarial loss value. Accordingly, the adversarial loss value corresponding to each denoised training image is determined by a preset discriminator, including: Perform the following processing on each denoised training image: The first discriminator performs global noise judgment on the denoised training images to obtain the global adversarial loss value. The second discriminator performs local noise detection on the denoised training images to obtain the local adversarial loss value. The adversarial loss value is obtained by weighting the global adversarial loss value and the local adversarial loss value.

[0168] Specifically, the first discriminator performs global noise assessment on the denoised training images to obtain a global adversarial loss value, including: The first discriminator determines the noise level of the training images and uses this noise level as the global adversarial loss value.

[0169] Accordingly, the second discriminator performs local noise assessment on the denoised training images to obtain local adversarial loss values, including: The denoising training images are divided into multiple local regions; The second discriminator determines the regional noise level value in each local area and identifies the regional noise level value as the local adversarial loss value.

[0170] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0171] The apparatus described above is used to implement the corresponding image denoising method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0172] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an image denoising device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image denoising method as described in any of the above embodiments.

[0173] Figure 10 This is a schematic diagram of the structure of an image denoising device provided in an embodiment of this application, as shown below. Figure 10 As shown, the device includes a processor 1001, a memory 1002, an input device 1003, and an output device 1004; the number of processors 1001 in the device can be one or more. Figure 10 Taking a processor 1001 as an example; the processor 1001, memory 1002, input device 1003, and output device 1004 in the device can be connected via a bus or other means. Figure 10Taking a bus connection as an example, the memory 1002, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules for implementing the image denoising method in this embodiment. The processor 1001 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 1002, thereby implementing the aforementioned image denoising method. The input device 1003 can be configured to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 1004 may include a display screen or other display device.

[0174] The apparatus described above is used to implement the corresponding image denoising method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0175] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a non-volatile storage medium containing computer-executable instructions, which, when executed by a computer processor, are configured to perform an image denoising method described in the above embodiments, including: acquiring an image to be denoised; The image to be denoised is input into the trained denoising model for scaling and extraction processing to obtain a noisy image of the image to be denoised. The pixel values ​​of each pixel in the image to be denoised are subtracted from the corresponding pixels in the noisy image to obtain the denoised image.

[0176] It is worth noting that in the above-described embodiments of the image denoising device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not configured to limit the protection scope of the embodiments of this application.

[0177] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is configured to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can execute the image denoising method described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated further here.

Claims

1. A method for image denoising, characterized in that, include: Obtain the image to be denoised; The image to be denoised is input into the trained denoising model, and the denoising model performs scaling and extraction processing on the image to be denoised to obtain the noisy image of the image to be denoised. The pixel values ​​of each pixel in the image to be denoised are compared with the corresponding pixels in the noisy image to obtain the denoised image.

2. The image denoising method according to claim 1, characterized in that, The denoising model includes an encoder and a decoder; The step of scaling and extracting the image to be denoised using the denoising model to obtain a noisy image of the image to be denoised includes: The encoder performs downsampling and noise feature extraction on the image to be denoised by a first multiple to obtain a feature map. The size of the image to be denoised is equal to a second multiple of the feature map, and the square of the first multiple is equal to the second multiple. After the feature map is enlarged by the decoder according to the second factor, noise feature splicing, noise feature extraction and noise feature fusion are performed to obtain the noise image.

3. The image denoising method according to claim 2, characterized in that, The encoder includes a downsampling layer and a noise feature extraction layer; The step of downsampling and noise feature extraction of the image to be denoised by a first factor through the encoder to obtain a feature map includes: The image to be denoised is downsampled by the first factor through the downsampling layer to obtain a reduced image; The feature map is obtained by extracting global texture features from the reduced image through the noise feature extraction layer.

4. The image denoising method according to claim 2, characterized in that, The decoder includes a transposed convolutional layer, a splicing layer, a feature extraction layer, and a feature fusion layer; The process of enlarging the feature map by the second factor using the decoder to perform noise feature stitching, noise feature extraction, and noise feature fusion to obtain the noise image includes: The transposed convolutional layer enlarges the size of the feature map to be the same as the image to be denoised by the second multiple. The noise features in the magnified feature map are spliced ​​together by the splicing layer to obtain a noise spliced ​​map. The noise features are extracted from the noise mosaic image by the feature extraction layer, and the extracted noise features are fused by the feature fusion layer to obtain the noise image.

5. The image denoising method according to claim 1, characterized in that, The step of performing pixel value processing on each pixel in the image to be denoised and the corresponding pixel in the noisy image to obtain the denoised image includes: Determine the pixel difference between the pixel value of each pixel in the image to be denoised and the pixel value of the corresponding pixel in the noisy image; The pixel values ​​at each pixel position are adjusted to the corresponding pixel differences to obtain the denoised image.

6. The image denoising method according to claim 1, characterized in that, The denoising model was trained in the following way: Obtain a training dataset, which includes multiple sets of metadata, each set of metadata including multiple noisy training images and their corresponding actual clear images; A predetermined number of metadata groups are input into the denoising model to be trained, and the denoising model to be trained outputs the noise prediction image corresponding to each noise training image. Each pixel in the noisy training image is compared with the corresponding pixel in the corresponding noisy prediction image to obtain the corresponding denoised training image. The multimodal loss value corresponding to each denoised training image is determined based on the multimodal loss function. Each multimodal loss value represents the noise of the corresponding denoised training image in multiple dimensions. The denoising model to be trained is trained based on the multimodal loss value to obtain the trained denoising model.

7. The image denoising method according to claim 6, characterized in that, The multimodal loss values ​​include image loss values ​​and adversarial loss values; The step of determining the multimodal loss value corresponding to each denoised training image based on the multimodal loss function includes: The adversarial loss value corresponding to each denoised training image is determined by a preset discriminator; The image loss value between each denoised training image and the corresponding actual clear image is determined based on the preset image loss function; For each denoised training image, the corresponding adversarial loss value and the corresponding image loss value are weighted to obtain the corresponding multimodal loss value.

8. The image denoising method according to claim 7, characterized in that, The discriminator includes a first discriminator and a second discriminator; the adversarial loss value includes a global adversarial loss value and a local adversarial loss value; The step of determining the adversarial loss value corresponding to each denoised training image through a preset discriminator includes: Perform the following processing on each denoised training image: The first discriminator performs global noise judgment on the denoised training image to obtain a global adversarial loss value. The second discriminator performs local noise detection on the denoised training image to obtain a local adversarial loss value. The adversarial loss value is obtained by weighting the global adversarial loss value and the local adversarial loss value.

9. The image denoising method according to claim 8, characterized in that, The step of performing global noise judgment on the denoised training image through the first discriminator to obtain a global adversarial loss value includes: The first discriminator determines the noise level of the denoised training images and uses the noise level as the global adversarial loss value.

10. The image denoising method according to claim 8, characterized in that, The step of judging local noise in the denoised training image through the second discriminator to obtain a local adversarial loss value includes: The denoised training image is divided into multiple local regions; The second discriminator determines the regional noise level value in each local region and identifies the regional noise level value as the local adversarial loss value.

11. An image denoising device, characterized in that, include: Acquisition module, noise prediction module, and noise reduction module; The acquisition module is configured to acquire the image to be denoised; The noise prediction module is configured to input the image to be denoised into a trained denoising model, and then use the denoising model to scale and extract the image to be denoised to obtain a noisy image of the image to be denoised. The denoising module is configured to perform pixel value processing on each pixel in the image to be denoised and the corresponding pixel in the noise image to obtain a denoised image.

Images