Image noise reduction model training method, image noise reduction method and system
By introducing a time embedding unit and a merged operation layer into the image noise reduction model, combined with the degree of noise damage, the problems of difficulty in detail and texture recovery and low noise reduction efficiency in the image noise reduction process in the prior art are solved, and a more efficient image noise reduction effect is achieved.
Patent Information
- Application Number
- CN202411633207.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-03-25
AI Technical Summary
The prior art is difficult to effectively restore image details and textures during image noise reduction, and the noise reduction efficiency is low.
An image noise reduction model training method is adopted to train an image noise reduction model including an encoder and a decoder by obtaining noise images, noiseless images and noise damage as data training sets. A time embedding unit and a merging operation layer are introduced in the decoder to receive and merge the degree of noise damage, thereby generating a noise-free image with more details and textures.
The image noise reduction model is realized with more efficient noise reduction capabilities and better image recovery effects. The generated noise reduction images have more details and natural textures, and the noise reduction efficiency is also improved.
Smart Images
Figure CN119693261A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image denoising model training method, an image denoising method, a system and a storage medium. Background Art
[0002] In today's digital age, images, as an important carrier of information transmission, are widely used in many key fields. In the photography industry, high-quality images can better record beautiful moments; in the field of medical imaging, clear images are crucial for accurate diagnosis of diseases; security monitoring systems rely on clear images to effectively identify and track targets.
[0003] However, in the process of image acquisition, storage and transmission, various noises are inevitably introduced, such as Gaussian noise, salt and pepper noise, etc., which seriously affect the image quality. Image denoising technology aims to effectively remove various types of noise in images. The denoised images have higher clarity and fineness, richer details and more realistic colors. This not only improves the visual effect of the image, but also provides a more reliable foundation for image analysis, processing and application, greatly expanding the application value and potential of the image. For example, the use of convolutional neural networks to automatically learn the features of images can effectively remove noise; the use of adversarial networks to remove noise can generate more realistic denoised images.
[0004] Existing image denoising algorithms can remove noise and improve image quality to a certain extent, but the denoised image still lacks a lot of details and textures, and the image denoising efficiency is low. Summary of the invention
[0005] In order to solve the defects of the prior art, the present invention provides an image denoising model training method, an image denoising method, a system and a storage medium.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] In a first aspect, an embodiment of the present invention provides an image denoising model training method, comprising the following steps:
[0008] Obtain a data training set, including noisy images, noise-free images, and noise damage levels;
[0009] Training the image denoising model using the data training set;
[0010] The image denoising model comprises: an encoder and a decoder, the decoder comprises a plurality of decoding modules, each decoding module comprises: at least two convolutional layers, at least two activation function layers, at least one merging operation layer and at least one time embedding unit, the time embedding unit is used to receive the noise damage degree;
[0011] After the noise image is processed by the convolution layer and the activation function layer of the decoder each time, the merging operation layer merges the time embedding unit into the processed image, and then makes it pass through the convolution layer and the activation function layer in sequence.
[0012] In some embodiments, the time embedding unit is generated by performing an embedding operation on the time steps.
[0013] In some embodiments, the image denoising model further includes: a downsampling and reorganization module and an upsampling and reorganization module;
[0014] The downsampling and reorganization module is used to divide the noisy image into a number of non-overlapping small blocks, and obtain the same position in each block respectively to form a new image, and then stack all the new images to the channel dimension, and then input them into the encoder for processing;
[0015] The up-sampling and reorganization module restores the image processed by the decoder to an image with the same resolution as the noisy image.
[0016] In some embodiments, the combined operation layer is a stacked layer or a fused layer.
[0017] In some embodiments, the training loss function of the image denoising model includes a mean square error loss function, a generative adversarial loss function, and a perceptual loss function.
[0018] In a second aspect, an embodiment of the present invention provides an image noise reduction method, comprising the following steps:
[0019] Acquire an image to be denoised;
[0020] The image to be denoised is input into an image denoising model trained by any training method of the first aspect to perform image denoising processing to obtain a denoised image, wherein the time embedding unit has a preset noise destruction degree.
[0021] In some embodiments, it also includes:
[0022] The noise damage degree is obtained, and it is input into the image denoising model together with the image to be denoised to perform image denoising processing to obtain a denoised image, wherein the noise damage degree is used to replace the preset noise damage degree of the time embedding unit.
[0023] In a third aspect, an embodiment of the present invention provides an image noise reduction system, including:
[0024] An image acquisition module, used for acquiring an image to be denoised;
[0025] An image denoising module is used to input the image to be denoised into an image denoising model trained by any training method of the first aspect to perform image denoising processing to obtain a denoised image, wherein the time embedding unit stores a preset noise destruction degree.
[0026] In some embodiments, it also includes:
[0027] A noise acquisition module is used to obtain the degree of noise damage;
[0028] At this time, the image denoising module is used to input the noise destruction degree together with the image to be denoised into the image denoising model for image denoising processing to obtain a denoised image, wherein the noise destruction degree is used to replace the preset noise destruction degree of the time embedding unit.
[0029] In a fourth aspect, an embodiment of the present invention provides a storage medium having instructions stored thereon, and when the instructions are executed by a processor, the image denoising method described in the second aspect is implemented.
[0030] The present invention provides an image denoising model training method, an image denoising method, a system and a storage medium. Compared with the prior art, the present invention has the following technical effects:
[0031] 1. Using noisy images, noise-free images and noise damage levels as data training sets, an image denoising model including an encoder, an intermediate module and a decoder is trained, and the decoder includes a convolution layer, an activation function layer, a merging operation layer, a time embedding unit and an upsampling layer, wherein the time embedding unit is used to receive the noise damage level, and merge the noise damage level into the image in the image denoising process through the merging operation layer, so as to realize that the image denoising model has the ability to generate texture and denoise, so that the trained image denoising model has better image denoising ability and efficiency, so that after the noise image is denoised by the image denoising model, a noise-free image with more details and texture is generated.
[0032] 2. Add upper and lower pixel reorganization modules before and after the image denoising model to rearrange the image data. By cutting off the original connection between pixels, block noise is converted into point noise, so that only point noise needs to be removed to achieve denoising. A denoising destruction strength parameter is introduced in the decoder part to achieve controllable denoising. Upsampling is added at the end of the model to restore the reorganized image to its original resolution, thereby obtaining the final denoised image, thereby achieving a better image denoising generation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description only involve some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0034] Figure 1 This is a flow chart of an image denoising model training method according to an embodiment of the present invention;
[0035] Figure 2 Schematic diagram of noise images corresponding to different noise damage degrees;
[0036] Figure 3 Schematic diagram of the process of generating a temporal embedding unit;
[0037] Figure 4 Schematic diagram of stacking operation on time embedding unit;
[0038] Figure 5 This is a schematic diagram of the state change of an image before and after being processed by the up- and down-sampling reorganization module;
[0039] Figure 6 This is a flow chart of an image noise reduction method according to an embodiment of the present invention;
[0040] Figure 7 This is a schematic diagram of the structure of an image noise reduction system according to an embodiment of the present invention;
[0041] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention;
[0042] Fig. 9 Schematic diagram of the structure of the image denoising model. DETAILED DESCRIPTION
[0043] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, other solutions obtained by ordinary technicians in this field without creative work should belong to the protection scope of the present invention.
[0044] Example 1
[0045] Figure 1 A flow chart of an image denoising model training method according to an embodiment of the present invention is shown. Figure 1 It can be seen that the image denoising model training method includes the following steps:
[0046] S10. Obtain a data training set, including a noisy image, a noise-free image, and a noise damage degree.
[0047] S12: training the image denoising model using the data training set.
[0048] Preferably, 100,000 high-definition images are collected from open source data sets div2k and LSDIR, or common websites unsplash, pexels, dreamstim, stock_adobe, shutterstock, and istockphoto, and the images are screened and cropped to produce a data training set.
[0049] For example, eliminate low-definition, compressed, noisy, blurred images, etc., and only select high-definition images with a small background area. For images with a large background area, crop the foreground part and retain more effective information images as noise-free images.
[0050] Then, the added noise is determined to be Gaussian noise I, with a mean of 0 and a variance of 1, and the maximum degree of image damage is determined. Then, a noisy image is generated based on the noise-free image, Gaussian noise I and the degree of noise damage. Finally, a data training set containing the noise image, the noise-free image and the degree of noise damage is generated.
[0051] For example, 100 samples are sampled in a linear sampling manner as the ratio of the noise-free image M. The maximum degree of damage is 0.15. 100 samples are sampled in a linear sampling manner from 0 to 0.15. Assuming that one of the samples has a value of 0.1, the final noise image is M*(1-0.1)+I*0.1). Figure 2 As shown, the degree of noise damage increases from left to right.
[0052] The image denoising model includes: an encoder, an intermediate module and a decoder. For example, the encoder is a U-Net encoder, and the encoder includes multiple encoding modules and multiple AvgPool pooling layers, preferably 4 encoding modules, each encoding module includes: a convolution layer and a relu activation function layer.
[0053] The decoder includes multiple decoding modules and multiple Bilinear Upsample upsampling layers, preferably 4 decoding modules, each decoding module includes: at least two convolutional layers, at least two activation function layers, at least one merging operation layer and at least one time embedding unit. The merging operation layer is a stacking layer or a fusion layer, and the time embedding unit is generated by embedding the time step and is used to receive the noise damage degree. Figure 3As shown in the figure, the time step time_step is processed by the Embedding layer, and then its dimension is expanded by the Linear fully connected layer, and then it is activated by the SiLU function, and finally processed by a Linear fully connected layer to obtain the time embedding unit Time_Embedding.
[0054] The time embedding unit is used to provide the image denoising model with the noise damage level corresponding to the current noise image. Figure 4 As shown, each time the noise image is processed by the conv convolution layer and the relu activation function layer of the decoder, the merging operation layer, such as the Concat stacking layer, merges the time embedding unit Time_Embedding into the processed image, and then makes it pass through the conv convolution layer and the relu activation function layer in sequence.
[0055] Preferably, the training loss function of the image denoising model includes a mean square error loss function, a generative adversarial loss function and a perceptual loss function, and the ratio of the three is 1:1:1, and it is best to use them together to train and optimize the image denoising model.
[0056] This embodiment uses a noisy image, a noise-free image and a noise damage degree as a data training set, and then uses the data training set to train an image denoising model including an encoder, an intermediate module and a decoder, and the decoder includes a convolution layer, an activation function layer, a merging operation layer, a time embedding unit and an upsampling layer, wherein the time embedding unit is used to receive the noise damage degree, and merge the noise damage degree into the image in the image denoising process through the merging operation layer, so as to realize that the image denoising model has the ability to generate texture and denoise, so that the trained image denoising model has better image denoising ability and efficiency, and then the noisy image is denoised by the image denoising model to generate a noise-free image with more details and texture.
[0057] In another embodiment, the image denoising model further includes: a downsampling and reorganization module and an upsampling and reorganization module;
[0058] The downsampling and reorganization module is used to divide the noise image into a number of non-overlapping small blocks, and obtain the same position in each block respectively to form a new image, and then stack all the new images to the channel dimension, and then input them into the encoder for processing. In other words, the downsampling and reorganization module is used to rearrange the image data, and by cutting off the original connection between the pixels, the block noise is converted into point noise, so that only the point noise needs to be removed to achieve noise reduction. The image processed by the downsampling and reorganization module still retains all the pixel information of the original image, and only the pixels are reorganized to restore the original image.
[0059] The up-sampling and reorganization module restores the image processed by the decoder to an image with the same resolution as the noisy image.
[0060] That is, the downsampling and reconstruction module breaks the connectivity information between the pixels of the original noisy image. The encoder gradually extracts high-level features of the image through continuous convolution and downsampling operations to understand the global information of the image. The decoder gradually restores the resolution of the image through upsampling operations, combines the corresponding feature map from the encoder, fuses low-level and high-level features, and integrates the generated denoising degree into the convolution part to achieve the effect of controlling the denoising generation. Finally, the upsampling and reconstruction module is added to restore the image information, thereby obtaining the final denoised image.
[0061] like Figure 5 As shown in the figure, the downsampling and reorganization module divides the input image into several non-overlapping 4x4 blocks, and obtains the same position in each block to form a new image, and then stacks these new images to the channel dimension. In this way, the image can be losslessly downsampled when input, and the original information connection between pixels is separated, thereby converting the noise in the real world into the point noise domain required for model training, thereby achieving the effect of denoising. The upsampling and reorganization module performs the opposite restoration processing on the image after the previous reorganization processing, performs upsampling processing, and outputs an image with the same resolution size as the input as the final denoised image.
[0062] That is to say, this embodiment adds upper and lower pixel reorganization modules before and after the image denoising model of Embodiment 1, respectively, for rearranging image data, and converting block noise into point noise by cutting off the original connection between pixel points, so that only the point noise needs to be removed to achieve denoising, and introduces a denoising destruction strength parameter in the decoder part, so as to achieve a controllable degree of denoising, and adds upsampling at the end of the model to restore the reorganized image to its original resolution, thereby obtaining the final denoising map, and further achieving a better image denoising generation effect.
[0063] Example 2
[0064] Figure 6 A flow chart of an image noise reduction method according to an embodiment of the present invention is shown. Figure 6 It can be seen that the image denoising method comprises the following steps:
[0065] S60: Obtain an image to be denoised.
[0066] S62: Input the image to be denoised into the image denoising model trained by the image denoising model training method of the aforementioned embodiment to perform image denoising processing to obtain a denoised image.
[0067] The time embedding unit has a preset noise destruction degree, the range of which is 0.1-0.9, and the optimal value is 0.5 or 0.6.
[0068] In another embodiment, it is also necessary to obtain the degree of noise destruction and input it together with the image to be denoised into the image denoising model for image denoising processing to obtain a denoised image, wherein the degree of noise destruction is used to replace the preset noise destruction degree of the time embedding unit.
[0069] This embodiment introduces a noise reduction destruction strength parameter in the decoder part. Either a preset noise destruction degree or an input noise destruction degree can be used to achieve a controllable noise reduction degree, thereby achieving a better image noise reduction generation effect. The denoised image has a more natural texture without losing details.
[0070] Example 3
[0071] Figure 7 FIG. 1 shows a schematic diagram of the structure of an image noise reduction system according to an embodiment of the present invention. Figure 7 It can be seen that the image denoising system includes: an image acquisition module 71 and an image denoising module 72 .
[0072] The image acquisition module 71 is used to acquire the image to be denoised;
[0073] The image denoising module 72 is used to input the image to be denoised into the image denoising model trained by the image denoising model training method of the aforementioned embodiment to perform image denoising processing to obtain a denoised image, wherein the time embedding unit stores a preset noise destruction degree.
[0074] In another embodiment, the image noise reduction system further includes: a noise acquisition module 73 .
[0075] The noise acquisition module 73 is used to obtain the degree of noise damage;
[0076] At this time, the image denoising module 72 is used to input the noise destruction degree together with the image to be denoised into the image denoising model for image denoising processing to obtain a denoised image, wherein the noise destruction degree is used to replace the preset noise destruction degree of the time embedding unit.
[0077] This embodiment introduces a noise reduction destruction strength parameter in the decoder part. Either a preset noise destruction degree or an input noise destruction degree can be used to achieve a controllable noise reduction degree, thereby achieving a better image noise reduction generation effect. The denoised image has a more natural texture without losing details.
[0078] Example 4
[0079] An embodiment of the present invention provides a readable storage medium, comprising: computer program instructions, which, when executed by at least one processor of an electronic device, enable the electronic device to implement the image denoising method provided by any of the above method embodiments.
[0080] Example 5
[0081] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Figure 8 It can be seen that an electronic device 50 includes: a memory 52 and a processor 54;
[0082] The memory 52 may be an independent physical unit, and may be connected to the processor 54 via a bus 56. The memory 52 and the processor 54 may also be integrated together and implemented via hardware.
[0083] The memory 52 is used to store program instructions, and the processor 54 calls the program instructions to execute the image noise reduction method provided by any of the above method embodiments.
[0084] Optionally, when part or all of the methods of the above embodiments are implemented by software, the above electronic device 50 may also only include a processor 54. The memory 52 for storing programs is located outside the electronic device 50, and the processor 54 is connected to the memory through circuits / wires to read and execute the programs stored in the memory.
[0085] The processor 54 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.
[0086] The processor 54 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0087] The memory 52 may include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory may also include a combination of the above types of memory.
[0088] Example 6
[0089] An embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the image denoising method of the above embodiment is implemented.
[0090] Example 7
[0091] Preferably, if Fig. 9 As shown in the figure, the above image denoising model is improved from the U-Net model, and the image denoising model includes a downsampling and reorganization module (referred to as "downsampling" in the figure), a Space to Depth module, a new Unet structure module, a Depth to Space module and an upsampling and reorganization module (referred to as "upsampling" in the figure).
[0092] Among them, the new Unet structure module includes: encoder, intermediate module, decoder, the encoder includes 4 encoding modules (blue rectangles in the figure) and 4 AvgPool pooling layers, each encoding module includes: 3 conv convolutional layers and 3 relu activation function layers (as shown in the detailed structure of the blue rectangle in the figure). The decoder includes 4 decoding modules (red rectangles in the figure) and 4 Bilinear Upsample upsampling layers, each decoding module (as shown in the detailed structure of the red rectangle in the figure) includes: 3 conv convolutional layers, 3 relu activation function layers, 2 concat merging operation layers and 2 Time_Embedding time embedding units (not shown in the figure). The functions of each part are similar to the above description and will not be repeated.
[0093] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A method for training an image denoising model, characterized in that: The steps include: Obtain a data training set, including noisy images, noise-free images, and noise damage levels; Training the image denoising model using the data training set; The image denoising model comprises: an encoder and a decoder, the decoder comprises a plurality of decoding modules, each decoding module comprises: at least two convolutional layers, at least two activation function layers, at least one merging operation layer and at least one time embedding unit, the time embedding unit is used to receive the noise damage degree; After the noise image is processed by the convolution layer and the activation function layer of the decoder each time, the merging operation layer merges the time embedding unit into the processed image, and then makes it pass through the convolution layer and the activation function layer in sequence.
2. The image denoising model training method according to claim 1, characterized in that: The time embedding unit is generated by performing an embedding operation on the time steps.
3. The image denoising model training method according to claim 1 or 2, characterized in that: The image denoising model further includes: a downsampling and reorganization module and an upsampling and reorganization module; The downsampling and reorganization module is used to divide the noisy image into a number of non-overlapping small blocks, and obtain the same position in each block respectively to form a new image, and then stack all the new images to the channel dimension, and then input them into the encoder for processing; The up-sampling and reorganization module restores the image processed by the decoder to an image with the same resolution as the noisy image.
4. The image denoising model training method according to claim 3, characterized in that: The merging operation layer is a stacking layer or a fusion layer.
5. The image denoising model training method according to claim 4, characterized in that: The training loss functions of the image denoising model include a mean square error loss function, a generative adversarial loss function and a perceptual loss function.
6. An image denoising method, characterized in that: The steps include: Acquire an image to be denoised; The image to be denoised is input into an image denoising model trained by any training method according to claims 1-5 to perform image denoising processing to obtain a denoised image, wherein the time embedding unit has a preset noise destruction degree.
7. The image denoising method according to claim 6, characterized in that: Also includes: The noise damage degree is obtained, and it is input into the image denoising model together with the image to be denoised to perform image denoising processing to obtain a denoised image, wherein the noise damage degree is used to replace the preset noise damage degree of the time embedding unit.
8. An image noise reduction system, characterized in that: include: An image acquisition module, used for acquiring an image to be denoised; An image denoising module is used to input the image to be denoised into an image denoising model trained by any training method of claims 1-5 to perform image denoising processing to obtain a denoised image, wherein the time embedding unit has a preset noise destruction degree.
9. The image noise reduction system according to claim 8, characterized in that: Also includes: A noise acquisition module is used to obtain the degree of noise damage; At this time, the image denoising module is used to input the noise destruction degree together with the image to be denoised into the image denoising model for image denoising processing to obtain a denoised image, wherein the noise destruction degree is used to replace the preset noise destruction degree of the time embedding unit.
10. A storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the image noise reduction method according to claim 6 or 7 is implemented.