Image denoising model processing method and device, and electronic equipment
By extracting the edge feature differences between the teacher and student models in the image denoising model, calculating the reconstruction loss value and adjusting the parameters, the problems of difficult model deployment on edge computing devices and lack of target in global distillation methods are solved, achieving a more efficient image denoising effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU HUYA INFORMATION TECH CO LTD
- Filing Date
- 2022-12-01
- Publication Date
- 2026-05-12
AI Technical Summary
Existing image denoising models are difficult to deploy on edge computing devices, and global knowledge distillation methods lack clear learning objectives, which limits the performance improvement of student models.
By extracting the differences in edge feature images between the teacher model and the student model, calculating the reconstruction loss value, and adjusting the parameters of the student model according to the differences, the learning focuses on image edge features, thus narrowing the learning scope.
It improves the image denoising capability of student models, reduces the difficulty of model deployment and computational load, and enhances the efficiency of knowledge distillation learning.
Smart Images
Figure CN115908827B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and more specifically, to an image denoising model processing method, apparatus, and electronic device. Background Technology
[0002] Images play a vital role in human society, but their formation, transmission, and storage are inevitably affected by external factors, resulting in noticeable noise. Thanks to the rapid development of deep learning, related technologies have been widely applied to image denoising tasks, achieving remarkable results. However, most high-performance image denoising algorithms involve a large number of network parameters. Achieving fast inference speeds requires deploying these algorithms on high-performance but expensive devices, increasing the cost of application. Furthermore, edge computing devices, such as mobile phones and cameras, are limited by their computing and storage capabilities, making it impossible to deploy these large-parameter image denoising models. Reducing the number of parameters and computational load in image denoising models can significantly reduce deployment difficulty. However, while constructing image denoising models with fewer parameters can lower deployment costs and improve inference speed, it inevitably reduces the model's denoising capabilities.
[0003] Therefore, some image denoising solutions employ knowledge distillation to train models. This involves first constructing a teacher model with a large number of parameters and excellent denoising capabilities, and then constructing a student model with fewer parameters but relatively weaker denoising capabilities. During training, the student model gradually fits the output of the teacher model, thus gradually bringing its performance closer to that of the teacher model. However, current knowledge distillation methods in image denoising are primarily based on complete feature maps or the complete model output. This global distillation learning leaves the student model without a clear learning objective, thereby reducing the benefits of knowledge distillation. Summary of the Invention
[0004] To overcome the aforementioned shortcomings in the prior art, the purpose of this application is to provide an image denoising model processing method, comprising:
[0005] Obtain the trained teacher model and the student model to be trained. The teacher model and the student model are used to perform denoising processing on the image. The structural complexity of the student model is less than that of the teacher model.
[0006] Obtain training samples, which include images to be processed that need to be noisy;
[0007] The images to be processed in the training samples are input into the teacher model and the student model respectively to obtain the first processing result output by the teacher model and the second processing result output by the student model;
[0008] Extract a first edge feature image corresponding to the edge position of the image content from the first processing result, and extract a second edge feature image corresponding to the edge position of the image content from the second processing result;
[0009] The first reconstruction loss value is calculated based on the difference between the first edge feature image and the second edge feature image;
[0010] The model parameters of the student model are adjusted based on the first reconstruction loss value.
[0011] In one possible implementation, the training samples further include a noise-removed reference image corresponding to the image to be processed; the method further includes:
[0012] Based on the reference image and the second processing result, the reconstruction loss is calculated to obtain the second reconstruction loss value;
[0013] The step of adjusting the model parameters of the student model based on the first reconstruction loss value includes:
[0014] The model parameters of the student model are adjusted based on the first reconstruction loss value and the second reconstruction loss value.
[0015] In one possible implementation, the training samples further include a noise-removed reference image corresponding to the image to be processed; the steps of extracting a first edge feature image corresponding to the edge position of the image content from the first processing result and extracting a second edge feature image corresponding to the edge position of the image content from the second processing result include:
[0016] Image content edge recognition is performed on the reference images in the training samples to obtain the edge contour position information of the image content;
[0017] The first edge feature image is obtained by extracting the edge contour position information from the first processing result, and the second edge feature image is obtained by extracting the edge contour position information from the second processing result.
[0018] In one possible implementation, the edge contour location information includes an edge contour mask image, wherein the value of the pixel corresponding to the edge position in the edge contour mask image is 1, and the value of the pixel corresponding to the non-edge position is 0.
[0019] The steps of extracting the first edge feature image from the first processing result based on the edge contour position information, and extracting the second edge feature image from the second processing result based on the edge contour position information, include:
[0020] The first processing result is multiplied pixel by pixel with the edge contour mask image to obtain the first edge feature image;
[0021] The second processing result is multiplied pixel by pixel with the edge contour mask image to obtain the second edge feature image.
[0022] In one possible implementation, the step of calculating the first reconstruction loss value based on the difference between the first edge feature image and the second edge feature image includes:
[0023] A difference feature map is calculated based on the difference values between each pixel in the first edge feature image and the second edge feature image.
[0024] The first processing result is multiplied pixel by pixel with the difference feature map to obtain the first weighted result;
[0025] The second processing result is multiplied pixel by pixel with the difference feature map to obtain the second weighted result;
[0026] The reconstruction loss is calculated based on the first weighted result and the second weighted result to obtain the first reconstruction loss value.
[0027] In one possible implementation, the step of calculating the difference feature map based on the difference values between each pixel in the first edge feature image and the second edge feature image includes:
[0028] The absolute value of the difference between each corresponding pixel in the first edge feature image and the second edge feature image is calculated to obtain the difference feature map.
[0029] In one possible implementation, the step of inputting the image to be processed from the training samples into the teacher model and the student model respectively, and obtaining a first processing result output by the teacher model and a second processing result output by the student model, includes:
[0030] The images to be processed in the training samples are input into the teacher model and the student model respectively. The image denoising result output by the teacher model is used as the first processing result, and the image denoising result output by the student model is used as the second processing result.
[0031] Alternatively, the images to be processed in the training samples can be input into the teacher model and the student model respectively, and the intermediate feature map extracted by the teacher model when performing image denoising can be used as the first processing result, and the intermediate feature map extracted by the student model when performing image denoising can be used as the second processing result.
[0032] Another objective of this application is to provide an image denoising model processing apparatus, the image denoising model processing apparatus comprising:
[0033] The model acquisition module is used to acquire the trained teacher model and the student model to be trained. The teacher model and the student model are used to perform denoising processing on the image. The structural complexity of the student model is less than that of the teacher model.
[0034] The sample acquisition module is used to acquire training samples, which include the image to be processed that has noise and the reference image after noise removal.
[0035] The image processing module is used to input the images to be processed in the training samples into the teacher model and the student model respectively, and obtain the first processing result output by the teacher model and the second processing result output by the student model;
[0036] An edge feature extraction module is used to extract a first edge feature image corresponding to the edge position of the image content from the first processing result, and to extract a second edge feature image corresponding to the edge position of the image content from the second processing result;
[0037] The loss calculation module is used to calculate a first reconstruction loss value based on the difference between the first edge feature image and the second edge feature image;
[0038] The model adjustment module is used to adjust the model parameters of the student model based on the first reconstruction loss value.
[0039] Another objective of this application is to provide an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions, when executed by the processor, implement the image denoising model processing method provided in this application.
[0040] Another objective of this application is to provide a machine-readable storage medium storing machine-executable instructions that, when executed by one or more processors, implement the image denoising model processing method provided in this application.
[0041] Compared with the prior art, this application has the following beneficial effects:
[0042] The image denoising model processing method, apparatus, and electronic device provided in this application calculate the reconstruction loss value of the student model based on the edge feature differences of the processing results output by the teacher model and the student model during the training of the student model. This enables image denoising knowledge distillation based on edge feature differences during the training process of the student model, allowing the student model to focus its learning on the edge features in the image, narrowing the learning scope of the student model, thereby improving the improvement efficiency brought by knowledge distillation learning. Attached Figure Description
[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is one of the step flowcharts of the image denoising model processing method provided in the embodiments of this application;
[0045] Figure 2 This is the second schematic diagram of the steps of the image denoising model processing method provided in the embodiments of this application;
[0046] Figure 3 This is a schematic diagram of data transfer during the training process provided in an embodiment of this application;
[0047] Figure 4 A schematic diagram of an electronic device provided in an embodiment of this application;
[0048] Figure 5 This is a schematic diagram of the functional modules of the image denoising model processing device provided in the embodiments of this application. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0050] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0051] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0052] In the description of this application, it should be noted that the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0053] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0054] The inventors discovered that in some image denoising scenarios, most areas of the image to be processed are regions with relatively flat data changes, such as the blue sky or a smooth, textureless wall. These flat areas have relatively simple features, and even smaller models can learn the denoising capabilities for these areas well. However, in high-frequency image content edge regions (e.g., the boundary between foreground and background images), feature changes are often significant, making it difficult for smaller models to learn the denoising capabilities for these areas. Therefore, focusing on the features of edge regions during distillation learning can narrow the learning scope of the student model, thus achieving better results than global distillation learning.
[0055] In view of this, this embodiment provides an image denoising model processing scheme based on edge feature differences. The scheme provided in this embodiment will be described in detail below.
[0056] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of an image denoising model processing method provided in this embodiment. The method may include the following steps.
[0057] Step S110: Obtain the trained teacher model and the student model to be trained.
[0058] In this embodiment, the teacher model and the student model are used to perform denoising processing on the image. The structural complexity of the student model is less than that of the teacher model, and the learning ability of the teacher model is stronger than that of the student model.
[0059] For example, in one instance, the teacher model can be a MIRNet model with a large number of convolutional channels, and the student model can be a MIRNet_s model with a smaller number of convolutional channels. The MIRNet_s model can be obtained by halving the number of channels in all convolutions of the MIRNet model. It should be noted that in other implementations of this embodiment, the teacher model and the student model can also be machine learning models with other structures, which are not specifically limited here.
[0060] In this embodiment, the teacher model can be trained in advance using multiple training samples to obtain the trained teacher model.
[0061] Step S120: Obtain training samples, which include images to be processed that need to be noisy.
[0062] In this embodiment, multiple training samples can be obtained, and these training samples are used to train the student model to be trained.
[0063] Step S130: Input the images to be processed in the training samples into the teacher model and the student model respectively to obtain the first processing result output by the teacher model and the second processing result output by the student model.
[0064] In this embodiment, the image to be processed from the same training sample can be input into the teacher model and the student model respectively, and the teacher model and the student model can process the image to be processed to obtain a first processing result output by the teacher model and a second processing result output by the student model.
[0065] Specifically, in one possible implementation, the image to be processed in the training samples can be input into the teacher model and the student model respectively, and the image denoising result output by the teacher model can be used as the first processing result, and the image denoising result output by the student model can be used as the second processing result.
[0066] In another possible implementation, the images to be processed from the training samples can be input into the teacher model and the student model respectively. The intermediate feature map extracted by the teacher model during image denoising is used as the first processing result, and the intermediate feature map extracted by the student model during image denoising is used as the second processing result. Further, since the number of feature channels in the corresponding network layers of the teacher model and the student model may differ, in this embodiment, after obtaining the intermediate feature map extracted by the student model during image denoising, it can be processed through an adaptive layer (e.g., a 1*1 convolutional network) to ensure that the number of feature channels in the first processing result and the second processing result are consistent.
[0067] Step S140: Extract a first edge feature image corresponding to the edge position of the image content from the first processing result, and extract a second edge feature image corresponding to the edge position of the image content from the second processing result.
[0068] In this embodiment, in order to perform knowledge distillation on the student model specifically for edge features, a first edge feature image corresponding to the edge position of the image content can be extracted from the first processing result, and a second edge feature image corresponding to the edge position of the image content can be extracted from the second processing result.
[0069] Step S150: Calculate the first reconstruction loss value based on the difference between the first edge feature image and the second edge feature image.
[0070] In this embodiment, a first reconstruction loss value can be calculated based on the difference between the first edge feature image and the second edge feature image. The first reconstruction loss value can indicate the difference in action execution between the teacher model and the student model when performing image denoising.
[0071] Step S160: Adjust the model parameters of the student model according to the first reconstruction loss value.
[0072] In this embodiment, the model parameters of the student model can be adjusted to minimize the first reconstruction loss value, so that the feature extraction and image adjustment actions performed by the student model when performing image denoising are more similar to those of the teacher model, thereby achieving knowledge distillation.
[0073] Based on the above design, in the image denoising model processing method provided in this embodiment, by calculating the reconstruction loss value of the student model based on the edge feature differences of the processing results output by the teacher model and the student model during the training of the student model, the knowledge distillation of image denoising based on edge feature differences is realized during the training process of the student model. This allows the student model to focus its learning on the edge features in the image, narrowing the learning range of the student model, thereby improving the improvement efficiency brought by knowledge distillation learning.
[0074] It is understood that, in this embodiment, steps S130 to S160 can be repeatedly executed multiple times to train the student model for multiple training samples until the first reconstruction loss value and / or other loss values related to the first reconstruction loss value are within a preset range, or the number of training iterations reaches a preset threshold, ultimately resulting in a trained student model. After training, the student model can independently perform image denoising.
[0075] In one possible implementation, please refer to Figure 2 The method provided in this embodiment may further include the following steps.
[0076] Step S210: Calculate the reconstruction loss based on the reference image and the second processing result to obtain the second reconstruction loss value.
[0077] In step S160, the model parameters of the student model can be adjusted based on the first reconstruction loss value and the second reconstruction loss value.
[0078] Specifically, in this embodiment, the loss value when adjusting the parameters of the student model can be a weighted sum L of the first reconstruction loss value L1 and the second reconstruction loss value L2. total For example, L total = aL1 + bL2, where a = b = 1.0.
[0079] By combining the first reconstruction loss value and the second reconstruction loss value to adjust the parameters of the student model, the image denoising results of the student model can be made more accurate.
[0080] In one possible implementation, step S140 may include the following steps.
[0081] Step S141: Perform image content edge recognition on the reference images in the training samples to obtain the edge contour position information of the image content.
[0082] In this embodiment, image content edge recognition can be performed on the reference images in the training samples using a preset edge detection algorithm. For example, the Sobel edge detection algorithm can be used to perform image content edge recognition on the reference images in the training samples to obtain the edge contour position information. Alternatively, a pre-trained semantic segmentation model can be used to perform image content recognition on the reference images to obtain the edge contour position information.
[0083] In one example, the edge contour location information includes an edge contour mask image, in which the value of the pixel corresponding to the edge position is 1, and the value of the pixel corresponding to the non-edge position is 0.
[0084] Step S142: Extract the first edge feature image from the first processing result based on the edge contour position information, and extract the second edge feature image from the second processing result based on the edge contour position information.
[0085] In this embodiment, the edges of the image content in the reference image are the same as the edges of the image content in the image to be processed. Therefore, by combining the edge contour position information, features corresponding to the edges of the image content can be extracted from the first processing result and the second processing result, respectively, to obtain the first edge feature image and the second edge feature image.
[0086] Specifically, in this embodiment, since the pixel value corresponding to the edge position in the edge contour mask image is 1, and the pixel value corresponding to the non-edge position is 0, the first processing result can be multiplied pixel by pixel with the edge contour mask image. This removes the data corresponding to the non-edge positions in the first processing result, thereby obtaining the first edge feature image. Similarly, the second processing result can be multiplied pixel by pixel with the edge contour mask image. This removes the data corresponding to the non-edge positions in the second processing result, thereby obtaining the second edge feature image.
[0087] In one possible implementation, step S150 may include the following sub-steps.
[0088] Step S151: Calculate the difference feature map based on the difference values between each pixel in the first edge feature image and the second edge feature image.
[0089] Specifically, in this embodiment, the absolute value of the difference between each corresponding pixel in the first edge feature image and the second edge feature image can be calculated to obtain the difference feature map.
[0090] For example, the difference feature map D (x,y) It can be calculated using the following formula:
[0091]
[0092] Where (x, y) are the image coordinates, T (x,y) For the first edge feature image, S (x,y) This is the second edge feature image.
[0093] Step S152: Multiply the first processing result by pixels with the difference feature map to obtain the first weighted result.
[0094] By multiplying the first processing result by pixels with the difference feature map, the edge contour features that differ from the second processing result in the first processing result can be made more prominent.
[0095] Step S153: Multiply the second processing result by pixels with the difference feature map to obtain the second weighted result.
[0096] By multiplying the second processing result by pixels with the difference feature map, the edge contour features that differ from the first processing result in the second processing result can be made more prominent.
[0097] Step S154: Calculate the reconstruction loss based on the first weighted result and the second weighted result to obtain the first reconstruction loss value.
[0098] Since the features located at the edge of the image content and showing differences in the first processing result and the second processing result are amplified in steps S152 and S153, the first reconstruction loss value calculated based on the first weighted result and the second weighted result can better reflect the differences in processing actions between the teacher model and the student model, thereby improving the efficiency of knowledge distillation learning when adjusting the student model based on the first reconstruction loss value.
[0099] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of data transfer during the model training process provided in this embodiment.
[0100] In step S120, the training samples can be obtained. The training samples include the image to be processed and a reference image corresponding to the image to be processed. The image to be processed is an image with image noise, and the reference image is an image with image noise removed.
[0101] In step S130, the image to be processed can be input into the teacher model and the student model respectively to obtain the first processing result and the second processing result.
[0102] In step S140, edge recognition can be performed on the reference image using an edge detection algorithm to obtain the edge contour mask image. Then, the first processing result and the second processing result are multiplied pixel-wise with the edge contour image to obtain the first edge feature image and the second edge feature image.
[0103] In step S150, the difference between the first edge feature image and the second edge feature image can be calculated to obtain a difference feature map. Then, the first edge feature image and the second edge feature image are multiplied pixel by pixel with the difference feature image to obtain the first weighted result and the second weighted result. Finally, the first weighted result and the second weighted result are used to calculate the reconstruction loss to obtain the first reconstruction loss value.
[0104] In addition, in step S210, the reference image and the second processing result can be used to perform reconstruction loss calculation to obtain the second reconstruction loss value.
[0105] Finally, in step S160, the model parameters of the student model are adjusted based on the first reconstruction loss value and the second reconstruction loss value.
[0106] Based on the same inventive concept, this embodiment also provides an electronic device capable of running the image denoising model processing method provided in this embodiment. This electronic device may include devices with image processing capabilities such as servers, personal computers, laptops, and workstations. Please refer to... Figure 4 , Figure 4 This is a block diagram of the electronic device 100. The electronic device 100 includes an image denoising model processing device 110, a machine-readable storage medium 120, and a processor 130.
[0107] The machine-readable storage medium 120 and the processor 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The image denoising model processing device 110 includes at least one software function module that can be stored in the machine-readable storage medium 120 in the form of software or firmware or embedded in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the machine-readable storage medium 120, such as the software function modules and computer programs included in the image denoising model processing device 110.
[0108] The machine-readable storage medium 120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store programs, and the processor 130 executes these programs / executable image denoising model processing methods provided in this embodiment after receiving execution instructions.
[0109] The processor 130 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.
[0110] Please refer to Figure 5 This embodiment also provides an image denoising model processing device 110, which includes at least one functional module that can be stored in a machine-readable storage medium 120 in software form. Functionally, the image denoising model processing device 110 may include a model acquisition module 111, a sample acquisition module 112, an image processing module 113, an edge feature extraction module 114, a loss calculation module 115, and a model adjustment module 116.
[0111] The model acquisition module 111 is used to acquire the trained teacher model and the student model to be trained. The teacher model and the student model are used to perform denoising processing on the image. The structural complexity of the student model is less than that of the teacher model.
[0112] In this embodiment, the model acquisition module 111 can be used to perform... Figure 1For a detailed description of the model acquisition module 111 shown in step S110, please refer to the description of step S110.
[0113] The sample acquisition module 112 is used to acquire training samples, which include the image to be processed that has noise and the reference image after noise removal.
[0114] In this embodiment, the sample acquisition module 112 can be used to perform... Figure 1 For a detailed description of the sample acquisition module 112 shown in step S120, please refer to the description of step S120.
[0115] The image processing module 113 is used to input the image to be processed in the training samples into the teacher model and the student model respectively, and obtain the first processing result output by the teacher model and the second processing result output by the student model;
[0116] In this embodiment, the image processing module 113 can be used to perform... Figure 1 For a detailed description of the image processing module 113 shown in step S130, please refer to the description of step S130.
[0117] The edge feature extraction module 114 is used to extract a first edge feature image corresponding to the edge position of the image content from the first processing result, and to extract a second edge feature image corresponding to the edge position of the image content from the second processing result;
[0118] In this embodiment, the edge feature extraction module 114 can be used to perform... Figure 1 For a detailed description of the edge feature extraction module 114 shown in step S140, please refer to the description of step S140.
[0119] The loss calculation module 115 is used to calculate a first reconstruction loss value based on the difference between the first edge feature image and the second edge feature image;
[0120] In this embodiment, the loss calculation module 115 can be used to perform... Figure 1 For a detailed description of the loss calculation module 115 shown in step S150, please refer to the description of step S150.
[0121] The model adjustment module 116 is used to adjust the model parameters of the student model according to the first reconstruction loss value.
[0122] In this embodiment, the model adjustment module 116 can be used to perform... Figure 1 For a detailed description of the model adjustment module 116 shown in step S160, please refer to the description of step S160.
[0123] In summary, this application provides an image denoising model processing method, apparatus, and electronic device. By calculating the reconstruction loss value of the student model based on the edge feature differences of the processing results output by the teacher model and the student model during the training of the student model, the image denoising knowledge distillation based on edge feature differences is realized during the training process of the student model. This allows the student model to focus its learning on the edge features in the image, narrowing the learning scope of the student model and thus improving the efficiency of knowledge distillation learning.
[0124] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0125] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0126] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0127] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0128] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. An image denoising model processing method, characterized in that, The method includes: Obtain the trained teacher model and the student model to be trained. The teacher model and the student model are used to perform denoising processing on the image. The structural complexity of the student model is less than that of the teacher model. Obtain training samples, which include a noisy image to be processed and a noise-removed reference image corresponding to the image to be processed; The images to be processed in the training samples are input into the teacher model and the student model respectively to obtain the first processing result output by the teacher model and the second processing result output by the student model; Extract a first edge feature image corresponding to the edge position of the image content from the first processing result, and extract a second edge feature image corresponding to the edge position of the image content from the second processing result; Based on the difference values between each pixel in the first edge feature image and the second edge feature image, a difference feature map is calculated; the first processing result is multiplied by the difference feature map pixel by pixel to obtain a first weighted result; the second processing result is multiplied by the difference feature map pixel by pixel to obtain a second weighted result; a reconstruction loss is calculated based on the first weighted result and the second weighted result to obtain a first reconstruction loss value; Based on the reference image and the second processing result, the reconstruction loss is calculated to obtain the second reconstruction loss value; The model parameters of the student model are adjusted based on the first reconstruction loss value and the second reconstruction loss value.
2. The method according to claim 1, characterized in that, The training samples also include noise-removed reference images corresponding to the image to be processed; the steps of extracting a first edge feature image corresponding to the edge position of the image content from the first processing result and extracting a second edge feature image corresponding to the edge position of the image content from the second processing result include: Image content edge recognition is performed on the reference images in the training samples to obtain the edge contour position information of the image content; The first edge feature image is obtained by extracting the edge contour position information from the first processing result, and the second edge feature image is obtained by extracting the edge contour position information from the second processing result.
3. The method according to claim 2, characterized in that, The edge contour position information includes an edge contour mask image, in which the value of the pixel corresponding to the edge position is 1, and the value of the pixel corresponding to the non-edge position is 0. The steps of extracting the first edge feature image from the first processing result based on the edge contour position information, and extracting the second edge feature image from the second processing result based on the edge contour position information, include: The first processing result is multiplied pixel by pixel with the edge contour mask image to obtain the first edge feature image; The second processing result is multiplied pixel by pixel with the edge contour mask image to obtain the second edge feature image.
4. The method according to claim 1, characterized in that, The step of calculating a difference feature map based on the difference values between each pixel in the first edge feature image and the second edge feature image includes: The absolute value of the difference between each corresponding pixel in the first edge feature image and the second edge feature image is calculated to obtain the difference feature map.
5. The method according to claim 1, characterized in that, The step of inputting the image to be processed from the training samples into the teacher model and the student model respectively, and obtaining the first processing result output by the teacher model and the second processing result output by the student model, includes: The images to be processed in the training samples are input into the teacher model and the student model respectively. The image denoising result output by the teacher model is used as the first processing result, and the image denoising result output by the student model is used as the second processing result. Alternatively, the images to be processed in the training samples can be input into the teacher model and the student model respectively, and the intermediate feature map extracted by the teacher model when performing image denoising can be used as the first processing result, and the intermediate feature map extracted by the student model when performing image denoising can be used as the second processing result.
6. An image denoising model processing device, characterized in that, The image denoising model processing device includes: The model acquisition module is used to acquire the trained teacher model and the student model to be trained. The teacher model and the student model are used to perform denoising processing on the image. The structural complexity of the student model is less than that of the teacher model. The sample acquisition module is used to acquire training samples, which include the image to be processed that has noise and the reference image after noise removal. The image processing module is used to input the images to be processed in the training samples into the teacher model and the student model respectively, and obtain the first processing result output by the teacher model and the second processing result output by the student model; An edge feature extraction module is used to extract a first edge feature image corresponding to the edge position of the image content from the first processing result, and to extract a second edge feature image corresponding to the edge position of the image content from the second processing result; The loss calculation module is used to calculate a difference feature map based on the difference values between each pixel in the first edge feature image and the second edge feature image; multiply the first processing result by the difference feature map pixel by pixel to obtain a first weighted result; multiply the second processing result by the difference feature map pixel by pixel to obtain a second weighted result; and calculate the reconstruction loss based on the first weighted result and the second weighted result to obtain a first reconstruction loss value. The loss calculation module is also used to perform reconstruction loss calculation based on the reference image and the second processing result to obtain a second reconstruction loss value; The model adjustment module is used to adjust the model parameters of the student model based on the first reconstruction loss value and the second reconstruction loss value.
7. An electronic device, characterized in that, The method includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, which, when executed by the processor, implement the method according to any one of claims 1-5.
8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when executed by one or more processors, implement the method according to any one of claims 1-5.