Image denoising method, device, electronic device and storage medium

Abstract

The present application relates to an image denoising method, device, electronic device and storage medium. The above method includes extracting a first feature and a second feature of a first image; the first feature indicates a feature in the first image whose feature change degree is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature change degree is higher than the first threshold; denoising the second feature to obtain a third feature; extracting features from a first reference image generated based on the first feature to obtain a feature extraction result; the feature extraction result includes a fourth feature, and the fourth feature indicates a feature in the first reference image whose feature change degree is higher than a second threshold; based on the fourth feature, the third feature is fused with the first reference image to obtain a second image, and the second image represents the denoising result of the first image. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The present application can better balance the denoising effect and denoising efficiency.

Description

Technical Field

[0001] The present application relates to the field of Internet communication technology, and in particular to an image denoising method, device, electronic device and storage medium. Background Art

[0002] Images are a carrier of visual information, allowing people to gain insights. Typically, noise interferes with image generation and transmission, complicating computer image processing. Related technologies often use fixed image denoising parameters to denoise images, but this approach does not yield effective denoising results. Therefore, a more effective image denoising solution is needed. Summary of the Invention

[0003] In order to solve at least one of the above-mentioned technical problems, the present application provides an image denoising method, apparatus, electronic device, and storage medium:

[0004] According to a first aspect of the present application, a method for image denoising is provided, the method comprising:

[0005] Extracting a first feature and a second feature of a first image; wherein the first feature indicates a feature in the first image whose feature variation degree is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature variation degree is higher than the first threshold;

[0006] Performing denoising on the second feature to obtain a third feature;

[0007] performing feature extraction on a first reference image generated based on the first feature to obtain a feature extraction result; the feature extraction result includes a fourth feature, the fourth feature indicating a feature in the first reference image where a degree of feature change is greater than a second threshold;

[0008] Based on the fourth feature, the third feature is fused with the first reference image to obtain a second image, where the second image represents a denoising result of the first image.

[0009] According to a second aspect of the present application, an image denoising device is provided, characterized in that the device comprises:

[0010] A first feature extraction module is configured to extract a first feature and a second feature of the first image; the first feature indicates a feature in the first image whose feature variation is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature variation is higher than the first threshold;

[0011] Denoising processing module: used for performing denoising processing on the second feature to obtain a third feature;

[0012] a second feature extraction module configured to perform feature extraction on a first reference image generated based on the first feature to obtain a feature extraction result; the feature extraction result including a fourth feature, the fourth feature indicating a feature in the first reference image where a degree of feature change is greater than a second threshold;

[0013] Fusion module: used to fuse the third feature with the first reference image based on the fourth feature to obtain a second image, where the second image represents a denoising result of the first image.

[0014] According to a third aspect of the present application, an electronic device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the image denoising method as described in the first aspect.

[0015] According to a fourth aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the image denoising method as described in the first aspect.

[0016] According to a fifth aspect of the present application, a computer program product is provided, comprising at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is loaded and executed by a processor to implement the image denoising method as described in the first aspect.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0018] The implementation of this application has the following beneficial effects:

[0019] The present application can better balance the denoising effect and denoising efficiency. The first reference image generated based on the first feature can, to a certain extent, better present the general outline and contour of the first image. The third feature obtained by denoising the second feature can, to a certain extent, better reflect the details of the first image. The fourth feature extracted from the first reference image can, to a certain extent, better reflect the details of the first reference image. Based on the fourth feature, the detail features of the first image (corresponding to the third feature) and the close-up image of the first image (corresponding to the first reference image) are fused to improve the denoising effect. At the same time, the process of extracting the first feature and the second feature from the first image based on the first threshold and generating the first reference image based on the first feature provides an approximate image for the integration of detail features. This process ensures the efficiency of obtaining the approximate image and also ensures the efficiency of obtaining the second image representing the denoising result of the first image.

[0020] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A schematic diagram of an application environment according to an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram showing a flow chart of an image denoising method according to an embodiment of the present application;

[0024] Figure 3 A schematic diagram showing a process flow of a denoising process according to an embodiment of the present application;

[0025] Figure 4 A schematic diagram showing a process of integrating features into an image according to an embodiment of the present application;

[0026] Figure 5 A schematic diagram showing a flow chart of a model training process according to an embodiment of the present application;

[0027] Figure 6 A comparison diagram of model performance according to an embodiment of the present application is shown;

[0028] Figure 7 A schematic diagram illustrating an application of Haar wavelet transform according to an embodiment of the present application is shown;

[0029] Figure 8 A block diagram of an image denoising device according to an embodiment of the present application is shown;

[0030] Figure 9 A schematic diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0033] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0036] In addition, numerous specific details are provided in the following detailed description to better illustrate the present application. Those skilled in the art will appreciate that the present application can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present application.

[0037] See also Figure 1 , Figure 1 A schematic diagram of an application environment according to an embodiment of the present application is shown, and the application environment may include a client 10 and a server 20. The client 10 and the server 20 may be directly or indirectly connected via wired or wireless communication. A related object (such as a user, a simulator) may send an image denoising request indicating a first image to the server 20 via the client 10. In response to the received image denoising request, the server 20 extracts the first feature and the second feature of the first image; then, denoises the second feature to obtain a third feature; further, extracts features from the first reference image generated based on the first feature to obtain a fourth feature; finally, based on the fourth feature, the third feature is fused with the first reference image to obtain the second image. It should be noted that Figure 1 Just an example.

[0038] The client 10 may be a physical device such as a smartphone, a computer (e.g., a desktop computer, tablet computer, laptop computer), an augmented reality (AR) / virtual reality (VR) device, a digital assistant, an intelligent voice interaction device (e.g., a smart speaker), a smart wearable device, a smart home appliance, or an in-vehicle terminal. It may also be software running on a physical device, such as a computer program. The operating system supported by the client may be Android, iOS (a mobile operating system developed by Apple), Linux, or Microsoft Windows.

[0039] The server side 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can include a network communication unit, a processor, a memory, etc. The server side can provide background services for the corresponding client.

[0040] In practical applications, the client 10 may independently perform the above-mentioned image denoising steps on the first image ("extracting the first feature and the second feature of the first image" to "based on the fourth feature, fusing the third feature with the first reference image to obtain the second image"), or the client 10 and the server 20 may interact to perform the above-mentioned image denoising steps on the first image. The image denoising solution provided in the embodiment of the present application can be applied to mobile terminals. Taking a smartphone as an example, the image denoising solution provided in the embodiment of the present application can be used in the image preprocessing process of the camera module therein.

[0041] The image denoising solution provided in the embodiments of the present application can use technologies related to artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology (CV), speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0042] Among them, computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0043] It should be noted that for images that are associated with user information, when the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0044] Figure 2 A flow chart of an image denoising method according to an embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:

[0045] S201: Extracting a first feature and a second feature of a first image; wherein the first feature indicates a feature in the first image whose feature variation is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature variation is higher than the first threshold;

[0046] In an embodiment of the present application, the server or client extracts the first feature and the second feature of the first image. The first feature and the second feature are extracted from the first image based on a first threshold. The first image is the image to be denoised. The first feature indicates a feature in the first image where the degree of feature change is lower than or equal to the first threshold. The first feature can characterize an area in the first image where pixel features change slowly. This area is generally a non-edge area. Accordingly, the first feature can be a feature reflecting the general outline and contour of the first image. The second feature indicates a feature in the first image where the degree of feature change is higher than the first threshold. The second feature can characterize an area in the first image where pixel features change dramatically. This area is generally an edge area. Accordingly, the second feature can be a feature reflecting the noisy details of the first image. The degree of feature change here can be for color, grayscale, etc.

[0047] Exemplarily, the first image can be subjected to a frequency domain transformation based on a first threshold to obtain a first feature and a second feature. The frequency domain transformation performed can be a wavelet transform, specifically a Haar wavelet transform. Taking the wavelet transform as an example, after the noisy image is subjected to the wavelet transform, the noise features are mainly distributed in the high-frequency sub-band, while the low-frequency sub-band contains important detail features of the original image. A Haar wavelet transform is performed on the first image to obtain low-frequency features and high-frequency features. For example, the RGB three-channel separation is performed on the first image to generate an RGB three-channel image. See Figure 7 The RGB three-channel image is convolved with a preset Haar wavelet operator to obtain a 3×12 channel band feature. From the 3×12 channel band feature, a 3-channel band feature is extracted as a low-frequency feature, and three 3-channel band features are extracted as high-frequency features. The first feature can be considered a low-frequency feature, and the second feature can be considered a high-frequency feature. Frequency domain transformation improves the convenience and efficiency of feature extraction.

[0048] S202: performing denoising processing on the second feature to obtain a third feature;

[0049] In an embodiment of the present application, the server or client performs denoising on the second feature to obtain a third feature. The third feature may be a detail feature reflecting the first image. The detail feature may indicate adjacent areas in the first image where the contrast of relevant indicators is greater than a contrast threshold, and texture features in the first image where the frequency of occurrence is lower than a frequency threshold, etc. Relevant indicators include but are not limited to color and grayscale. Non-noise features in the second feature may be extracted based on a relevant threshold, and the non-noise features may be used as the third feature. The relevant threshold may be set based on historical experience; it may also be an adaptive threshold. in represents the adjustment parameter, It represents the estimated noise variance of the channel frequency band corresponding to the second feature, and σ2 represents the characteristic standard deviation of each sub-band of the channel frequency band corresponding to the second feature.

[0050] In an exemplary embodiment, Figure 3 As shown, the denoising process is performed on the second feature to obtain the third feature, including:

[0051] S301: Extracting candidate noise features and candidate non-noise features from the second features;

[0052] S302: Determine a second association relationship between the candidate noise feature and the candidate non-noise feature;

[0053] S303: extracting residual non-noise features from the candidate noise features based on the second association relationship, and extracting residual noise features from the candidate non-noise features to obtain target non-noise features;

[0054] S304: Obtain the third feature based on the residual non-noise feature and the target non-noise feature.

[0055] Considering that the extracted non-noise features may still contain noise features, and the extracted noise features may still contain non-noise features, in order to improve the denoising effect, residual features can be extracted based on the correlation between the extracted non-noise features and the extracted noise features, thereby ensuring the integrity and purity of the obtained non-noise features.

[0056] Taking the second feature X as an example, candidate noise features Xi and candidate non-noise features Xj are extracted from the second feature X. The second association relationship can characterize the intersection features of the candidate noise features Xi and the candidate non-noise features Xj. Since the intersection features can include a first type of intersection features indicating noise features and a second type of intersection features indicating non-noise features, the second association relationship includes an association relationship 1 characterizing the first type of intersection features and an association relationship 2 characterizing the second type of intersection features. Based on this, the residual noise feature Xji can be extracted from the candidate non-noise feature Xj based on the association relationship 1 to obtain the target non-noise feature. The target non-noise feature can be represented by Xj-Xji. The residual non-noise feature Xij can be extracted from the candidate noise feature Xi based on the association relationship 2, and then the third feature can be obtained based on the target non-noise feature Xj-Xji and the residual non-noise feature Xij. The third feature can be represented by Xj-Xji+Xij.

[0057] In addition, 1) the target noise feature can be represented by Xi-Xij, and then the correlation between the target noise feature and the third feature is determined. Then, based on the correlation, the residual non-noise feature is extracted from the target noise feature, and the residual noise feature is extracted from the third feature, and then the third feature is updated based on the extracted third feature and the residual non-noise feature. The relevant steps for updating the third feature here can be repeated to further improve the integrity and purity of the obtained non-noise feature. 2) Before determining the correlation between the noise feature and the non-noise feature, the noise feature and the non-noise feature can be enhanced separately, so as to improve the accuracy of the determined correlation, thereby ensuring the accuracy of positioning and extracting residual features. 3) Considering that the first feature also contains noise features, the denoising process for the second feature here can also be used for the first feature.

[0058] S203: performing feature extraction on a first reference image generated based on the first feature to obtain a feature extraction result; the feature extraction result includes a fourth feature, the fourth feature indicating a feature in the first reference image where a degree of feature change is greater than a second threshold;

[0059] In an embodiment of the present application, a server or client performs feature extraction on a first reference image generated based on a first feature to obtain a feature extraction result. The first reference image is generated based on the first feature and can be considered an approximation of the first image. In practical applications, the first reference image can be an RGB three-channel image.

[0060] The fourth feature indicates a feature in the first reference image whose degree of feature change is higher than the second threshold. The fourth feature can characterize an area in the first reference image where pixel features change dramatically, which is generally an edge area. Accordingly, the fourth feature can be a noisy detail feature reflecting the first reference image. The degree of feature change here can be for color, grayscale, etc. The process of extracting the fourth feature of the first reference image here can refer to the relevant records of the aforementioned step S201 "Extracting the second feature of the first image" and will not be repeated here. It should be noted that the "second threshold" here can be the same as or different from the above-mentioned "first threshold".

[0061] S204: Based on the fourth feature, fuse the third feature with the first reference image to obtain a second image, where the second image represents a denoising result of the first image.

[0062] In an embodiment of the present application, the server or client fuses the third feature with the first reference image based on the fourth feature to obtain the second image. If the second feature reflects the noisy detail feature of the first image, the first reference image is an approximate image of the first image. Then, the third feature reflects the detail feature of the first image, and fusing the third feature with the first reference image is fusing the detail feature with the approximate image, which can ensure that the first image is effectively restored while removing the related noise of the first image. The fourth feature is used when fusing the third feature with the first reference image. It can be understood that the third feature is integrated into the first reference image with the fourth feature as the target to obtain the second image. The fourth feature can be used to indicate the point where the third feature is to be integrated into the first reference image, thereby ensuring the integration effect of the detail feature. Technology related to spatial projection can be used here. The second image is a denoising result that represents the first image. The second image is an image with better image quality obtained after denoising the first image.

[0063] In an exemplary embodiment, the feature extraction result further includes a fifth feature, wherein the fifth feature indicates a feature in the first reference image whose feature change degree is less than or equal to the second threshold value, and the fourth feature and the fifth feature are obtained based on frequency domain transformation, such as Figure 4 As shown, the method of fusing the third feature with the first reference image based on the fourth feature to obtain the second image includes:

[0064] S401: Determine a first association relationship between the fourth feature and the third feature;

[0065] S402: Based on the first association relationship and the second threshold, perform frequency domain inverse transformation on the third feature, the fourth feature, and the fifth feature to obtain the second image.

[0066] The fifth feature indicates a feature in the first reference image whose degree of feature change is lower than or equal to the second threshold. The fifth feature can characterize an area in the first reference image where pixel features change slowly, which is generally a non-edge area. Accordingly, the fifth feature can be a feature that reflects the general outline and contour of the first reference image. The first reference image can be subjected to a frequency domain transformation based on the second threshold to obtain the fourth and fifth features. The frequency domain transformation performed can be a wavelet transform, specifically a Haar wavelet transform. The fourth feature can be regarded as a high-frequency feature, and the fifth feature can be regarded as a low-frequency feature. The degree of feature change here can be for color, grayscale, etc. The process of extracting the fourth and fifth features of the first reference image here can refer to the relevant records of the aforementioned step S201 "Extracting the first and second features of the first image" and will not be repeated here. It should be noted that the "second threshold" here can be the same as or different from the above-mentioned "first threshold".

[0067] The third feature reflects the detail features of the first image, and the fourth feature reflects the noisy detail features of the first reference image. Determining the association relationship between the fourth feature and the third feature can be done by establishing a global association (mapping) relationship between the fourth feature and the third feature; determining a local feature within the fourth feature and establishing a local-global association (mapping) relationship between the local feature and the third feature; determining a local feature within the third feature and establishing a local-global association (mapping) relationship between the local feature and the fourth feature; or determining local feature 1 within the fourth feature and local feature 2 within the third feature and establishing a local-local association (mapping) relationship between local feature 1 and local feature 2. This association (mapping) relationship can serve as a basis for integrating the third feature into the first reference image.

[0068] For example, if the image features include color features, grayscale features, and texture features, color feature i can be extracted from the third feature, and color feature j can be extracted from the fourth feature. Then, based on factors such as the indicated image position, an association (mapping) relationship is established between the relevant features in color feature i and the relevant features in color feature j. The same applies to grayscale features and texture features.

[0069] On the basis of the known first correlation relationship and the second threshold, the fusion of the third feature and the first reference image can be achieved through inverse frequency domain transformation: the third feature, the fourth feature and the fifth feature are subjected to inverse frequency domain transformation. The inverse frequency domain transformation performed can be an inverse wavelet transform, specifically an inverse Haar wavelet transform. When integrating the detail features of the first image into the first reference image, performing an inverse frequency domain transformation based on the correlation relationship and the threshold can improve the efficiency and convenience of obtaining the second image and ensure the quality of the obtained second image. The process of inverse frequency domain transformation can be regarded as the inverse process of the frequency domain transformation described above. If N (for example, 3) is the target channel number, the channel numbers corresponding to the third feature, the fourth feature and the fifth feature are adjusted based on the target channel number; then, the features after the adjusted channel number are subjected to deconvolution operation with the preset Haar wavelet operator to obtain the second image.

[0070] In an exemplary embodiment, the fusing of the third feature and the first reference image based on the fourth feature to obtain the second image may include the following steps: fusing the third feature and the second reference image based on the sixth feature to obtain the second image; wherein the second reference image is obtained by adjusting the resolution of the first reference image based on a reference resolution, and the reference resolution is the resolution of the first image; the sixth feature is a feature in which the degree of feature change in the second reference image is higher than a third threshold value.

[0071] Considering that the resolution (size) of the first reference image is smaller than that of the first image, adjusting the resolution of the first reference image can improve the correlation and matching between the noisy detail features of the adjusted first reference image and the detail features of the first image, thereby improving the integration of the detail features of the first image. The reference resolution is the resolution of the first image, and the resolution of the first reference image can be adjusted to the reference resolution to achieve resolution restoration. The resolution of the first reference image is adjusted to obtain the corresponding second reference image. For example, an adjustment ratio can be determined based on the resolution of the first reference image and the resolution of the first image, and then the first reference image can be amplified according to the adjustment ratio to obtain the second reference image. Techniques related to spatial projection can also be used here. The sixth feature indicates a feature in the second reference image where the degree of feature change exceeds a third threshold. The sixth feature can represent an area in the second reference image where pixel features vary dramatically, typically an edge area. Accordingly, the sixth feature can reflect the noisy detail features of the second reference image. The degree of feature change here can be for color, grayscale, etc. The process of extracting the sixth feature of the second reference image can refer to the aforementioned description of step S201, "Extracting the second feature of the first image," and will not be repeated here. It should be noted that the “third threshold” here may be the same as or different from the above “first threshold”.

[0072] In addition, the process of "based on the sixth feature, fusing the third feature with the second reference image to obtain the second image" can also be combined with the relevant records of the aforementioned steps S401-S402. That is, the third correlation relationship between the sixth feature and the third feature is determined; then, based on the third correlation relationship and the third threshold, the third feature, the sixth feature and the seventh feature are subjected to inverse frequency domain transformation to obtain the second image. The seventh feature indicates a feature in the second reference image where the degree of feature change is less than or equal to the third threshold. The seventh feature can characterize an area in the second reference image where pixel features change slowly, which is generally a non-edge area. Accordingly, the seventh feature can be a feature that reflects the general outline and contour of the second reference image. The second reference image can be subjected to frequency domain transformation based on the third threshold to obtain the sixth and seventh features. The degree of feature change here can be for color, grayscale, etc. The frequency domain transformation performed can be a wavelet transform, specifically a Haar wavelet transform.

[0073] In an exemplary embodiment, the relevant steps involved in the aforementioned steps S201-S204 can be implemented based on a denoising network. The denoising network includes a first network, a second network, a third network, and a fourth network. The extracting the first feature and the second feature of the first image (corresponding to step S201) may include the following steps: inputting the first image into the first network to obtain the first feature and the second feature. The denoising process of the second feature to obtain the third feature (corresponding to step S202) may include the following steps: inputting the second feature into the second network to obtain the third feature. The feature extraction of the first reference image generated based on the first feature to obtain the feature extraction result (corresponding to step S203) may include the following steps: inputting the first feature into the third network to obtain the feature extraction result. The fusing the third feature with the first reference image based on the fourth feature to obtain the second image (corresponding to step S204) may include the following steps: inputting the fourth feature, the third feature, and the first reference image into the fourth network to obtain the second image.

[0074] The denoising network is a trained model with high generalization capabilities. Using the denoising network for image denoising can improve the adaptability and reliability of image denoising. Model training is ongoing, and the denoising network is constantly updated. It can be understood that the denoising network used for image denoising here can be the result of a previous training session or the basis for the next training session. The following describes the model training process, using the denoising network as the basis for the next training session as an example:

[0075] like Figure 5 As shown, the process includes:

[0076] S501: Extracting first sample features and second sample features of a first sample image using the first network;

[0077] S502: Using the second network to perform denoising on the second sample feature to obtain a third sample feature;

[0078] S503: Using the third network to perform feature extraction on the sample reference image generated based on the first sample feature to obtain a fourth sample feature;

[0079] S504: Using the fourth network, fusing the third sample feature with the sample reference image based on the fourth sample feature to obtain a second sample image;

[0080] S505: Based on the correlation between the first sample image and the second sample image, adjust the parameters of the target network until the denoising network meets the convergence condition; the target network includes at least one of the first network, the second network, the third network and the fourth network.

[0081] The process of extracting the first sample features and the second sample features of the first sample image by the first network can refer to the relevant records of the aforementioned step S201 and will not be repeated here. The first network may include at least one first subspace encoding unit, which performs a wavelet transform on the input data. When the first network includes at least two first subspace encoding units, the at least two first subspace encoding units may be arranged in series. In practical applications, the sample reference image generated by the first sample features may be an RGB three-channel image. Accordingly, the first network may use a lighter-weight network structure design, such as reducing the parameter design of the network structure.

[0082] The process of performing denoising on the second sample features by the second network to obtain the third sample features can be described with reference to the relevant description of the aforementioned step S202 and will not be repeated here. The second network may include at least one dual-branch feature extraction unit, which is used to determine the correlation between noise features and non-noise features. The second network may also include at least one self-redundant unit, which is used to perform feature enhancement processing on the features to be involved in the correlation determination. The second network may specifically include at least one extraction submodule, which is composed of "a dual-branch feature extraction unit" and "at least two self-redundant units." When the second network includes at least two extraction submodules, the at least two extraction submodules can be arranged in series. In practical applications, the first sample features can also be input into the second network, and the correlation determined by the second network (between the first sample features and the second sample features, or between the first sample features and the non-noise features extracted from the second sample features) is used to update the first sample features. The updated first sample features are then input into the third network to generate a sample reference image. This can increase the information content of the sample reference image and improve the subsequent integration of the third features.

[0083] The process of extracting features from the sample reference image generated based on the first sample feature by the third network to obtain the fourth sample feature can be referred to the relevant description of the aforementioned step S203 and will not be repeated here. The process of fusing the third sample feature with the sample reference image based on the fourth sample feature by the fourth network to obtain the second sample image can be referred to the relevant description of the aforementioned step S204 and will not be repeated here. The fourth network may include at least one second subspace encoding unit and a subspace projection unit. The at least one second subspace encoding unit is used to extract the fourth sample feature of the sample reference image. The subspace projection unit is used to integrate the third sample feature into the sample reference image with the fourth sample feature as the target to obtain the second sample image. The second subspace encoding unit performs a wavelet transform on the input data. When the second network includes at least two second subspace encoding units, the at least two second subspace encoding units can be arranged in series.

[0084] By guiding parameter adjustment with correlation, a model with better denoising performance can be obtained with the goal of achieving a better denoising effect. In practical applications, the first network, the second network, and the third network can be regarded as the first module, and the fourth network can be regarded as the second module. Accordingly, the denoising network includes the first module and the second module. If the correlation between the first sample image and the second sample image is correlation 1, the correlation 2 between the first sample image and the sample reference image can be determined, and the correlation 3 between the sample reference image and the second sample image can be determined. Based on the correlations 1-3, the parameters of the target network can be adjusted until the denoising network meets the convergence conditions.

[0085] Furthermore, the model can be trained using paired sample images, wherein the paired sample images include a sample noisy image and a sample denoised image, and the paired sample images can come from the image to be denoised and the denoising result of the application of the relevant image denoising scheme. Guiding parameter adjustment with the correlation between the sample denoised image and the second sample image can improve the convenience of locating the quality of the denoising effect and improve the efficiency of parameter adjustment. Specifically: the first sample image is a sample noisy image, and adjusting the parameters of the target network based on the correlation between the first sample image and the second sample image until the denoising network meets the convergence condition can include the following steps: adjusting the parameters of the target network based on the correlation between the second sample image and the sample denoised image corresponding to the sample noisy image until the denoising network meets the convergence condition.

[0086] In addition, 1) multiple sample images can be used during model training. Before inputting the model, sample images can be augmented through rotation (e.g., 90°, 180°, 270°), cropping (e.g., cropping based on the image center and then scaling it back to its original size), and fusion (e.g., randomly selecting two images and fusing them in their RGB channels). This increases the diversity of the training data, thereby improving the model's generalization and denoising capabilities. The sample images can be sourced from datasets that provide realistic noisy images: the SIDD and DND datasets. The sample images selected from the SIDD dataset include 320 training image pairs and 40 validation image pairs. These images were captured from 10 scenes under varying lighting conditions using the camera modules of five popular smartphones. The denoised images in these pairs are generated by a system program and are high-resolution. These images can be cropped into approximately 30,000 small patches for training and 1,280 patches for validation.

[0087] The sample images selected from the DND dataset include 50 image pairs. For each image pair, the denoised image is captured at a basic low ISO level, and the noisy image is captured at a higher ISO and appropriately adjusted exposure time. The denoised image is also carefully post-processed, including small camera offset adjustments, linear intensity scaling, and removal of low-frequency offsets. It should be noted that the sample images selected from the DND dataset were captured by consumer-grade cameras with different sensor sizes.

[0088] 2) During the parameter adjustment process guided by correlation, the loss function value can be calculated based on the correlation, and then the parameters of the target network can be adjusted based on the loss function value until the denoising network meets the convergence conditions. It can be understood that the loss function can be used to constrain the model. During the parameter adjustment process, the Adam optimizer can be used. In practical applications, the loss function can be expressed as follows:

[0089] L=L t +αL 1(c) +βL 1(t) +L G +L s

[0090] Among them, α can be taken as 0.6 and β can be taken as 0.4. t represents the baseline image loss, L 1(c) , L 1(t) , L G and L s Denotes the denoising image loss. L t The downsampling result X of the first type of reference image T1 and the original image can be taken s The L2 distance between them. 1(c) The L1 distance between the denoising result X and the first type of reference image T1 can be taken. 1(t) The L1 distance between the denoising result X and the second type of reference image T2 can be taken. G The L1 distance of the gradient constraint between the denoising result X and the second type of reference image T2 can be taken: L G =||Δ x / y T2-Δ x / y X1, gradient constraint refers to the gradient size constraint of adjacent pixels in the image along the horizontal and vertical directions. L s Structural similarity between the denoising result X and the second type of benchmark image T2: L s =1-SSIM(T2, X). Exemplarily, the original image corresponds to the first image, the denoising result X corresponds to the second image, the first-type reference image T1 corresponds to the first reference image, and the second-type reference image T2 corresponds to the second reference image.

[0091] The learning rate of the Adam optimizer can be set to 0.0002, β1 can be set to 0.9, and β2 can be set to 0.999. Weight decay is used to reduce the number of iterations by half every 50,000 iterations. A total of 250,000 iterations are performed in one round of training.

[0092] 3) When evaluating a model's denoising performance, the image's peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) can be used. The complexity of each model can be calculated using the official code release to calculate the model's total number of parameters and GFloating-Point Operations per Second (GFLOPS). Note that for generative models or distillation-based models, the parameters and GFLOPs of the generator or teacher model are not included.

[0093] During the test, we tested the results of heavy-duty models (with more than 10M parameters) and lightweight models. The heavy-duty model can achieve better PSNR and SIMM, but requires a large number of parameters and computational cost. The lightweight model can achieve similar results to the heavy-duty model at a faster speed and with less computational cost. The performance comparison results of the models on the SIDD and DND datasets can be found in Figure 6 .

[0094] The figure also shows the model performance of the denoising network provided by the embodiment of the present application on the SIDD dataset and the DND dataset: with 2.68M parameters and 18.81GFlops, the measured peak signal-to-noise ratio reaches 39.47dB and the structural similarity reaches 0.957. It is worth noting that it exceeds the existing model by about 0.2dB. In addition, it only takes up 40% and 12% of Gflops respectively, which shows that the embodiment of the present application has good adaptability to the over-smoothing problem, allowing detailed features to be accurately preserved and relevant network parameters to be effectively utilized.

[0095] It can be seen from the technical solutions provided by the above embodiments of the present application that the embodiments of the present application can better balance the denoising effect and denoising efficiency. The embodiments of the present application also have good adaptability to the randomness and nonlinear characteristics of real noise. The first reference image generated based on the first feature can, to a certain extent, better present the general outline and contour of the first image. The third feature obtained by denoising the second feature can, to a certain extent, better reflect the details of the first image. The fourth feature extracted from the first reference image can, to a certain extent, better reflect the details of the first reference image. Based on the fourth feature, the detail features of the first image (corresponding to the third feature) and the close-up image of the first image (corresponding to the first reference image) are fused to improve the denoising effect. At the same time, the process of extracting the first feature and the second feature from the first image based on the first threshold and generating the first reference image based on the first feature provides an approximate image for the integration of detail features. This process ensures the efficiency of obtaining the approximate image and also ensures the efficiency of obtaining the second image representing the denoising result of the first image.

[0096] The present application also provides an image denoising device, such as Figure 8 As shown, the image denoising device 80 includes:

[0097] First feature extraction module 801: configured to extract first and second features of a first image; the first feature indicates a feature in the first image whose feature variation is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature variation is higher than the first threshold;

[0098] De-noising module 802: configured to perform denoising on the second feature to obtain a third feature;

[0099] A second feature extraction module 803 is configured to perform feature extraction on a first reference image generated based on the first feature to obtain a feature extraction result; the feature extraction result includes a fourth feature, the fourth feature indicating a feature in the first reference image where a degree of feature change is greater than a second threshold;

[0100] Fusion module 804: used to fuse the third feature with the first reference image based on the fourth feature to obtain a second image, where the second image represents a denoising result of the first image.

[0101] It should be noted that the device and method embodiments in the device embodiment are based on the same inventive concept.

[0102] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0103] The present application also provides a computer-readable storage medium having at least one instruction or at least one program stored therein, which is loaded and executed by a processor to implement the above method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0104] An embodiment of the present application also provides an electronic device, which includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the above method.

[0105] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0106] Figure 9 1 shows a block diagram of an electronic device according to an embodiment of the present application. For example, the electronic device 1900 can be provided as a server. Figure 9 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0107] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0108] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0109] The present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0110] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0111] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0112] The computer program instructions for performing the operation of the present application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C+, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer and partially on a remote computer, or executed completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as by using an Internet service provider to connect to the Internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present application.

[0113] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0114] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0115] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0116] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the above-mentioned module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions specified in the box can also occur in an order different from the order specified in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a special hardware-based system that performs the specified function or action, or can be implemented by a combination of special hardware and computer instructions.

[0117] While various embodiments of the present application have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image denoising method, characterized in that: Applied to a denoising network, the denoising network includes a first network, a second network, a third network, and a fourth network, the method includes: Extracting a first feature and a second feature of a first image using the first network; the first feature indicates a feature in the first image whose feature variation is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature variation is higher than the first threshold; Performing denoising on the second feature using the second network to obtain a third feature; performing feature extraction on a first reference image generated based on the first feature using the third network to obtain a feature extraction result; the feature extraction result includes a fourth feature and a fifth feature, the fourth feature indicating a feature in the first reference image whose degree of feature change is greater than a second threshold, the fifth feature indicating a feature in the first reference image whose degree of feature change is less than or equal to the second threshold, the fourth feature and the fifth feature being obtained based on frequency domain transformation; The fourth network is used to determine a first correlation relationship between the fourth feature and the third feature, and based on the first correlation relationship and the second threshold, the third feature, the fourth feature and the fifth feature are inversely transformed in the frequency domain to obtain a second image, where the second image represents a denoising result of the first image, and the first correlation relationship is a basis for integrating the third feature into the first reference image.

2. The method according to claim 1, characterized in that The method further comprises: determining a third correlation relationship between the sixth feature and the third feature using the fourth network, and performing a frequency domain inverse transform on the third feature, the sixth feature, and the seventh feature based on the third correlation relationship and a third threshold, to obtain the second image; The second baseline image is obtained by adjusting the resolution of the first baseline image based on a reference resolution, the reference resolution is the resolution of the first image, the sixth feature is a feature in which the degree of feature change in the second baseline image is higher than the third threshold, and the seventh feature indicates a feature in which the degree of feature change in the second baseline image is lower than or equal to the third threshold.

3. The method according to claim 1 or 2, characterized in that The denoising process of the second feature by using the second network to obtain the third feature includes: Extracting candidate noise features and candidate non-noise features from the second features using the second network; Determining, using the second network, a second association relationship between the candidate noise feature and the candidate non-noise feature, wherein the second association relationship represents an intersection feature of the candidate noise feature and the candidate non-noise feature, the intersection feature including a first type of intersection feature indicating a noise feature and a second type of intersection feature indicating a non-noise feature; extracting residual non-noise features from the candidate noise features and residual noise features from the candidate non-noise features based on the second association relationship using the second network to obtain target non-noise features; The third feature is obtained by using the second network based on the residual non-noise feature and the target non-noise feature.

4. The method according to claim 1, wherein The method further comprises: Inputting the first sample image into the denoising network to obtain a second sample image; Based on the correlation between the first sample image and the second sample image, the parameters of the target network are adjusted until the denoising network meets the convergence conditions; the target network includes at least one of the first network, the second network, the third network and the fourth network.

5. The method according to claim 4, characterized in that The adjusting the parameters of the target network based on the correlation between the first sample image and the second sample image until the denoising network meets the convergence condition includes: Based on the correlation between the second sample image and the denoised image corresponding to the first sample image, the parameters of the target network are adjusted until the denoising network meets the convergence condition.

6. An image denoising device, characterized in that: The apparatus is configured in a denoising network, the denoising network including a first network, a second network, a third network and a fourth network, and includes: A first feature extraction module is configured to extract a first feature and a second feature of the first image using the first network; the first feature indicates a feature in the first image whose feature variation is lower than or equal to a first threshold, and the second feature indicates a feature in the first image whose feature variation is higher than the first threshold; Denoising processing module: used to perform denoising processing on the second feature using the second network to obtain a third feature; a second feature extraction module configured to perform feature extraction on a first reference image generated based on the first feature using the third network, obtaining a feature extraction result; the feature extraction result comprising a fourth feature and a fifth feature, the fourth feature indicating a feature in the first reference image whose feature change degree is higher than a second threshold, the fifth feature indicating a feature in the first reference image whose feature change degree is lower than or equal to the second threshold, the fourth feature and the fifth feature being obtained based on frequency domain transformation; A fusion module is configured to determine a first correlation relationship between the fourth feature and the third feature using the fourth network, and to perform an inverse frequency domain transform on the third feature, the fourth feature, and the fifth feature based on the first correlation relationship and the second threshold value to obtain the second image, wherein the second image represents the denoising result of the first image, and the first correlation relationship is the basis for integrating the third feature into the first reference image.

7. The device according to claim 6, characterized in that The apparatus is further configured to: determine, using the fourth network, a third correlation relationship between the sixth feature and the third feature, and perform a frequency domain inverse transform on the third feature, the sixth feature, and the seventh feature based on the third correlation relationship and a third threshold value to obtain the second image; The second baseline image is obtained by adjusting the resolution of the first baseline image based on a reference resolution, the reference resolution is the resolution of the first image, the sixth feature is a feature in which the degree of feature change in the second baseline image is higher than the third threshold, and the seventh feature indicates a feature in which the degree of feature change in the second baseline image is lower than or equal to the third threshold.

8. The device according to claim 6 or 7, characterized in that The denoising processing module is further configured to: extract candidate noise features and candidate non-noise features from the second features using the second network; Determining, using the second network, a second association relationship between the candidate noise feature and the candidate non-noise feature, wherein the second association relationship represents an intersection feature of the candidate noise feature and the candidate non-noise feature, the intersection feature including a first type of intersection feature indicating a noise feature and a second type of intersection feature indicating a non-noise feature; extracting residual non-noise features from the candidate noise features and residual noise features from the candidate non-noise features based on the second association relationship using the second network to obtain target non-noise features; The third feature is obtained by using the second network based on the residual non-noise feature and the target non-noise feature.

9. The device according to claim 6, characterized in that The device is also used for: Inputting the first sample image into the denoising network to obtain a second sample image; Based on the correlation between the first sample image and the second sample image, the parameters of the target network are adjusted until the denoising network meets the convergence conditions; the target network includes at least one of the first network, the second network, the third network and the fourth network.

10. The device according to claim 9, characterized in that Adjusting the parameters of the target network until the denoising network meets the convergence condition based on the correlation between the first sample image and the second sample image includes: adjusting the parameters of the target network until the denoising network meets the convergence condition based on the correlation between the second sample image and the denoised image corresponding to the first sample image.

11. An electronic device, characterized in that: The electronic device includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the image denoising method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the image denoising method according to any one of claims 1 to 5.

13. A computer program product, characterized in that The computer program product includes at least one instruction or at least one program segment, and the at least one instruction or at least one program segment is loaded and executed by a processor to implement the image denoising method according to any one of claims 1 to 5.

Images