Image processing method, image denoising model training method, device and equipment

Abstract

The present disclosure provides an image processing method, a training method and device of an image denoising model, and equipment, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision, digital image processing, video image processing and the like. The implementation scheme is: noise level evaluation is performed on a to-be-processed image to obtain noise level information of the to-be-processed image; and based on the to-be-processed image and the noise level information, a first operation is performed to obtain a denoised image, the first operation comprising: predicting position information of at least one noise point based on the noise level information; and performing denoising processing on the to-be-processed image based on the position information of the at least one noise point to obtain the denoised image.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning, computer vision, digital image processing, and video image processing. Specifically, it relates to an image processing method, a training method for an image denoising model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] Image denoising is an important technique in image processing, primarily used to eliminate noise in images and improve their clarity and quality. Noise can appear at every pixel of an image and can originate from various sources, such as camera sensors, interference during transmission, and compression. In recent years, with the development of deep learning technology, more and more researchers have begun to use deep neural networks to remove image noise.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides an image processing method, an image denoising model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this disclosure, an image processing method is provided, comprising: evaluating the noise level of an image to be processed to obtain noise level information of the image to be processed, wherein the image to be processed includes at least one noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the image to be processed; and performing a first operation based on the image to be processed and the noise level information to obtain a denoised image, the first operation comprising: predicting the location information of the at least one noise point based on the noise level information; and performing denoising processing on the image to be processed based on the location information of the at least one noise point to obtain a denoised image.

[0007] According to another aspect of this disclosure, a training method for an image denoising model is provided. The method includes: evaluating the noise level of an original noisy image to obtain noise level information of the original noisy image, wherein the original noisy image includes at least one noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the original noisy image; randomly generating simulated noise based on the noise level information; superimposing the simulated noise onto the original noisy image to obtain a training image, wherein the distribution ratio of the noise points contained in the simulated noise in the training image is the same as the distribution ratio of the at least one noise point in the original noisy image; inputting the training image and the noise level information into an image denoising model to obtain a predicted image output by the image denoising model; calculating a comprehensive loss based on the predicted image and the original noisy image; and adjusting at least one parameter of the image denoising model based on the comprehensive loss.

[0008] According to another aspect of this disclosure, an image processing apparatus is provided, comprising: a first acquisition unit configured to perform noise level evaluation on an image to be processed to acquire noise level information of the image to be processed, wherein the image to be processed includes at least one noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the image to be processed; and an execution unit configured to perform a first operation through a subordinate subunit based on the image to be processed and the noise level information to obtain a denoised image, the execution unit comprising: a prediction subunit configured to predict the location information of the at least one noise point based on the noise level information; and a processing subunit configured to perform denoising processing on the image to be processed based on the location information of the at least one noise point to obtain a denoised image.

[0009] According to another aspect of this disclosure, a training apparatus for an image denoising model is provided. The apparatus includes: a first acquisition unit configured to evaluate the noise level of an original noisy image to obtain noise level information of the original noisy image, wherein the original noisy image includes at least one noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the original noisy image; a generation unit configured to randomly generate simulated noise based on the noise level information; a second acquisition unit configured to superimpose the simulated noise onto the original noisy image to obtain a training image, wherein the distribution ratio of the noise points contained in the simulated noise in the training image is the same as the distribution ratio of the at least one noise point in the original noisy image; an input unit configured to input the training image and the noise level information into the image denoising model to obtain a predicted image output by the image denoising model; a calculation unit configured to calculate a comprehensive loss based on the predicted image and the original noisy image; and an adjustment unit configured to adjust at least one parameter of the image denoising model based on the comprehensive loss.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described image processing method or image denoising model training method.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the above-described image processing method or image denoising model training method.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements the above-described image processing method or image denoising model training method.

[0013] According to one or more embodiments of this disclosure, noise level information in a noisy image is estimated, and the location information of noise points in the noisy image is predicted based on the noise level information, thereby generating a denoised image based on the prediction result. This allows pre-estimated noise level information to be introduced into the image denoising process, improving the denoising effect.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0017] Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;

[0018] Figure 3 A flowchart illustrating obtaining a denoised image based on an image to be processed and noise level information according to an embodiment of the present disclosure is shown.

[0019] Figure 4A flowchart illustrating the denoised image obtained from the output of a feature integration network according to an embodiment of the present disclosure is shown.

[0020] Figure 5 A structural block diagram of an image denoising model according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 6 A structural block diagram of a first extraction subnetwork and a first integration subnetwork according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 7 A flowchart illustrating a training method for an image denoising model according to an embodiment of the present disclosure is shown;

[0023] Figure 8 A flowchart illustrating the calculation of the comprehensive loss according to an embodiment of the present disclosure is shown;

[0024] Figure 9 A flowchart illustrating model training and model application according to embodiments of the present disclosure is shown;

[0025] Figure 10 The image denoising result obtained according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 11 A structural block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown;

[0027] Figure 12 A structural block diagram of a training apparatus for an image denoising model according to an embodiment of the present disclosure is shown;

[0028] Figure 13 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0031] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0032] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0033] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0034] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of the above-described image processing methods or training methods for image denoising models.

[0035] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0036] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0037] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to upload images to be processed. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0038] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0039] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0040] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0041] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0042] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0043] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0044] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0045] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0046] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0047] According to some embodiments, such as Figure 2 As shown, an image processing method is provided, including: step S201, evaluating the noise level of the image to be processed to obtain noise level information of the image to be processed, wherein the image to be processed includes at least one noise point, and the noise level information is used to indicate the distribution ratio of at least one noise point in the image to be processed; and step S202, performing a first operation based on the image to be processed and the noise level information to obtain a denoised image, the first operation including: step S2021, predicting the location information of at least one noise point based on the noise level information; and step S2022, performing denoising processing on the image to be processed based on the location information of at least one noise point to obtain a denoised image.

[0048] Therefore, by estimating the noise level information in a noisy image and predicting the location information of noise points in the image based on the noise level information, a denoised image can be generated based on the prediction results. This allows the pre-estimated noise level information to be introduced into the image denoising process, improving the denoising effect.

[0049] In some embodiments, the image to be processed may be an image containing at least one noise point.

[0050] In some embodiments, the noise contained in the image to be processed can be salt-and-pepper noise, or other noise with similar properties to salt-and-pepper noise, such as Poisson noise or Gaussian noise. The method of this disclosure will be described below using salt-and-pepper noise as an example.

[0051] In some embodiments, the noise level can be assessed by counting the number of noise points in the image to be processed and determining their proportion among all pixels in the image to be processed, thereby determining the noise level information of the image to be processed.

[0052] In some embodiments, noise level assessment can also be performed by setting a window detector to estimate the salt-and-pepper noise ratio of the noisy image block by block.

[0053] In some embodiments, evaluating the noise level of the image to be processed to obtain noise level information of the image to be processed includes: using a sliding window with a preset length and width, taking each pixel in the image to be processed as the center, counting the noise points within the area selected by the sliding window corresponding to that pixel to obtain the noise statistics information corresponding to that pixel, wherein the length and width of the sliding window are smaller than the length and width of the image to be processed, respectively; and determining the noise level information of the image to be processed based on the noise statistics information corresponding to each pixel in the image to be processed.

[0054] Therefore, by using a sliding window, the noise distribution in multiple regions of a noisy image can be statistically analyzed, thereby obtaining richer noise information and improving the subsequent image denoising effect.

[0055] In some embodiments, the noise ratio can be statistically analyzed one by one, using each pixel in the image to be processed as the center of the window, based on the sliding window described above.

[0056] In some embodiments, the number of noise points and / or the proportion of noise points in each sliding window can be recorded separately as noise statistics.

[0057] In some embodiments, the proportion data of all noise points can be directly used as the noise level information of the image to be processed.

[0058] In some embodiments, the average value of all noise point proportions can be calculated to serve as the noise level information of the image to be processed.

[0059] In some embodiments, noise points can be detected by detecting whether the pixel value of a certain pixel is greater than a first preset threshold or less than a second preset threshold; if the above conditions are met, it is determined to be a noise point.

[0060] For the detection of salt-and-pepper noise, in some exemplary embodiments, the salt-and-pepper noise model can be denoted as n, and the noise-free image as x, i.e.:

[0061]

[0062] Where α represents the proportion of salt-and-pepper noise in the noisy image, and x(i,j) represents the pixel value corresponding to the (i,j)th pixel in the noise-free image.

[0063] Define a coordinate system centered at (i, j) and of size ω. h ×ω w Sliding window W i，j Starting from the top left corner of the noisy image y, iterate row by row, calculate the proportion α of values ​​of 255 or 0 within the sliding window, and record the current sliding window number (blocks). Following this calculation rule, calculate the noise proportion α of the entire image. total ,Right now:

[0064]

[0065] Where, ω h ω represents the height of the sliding window. w This indicates the width of the sliding window.

[0066] In some embodiments, after evaluating the noise level information, a pixel value threshold can be determined as a filter parameter based on the noise level. The filter can be, for example, a median filter. The filter can further determine the location of the noise point based on the pixel value threshold, and adjust the pixel value of the noise point based on the location of the noise point and the pixel values ​​within a certain pixel range around the noise point (for example, adjust the pixel value of the point to the median or mean of the pixel values ​​within a certain pixel range around the point), thereby obtaining the denoised image.

[0067] In some embodiments, the first operation described above can also be implemented based on a trained image denoising model. The image denoising model is trained based on a sample noise image, a superimposed image, and first noise level information of the sample noise image. The superimposed image is obtained by superimposing simulated noise on the sample noise image, and the distribution ratio of the simulated noise is determined based on the first noise level information.

[0068] Therefore, by applying a denoising model to perform the above operations, this model is trained using noisy images with simulated noise levels added as training samples and the original noisy images as label samples. This effectively addresses the problem of significant differences between the statistical distribution of noise in the test data and the prior image data compared to the training data, without requiring noise-free image samples.

[0069] In some embodiments, the above-described process of predicting the location information of at least one noise point based on noise level information, and denoising the image to be processed based on the location information of at least one noise point to obtain a denoised image, can be implemented by an image denoising model. Specifically, the image to be processed and the noise level information can be input into the image denoising model together, so that the image denoising model outputs a denoised image of the image. The noise level information can be a noise ratio value obtained statistically based on the above method or an array composed of the noise ratio of each window.

[0070] In some embodiments, the image denoising model described above can be trained based on a denoising convolutional neural network (DnCNN).

[0071] In some embodiments, the above-described image denoising model can also be trained based on the following image denoising model.

[0072] In some embodiments, the image denoising model may include a first feature extraction network and a superposition network. Obtaining a denoised image based on the image to be processed and noise level information includes: inputting the image to be processed and noise level information as first input data into the first feature extraction network to obtain a first feature map output by the first feature extraction network; and superimposing the first feature map and the image to be processed based on the superposition network to obtain a denoised image.

[0073] Therefore, by superimposing the image to be processed after feature extraction, residual operations on the extracted features can be achieved, enabling the model to extract richer information and improve the denoising effect during the application stage; during the model training stage, it helps to alleviate the gradient vanishing problem and accelerate the training process.

[0074] In some embodiments, the first feature extraction network may be a network built on a convolutional neural network, such as one or more convolutional layers.

[0075] In some embodiments, the overlay network is used to overlay the first feature map output by the first feature extraction network and the image to be processed to obtain a denoised image.

[0076] In some embodiments, the image denoising model may further include a second feature extraction network disposed before the first feature extraction network and a feature integration network disposed after the overlay network.

[0077] Furthermore, in some embodiments, such as Figure 3As shown, obtaining a denoised image based on the image to be processed and noise level information may include: step S301, inputting the image to be processed and noise level information into a second feature extraction network to obtain a second feature map output by the second feature extraction network; step S302, inputting the second feature map as first input data into a first feature extraction network to obtain a first feature map output by the first feature extraction network; step S303, superimposing the first feature map and the second feature map based on a superposition network to obtain a third feature map; and step S304, inputting the third feature map into a feature integration network to obtain a denoised image output by the feature integration network.

[0078] Therefore, by using the second feature extraction network and the feature integration network, richer feature information can be extracted, thereby further improving the image denoising effect.

[0079] In some embodiments, the second feature extraction network may be a network built on a convolutional neural network, such as one or more convolutional layers.

[0080] In some embodiments, the feature integration network can be a network built on a convolutional neural network, such as one or more convolutional layers.

[0081] In some embodiments, the second feature map extracted by the second feature extraction network can be input into the first feature extraction network as the first input data to obtain a first feature map; subsequently, the first feature map and the second feature map are superimposed through a superposition network to obtain a third feature map; and feature integration is performed on the third feature map based on a feature integration network to obtain a denoised image. This enables a more in-depth extraction of image features, further improving the image denoising results.

[0082] In some embodiments, the feature integration network includes a cascaded first integration subnetwork and a second integration subnetwork, the first integration subnetwork including a first branch and a second branch, and the second integration subnetwork including at least a convolutional layer.

[0083] And, as Figure 4 As shown, inputting the third feature map into the feature integration network to obtain the denoised image output by the feature integration network can include:

[0084] Step S401: Extract features from the third feature map based on the first branch of the first integrated sub-network to obtain the fourth feature map output by the first branch;

[0085] Step S402: Based on the second branch of the first integrated sub-network, perform channel-wise dimensionality upscaling on the third feature map; and

[0086] Step S403: Based on the second branch of the first integrated sub-network, feature extraction is performed on the upgraded third feature map, and channel dimensionality reduction is performed on the extracted feature map to obtain the fifth feature map. The fifth feature map has the same number of channels as the third feature map.

[0087] Step S404: Integrate the fourth and fifth feature maps to obtain the sixth feature map; and

[0088] Step S405: Input the sixth feature map into the second integration sub-network to obtain the denoised image output by the second integration sub-network.

[0089] Therefore, by applying the first fusion subnetwork with two branches, the network's representational ability can be further enhanced, richer noise distribution information can be extracted, and the image denoising effect can be improved.

[0090] In some embodiments, the first branch may include one or more convolutional layers.

[0091] In some embodiments, the first branch may include one or more separable convolutional networks, such as a 3×3 depth separable convolutional network. This can further enhance the network's representational capabilities, extract richer noise distribution information, and improve image denoising performance.

[0092] In some embodiments, the dimensionality increase and dimensionality decrease operations in the second branch can be implemented based on a 1×1 convolution kernel, respectively.

[0093] In some embodiments, feature extraction in the second branch can be implemented based on one or more convolutional layers.

[0094] In some embodiments, feature extraction in the second branch can be based on one or more separable convolutional networks, such as a 3×3 depth separable convolutional network. This further enhances the network's representational capabilities, extracts richer noise distribution information, and improves image denoising performance.

[0095] In some embodiments, the second integrated subnetwork may be implemented based on one or more convolutional layers.

[0096] In some embodiments, the second integrated subnetwork may be implemented based on one or more convolutional layers and an activation function, wherein the activation function may be, for example, a ReLU function.

[0097] In some embodiments, the first feature extraction network includes N cascaded first extraction subnetworks and M second extraction subnetworks, wherein each first extraction subnetwork includes a third branch and a fourth branch, and each second extraction subnetwork includes at least a convolutional layer, and N and M are both positive integers.

[0098] Furthermore, the process of inputting the first input data into the first feature extraction network to obtain the first feature map output by the first feature extraction network may include: using the nth first extraction sub-network among N first extraction sub-networks, performing the following operations on the output feature map of the (n-1)th first extraction sub-network to obtain the output feature map of the nth first extraction sub-network: performing feature extraction on the output feature map of the (n-1)th first extraction sub-network based on the third branch of the nth first extraction sub-network to obtain the seventh feature map; performing channel upscaling operation on the output feature map of the (n-1)th first extraction sub-network based on the fourth branch of the nth first extraction sub-network; and performing a channel upscaling operation on the output feature map of the (n-1)th first extraction sub-network based on the nth first... The fourth branch of the extraction sub-network is used to extract features from the output feature map after dimensionality increase, and channel dimensionality reduction is performed on the extracted feature map to obtain the eighth feature map. The eighth feature map has the same number of channels as the output feature map of the (n-1)th first extraction sub-network. The seventh and eighth feature maps are then integrated to obtain the output feature map of the nth first extraction sub-network, where n is a positive integer less than or equal to N. When n equals 1, the output feature map of the (n-1)th first extraction sub-network is the first input data. M second extraction sub-networks are used to extract features from the output feature map of the Nth first extraction sub-network to obtain the first feature map.

[0099] Therefore, by applying a first extraction subnetwork with two branches, it is possible to extract multi-dimensional features such as mid-level and high-level features of the input image, better understand the abstract information and key details in the image, and thus further improve the image denoising effect.

[0100] In some embodiments, the third branch may include one or more convolutional layers.

[0101] In some embodiments, the third branch may include one or more separable convolutional networks, such as a 3×3 depth separable convolutional network. This can further enhance the network's representational capabilities, extract richer noise distribution information, and improve image denoising performance.

[0102] In some embodiments, the dimensionality increase and decrease operations in the fourth branch can be implemented based on a 1×1 convolution kernel, respectively.

[0103] In some embodiments, feature extraction in the fourth branch can be implemented based on one or more convolutional layers.

[0104] In some embodiments, feature extraction in the fourth branch can be based on one or more separable convolutional networks, such as a 3×3 depth separable convolutional network. This further enhances the network's representational capabilities, extracts richer noise distribution information, and improves image denoising performance.

[0105] In some embodiments, the second extraction subnetwork may be implemented based on one or more convolutional layers.

[0106] In some embodiments, N and M can be any positive integer. Understandably, N and M can be determined based on actual needs.

[0107] In some exemplary embodiments, N can be 3 and M can be 1.

[0108] Figure 5 A structural block diagram of an image denoising model according to an exemplary embodiment of the present disclosure is shown. Figure 6 A structural block diagram of a first extraction subnetwork and a first integration subnetwork according to an exemplary embodiment of the present disclosure is shown.

[0109] In some exemplary embodiments, such as Figure 5 As shown, the image to be processed and its corresponding noise level information are input into the second feature extraction network 510 (i.e., a convolutional layer) for feature extraction to obtain a second feature map. Subsequently, the second feature map is input into three cascaded first extraction sub-networks (including first extraction sub-networks 520-1, 520-2, and 520-3) to obtain an output feature map. This output feature map is then further input into the second extraction sub-network 530 (i.e., a convolutional layer) to obtain a first feature map. Based on the superposition network 540, the first and second feature maps are superimposed to obtain a third feature map. The third feature map is then input into the first integration sub-network 550, and the obtained output feature map is further input into the second integration sub-network 560 (including a convolutional layer and a ReLU activation function layer) to obtain the final output denoised image.

[0110] In some exemplary embodiments, each of the first extraction sub-network 520 and the first integration sub-network 550 may have the same network structure, such as... Figure 6 As shown, each of the above sub-networks includes two branches. After the input feature map is input into this sub-network, a 3×3 depthwise separable convolutional network on one branch extracts features from the input feature map, thus obtaining the output feature map of that branch. Simultaneously, on the other branch, a 1×1 convolutional kernel is first used to increase the dimensionality of the feature map, thereby increasing the number of channels in the input feature map. Then, a 3×3 depthwise separable convolutional network is used to extract features from the feature map with increased channels. Finally, another 1×1 convolutional kernel is used to reduce the dimensionality of the extracted feature map, adjusting its number of channels to match the input feature map. Figure 1The feature maps of the two branches are concatenated or superimposed to obtain the output feature map of the sub-network.

[0111] Therefore, through the aforementioned sub-networks, multi-dimensional features such as mid-level and high-level features of the input image can be extracted, enabling the model to better understand the abstract information and key details in the image, thereby further improving the image denoising effect.

[0112] In some embodiments, such as Figure 7 As shown, a training method for an image denoising model is provided. The method includes: step S701, evaluating the noise level of the original noisy image to obtain noise level information of the original noisy image, wherein the original noisy image includes at least one noise point, and the noise level information is used to indicate the distribution ratio of at least one noise point in the original noisy image; step S702, randomly generating simulated noise based on the noise level information; step S703, superimposing the simulated noise on the original noisy image to obtain a training image, wherein the distribution ratio of the noise points contained in the simulated noise in the training image is the same as the distribution ratio of at least one noise point in the original noisy image; step S704, inputting the training image and the noise level information into the image denoising model to obtain a predicted image output by the image denoising model; step S705, calculating a comprehensive loss based on the predicted image and the original noisy image; and step S706, adjusting at least one parameter of the image denoising model based on the comprehensive loss.

[0113] Therefore, by first evaluating the noise level of a noisy image with an unknown noise level, the noise proportion of the current noisy image is estimated. Then, based on the current noisy image, simulated noisy images with corresponding noise levels are added as training samples, and the original noisy image is used as the label sample. The loss function is calculated and the model is trained. This effectively solves the problem of significant differences between the statistical distribution of noise in the test data and the prior image data relative to the training data, without requiring noise-free image samples.

[0114] In related technologies, when denoising a noisy image, noise-free images and noisy images are often paired for training. After training, the trained model is directly used on the test set to denoise the noisy image. Specifically, for example, images generated by adding noise to a noise-free image are used as training samples, while noise-free images or images generated by appropriately processing noise-free images are used as label samples for model training.

[0115] However, in real-world applications, it is often difficult to obtain noise-free image samples with the same distribution as the training dataset, which affects the reconstruction results of noisy images to some extent. Furthermore, such learning strategies are easily affected by "domain disparity," meaning there is a significant difference between the noise statistical distribution of the test data and the image prior relative to the training data. This problem becomes even more pronounced when processing signal-dependent noise in real-world images.

[0116] The model training method provided in the above embodiments of this disclosure can utilize the noisy image itself as a label sample without requiring noise-free image samples, and further add noise (salt-and-pepper noise) to the noisy image as a training sample for the network. Therefore, it can effectively solve the problem of significant differences between the statistical distribution of noise in the test data and the prior image data relative to the training data, even without requiring noise-free image samples.

[0117] In some embodiments, the original noisy image may be an image containing at least one noise point.

[0118] In some embodiments, the noise contained in the original noisy image may be salt-and-pepper noise, or other noise with similar properties to salt-and-pepper noise, such as Poisson noise or Gaussian noise. The method of this disclosure will be described below using salt-and-pepper noise as an example.

[0119] In some embodiments, the noise level can be assessed by counting the number of noise points in the original noisy image and determining its proportion among all pixels in the original noisy image, thereby determining the noise level information of the original noisy image.

[0120] In some embodiments, noise level assessment can also be performed by setting a window detector to estimate the salt-and-pepper noise ratio of the noisy image block by block.

[0121] In some embodiments, evaluating the noise level of the original noisy image to obtain noise level information of the original noisy image includes: using a sliding window with a preset length and width, statistically analyzing the noise points within the area selected by the sliding window corresponding to each pixel in the original noisy image, to obtain noise statistics information corresponding to that pixel, wherein the length and width of the sliding window are smaller than the length and width of the original noisy image, respectively; and determining the noise level information of the original noisy image based on the noise statistics information corresponding to each pixel in the original noisy image.

[0122] Therefore, by using a sliding window, the noise distribution in multiple regions of a noisy image can be statistically analyzed, thereby obtaining richer noise information and improving the subsequent image denoising effect.

[0123] In some embodiments, the noise ratio can be statistically analyzed one by one, using each pixel in the image to be processed as the center of the window, based on the sliding window described above.

[0124] In some embodiments, the number of noise points and / or the proportion of noise points in each sliding window can be recorded separately as noise statistics.

[0125] In some embodiments, the proportion data of all noise points can be directly used as the noise level information of the image to be processed.

[0126] In some embodiments, the average value of all noise point proportions can be calculated to serve as the noise level information of the image to be processed.

[0127] In some embodiments, noise points can be detected by detecting whether the pixel value of a certain pixel is greater than a first preset threshold or less than a second preset threshold; if the above conditions are met, it is determined to be a noise point.

[0128] For the detection of salt-and-pepper noise, in some exemplary embodiments, the salt-and-pepper noise model can be denoted as n, and the noise-free image as x, i.e.:

[0129]

[0130] Where α represents the proportion of salt-and-pepper noise in the noisy image, and x(i,j) represents the pixel value corresponding to the (i,j)th pixel in the noise-free image.

[0131] Define a coordinate system centered at (i, j) and of size ω. h ×ω w Sliding window W i，j Starting from the top left corner of the noisy image y, iterate row by row, calculate the proportion α of values ​​of 255 or 0 within the sliding window, and record the current sliding window number (blocks). Following this calculation rule, calculate the noise proportion α of the entire image. total ,Right now:

[0132]

[0133] Where, ω h ω represents the height of the sliding window. w This indicates the width of the sliding window.

[0134] In some embodiments, the image denoising model described above can be applied to a denoising convolutional neural network (DnCNN).

[0135] In some embodiments, the image denoising model described above may also have the structure described below.

[0136] In some embodiments, the image denoising model may include a first feature extraction network and a superposition network. Inputting the training image and noise level information into the image denoising model to obtain the predicted image output by the image denoising model includes: inputting the training image and noise level information as first input data into the first feature extraction network to obtain a first feature map output by the first feature extraction network; and superimposing the first feature map and the training image based on the superposition network to obtain the predicted image.

[0137] In some embodiments, the image denoising model includes a second feature extraction network, a first feature extraction network, a stacking network, and a feature integration network. Inputting a training image and noise level information into the image denoising model to obtain a predicted image output by the image denoising model includes: inputting the training image and noise level information into the second feature extraction network to obtain a second feature map output by the second feature extraction network; inputting the second feature map as first input data into the first feature extraction network to obtain a first feature map output by the first feature extraction network; stacking the first feature map and the second feature map based on the stacking network to obtain a third feature map; and inputting the third feature map into the feature integration network to obtain a predicted image output by the feature integration network.

[0138] In some embodiments, the feature integration network includes a cascaded first integration subnetwork and a second integration subnetwork. The first integration subnetwork includes a first branch and a second branch, and the second integration subnetwork includes at least a convolutional layer. Inputting a third feature map into the feature integration network to obtain a predicted image output by the feature integration network includes: performing feature extraction on the third feature map based on the first branch of the first integration subnetwork to obtain a fourth feature map output by the first branch; performing the following operations on the third feature map based on the second branch of the first integration subnetwork: performing channel dimensionality increase on the third feature map; and performing feature extraction on the dimensionality increase third feature map, and performing channel dimensionality reduction on the extracted feature map to obtain a fifth feature map, the fifth feature map having the same number of channels as the third feature map; integrating the fourth feature map and the fifth feature map to obtain a sixth feature map; and inputting the sixth feature map into the second integration subnetwork to obtain a predicted image output by the second integration subnetwork.

[0139] In some embodiments, the first feature extraction network includes N cascaded first extraction sub-networks and M second extraction sub-networks, wherein each first extraction sub-network includes a third branch and a fourth branch, each second extraction sub-network includes at least a convolutional layer, N and M are both positive integers, and wherein inputting first input data into the first feature extraction network to obtain a first feature map output by the first feature extraction network includes: using the nth first extraction sub-network among the N first extraction sub-networks, performing the following operation on the output feature map of the (n-1)th first extraction sub-network to obtain the output feature map of the nth first extraction sub-network: performing feature extraction on the output feature map of the (n-1)th first extraction sub-network based on the third branch of the nth first extraction sub-network to obtain a seventh feature map; based on the nth first extraction sub-network, performing feature extraction on the output feature map of the (n-1)th first extraction sub-network to obtain a seventh feature map; performing feature extraction on the output feature map of the (n-1)th first extraction sub-network based on the third branch of the n ... The fourth branch of the network performs channel-level dimensionality upscaling on the output feature map of the (n-1)th first extraction sub-network; based on the fourth branch of the nth first extraction sub-network, it performs feature extraction on the dimensionality-upgraded output feature map and performs channel-level dimensionality downscaling on the extracted feature map to obtain the eighth feature map, which has the same number of channels as the output feature map of the (n-1)th first extraction sub-network; and it integrates the seventh and eighth feature maps to obtain the output feature map of the nth first extraction sub-network, where n is a positive integer less than or equal to N. When n equals 1, the output feature map of the (n-1)th first extraction sub-network is the first input data; and it uses M second extraction sub-networks to perform feature extraction on the output feature map of the Nth first extraction sub-network to obtain the first feature map.

[0140] The structure of each part in the above image denoising model is similar to the structure of each part in the model used in the above image processing method, and will not be described in detail here.

[0141] In some embodiments, after obtaining the noise level information of the original noisy image, simulated noise is generated based on the noise level information and superimposed on the original noisy image to serve as a training image.

[0142] In some embodiments, image noise with the same or similar noise ratio as the overall noise ratio in the image can be randomly generated; alternatively, noise can be generated for each window based on the noise ratio in each window of the image, so that the simulated noise ratio in each window is consistent with its corresponding noise ratio.

[0143] In some exemplary embodiments, the salt-and-pepper noise ratio α of the original noise image is obtained through the above noise level evaluation. total Then, randomly generated noise of proportion α is added to the original noisy image y. total Salt and pepper noise n, generating training image y new ,Right now:

[0144] y new =y+n

[0145] Subsequently, the training image and noise level information can be input into the image denoising model to obtain the predicted image output by the image denoising model.

[0146] Subsequently, a comprehensive loss can be calculated based on the predicted image and the original noisy image.

[0147] In some embodiments, such as Figure 8 As shown, the calculation of the comprehensive loss based on the predicted image and the original noisy image includes: step S801, calculating a first loss and a second loss based on the predicted image and the original noisy image, wherein the first loss is determined based on the pixel value difference of corresponding pixels between the predicted image and the original noisy image, and the second loss is determined based on the feature distribution difference between the predicted image and the original noisy image; step S802, determining the first weight corresponding to the first loss and the second weight corresponding to the second loss; and step S803, weighting the first loss and the second loss based on the first weight and the second weight to obtain the comprehensive loss.

[0148] Therefore, by calculating the pixel-level loss and the feature distribution loss separately and then weighting them to obtain the comprehensive loss, the trained model can generate images that are closer to the perceptual features of noisy images while preserving image details.

[0149] In some exemplary embodiments, the first loss may be determined based on the difference in pixel values ​​of corresponding pixels between the predicted image and the original noisy image. For example, the first loss may be calculated based on the absolute value error loss function (L1 loss function).

[0150] In some exemplary embodiments, the second loss may be determined based on the difference in feature distribution between the predicted image and the original noisy image. For example, the second loss may be calculated based on the Kullback-Leibler divergence (KL divergence) loss function.

[0151] In some exemplary embodiments, the L1 loss between the generated image and the noisy image can be used as a pixel-level loss, while the KL divergence loss between the perceptual features of the generated image and the noisy image in the network can be added as a perceptual feature-level loss. This can guide the generated image to more closely approximate the perceptual features of the noisy image while preserving image details. The expression for the loss function is as follows:

[0152] L total =λ L1 ·L L1 (F(y new ), y)+λ perceptual ·Lperceptual (F(y new ), y)

[0153] Where, λ L1 and λ perceptual These are the weights of the L1 loss for pixel values ​​and the KL loss for perceptual features, used to control the relative importance between the two.

[0154] The L1 loss function for pixel values ​​can be expressed as follows:

[0155]

[0156] The KL divergence loss for perceptual features can be defined by calculating the difference in KL divergence between the feature representations of the generated image and the noisy image in the network. The expression for the KL divergence loss for perceptual features is as follows:

[0157] L perceptual (F(y new ), y)=1-KL(F(y new ), y)

[0158] Wherein, KL(·), namely Kullback-Leibler divergence, represents the difference between two feature distributions. The larger the value, the closer the two feature distributions are.

[0159] y new This is the set of noisy images after adding the corresponding salt and pepper noise, where N is the number of pixels in the noisy images. It is the set of denoised images; F(·) is the processing of the image denoising model; F(y new y(i,j) represents the pixel value corresponding to the (i,j)th pixel in the denoised image; y(i,j) represents the pixel value corresponding to the (i,j)th pixel in the original noisy image.

[0160] In some embodiments, the above model training method may further include: adjusting the first weight and the second weight based on the comprehensive loss.

[0161] Therefore, by adjusting the first and second weights based on the comprehensive loss while adjusting the model parameters, it is possible to further accelerate model training and improve the training effect of the model, so that the trained model can generate images that are closer to the perceptual features of noisy images while preserving image details.

[0162] In some embodiments, the initial learning rate during model training can be adjusted based on the estimated noise ratio described above. For example, if the noise ratio is less than 0.5, the initial learning rate is set to 1x10^-4; if the noise ratio is greater than 0.5, the initial learning rate is set to 1x10^-3.

[0163] In some embodiments, an Adam optimizer with preset parameters can be used to adjust the initial learning rate during training.

[0164] In some embodiments, the current learning rate can be halved after a preset number of iterations until training ends.

[0165] In some exemplary embodiments, an Adam optimizer with parameters beta1 = 0.8 and beta2 = 0.88 is used, and the learning rate is halved every 10,000 iterations.

[0166] In some embodiments, the training of the model is considered to have ended when the overall loss is less than a preset threshold or the number of training iterations reaches a preset threshold.

[0167] Figure 9 A flowchart illustrating model training and model application according to embodiments of the present disclosure is shown.

[0168] In some exemplary embodiments, such as Figure 9 As shown, the model training and application process of the embodiments of this disclosure includes: Step S901: In the model training stage, the original noisy image is input into the noise estimation module to evaluate the noise level of the image and obtain the noise level information of the image; Step S902: Based on the noise level information, simulated noise is randomly generated; Step S903: The simulated noise is superimposed on the original noisy image to obtain a training image; Step S904: The training image and noise level information are input into the image denoising model to obtain the predicted image output by the image denoising model; Step S905: Based on the predicted image and the original noisy image, a comprehensive loss is calculated; Step S906: Based on the comprehensive loss, at least one parameter of the image denoising model is adjusted; Step S907: After the model completes training and goes online, for the obtained image to be processed, the image is first input into the noise estimation module to evaluate the noise level of the image and obtain the noise level information of the image; Step S908: The image to be processed and the noise level information are input into the trained image denoising model to obtain the denoised image output by the image denoising model.

[0169] Figure 10 The image denoising result obtained according to an exemplary embodiment of the present disclosure is shown.

[0170] According to embodiments of this disclosure, denoising experiments were conducted on noisy images with different noise ratios. For example... Figure 10As shown, noisy image 1001 and noisy image 1003 are salt-and-pepper noise images with salt-and-pepper noise ratios of 0.3 and 0.95, respectively, and their corresponding denoising results are denoised image 1002 and denoised image 1004, respectively. It can be seen that, according to the embodiments of this disclosure, noisy images can be accurately denoised, thereby obtaining accurately denoised and reconstructed images.

[0171] In some embodiments, such as Figure 11 An image processing apparatus 1100 is provided, comprising:

[0172] The first acquisition unit 1110 is configured to perform noise level evaluation on the image to be processed to obtain noise level information of the image to be processed, wherein the image to be processed includes at least one noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the image to be processed; and

[0173] Execution unit 1120 is configured to perform a first operation through subordinate subunits based on the image to be processed and noise level information to obtain a denoised image. The execution unit includes:

[0174] Prediction subunit 1121 is configured to predict the location information of at least one noise point based on noise level information; and

[0175] The processing subunit 1122 is configured to perform denoising processing on the image to be processed based on the location information of at least one noise point to obtain a denoised image.

[0176] The methods performed by units 1110, 1120, and subunits 1121 and 1122 in the image processing apparatus 1100 are similar to steps S201 to S202 and steps S2021 to S2022 in the above-mentioned image processing method, and will not be described again here.

[0177] In some embodiments, such as Figure 12 As shown, a training device 1200 for an image denoising model is provided, comprising:

[0178] The first acquisition unit 1210 is configured to perform noise level evaluation on the original noise image to obtain noise level information of the original noise image, wherein the original noise image includes at least one noise point, and the noise level information is used to indicate the distribution ratio of at least one noise point in the original noise image.

[0179] The generation unit 1220 is configured to randomly generate simulated noise based on noise level information;

[0180] The second acquisition unit 1230 is configured to superimpose simulated noise onto the original noise image to obtain a training image, wherein the distribution ratio of noise points contained in the simulated noise in the training image is the same as the distribution ratio of at least one noise point in the original noise image.

[0181] The input unit 1240 is configured to input the training image and noise level information into the image denoising model to obtain the predicted image output by the image denoising model.

[0182] The computing unit 1250 is configured to calculate a comprehensive loss based on the predicted image and the original noisy image; and

[0183] The adjustment unit 1260 is configured to adjust at least one parameter of the image denoising model based on the comprehensive loss.

[0184] The methods performed by units 1210 to 1260 in the image denoising model training device 1200 are similar to steps S701 to S706 in the image processing method described above, and will not be repeated here.

[0185] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0186] refer to Figure 13 The present invention describes a structural block diagram of an electronic device 1300 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0187] like Figure 13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. The RAM 1303 may also store various programs and data required for the operation of the electronic device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0188] Multiple components in electronic device 1300 are connected to I / O interface 1305, including: input unit 1306, output unit 1307, storage unit 1308, and communication unit 1309. Input unit 1306 can be any type of device capable of inputting information to electronic device 1300. Input unit 1306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 1307 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1308 may include, but is not limited to, a hard disk and an optical disk. The communication unit 1309 allows the electronic device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0189] The computing unit 1301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the image processing methods or image denoising model training methods described above. For example, in some embodiments, the image processing methods or image denoising model training methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the image processing methods or image denoising model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured by any other suitable means (e.g., by means of firmware) to perform the above-described image processing method or the training method of the image denoising model.

[0190] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0191] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0192] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0193] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0194] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0195] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0196] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0197] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. An image processing method, comprising: A noise level assessment is performed on the image to be processed to obtain noise level information of the image to be processed, wherein the image to be processed includes at least one noise point, the at least one noise point being a salt-and-pepper noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the image to be processed. The noise level assessment of the image to be processed to obtain the noise level information of the image to be processed includes: Using a sliding window with preset length and width, noise points within the area selected by the sliding window corresponding to each pixel in the image to be processed are statistically analyzed to obtain noise statistics for that pixel. The length and width of the sliding window are respectively smaller than the length and width of the image to be processed. Based on the noise statistics of each pixel in the image to be processed, the noise level information of the image to be processed is determined. The noise level information includes an array consisting of the noise proportions of the sliding window corresponding to each pixel; and Based on the image to be processed and the noise level information, a first operation is performed to obtain a denoised image, the first operation including: Based on the noise level information, predict the location information of the at least one noise point; and Based on the location information of the at least one noise point, the image to be processed is denoised to obtain the denoised image; The first operation is implemented based on a trained image denoising model, which is trained based on a sample noise image, a superimposed image, and first noise level information of the sample noise image. The superimposed image is obtained by superimposing simulated noise on the sample noise image and is used as a training sample for training the image denoising model. The proportion of simulated noise in the sliding window corresponding to each pixel in the superimposed image is consistent with the noise proportion of the corresponding sliding window in the first noise level information. The simulated noise is salt and pepper noise. The sample noise image is used as a label sample for training the image denoising model.

2. The method according to claim 1, wherein, The image denoising model includes a first feature extraction network and a stacking network. Based on the image to be processed and the noise level information, the denoised image is obtained by: The image to be processed and the noise level information are input as first input data into the first feature extraction network to obtain a first feature map output by the first feature extraction network; and Based on the overlay network, the first feature map and the image to be processed are overlaid to obtain the denoised image.

3. The method according to claim 1, wherein, The image denoising model includes a second feature extraction network, a first feature extraction network, a stacking network, and a feature integration network. Obtaining the denoised image based on the image to be processed and the noise level information includes: The image to be processed and the noise level information are input into the second feature extraction network to obtain the second feature map output by the second feature extraction network. The second feature map is used as the first input data and input into the first feature extraction network to obtain the first feature map output by the first feature extraction network. Based on the overlay network, the first feature map and the second feature map are overlaid to obtain a third feature map; and The third feature map is input into the feature integration network to obtain the denoised image output by the feature integration network.

4. The method according to claim 3, wherein, The feature integration network includes a cascaded first integration subnetwork and a second integration subnetwork. The first integration subnetwork includes a first branch and a second branch. The second integration network includes at least a convolutional layer. The step of inputting the third feature map into the feature integration network to obtain the denoised image output by the feature integration network includes: Based on the first branch of the first integrated sub-network, feature extraction is performed on the third feature map to obtain the fourth feature map output by the first branch; Based on the second branch of the first integrated subnetwork, the following operation is performed on the third feature map: Perform channel-level dimensionality upscaling on the third feature map; and Feature extraction is performed on the third feature map after dimensionality increase, and channel dimensionality reduction is performed on the extracted feature map to obtain a fifth feature map, which has the same number of channels as the third feature map; The fourth and fifth feature maps are integrated to obtain a sixth feature map; and The sixth feature map is input into the second integrated sub-network to obtain the denoised image output by the second integrated sub-network.

5. The method according to any one of claims 2 to 4, wherein, The first feature extraction network comprises N cascaded first extraction subnetworks and M second extraction subnetworks, wherein each first extraction subnetwork includes a third branch and a fourth branch, and each second extraction subnetwork includes at least a convolutional layer, wherein N and M are both positive integers, and wherein... Inputting the first input data into the first feature extraction network to obtain the first feature map output by the first feature extraction network includes: Using the nth first extraction subnetwork among the N first extraction subnetworks, perform the following operation on the output feature map of the (n-1)th first extraction subnetwork to obtain the output feature map of the nth first extraction subnetwork: Based on the third branch of the nth first extraction sub-network, feature extraction is performed on the output feature map of the (n-1)th first extraction sub-network to obtain the seventh feature map; Based on the fourth branch of the nth first extraction sub-network, the output feature map of the (n-1)th first extraction sub-network is subjected to channel dimensionality upscaling operation; Based on the fourth branch of the nth first extraction sub-network, feature extraction is performed on the dimensionality-upgraded output feature map, and channel-wise dimensionality reduction is performed on the extracted feature map to obtain an eighth feature map. This eighth feature map has the same number of channels as the output feature map of the (n-1)th first extraction sub-network. The seventh and eighth feature maps are integrated to obtain the output feature map of the nth first extraction sub-network, where n is a positive integer less than or equal to N, and when n equals 1, the nth... The output feature map of the first extraction sub-network is the first input data; The first feature map is obtained by using the M second extraction sub-networks to extract features from the output feature map of the Nth first extraction sub-network.

6. A method for training an image denoising model, the method comprising: A noise level assessment is performed on the original noisy image to obtain noise level information of the original noisy image, wherein the original noisy image includes at least one noise point, the at least one noise point being a salt-and-pepper noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the original noisy image. The noise level assessment of the original noisy image to obtain the noise level information of the original noisy image includes: Using a sliding window with preset length and width, noise points within the area selected by the sliding window corresponding to each pixel in the original noise image are statistically analyzed to obtain noise statistics for that pixel. The length and width of the sliding window are respectively smaller than the length and width of the original noise image. Based on the noise statistics of each pixel in the original noisy image, the noise level information of the original noisy image is determined. The noise level information includes an array composed of the noise ratio of the sliding window corresponding to each pixel. Based on the noise level information, simulated noise is generated randomly; Simulated noise is superimposed on the original noise image to obtain a training image. The distribution ratio of noise points contained in the simulated noise in the training image is the same as the distribution ratio of the at least one noise point in the original noise image. The proportion of simulated noise in the sliding window corresponding to each pixel in the training image is consistent with the noise proportion of the corresponding sliding window in the noise level information. The simulated noise is salt and pepper noise. The training image and the noise level information are input into the image denoising model to obtain the predicted image output by the image denoising model; Based on the predicted image and the original noisy image, calculate the comprehensive loss; and Based on the comprehensive loss, at least one parameter of the image denoising model is adjusted.

7. The method according to claim 6, wherein, The image denoising model includes a first feature extraction network and a stacking network. The step of inputting the training image and the noise level information into the image denoising model to obtain the predicted image output by the image denoising model includes: The training image and the noise level information are input as first input data into the first feature extraction network to obtain a first feature map output by the first feature extraction network; and Based on the overlay network, the first feature map and the training image are overlaid to obtain the predicted image.

8. The method according to claim 7, wherein, The image denoising model includes a second feature extraction network, a first feature extraction network, a stacking network, and a feature integration network. The step of inputting the training image and the noise level information into the image denoising model to obtain the predicted image output by the image denoising model includes: The training image and the noise level information are input into the second feature extraction network to obtain the second feature map output by the second feature extraction network. The second feature map is used as the first input data and input into the first feature extraction network to obtain the first feature map output by the first feature extraction network. Based on the overlay network, the first feature map and the second feature map are overlaid to obtain a third feature map; and The third feature map is input into the feature integration network to obtain the predicted image output by the feature integration network.

9. The method according to claim 8, wherein, The feature integration network includes a cascaded first integration subnetwork and a second integration subnetwork. The first integration subnetwork includes a first branch and a second branch. The second integration subnetwork includes at least a convolutional layer. The step of inputting the third feature map into the feature integration network to obtain the predicted image output by the feature integration network includes: Based on the first branch of the first integrated sub-network, feature extraction is performed on the third feature map to obtain the fourth feature map output by the first branch; Based on the second branch of the first integrated subnetwork, the following operation is performed on the third feature map: Perform channel-level dimensionality upscaling on the third feature map; and Feature extraction is performed on the third feature map after dimensionality increase, and channel dimensionality reduction is performed on the extracted feature map to obtain a fifth feature map, which has the same number of channels as the third feature map; The fourth and fifth feature maps are integrated to obtain a sixth feature map; and The sixth feature map is input into the second integrated subnetwork to obtain the predicted image output by the second integrated subnetwork.

10. The method according to any one of claims 7 to 9, wherein, The first feature extraction network comprises N cascaded first extraction subnetworks and M second extraction subnetworks, wherein each first extraction subnetwork includes a third branch and a fourth branch, and each second extraction subnetwork includes at least a convolutional layer, wherein N and M are both positive integers, and wherein... Inputting the first input data into the first feature extraction network to obtain the first feature map output by the first feature extraction network includes: Using the nth first extraction subnetwork among the N first extraction subnetworks, perform the following operation on the output feature map of the (n-1)th first extraction subnetwork to obtain the output feature map of the nth first extraction subnetwork: Based on the third branch of the nth first extraction sub-network, feature extraction is performed on the output feature map of the (n-1)th first extraction sub-network to obtain the seventh feature map; Based on the fourth branch of the nth first extraction sub-network, the output feature map of the (n-1)th first extraction sub-network is subjected to channel dimensionality upscaling operation; Based on the fourth branch of the nth first extraction sub-network, feature extraction is performed on the dimensionality-upgraded output feature map, and channel-wise dimensionality reduction is performed on the extracted feature map to obtain an eighth feature map. This eighth feature map has the same number of channels as the output feature map of the (n-1)th first extraction sub-network. The seventh and eighth feature maps are integrated to obtain the output feature map of the nth first extraction sub-network, where n is a positive integer less than or equal to N, and when n equals 1, the nth... The output feature map of the first extraction sub-network is the first input data; The first feature map is obtained by using the M second extraction sub-networks to extract features from the output feature map of the Nth first extraction sub-network.

11. The method according to claim 6, wherein, The calculation of the comprehensive loss based on the predicted image and the original noisy image includes: Based on the predicted image and the original noisy image, a first loss and a second loss are calculated, wherein the first loss is determined based on the pixel value difference of corresponding pixels between the predicted image and the original noisy image, and the second loss is determined based on the feature distribution difference between the predicted image and the original noisy image; Determine the first weight corresponding to the first loss and the second weight corresponding to the second loss; and The first loss and the second loss are weighted based on the first weight and the second weight to obtain the comprehensive loss.

12. The method of claim 11, further comprising: Based on the comprehensive loss, the first weight and the second weight are adjusted.

13. An image processing apparatus, comprising: The first acquisition unit is configured to perform noise level evaluation on the image to be processed to obtain noise level information of the image to be processed, wherein the image to be processed includes at least one noise point, the at least one noise point is a salt-and-pepper noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the image to be processed. The noise level evaluation of the image to be processed to obtain the noise level information of the image to be processed includes: Using a sliding window with preset length and width, noise points within the area selected by the sliding window corresponding to each pixel in the image to be processed are statistically analyzed to obtain noise statistics for that pixel. The length and width of the sliding window are respectively smaller than the length and width of the image to be processed. Based on the noise statistics of each pixel in the image to be processed, the noise level information of the image to be processed is determined. The noise level information includes an array composed of the noise proportions of the sliding window corresponding to each pixel; and An execution unit is configured to perform a first operation through subordinate subunits based on the image to be processed and the noise level information to obtain a denoised image. The execution unit includes: A prediction subunit is configured to predict the location information of the at least one noise point based on the noise level information; and The processing subunit is configured to perform denoising processing on the image to be processed based on the location information of the at least one noise point, so as to obtain the denoised image; The first operation is implemented based on a trained image denoising model, which is trained based on a sample noise image, a superimposed image, and first noise level information of the sample noise image. The superimposed image is obtained by superimposing simulated noise on the sample noise image and is used as a training sample for training the image denoising model. The proportion of simulated noise in the sliding window corresponding to each pixel in the superimposed image is consistent with the noise proportion of the corresponding sliding window in the first noise level information. The simulated noise is salt and pepper noise. The sample noise image is used as a label sample for training the image denoising model.

14. A training apparatus for an image denoising model, the apparatus comprising: The first acquisition unit is configured to perform a noise level assessment on the original noisy image to obtain noise level information of the original noisy image, wherein the original noisy image includes at least one noise point, the at least one noise point is a salt-and-pepper noise point, and the noise level information is used to indicate the distribution ratio of the at least one noise point in the original noisy image. The first acquisition unit is further configured to: Using a sliding window with preset length and width, noise points within the area selected by the sliding window corresponding to each pixel in the original noise image are statistically analyzed to obtain noise statistics for that pixel. The length and width of the sliding window are respectively smaller than the length and width of the original noise image. Based on the noise statistics of each pixel in the original noisy image, the noise level information of the original noisy image is determined. The noise level information includes an array composed of the noise ratio of the sliding window corresponding to each pixel. The generation unit is configured to randomly generate simulated noise based on the noise level information; The second acquisition unit is configured to superimpose simulated noise onto the original noise image to obtain a training image. The distribution ratio of noise points contained in the simulated noise in the training image is the same as the distribution ratio of the at least one noise point in the original noise image. The proportion of simulated noise in the sliding window corresponding to each pixel in the training image is consistent with the noise proportion of the corresponding sliding window in the noise level information. The simulated noise is salt and pepper noise. The input unit is configured to input the training image and the noise level information into the image denoising model to obtain the predicted image output by the image denoising model; The computing unit is configured to calculate a comprehensive loss based on the predicted image and the original noisy image; and The adjustment unit is configured to adjust at least one parameter of the image denoising model based on the comprehensive loss.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

17. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-12.

Images