Image denoising method and device

Abstract

The application provides an image denoising method and device, comprising: training an initial network model to obtain a target network model; specifically, inputting a sample image into a feature extraction channel of a network channel in a sub-network to train parameters of the feature extraction channel to obtain initial feature extraction parameters; inputting the sample image into the feature extraction channel of the network channel and a denoising main channel in the sub-network to train parameters of the denoising main channel to obtain initial denoising parameters; inputting the sample image into the sub-network to adjust the initial feature extraction parameters and the initial denoising parameters of the sub-network to obtain candidate feature extraction parameters and candidate denoising parameters; inputting the sample image into the initial network model to adjust the candidate feature extraction parameters and the candidate denoising parameters of each sub-network of the initial network model to obtain target feature extraction parameters and target denoising parameters. Through the technical scheme, the image can be denoised.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image noise reduction method and device. Background Technology

[0002] In low-light, high-gain scenarios, when imaging devices (such as cameras) capture images, a lot of noise appears in the images, resulting in poor image quality, affecting the intuitive perception of the face, and making the facial features unclear.

[0003] To remove noise from an image, i.e., to perform image denoising, image denoising algorithms can be used, such as BM3D (Block-Matching and 3D filtering). The working principle of BM3D is as follows: the image is segmented into small pixel patches. After selecting a reference patch, similar small patches are found to form 3D blocks. All similar blocks undergo 3D transformation. The transformed 3D blocks are then thresholded and shrunk, which is also a noise removal process. An inverse 3D transformation is performed, and all 3D blocks are weighted and averaged to restore the image. The core of BM3D lies in employing different denoising strategies, searching for similar blocks and filtering them in the transform domain to obtain evaluation values, and finally weighting each point in the image to achieve the denoising effect.

[0004] However, image denoising algorithms struggle to handle facial noise under varying lighting conditions, resulting in poor edge fidelity preservation of facial features. For instance, facial images exhibit significant noise variations and morphological distributions under different imaging devices and lighting conditions. When using image denoising algorithms, details at the boundaries of facial features are often mistakenly removed as noise, leading to poor protection of facial details. Summary of the Invention

[0005] This application provides an image denoising method. The initial network model to be trained includes K sub-networks. For each sub-network, the sub-network includes M network channels. Each network channel includes a denoising main channel and a feature extraction channel. K and M are both positive integers. The method includes:

[0006] The initial network model is trained to obtain the trained target network model;

[0007] After obtaining and deploying the target network model, the image to be denoised is acquired, and the image to be denoised is input into the target network model to obtain the target image with image denoising completed.

[0008] The initial network model is trained to obtain a trained target network model, including:

[0009] For each sub-network, a sample image is input to the feature extraction channel of the network channel in the sub-network to train the parameters of the feature extraction channel and obtain initial feature extraction parameters; the sample image is input to the feature extraction channel and the main noise reduction channel of the network channel in the sub-network to train the parameters of the main noise reduction channel and obtain initial noise reduction parameters.

[0010] The sample image is input into the sub-network to adjust the initial feature extraction parameters and initial noise reduction parameters of the sub-network, thereby obtaining candidate feature extraction parameters and candidate noise reduction parameters;

[0011] The sample image is input into the initial network model to adjust the candidate feature extraction parameters and candidate denoising parameters of each sub-network of the initial network model, so as to obtain the target feature extraction parameters and target denoising parameters. The target network model is generated based on the target feature extraction parameters and target denoising parameters.

[0012] This application provides an image noise reduction device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image noise reduction method disclosed in the above example of this application.

[0013] As can be seen from the above technical solutions, the embodiments of this application propose a face image denoising method based on deep learning technology. This method can achieve image denoising based on a target network model, without requiring an image denoising algorithm. When denoising based on the target network model, it can take into account facial noise under different lighting conditions, and it has a good edge fidelity preservation effect for facial features. Even if the noise in the face image varies greatly and the noise morphology distribution is different, it will not remove the details of the facial feature boundaries as noise, thus better protecting facial details. By denoising the image, image quality can be significantly improved, noise in the image can be reduced, facial features can be made clearer, and the image display effect can be improved. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0015] Figure 1 This is a schematic diagram of the target network model in one embodiment of this application;

[0016] Figure 2This is a schematic diagram of the sub-network structure in one embodiment of this application;

[0017] Figure 3 This is a schematic diagram of the sub-network structure in one embodiment of this application;

[0018] Figure 4 This is a schematic diagram of the hourglass module in one embodiment of this application;

[0019] Figure 5 This is a schematic diagram of the structure of a multiple residual layer in one embodiment of this application;

[0020] Figure 6 This is a schematic diagram of the target network model in one embodiment of this application;

[0021] Figure 7 This is a flowchart illustrating an image noise reduction method according to one embodiment of this application;

[0022] Figure 8 This is a hardware structure diagram of an image noise reduction device according to one embodiment of this application. Detailed Implementation

[0023] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0025] When using image denoising algorithms to reduce image noise, it is difficult to take into account facial noise under different lighting conditions, resulting in poor edge fidelity preservation of facial features. For example, under different lighting conditions, the noise in facial images varies greatly, and the noise morphology and distribution are also different. When using image denoising algorithms, it is easy to treat the details of the facial feature boundaries as noise and remove them, resulting in poor preservation of facial details.

[0026] To address the aforementioned problems, this application proposes an image denoising method based on deep learning technology, specifically employing deep learning algorithms to achieve image denoising. To achieve image denoising using deep learning algorithms, this application involves the following processes: 1. Designing a network model. This network model can be based on a deep learning algorithm or another machine learning algorithm; there are no restrictions. 2. The network model training process, i.e., training the network model to obtain a trained network model. The network model before training is designated as the initial network model, and the trained network model is designated as the target network model. Image denoising can be achieved based on the target network model. During the network model training process, a loss function is designed to optimize and adjust the parameters of the initial network model, thereby training the target network model. 3. The image denoising process based on the target network model. That is, the image to be denoised can be input into the target network model, and the target network model outputs the denoised image.

[0027] First, regarding the structure of the initial network model or the target network model, the structure of the initial network model is the same as that of the target network model. The structure of the target network model will be used as an example in the following sections.

[0028] In this embodiment, the target network model is a network model used to denoise the face region in the image. It is a network model specially designed according to actual use cases, which can denoise face images under different illumination scenarios, thereby significantly improving image quality and reducing noise in the image.

[0029] In this embodiment, the target network model may include K sub-networks, where K is a positive integer, such as 1, 2, 3, 4, etc. There is no restriction on the value of K; it can be configured according to actual needs. For example, K can be 2 or 3. As the value of K increases, the complexity of the target network model also increases; that is, the complexity of the target network model increases with the increase of K.

[0030] See Figure 1 The diagram shows the structure of the target network model, which includes several sub-networks. Net-1 is the first sub-network of the target network model, Net-2 is the second sub-network of the target network model, ..., Net-K is the Kth sub-network of the target network model, which is the last sub-network.

[0031] from Figure 1As can be seen, the input data of Net-1 is the external input image, that is, the external input image of the target network model, which is the image that needs to be denoised. This image can be a Bayer format image, an RGB format image, a YUV format image, or other types of images; there are no restrictions. We will use a Bayer format image as an example below. Obviously, after obtaining the Bayer format image, if image denoising is needed, this image is input to Net-1. After receiving the Bayer format image, Net-1 processes it to obtain the processed image, which is called the target output image of Net-1, and then inputs the target output image of Net-1 to Net-2.

[0032] Net-2's input data is the target output image of Net-1. After receiving the target output image of Net-1, Net-2 processes the image to obtain a processed image, which is called Net-2's target output image. Net-2's target output image is then input to Net-3.

[0033] Similarly, after receiving the target output image from the previous sub-network, Net-K processes the image to obtain a processed image, which is called the target output image of Net-K. The target output image of Net-K is the final output image of the target network model, which is an image with image denoising completed. In other words, it successfully performs image denoising on the external input image and obtains an image with image denoising completed.

[0034] For each subnetwork of the target network model, the subnetwork can include M network channels, where M can be a positive integer, such as 1, 2, 3, 4, etc. There are no restrictions on the value of M; it can be configured according to actual needs. For example, M can be 2 or 3. As the value of M increases, the complexity of the subnetwork also increases; that is, the complexity of the subnetwork increases with the increase of M.

[0035] All subnetworks can have the same value of M, such as all subnetworks having an M of 3. Or, all subnetworks can have partially the same value of M, such as some subnetworks having an M of 3 and the remaining subnetworks having an M of 2. Or, all subnetworks can have completely different values ​​of M, such as Net-1 having an M of 1, Net-2 having an M of 2, and so on.

[0036] For example, the structure of each sub-network can be the same. For ease of description, this embodiment takes the structure of one sub-network as an example, such as the first sub-network (i.e., Net-1) as an example for explanation.

[0037] For the first sub-network, it can include M network channels. Each network channel includes a denoising main channel and a feature extraction channel. The feature extraction channel is used to extract classification features of various parts of the face (including parts of the body), and the denoising main channel is used to perform image denoising, such as image denoising based on the classification features extracted by the feature extraction channel. For ease of explanation, we will use 3 network channels as an example, and denote these 3 network channels as network channel a1, network channel a2, and network channel a3.

[0038] See Figure 2 The diagram shows the structure of the sub-network. Network channel a1 may include a denoising main channel b11 and a feature extraction channel b12, network channel a2 may include a denoising main channel b21 and a feature extraction channel b22, and network channel a3 may include a denoising main channel b31 and a feature extraction channel b32.

[0039] For each network channel of a sub-network, the original input image corresponding to that channel can be obtained and input into the denoising main channel and feature extraction channel corresponding to that network channel. Specifically, if the network channel is the first network channel of the sub-network, the image to be denoised for that sub-network is determined as the original input image corresponding to that network channel; if the network channel is not the first network channel of the sub-network, the image to be denoised for that sub-network is downsampled to obtain the original input image corresponding to that network channel. If the sub-network is the first sub-network of the target network model, the external input image of the target network model is used as the image to be denoised for that sub-network; if the sub-network is not the first sub-network of the target network model, the target output image of the previous sub-network is used as the image to be denoised for that sub-network.

[0040] For example, for Net-1, the external input image of the target network model is used as the image to be denoised in Net-1. For Net-2, the target output image of Net-1 (i.e., the output image obtained after image processing by Net-1) is used as the image to be denoised in Net-2. For Net-3, the target output image of Net-2 is used as the image to be denoised in Net-3, and so on, thus determining the image to be denoised for each sub-network. In the following process, the implementation process of Net-1 will be used as an example; the implementation methods of other sub-networks are similar.

[0041] Net-1 includes network channels a1, a2, and a3. Network channel a1 is the first network channel of Net-1, and the image to be denoised in Net-1 is used as the original input image for network channel a1. Network channel a2 is the second network channel of Net-1, and it performs downsampling processing on the image to be denoised in Net-1. For example, if the image to be denoised is downsampled once, the image after one downsampling is used as the original input image for network channel a2. Network channel a3 is the third network channel of Net-1, and it performs downsampling processing on the image to be denoised in Net-1. For example, if the image to be denoised is downsampled twice, the image after both downsamplings is used as the original input image for network channel a3. For example, the size of the image to be denoised is 256*256*4, where 256*256 represents the width and height of the image, and 4 represents the number of channels in the image. The size of the original input image corresponding to network channel a1 is 256*256*4, the size of the original input image corresponding to network channel a2 is 128*128*4, and the size of the original input image corresponding to network channel a3 is 64*64*4.

[0042] In summary, the original input image corresponding to network channel a1 can be obtained, denoted as original input image c1. Original input image c1 is then input to the denoising main channel b11 and feature extraction channel b12 corresponding to network channel a1. Similarly, the original input image corresponding to network channel a2 can be obtained, denoted as original input image c2. Original input image c2 is then input to the denoising main channel b21 and feature extraction channel b22 corresponding to network channel a2. Finally, the original input image corresponding to network channel a3 can be obtained, denoted as original input image c3. Original input image c3 is then input to the denoising main channel b31 and feature extraction channel b32 corresponding to network channel a3.

[0043] See Figure 3 The diagram shows the structure of the sub-network. For each feature extraction channel (e.g., feature extraction channel b12, feature extraction channel b22, feature extraction channel b32), the feature extraction channel includes a residual module and an hourglass module. For each denoising main channel (e.g., denoising main channel b11, denoising main channel b21, denoising main channel b31), the denoising main channel includes an encoding module (the number of encoding modules is at least one), a feature processing module, and a decoding module (the number of decoding modules is the same as the number of encoding modules).

[0044] In this process, after the original input image is fed into the feature extraction channel, residual processing and hourglass processing are performed on the original input image through the feature extraction channel to obtain the feature matrix corresponding to the original input image. This feature matrix is ​​then input into the denoising main channel. For example, the residual module can perform residual processing on the original input image to obtain a residual matrix; the hourglass module can then perform hourglass processing on this residual matrix to obtain the feature matrix corresponding to the original input image, and this feature matrix is ​​then input into the denoising main channel.

[0045] The process involves inputting the original input image into the denoising main channel, which then encodes the image to obtain coded features. Based on these coded features and the feature matrix, a feature code is obtained. This feature code is then decoded to obtain the original output features. For example, the encoding module performs convolution on the original input image to obtain initial features. Multiple residual processing is then applied to these initial features to obtain coded features corresponding to the original input image, which are then input into the feature processing module. The feature processing module superimposes the coded features and the feature matrix to obtain a feature code, which is then input into the decoding module. The decoding module performs multiple residual processing on the feature code to obtain the target features, which are then deconvolved by the decoding module to obtain the original output features.

[0046] The structure and function of each module will be explained below in the context of specific application scenarios.

[0047] 1. Residual Module. The residual module can be composed of residual networks. There are no restrictions on the structure of the residual network, as long as it can perform residual processing. Therefore, after the original input image is input to the residual module, the residual module can use the residual network to perform residual processing on the original input image to obtain the residual matrix.

[0048] 2. Hourglass Module. The hourglass module consists of N convolutional layers, where N can be a positive integer greater than 1. For example, N can be an odd number such as 3, 5, 7, 9, 11, or 13, or an even number such as 2, 4, 6, 8, or 10. After the residual matrix is ​​input into the hourglass module, the module uses the N convolutional layers to perform hourglass processing on the residual matrix, obtaining the feature matrix corresponding to the original input image. For instance, the hourglass processing involves performing convolution and deconvolution on the residual matrix using N convolutional layers to obtain the feature matrix.

[0049] In one possible implementation, taking N convolutional layers as an example, which is 9 convolutional layers, the hourglass module can include 9 convolutional layers. See also Figure 4The diagram shown is a structural schematic of an hourglass module. Of course, the number of convolutional layers in the hourglass module can be more or less. For example, if the hourglass module includes 7 convolutional layers, the 4th and 6th convolutional layers can be removed. If the hourglass module includes 5 convolutional layers, the 3rd, 4th, 6th and 7th convolutional layers can be removed.

[0050] In the above application scenario, the hourglass module uses N convolutional layers (e.g., 9 convolutional layers) to perform convolution and deconvolution processing on the residual matrix to obtain the feature matrix corresponding to the original input image, which may include:

[0051] The residual matrix is ​​input into the first convolutional layer, which then performs convolution on it to obtain matrix 1. For example, matrix 1 can be obtained by performing a convolutional filter on the residual matrix with a stride of 2.

[0052] Matrix 1 is input into the second convolutional layer, which then convolves it to obtain matrix 2. For example, matrix 2 can be obtained by performing a convolutional filter on matrix 1 with a stride of 2.

[0053] Matrix 2 is input into the third convolutional layer, which then convolves it to obtain matrix 3. For example, matrix 3 can be obtained by performing a convolutional filter on matrix 2 with a stride of 2.

[0054] Matrix 3 is input into the fourth convolutional layer, which then convolves it to obtain matrix 4. For example, matrix 4 can be obtained by performing a convolutional filter on matrix 3 with a stride of 1.

[0055] Matrix 4 is input into the fifth convolutional layer, which then convolves it to obtain matrix 5. For example, matrix 5 can be obtained by performing a convolutional filter on matrix 4 with a stride of 1.

[0056] Matrix 5 is input into the 6th convolutional layer, which convolves Matrix 5 to obtain Matrix 6. Then, Matrix 6 is deconvolved, and the deconvolved matrix (for example, the matrix obtained by deconvolving Matrix 6 with a stride of 2) is superimposed on Matrix 3. The superimposed matrix is ​​then convolved by the 7th convolutional layer to obtain Matrix 7.

[0057] After obtaining matrix 7 through the 7th convolutional layer, matrix 7 is deconvolved. The deconvolved matrix (such as the matrix obtained by deconvolving matrix 7 with a stride of 2) is superimposed on matrix 2. Then, the superimposed matrix is ​​convolved through the 8th convolutional layer to obtain matrix 8.

[0058] After obtaining matrix 8 through the 8th convolutional layer, matrix 8 is deconvolved. The deconvolved matrix (such as the matrix obtained by deconvolving matrix 8 with a stride of 2) is superimposed on matrix 1. Then, the superimposed matrix is ​​convolved through the 9th convolutional layer to obtain matrix 9.

[0059] After obtaining matrix 9 through the 9th convolutional layer, the feature matrix corresponding to the original input image can be determined based on matrix 9, such as using matrix 9 as the feature matrix corresponding to the original input image.

[0060] In the above embodiments, the size of matrix 9 is the same as that of matrix 1, such as w*h*c, where w represents the width, h represents the height, and c represents the number of channels. The size of matrix 8 is the same as that of matrix 2, such as (w / 2)*(h / 2)*(c*2). The size of matrix 7 is the same as that of matrix 3, such as (w / 4)*(h / 4)*(c*4). The sizes of matrices 4, 5, and 6 are the same, all of which are (w / 8)*(h / 8)*(c*8).

[0061] 3. Encoding Module. The encoding module consists of one convolutional layer and one multi-Res Block. After the original input image is fed into the convolutional layer of the encoding module, the original input image is processed by convolution. The features after convolution are recorded as initial features and fed into the multi-Res Block. The multi-Res Block performs multiple residual processing on the initial features to obtain the encoded features corresponding to the original input image, and then the encoded features are fed into the feature processing module.

[0062] For example, a multiple residual layer may include multiple convolutional layers (e.g., K convolutional layers). That is, the encoding module implements multiple residuals through K convolutional layers, where K is a positive integer greater than 1. K can be an odd number such as 3, 5, 7, or 9, or an even number such as 2, 4, 6, or 8. After the initial features are input into the multiple residual layer, the multiple residual layer can use K convolutional layers to perform multiple residual processing on the initial features to obtain the encoded features.

[0063] In one possible implementation, taking K convolutional layers as an example of 6 convolutional layers, see [link to relevant documentation]. Figure 5The diagram shows a schematic of a multiple residual layer. This multiple residual layer is designed for face reconstruction tasks. The input of convolution 1 and the output of convolution 2, the input of convolution 3 and the output of convolution 4, and the input of convolution 5 and the output of convolution 6 are added to the feature matrices using skip connections. To enhance the feature transferability of the entire multiple residual layer, the input of convolution 1 and the output of convolution 6, and the input of convolution 3 and the output of convolution 6, are also added to the feature matrices using skip connections.

[0064] In the above application scenarios, the multiple residual layer uses K convolutional layers (e.g., 6 convolutional layers) to perform multiple residual processing on the initial features to obtain the encoded features, which can include, but is not limited to, the following methods:

[0065] The first feature is obtained by convolving the initial feature using convolutional layer 1.

[0066] The second feature is obtained by convolving the first feature using convolutional layer 2.

[0067] The third feature is obtained by superimposing the initial feature and the second feature.

[0068] The third feature is convolved by the third convolutional layer to obtain the fourth feature;

[0069] The fourth feature is convolved by the fourth convolutional layer to obtain the fifth feature;

[0070] The third and fifth features are superimposed to obtain the sixth feature;

[0071] The sixth feature is convolved by the fifth convolutional layer to obtain the seventh feature.

[0072] The seventh feature is convolved by the sixth convolutional layer to obtain the eighth feature.

[0073] The sixth and eighth features are superimposed to obtain the ninth feature;

[0074] The third and ninth features are superimposed to obtain the tenth feature;

[0075] The initial feature and the tenth feature are superimposed to obtain the eleventh feature;

[0076] The encoding feature corresponding to the original input image is determined based on the eleventh feature. In other words, the eleventh feature is used as the encoding feature, and thus the encoding feature corresponding to the original input image is obtained.

[0077] 4. Feature Processing Module. The encoding module inputs encoded features to the feature processing module, and the feature extraction channel inputs the feature matrix to the feature processing module. Thus, the feature processing module can obtain feature codes based on the encoded features and the feature matrix. For example, for the feature processing module of the first network channel, its output, after passing through the residual module, serves as the input to the feature processing module of the second network channel. For the feature processing module of the second network channel, its output, after passing through the residual module, serves as the input to the feature processing module of the third network channel, and so on.

[0078] If the network channel is the first network channel of the sub-network, the feature processing module can superimpose the encoded features and the feature matrix to obtain the feature code corresponding to the network channel, and input the feature code to the decoding module, and input the feature code to the feature processing module of the next network channel.

[0079] If the network channel is not the first network channel of the sub-network, residual processing is performed on the feature encoding corresponding to the previous network channel to obtain residual features. For example, this feature encoding can be input to the residual module, which performs residual processing on the feature encoding to obtain residual features, and then inputs these residual features to the feature processing module. The feature processing module superimposes the encoded features, the feature matrix, and the residual features to obtain the feature encoding corresponding to the network channel, and inputs this feature encoding to the decoding module and then to the feature processing module of the next network channel.

[0080] In one possible implementation, the feature encoding can be determined using the following formula:

[0081] feature_encoding j =feature_encoding j-1 +encoding j +feature_face j

[0082] feature_encoding j The feature encoding of the j-th network channel is represented by `feature_encoding`. j-1 This represents the feature encoding of the (j-1)th network channel, which is the feature encoding of the previous network channel. j The feature_face represents the coding feature of the j-th network channel. j Let represent the feature matrix of the j-th network channel.

[0083] 5. Decoding Module. After obtaining the feature encoding, the decoding module performs multiple residual processing on the feature encoding to obtain the target features, and then performs deconvolution processing on the target features to obtain the original output features. For example, the decoding module can consist of one multi-residual layer and one deconvolution layer. After the feature encoding is input to the multi-residual layer of the decoding module, the feature encoding is processed by the multi-residual layer to obtain the target features corresponding to the original input image. The target features are then input to the deconvolution layer, and deconvolution processing is performed on the target features. The features after deconvolution processing are recorded as the original output features, which are the image features output by the main noise reduction channel.

[0084] For example, the multiple residual layer may include multiple convolutional layers (such as K convolutional layers). The multiple residual layer of the decoding module and the multiple residual layer of the encoding module may be the same, which will not be repeated here.

[0085] In summary, for each network channel of a sub-network, that channel can output original output features. Based on the original output features corresponding to each network channel, the target output image of that sub-network can be generated. For example, the original output features corresponding to each network channel can be upsampled to obtain upsampled features, and the upsampled features corresponding to M network channels can be fused to obtain the target output image.

[0086] In one possible implementation, after obtaining the target output image corresponding to the sub-network, if the sub-network is the last sub-network of the target network model, then the target output image corresponding to the sub-network is the image with completed image denoising output by the target network model. If the sub-network is not the last sub-network of the target network model, then the target output image corresponding to the sub-network is used to determine the original input image corresponding to each network channel of the next sub-network, i.e., as the image to be denoised.

[0087] See Figure 6 The diagram shown is a schematic representation of the subnetwork structure of the target network model. Figure 6 The subnetwork shown illustrates its structure. Each subnetwork can include three network channels, and each channel can include a denoising main channel and a feature extraction channel. The feature extraction channel includes a residual module and an hourglass module. The denoising main channel includes two encoding modules, a feature processing module, and two decoding modules. Two residual modules exist between the two feature processing modules.

[0088] The outputs of the first encoding module and the first decoding module of each network channel are cascaded and fused in a skip-step manner. The output images of the second and third network channels, obtained through the encoding modules, are upsampled to obtain an image of the same size as the first network channel. Finally, the output images of the three network channels are fused through upsampling to obtain the final output image.

[0089] For the main noise reduction channel, the three input dimensions are processed by their respective encoding modules to obtain feature code 1, feature code 2, and feature code 3. Each feature code is then processed by a decoding module to obtain its own output image. Simultaneously, feature code 1 is fused with feature code 2 through two residual modules, and feature code 2 is fused with feature code 3 through two residual modules. For the feature extraction channel, each input image, upon passing through the encoding module, is processed by a feature extraction module composed of a residual module and an hourglass module. The feature matrix output by the hourglass module is obtained, and this feature matrix is ​​added to and fused with the output of the second encoding module to obtain the feature code.

[0090] Second, regarding the training process of the network model, a sample dataset can be obtained. This dataset can include a large number of sample images (these sample images can include face images under different lighting conditions, and are face images with significant differences in noise). Each sample image has a corresponding label (the labels for the sample images can be found in subsequent embodiments). The sample images and labels can be input into the initial network model to train the initial network model and obtain the trained target network model. During the training process of the initial network model, a loss function is designed to optimize and adjust the parameters of the initial network model, thereby training the target network model.

[0091] In this embodiment, the initial network model to be trained includes K sub-networks. Each sub-network includes M network channels, and each network channel includes a denoising main channel and a feature extraction channel. K and M are both positive integers. Referring to the above embodiment, the structure of the initial network model is given (the structure of the initial network model is the same as the structure of the target network model). The initial network model can be trained to obtain the trained target network model. See also... Figure 7 The diagram shown illustrates the training process for the initial network model, which may include:

[0092] Step 701: For each sub-network of the initial network model, input the sample image into the feature extraction channel of the network channel in that sub-network to train the parameters of the feature extraction channel, obtain the initial feature extraction parameters, and configure the initial feature extraction parameters for that feature extraction channel.

[0093] In one possible implementation, different network channels within the same sub-network can correspond to the same initial feature extraction parameters; that is, the same initial feature extraction parameters are used as the initial feature extraction parameters for each network channel in that sub-network. Alternatively, network channels in different sub-networks can correspond to different or the same initial feature extraction parameters. For example, initial feature extraction parameters can be determined separately for each sub-network. For instance, inputting a sample image into the feature extraction channel of network channel Net-1 yields initial feature extraction parameter t11, which is used as the initial feature extraction parameter for each feature extraction channel of network channel Net-1. Similarly, inputting a sample image into the feature extraction channel of network channel Net-2 yields initial feature extraction parameter t12, which is used as the initial feature extraction parameter for each feature extraction channel of network channel Net-2, and so on. In practical applications, initial feature extraction parameter t12 may be the same as or different from initial feature extraction parameter t11.

[0094] In one possible implementation, the following steps can be used to train the parameters of the feature extraction channel. Of course, the following steps are just examples and are not limited thereto, as long as training can be completed based on sample images (which can be a large number of sample images, but we will take one sample image as an example later).

[0095] Step 7011: Input the sample image into the feature extraction channel of each network channel in the sub-network to obtain the feature matrix, that is, obtain the feature matrix corresponding to each network channel.

[0096] Taking Net-1 as an example, Net-1 includes network channels a1, a2, and a3. Network channel a1 includes feature extraction channel b12, network channel a2 includes feature extraction channel b22, and network channel a3 includes feature extraction channel b32. The sample image is the image to be denoised in Net-1. Based on this, the sample image is used as the original input image for feature extraction channel b12. The sample image is then downsampled by a factor of 2 and used as the original input image for feature extraction channel b22. Finally, the sample image is downsampled by a factor of 4 and used as the original input image for feature extraction channel b32.

[0097] Referring to the above embodiments, the feature extraction channel may include a residual module and an hourglass module. Therefore, after inputting the original input image into the feature extraction channel, the residual module can perform residual processing on the original input image to obtain a residual matrix, and the hourglass module can perform hourglass processing on the residual matrix to obtain the feature matrix corresponding to the network channel. In summary, the feature matrices corresponding to network channel a1, network channel a2, and network channel a3 can be obtained.

[0098] The hourglass module can include N convolutional layers, where N is a positive integer greater than 1. When the residual matrix is ​​processed by the hourglass module to obtain the feature matrix corresponding to the network channel, the hourglass processing can be based on the convolution and deconvolution processing of the residual matrix by the N convolutional layers to obtain the feature matrix.

[0099] Step 7012: Determine the first initial loss value corresponding to the network channel based on the feature matrix and sample image.

[0100] For example, the first initial loss value for network channel a1 is determined based on the feature matrix corresponding to network channel a1 and the original input image (i.e., the sample image) corresponding to network channel a1. For instance, a target feature map (the label corresponding to the sample image) can be pre-labeled; the target feature map is a feature matrix with no or very small feature loss. The difference between the feature matrix corresponding to network channel a1 and the target feature map can be calculated; this difference represents the face feature loss, which is the first initial loss value for network channel a1. The first initial loss value for network channel a2 is determined based on the feature matrix corresponding to network channel a2 and the original input image (i.e., the sample image after 2x downsampling), and so on, to obtain the first initial loss value for each network channel.

[0101] Step 7013: Determine the first target loss value based on the first initial loss value corresponding to each network channel.

[0102] For example, the first target loss value can be determined based on the sum of the first initial loss values ​​corresponding to all network channels in the sub-network. For instance, the sum of all the first initial loss values ​​can be used as the first target loss value.

[0103] For example, the loss function for the first target loss value can be seen in the following formula:

[0104]

[0105] In the above formula, i represents the i-th network channel of the sub-network, m represents the total number of network channels in the sub-network, and loss face_iLet loss represent the first initial loss value of the i-th network channel in the j-th sub-network. face_j This represents the first target loss value of the j-th subnetwork, i.e., the face feature loss value.

[0106] Step 7014: Adjust the parameters of the feature extraction channels based on the first target loss value to obtain the initial feature extraction parameters. For example, based on the first target loss value, algorithms such as gradient descent can be used to adjust the parameters of the feature extraction channels. There are no restrictions on this adjustment process. The goal of the adjustment is to make the first target loss value smaller and smaller until the first target loss value meets the convergence requirement.

[0107] For example, after adjusting the parameters of the feature extraction channel based on the first target loss value, the adjusted parameters can be obtained. If the parameters of the feature extraction channel have met the convergence requirements, the adjusted parameters are used as the initial feature extraction parameters. If the parameters of the feature extraction channel have not met the convergence requirements, steps 7011-7014 are repeated based on the adjusted parameters to continue adjusting the parameters of the feature extraction channel.

[0108] For example, when adjusting the parameters of each feature extraction channel in the sub-network based on the first target loss value, the adjusted parameters of each feature extraction channel can be the same, so that different feature extraction channels in the sub-network can correspond to the same initial feature extraction parameters.

[0109] For example, based on steps 7011-7014, each sub-network can be processed to obtain the initial feature extraction parameters corresponding to the feature extraction channels in each sub-network, which will not be elaborated here.

[0110] For example, after obtaining the initial feature extraction parameters corresponding to the feature extraction channel, the initial feature extraction parameters can be configured for the feature extraction channel, and subsequent processing can be performed under this configuration.

[0111] Step 702: For each sub-network of the initial network model, input the sample image to the feature extraction channel and the main noise reduction channel of the network channel in that sub-network to train the parameters of the main noise reduction channel, obtain the initial noise reduction parameters, and configure the initial noise reduction parameters for the main noise reduction channel.

[0112] In one possible implementation, different network channels within the same sub-network can correspond to the same initial denoising parameter; that is, the same initial denoising parameter can be used as the initial denoising parameter for each network channel in that sub-network. Network channels in different sub-networks can correspond to different or the same initial denoising parameters. For example, the initial denoising parameter can be determined separately for each sub-network. For instance, inputting a sample image into the feature extraction channel and the main denoising channel of Net-1 yields an initial denoising parameter t21, which serves as the initial denoising parameter t21 for the main denoising channel of each network channel in Net-1. Similarly, inputting a sample image into the feature extraction channel and the main denoising channel of Net-2 yields an initial denoising parameter t22, which serves as the initial denoising parameter t22 for the main denoising channel of each network channel in Net-1, and so on.

[0113] In one possible implementation, the following steps can be used to train the parameters of the main noise reduction channel. Of course, the following steps are just examples and are not limited thereto, as long as training can be completed based on sample images (which can be a large number of sample images, but we will take one sample image as an example later).

[0114] Step 7021: Input the sample image into the feature extraction channel of each network channel in the sub-network to obtain the feature matrix corresponding to the sample image, and input the feature matrix into the noise reduction main channel.

[0115] For example, for the feature extraction channel, initial feature extraction parameters have been configured. Based on this, a sample image can be input to the feature extraction channel. The residual module can perform residual processing on the original input image corresponding to the sample image to obtain a residual matrix. The hourglass module can then perform hourglass processing on the residual matrix to obtain a feature matrix, which is then input to the noise reduction main channel.

[0116] Step 7022: Input the sample image into the main noise reduction channel of each network channel in the sub-network to obtain the original output features corresponding to the sample image and the feature matrix. That is, obtain the original output features corresponding to each network channel, which are obtained based on the sample image and the feature matrix.

[0117] Taking Net-1 as an example, network channel a1 includes the main noise reduction channel b11, network channel a2 includes the main noise reduction channel b21, network channel a3 includes the main noise reduction channel b31, and the sample image is the image to be denoised in Net-1.

[0118] Based on this, the sample image is used as the original input image for the main noise reduction channel b11, and this original input image is input into the main noise reduction channel b11. The sample image is downsampled by 2 times and used as the original input image for the main noise reduction channel b21, and this original input image is input into the main noise reduction channel b21, and so on.

[0119] Referring to the above embodiments, the denoising main channel may include an encoding module, a feature processing module, and a decoding module. Therefore, after the original input image is input to the denoising main channel, the encoding module can perform convolution processing on the original input image to obtain initial features, and then perform multiple residual processing on the initial features to obtain the encoded features corresponding to the original input image. Furthermore, the feature processing module superimposes the encoded features and the feature matrix (i.e., the feature matrix input from the feature extraction channel to the denoising main channel) to obtain the feature encoding corresponding to the original input image. Finally, the decoding module performs multiple residual processing on the feature encoding to obtain the target features, and then performs deconvolution processing on the target features to obtain the original output features corresponding to the network channel. In summary, the original output features corresponding to network channel a1, network channel a2, and network channel a3 can be obtained.

[0120] Step 7023: Determine the second initial loss value corresponding to each network channel based on the original output features and sample images.

[0121] For example, the second initial loss value for network channel a1 is determined based on the original output features corresponding to network channel a1 and the original input image (i.e., the sample image) corresponding to network channel a1. For instance, the output feature map corresponding to the sample image can be pre-labeled (the output feature map is the label corresponding to the sample image). The output feature map represents output features with no or very small feature loss. The difference between the original output features and the output feature map corresponding to network channel a1 can be calculated; this difference represents the second initial loss value for network channel a1. The second initial loss value for network channel a2 is determined based on the original output features corresponding to network channel a2 and the original input image (i.e., the sample image after 2x downsampling), and so on, to obtain the second initial loss value for each network channel.

[0122] Step 7024: Determine the second target loss value based on the second initial loss value corresponding to each network channel.

[0123] For example, the second target loss value can be determined based on the sum of the second initial loss values ​​corresponding to all network channels in the sub-network. For instance, the sum of all the second initial loss values ​​can be used as the second target loss value.

[0124] For example, the loss function for the second target loss value can be seen in the following formula:

[0125]

[0126] In the above formula, i represents the i-th network channel of the sub-network, m represents the total number of network channels in the sub-network, and loss main_i Let loss represent the second initial loss value of the i-th network channel in the j-th sub-network. main_j This represents the second objective loss value of the j-th subnetwork, which is the loss value of the main channel.

[0127] Step 7025: Adjust the parameters of the main denoising channel based on the second target loss value to obtain the initial denoising parameters. For example, based on the second target loss value, algorithms such as gradient descent can be used to adjust the parameters of the main denoising channel. There are no restrictions on this adjustment process; the goal is to make the second target loss value smaller and smaller until it meets the convergence requirement. When adjusting the parameters of the main denoising channel, the initial feature extraction parameters of the feature extraction channel must remain unchanged.

[0128] For example, after adjusting the parameters of the main noise reduction channel based on the second target loss value, the adjusted parameters can be obtained. If the parameters of the main noise reduction channel have met the convergence requirements, the adjusted parameters can be used as the initial noise reduction parameters. If the parameters of the main noise reduction channel have not met the convergence requirements, steps 7021-7025 can be repeated based on the adjusted parameters to continue adjusting the parameters of the main noise reduction channel.

[0129] For example, when adjusting the parameters of each denoising main channel in the sub-network based on the second target loss value, the adjusted parameters of each denoising main channel can be the same, so that different denoising main channels in the sub-network can correspond to the same initial denoising parameters.

[0130] For example, based on steps 7021-7025, each sub-network can be processed to obtain the initial noise reduction parameters corresponding to the main noise reduction channel in each sub-network, which will not be elaborated here.

[0131] For example, after obtaining the initial noise reduction parameters corresponding to the main noise reduction channel, the initial noise reduction parameters can be configured for the main noise reduction channel, and subsequent processing can be performed under this configuration.

[0132] Step 703: For each sub-network of the initial network model, input the sample image into the sub-network to adjust the initial feature extraction parameters and initial denoising parameters of the sub-network, obtain candidate feature extraction parameters and candidate denoising parameters, configure the candidate feature extraction parameters for the feature extraction channel (i.e., replace the initial feature extraction parameters with the candidate feature extraction parameters), and configure the candidate denoising parameters for the main denoising channel (i.e., replace the initial denoising parameters with the candidate denoising parameters).

[0133] In one possible implementation, different network channels within the same sub-network can correspond to the same candidate feature extraction parameters, and different network channels within the same sub-network can correspond to the same candidate denoising parameters. Furthermore, network channels within different sub-networks can correspond to different or the same candidate feature extraction parameters, and different network channels within different sub-networks can correspond to different or the same candidate denoising parameters.

[0134] In one possible implementation, the following steps can be used to adjust the initial feature extraction parameters and the initial noise reduction parameters. Of course, the following steps are just examples and are not intended to limit the implementation.

[0135] Step 7031: Input the sample image into the feature extraction channel of each network channel in the sub-network to obtain the feature matrix; determine the third initial loss value corresponding to the network channel based on the feature matrix.

[0136] For example, the feature extraction channel may include a residual module and an hourglass module. Therefore, after the sample image is input into the feature extraction channel, the sample image can be processed by the residual module to obtain a residual matrix, and the residual matrix can be processed by the hourglass module to obtain the feature matrix corresponding to the network channel. This process can be referred to step 7011, and will not be repeated here.

[0137] For example, for each network channel, after obtaining the feature matrix corresponding to that network channel, a third initial loss value can be determined based on the feature matrix. For instance, a first structural similarity index loss value and a first minimum absolute value deviation loss value can be determined based on the feature matrix. Then, the third initial loss value can be determined based on the first structural similarity index loss value and the first minimum absolute value deviation loss value.

[0138] For example, the method for determining the third initial loss value can be seen in the following formula:

[0139]

[0140] In the above formula, i represents the i-th network channel of the sub-network, m represents the total number of network channels in the sub-network, and loss face_i This represents the third initial loss value corresponding to the i-th network channel. This represents the first structural similarity index loss value corresponding to the i-th network channel. This represents the first minimum absolute value deviation loss value corresponding to the i-th network channel. a and b are weighting coefficients, which can be configured values.

[0141] For example, a structural similarity index loss function can be pre-configured. There are no restrictions on this structural similarity index loss function. After obtaining the feature matrix corresponding to the network channel, the feature matrix can be substituted into the structural similarity index loss function to obtain the first structural similarity index loss value.

[0142] For example, a minimum absolute value deviation loss function can be pre-configured. There are no restrictions on this minimum absolute value deviation loss function. After obtaining the feature matrix corresponding to the network channel, the feature matrix can be substituted into the minimum absolute value deviation loss function to obtain the first minimum absolute value deviation loss value.

[0143] Obviously, after obtaining the first structural similarity index loss value and the first minimum absolute value deviation loss value corresponding to the network channel, the first structural similarity index loss value and the first minimum absolute value deviation loss value can be substituted into the above formula to obtain the third initial loss value corresponding to the network channel.

[0144] Step 7032: Input the sample image into the main noise reduction channel of each network channel in the sub-network to obtain the original output features; determine the fourth initial loss value corresponding to the network channel based on the original output features.

[0145] For example, the denoising main channel may include an encoding module, a feature processing module, and a decoding module. Therefore, after inputting the sample image into the denoising main channel, the encoding module performs convolution processing on the sample image to obtain initial features. Multiple residual processing is then applied to the initial features to obtain encoded features. The feature processing module superimposes these encoded features with the feature matrix (i.e., the feature matrix input from the feature extraction channel to the denoising main channel) to obtain feature encoding. The decoding module performs multiple residual processing on this feature encoding to obtain target features. Finally, the decoding module performs deconvolution processing on the target features to obtain the original output features corresponding to the network channel. This process can be referred to in step 7022 and will not be repeated here.

[0146] For example, for each network channel, after obtaining the original output features corresponding to the network channel, the fourth initial loss value corresponding to the network channel can be determined based on the original output features. For instance, the second structural similarity index loss value and the second minimum absolute value deviation loss value corresponding to the network channel can be determined based on the original output features, and the fourth initial loss value corresponding to the network channel can be determined based on the second structural similarity index loss value and the second minimum absolute value deviation loss value.

[0147] For example, the method for determining the fourth initial loss value can be seen in the following formula:

[0148]

[0149] In the above formula, i represents the i-th network channel of the sub-network, m represents the total number of network channels in the sub-network, and loss main_i This represents the fourth initial loss value corresponding to the i-th network channel. This represents the second structural similarity index loss value corresponding to the i-th network channel. This represents the second minimum absolute value deviation loss value corresponding to the i-th network channel. c and d are weighting coefficients, which can be configured values.

[0150] For example, a structural similarity index loss function can be pre-configured. There are no restrictions on this structural similarity index loss function. After obtaining the original output features corresponding to the network channel, the original output features can be substituted into the structural similarity index loss function to obtain the second structural similarity index loss value.

[0151] For example, a minimum absolute value deviation loss function can be pre-configured. There are no restrictions on this minimum absolute value deviation loss function. After obtaining the original output features corresponding to the network channel, the original output features can be substituted into the minimum absolute value deviation loss function to obtain the second minimum absolute value deviation loss value.

[0152] Obviously, after obtaining the second structural similarity index loss value and the second minimum absolute value deviation loss value corresponding to the network channel, the second structural similarity index loss value and the second minimum absolute value deviation loss value can be substituted into the above formula to obtain the fourth initial loss value corresponding to the network channel.

[0153] Step 7033: Determine the third target loss value based on the third initial loss value and the fourth initial loss value corresponding to each network channel. For example, the third target loss value can be determined based on the sum of the third initial loss value and the fourth initial loss value corresponding to all network channels in the sub-network.

[0154] For example, the loss function for the third target loss value can be seen in the following formula:

[0155] loss NET_j =loss face_j +loss main_j

[0156] In the above formula, loss NET_j Let loss represent the third objective loss value corresponding to the j-th subnetwork. face_j The loss represents the sum of the third initial loss values ​​for all network channels corresponding to the j-th sub-network. main_j This represents the sum of the fourth initial loss values ​​for all network channels corresponding to the j-th subnetwork.

[0157] Step 7034: Adjust the initial feature extraction parameters and initial denoising parameters of the sub-network based on the third objective loss value to obtain candidate feature extraction parameters and candidate denoising parameters. For example, based on the third objective loss value, algorithms such as gradient descent can be used to adjust the initial feature extraction parameters and initial denoising parameters of all network channels of the sub-network. There are no restrictions on this adjustment process; the goal is to make the third objective loss value smaller and smaller until the third objective loss value meets the convergence requirement.

[0158] For example, after adjusting the initial feature extraction parameters and initial denoising parameters based on the third target loss value, adjusted feature extraction parameters and adjusted denoising parameters can be obtained. If the convergence requirement is met, the adjusted feature extraction parameters can be used as candidate feature extraction parameters, and the adjusted denoising parameters can be used as candidate denoising parameters. If the convergence requirement is not met, steps 7031-7034 can be repeated based on the adjusted feature extraction parameters and adjusted denoising parameters to continue the parameter adjustment process.

[0159] For example, based on steps 7031-7034, each sub-network can be processed to obtain candidate feature extraction parameters and candidate noise reduction parameters in each sub-network, which will not be elaborated here.

[0160] For example, after obtaining the candidate feature extraction parameters and candidate denoising parameters, the candidate denoising parameters can be configured for the main denoising channel and the candidate feature extraction parameters can be configured for the feature extraction channel.

[0161] In one possible implementation, for step 7032, SSIM can be used when training the main noise reduction channel. main_i L1 main_i and mixed loss main_i Three loss functions are used alternately for training, for example: when using SSIM main_i During training, c=1, d=0; when using L1... main_iDuring training, c=0, d=1; when using a hybrid loss function... main_i During training, c+d=1, and c>0, d>0.

[0162] In one possible implementation, for step 7031, SSIM can be used when training the feature extraction channels. face_i L1 face_i and mixed loss face_i Three loss functions are used alternately for training, for example: when using SSIM face_i During training, a=1, b=0; when using L1... face_i During training, a = 0, b = 1; when using a hybrid loss function... face_i During training, a + b = 1, and a > 0, b > 0.

[0163] Step 704: Input the sample image into the initial network model to adjust the candidate feature extraction parameters and candidate denoising parameters of each sub-network of the initial network model, thereby obtaining the target feature extraction parameters and target denoising parameters. Generate the target network model based on the target feature extraction parameters and target denoising parameters. For example, configure the target feature extraction parameters for the feature extraction channel (i.e., replace the candidate feature extraction parameters with the target feature extraction parameters), and configure the target denoising parameters for the main denoising channel (i.e., replace the candidate denoising parameters with the target denoising parameters), to obtain the trained target network model. In other words, the network model configured with the target feature extraction parameters and target denoising parameters is used as the final output target network model.

[0164] In one possible implementation, different network channels within the same sub-network can correspond to the same target feature extraction parameters, and different network channels within the same sub-network can correspond to the same target denoising parameters. Furthermore, network channels within different sub-networks can correspond to different or the same target feature extraction parameters, and different network channels within different sub-networks can correspond to different or the same target denoising parameters.

[0165] In one possible implementation, the following steps can be used to adjust the candidate feature extraction parameters and candidate denoising parameters. Of course, the following steps are just examples and are not intended to limit the implementation.

[0166] Step 7041: For each sub-network of the initial network model, input the sample image to the feature extraction channel of each network channel in the sub-network to obtain the feature matrix (i.e. the feature matrix corresponding to the network channel), and determine the fifth initial loss value corresponding to the network channel based on the feature matrix.

[0167] For example, for each feature extraction channel of the network channel, the sample image can be processed by the residual module to obtain a residual matrix, and the residual matrix can be processed by the hourglass module to obtain the feature matrix corresponding to that network channel. This process can be referred to in step 7031, except that the initial feature extraction parameters of the feature extraction channel are updated to candidate feature extraction parameters for training.

[0168] For example, for each network channel, after obtaining the feature matrix corresponding to that network channel, a fifth initial loss value can be determined based on the feature matrix. For instance, the structural similarity index loss value and the minimum absolute value deviation loss value can be determined based on the feature matrix. Then, the fifth initial loss value can be determined based on the structural similarity index loss value and the minimum absolute value deviation loss value. The method for determining the fifth initial loss value is the same as the method for determining the third initial loss value, and will not be repeated here.

[0169] Step 7042: For each sub-network of the initial network model, input the sample image to the denoising main channel of each network channel in the sub-network to obtain the original output features (i.e. the original output features corresponding to the network channel), and determine the sixth initial loss value corresponding to the network channel based on the original output features.

[0170] For example, for the main denoising channel of each network channel, the encoding module performs convolution processing on the sample image to obtain initial features, and then performs multiple residual processing on the initial features to obtain encoded features. The feature processing module superimposes the encoded features and the feature matrix to obtain feature encoding. The decoding module performs multiple residual processing on the feature encoding to obtain target features, and then performs deconvolution processing on the target features to obtain the original output features corresponding to that network channel. This process is similar to step 7032, except that the initial denoising parameters of the main denoising channel are updated to candidate denoising parameters for training.

[0171] For example, for each network channel, after obtaining the original output features corresponding to that network channel, a sixth initial loss value can be determined based on those original output features. For instance, the structural similarity index loss value and the minimum absolute value deviation loss value corresponding to that network channel can be determined based on the original output features, and the sixth initial loss value corresponding to that network channel can be determined based on those structural similarity index loss value and the minimum absolute value deviation loss value. The method for determining the sixth initial loss value can be found in the method for determining the fourth initial loss value, and will not be repeated here.

[0172] Step 7043: For each subnetwork of the initial network model, determine the fourth target loss value for that subnetwork based on the fifth initial loss value and the sixth initial loss value corresponding to each network channel in that subnetwork. For example, the fourth target loss value can be determined based on the sum of the fifth and sixth initial loss values ​​corresponding to all network channels in that subnetwork.

[0173] Step 7044: Determine the fifth objective loss value based on the fourth objective loss value corresponding to each sub-network.

[0174] For example, the fifth objective loss value can be determined based on the sum of the fourth objective loss values ​​corresponding to all sub-networks of the initial network model. The loss function for the fifth objective loss value can be found in the following formula:

[0175]

[0176] In the above formula, loss NET This represents the fifth objective loss value, which is the loss value of the entire initial network model, i.e., the global network loss value. NET_j This represents the fourth objective loss value corresponding to the j-th subnetwork. The value of j can be in the range of 1-n, where n represents the total number of subnetworks in the initial network model.

[0177] Step 7045: Adjust the candidate feature extraction parameters and candidate denoising parameters of each sub-network of the initial network model based on the fifth objective loss value to obtain the target feature extraction parameters and target denoising parameters of each sub-network of the initial network model. For example, based on the fifth objective loss value, algorithms such as gradient descent can be used to adjust the candidate feature extraction parameters and candidate denoising parameters of all network channels of all sub-networks of the initial network model. There are no restrictions on this adjustment process; the goal is to make the fifth objective loss value smaller and smaller until the fifth objective loss value meets the convergence requirement.

[0178] For example, after adjusting the candidate feature extraction parameters and candidate denoising parameters based on the fifth target loss value, the adjusted feature extraction parameters and adjusted denoising parameters can be obtained. If the convergence requirement has been met, the adjusted feature extraction parameters can be used as the target feature extraction parameters, and the adjusted denoising parameters can be used as the target denoising parameters. If the convergence requirement has not been met, steps 7041-7045 can be repeated based on the adjusted feature extraction parameters and adjusted denoising parameters to continue the parameter adjustment process.

[0179] For example, after obtaining the target feature extraction parameters and the target noise reduction parameters, the target noise reduction parameters can be configured for the main noise reduction channel, and the target feature extraction parameters can be configured for the feature extraction channel.

[0180] Furthermore, the network model with configured target denoising parameters and target feature extraction parameters can be used as the target network model, that is, the model training process is completed, and the trained target network model is obtained.

[0181] When the target network model is required, it can be ported to the device for simulation and corresponding integration testing, thereby deploying the target network model on the device.

[0182] Third, in the image denoising process based on the target network model, after obtaining and deploying the target network model, the image to be denoised can be acquired and input into the target network model to obtain the target image that has been denoised, and then the target network model outputs the denoised image.

[0183] For example, the structure of the target network model can be found in [reference needed]. Figure 3 As shown, the target network model includes K sub-networks. For each sub-network, there are M network channels. Each network channel includes a denoising main channel and a feature extraction channel. The image to be denoised is the external input image, which is the image that needs to be denoised. Based on this, the image denoising process can include the following steps:

[0184] Step S11: For each network channel of the sub-network, obtain the original input image corresponding to that network channel, and input the original input image into the noise reduction main channel and feature extraction channel corresponding to that network channel.

[0185] Step S12: Perform residual processing and hourglass processing on the original input image through the feature extraction channel to obtain the feature matrix corresponding to the original input image, and input the feature matrix into the noise reduction main channel.

[0186] In one possible implementation, the feature extraction channel may include a residual module and an hourglass module. The residual module can perform residual processing on the original input image to obtain a residual matrix; the hourglass module can perform hourglass processing on the residual matrix to obtain the feature matrix corresponding to the original input image.

[0187] Step S13: Encode the original input image through the noise reduction main channel to obtain the encoding features corresponding to the original input image. Based on the encoding features and the feature matrix, obtain the feature encoding and decode the feature encoding to obtain the original output features corresponding to the network channel.

[0188] In one possible implementation, the main noise reduction channel may include an encoding module, a feature processing module, and a decoding module. The encoding module performs convolution processing on the original input image to obtain initial features. These initial features are then subjected to multiple residual processing to obtain encoded features corresponding to the original input image, which are then input to the feature processing module. The feature processing module can superimpose the encoded features and the feature matrix to obtain feature encoding, which is then input to the decoding module. The decoding module can perform multiple residual processing on this feature encoding to obtain target features, and then perform deconvolution processing on the target features to obtain the original output features. Thus, the original output features corresponding to the network channel are obtained.

[0189] Step S14: Generate the target output image of the sub-network based on the original output features corresponding to each network channel. For example, upsample the original output features corresponding to each network channel to obtain upsampled features, and fuse the upsampled features corresponding to M network channels to obtain the target output image.

[0190] In one possible implementation, after obtaining the target output image corresponding to the sub-network, if the sub-network is the last sub-network of the target network model, then the target output image corresponding to the sub-network is the image with completed image denoising output by the target network model. If the sub-network is not the last sub-network of the target network model, then the target output image corresponding to the sub-network is used to determine the original input image corresponding to each network channel of the next sub-network, i.e., as the image to be denoised.

[0191] For example, the implementation of steps S11-S14 is related to the structure of the target network model, as can be found in the introduction in point one. This implementation process will not be repeated in this embodiment.

[0192] As can be seen from the above technical solutions, this application proposes a face image denoising method based on deep learning technology. This method can achieve image denoising based on a target network model, without requiring an image denoising algorithm. When denoising based on the target network model, it can take into account facial noise under different illumination scenarios, and it has a good edge fidelity preservation effect for facial features. Even if the noise in the face image varies greatly and the noise morphology distribution is different, it will not remove the details of the facial feature boundaries as noise, thus protecting facial details well. By denoising the image, image quality can be significantly improved, noise in the image can be reduced, facial features can be made clearer, and the image display effect can be improved. In this embodiment, a deep learning-based image denoising neural network structure is proposed. It achieves dual network hybridization based on a multi-scale denoising main channel and a feature extraction channel. Multi-channel feature encoding is progressively fused in each level of sub-channel, and a face feature extraction network based on a multi-scale residual module and an hourglass module is proposed. This paper proposes a multi-level progressive convolutional neural network for multi-scale encoding and decoding of faces and facial features. It also proposes a loss function for face denoising based on deep learning techniques and a training optimization method for this loss function. In this embodiment, a hybrid loss function design using a multi-dimensional structural similarity index and a minimum absolute value deviation index is employed. Furthermore, each sub-network uses an alternating loss function, and the main denoising channel and feature extraction channel within the sub-network are trained alternately.

[0193] Based on the same concept as the methods described above, this application proposes an image noise reduction device, see [link to relevant documentation]. Figure 8 As shown, the image denoising device may include: a processor 81 and a machine-readable storage medium 82, the machine-readable storage medium 82 storing machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the image denoising method disclosed in the above example of this application.

[0194] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the image noise reduction method disclosed in the above examples of this application.

[0195] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0196] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0197] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0198] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0200] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0201] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0202] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image denoising method, characterized in that, The initial network model to be trained includes K sub-networks. For each sub-network, there are M network channels, each including a denoising main channel and a feature extraction channel. K and M are both positive integers. The method includes: The initial network model is trained to obtain the trained target network model; After obtaining and deploying the target network model, the image to be denoised is acquired, and the image to be denoised is input into the target network model to obtain the target image with image denoising completed. The initial network model is trained to obtain a trained target network model, including: For each sub-network, a sample image is input to the feature extraction channel of the network channel in the sub-network to train the parameters of the feature extraction channel and obtain initial feature extraction parameters; the sample image is input to the feature extraction channel and the main noise reduction channel of the network channel in the sub-network to train the parameters of the main noise reduction channel and obtain initial noise reduction parameters. The sample image is input into the sub-network to adjust the initial feature extraction parameters and initial noise reduction parameters of the sub-network, thereby obtaining candidate feature extraction parameters and candidate noise reduction parameters; The sample image is input into the initial network model to adjust the candidate feature extraction parameters and candidate denoising parameters of each sub-network of the initial network model, so as to obtain the target feature extraction parameters and target denoising parameters. The target network model is generated based on the target feature extraction parameters and target denoising parameters.

2. The method according to claim 1, characterized in that, Different network channels within the same sub-network correspond to the same initial feature extraction parameters, different network channels within the same sub-network correspond to the same candidate feature extraction parameters, and different network channels within the same sub-network correspond to the same target feature extraction parameters; different network channels within the same sub-network correspond to the same initial denoising parameters, different network channels within the same sub-network correspond to the same candidate denoising parameters, and different network channels within the same sub-network correspond to the same target denoising parameters. Network channels in different sub-networks correspond to different or the same initial feature extraction parameters; network channels in different sub-networks correspond to different or the same candidate feature extraction parameters; network channels in different sub-networks correspond to different or the same target feature extraction parameters. Network channels in different sub-networks correspond to different or the same initial denoising parameters, network channels in different sub-networks correspond to different or the same candidate denoising parameters, and network channels in different sub-networks correspond to different or the same target denoising parameters.

3. The method according to claim 1, characterized in that, The step of inputting sample images into the feature extraction channels of the network channels in the sub-network to train the parameters of the feature extraction channels and obtain initial feature extraction parameters includes: inputting sample images into the feature extraction channels of each network channel in the sub-network to obtain a feature matrix; Based on the feature matrix and the sample image, determine the first initial loss value corresponding to the network channel; A first target loss value is determined based on the first initial loss value corresponding to each network channel. The parameters of the feature extraction channel are adjusted based on the first target loss value to obtain the initial feature extraction parameters.

4. The method according to claim 1, characterized in that, The step involves inputting the sample image into the feature extraction channel and the main noise reduction channel of the network channel in the sub-network to train the parameters of the main noise reduction channel and obtain initial noise reduction parameters, including: The sample image is input to the feature extraction channel of each network channel in the sub-network to obtain the feature matrix corresponding to the sample image, and the feature matrix is ​​input to the noise reduction main channel; The sample image is input into the main noise reduction channel of each network channel in the sub-network to obtain the original output features corresponding to the sample image and the feature matrix; Based on the original output features and the sample image, determine the second initial loss value corresponding to the network channel; The second target loss value is determined based on the second initial loss value corresponding to each network channel. The parameters of the main noise reduction channel are adjusted based on the second target loss value to obtain the initial noise reduction parameters.

5. The method according to claim 1, characterized in that, The step of inputting sample images into the sub-network to adjust the initial feature extraction parameters and initial denoising parameters of the sub-network to obtain candidate feature extraction parameters and candidate denoising parameters includes: The sample image is input into the feature extraction channel of each network channel in the sub-network to obtain the feature matrix; the third initial loss value corresponding to the network channel is determined based on the feature matrix. The sample image is input into the main noise reduction channel of each network channel in the sub-network to obtain the original output features; the fourth initial loss value corresponding to the network channel is determined based on the original output features. A third target loss value is determined based on the third initial loss value and the fourth initial loss value corresponding to each network channel. The initial feature extraction parameters and initial denoising parameters of the sub-network are then adjusted based on the third target loss value to obtain candidate feature extraction parameters and candidate denoising parameters.

6. The method according to claim 5, characterized in that, The process of determining the third initial loss value corresponding to the network channel based on the feature matrix includes: Based on the feature matrix, the first structural similarity index loss value and the first minimum absolute value deviation loss value corresponding to the network channel are determined, and the third initial loss value corresponding to the network channel is determined based on the first structural similarity index loss value and the first minimum absolute value deviation loss value. The process of determining the fourth initial loss value corresponding to the network channel based on the original output features includes: Based on the original output features, determine the second structural similarity index loss value and the second minimum absolute value deviation loss value corresponding to the network channel. Based on the second structural similarity index loss value and the second minimum absolute value deviation loss value, determine the fourth initial loss value corresponding to the network channel.

7. The method according to claim 1, characterized in that, The step of inputting the sample image into the initial network model to adjust the candidate feature extraction parameters and candidate denoising parameters of each sub-network of the initial network model to obtain the target feature extraction parameters and target denoising parameters includes: For each sub-network, the sample image is input to the feature extraction channel of each network channel in the sub-network to obtain the feature matrix, and the fifth initial loss value corresponding to the network channel is determined based on the feature matrix; the sample image is input to the denoising main channel of each network channel in the sub-network to obtain the original output features, and the sixth initial loss value corresponding to the network channel is determined based on the original output features; the fourth target loss value corresponding to the sub-network is determined based on the fifth initial loss value and the sixth initial loss value corresponding to each network channel. A fifth target loss value is determined based on the fourth target loss value corresponding to each sub-network, and the candidate feature extraction parameters and candidate denoising parameters of each sub-network of the initial network model are adjusted based on the fifth target loss value to obtain the target feature extraction parameters and target denoising parameters.

8. The method according to any one of claims 3-5 and 7, characterized in that, The feature extraction channel includes a residual module and an hourglass module. The process of inputting the sample image into the feature extraction channel of each network channel in the sub-network to obtain a feature matrix includes: The residual module performs residual processing on the sample image to obtain a residual matrix; The residual matrix is ​​processed by the hourglass module to obtain the feature matrix corresponding to the sample image. The hourglass module includes N convolutional layers, where N is a positive integer greater than 1. The process of processing the residual matrix by the hourglass module to obtain the feature matrix corresponding to the sample image includes: the hourglass processing is performed on the residual matrix by convolution and deconvolution based on the N convolutional layers to obtain the feature matrix corresponding to the sample image.

9. The method according to any one of claims 4-5 and 7, characterized in that, The denoising main channel includes an encoding module, a feature processing module, and a decoding module. The step of inputting the sample image into the denoising main channel of each network channel in the sub-network to obtain the original output features includes: The initial features are obtained by performing convolution processing on the sample image through the encoding module, and the initial features are then subjected to multiple residual processing to obtain the encoded features corresponding to the sample image. The feature processing module superimposes the encoded features and the feature matrix to obtain the feature encoding corresponding to the sample image; wherein, the feature matrix is ​​provided by the feature extraction channel; The target feature is obtained by performing multiple residual processing on the feature encoding through the decoding module, and the original output feature is obtained by performing deconvolution processing on the target feature through the decoding module.

10. An image noise reduction device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the steps of the method according to any one of claims 1-9.

Images