An image denoising method, device, system, edge device and storage medium

Abstract

The embodiment of the application provides a kind of image denoising method, device, system, edge device and storage medium, the method comprises: obtaining to be denoised image;With preset multi-head attention mechanism module, the first feature matrix of to-be-denosed image is extracted, wherein, preset multi-head attention mechanism module includes self-attention mechanism and channel attention mechanism;First feature matrix is up-sampled, and second feature matrix is obtained;Based on second feature matrix, the denoised image corresponding to to-be-denosed image is generated. The technical scheme provided in the embodiment of the application can improve the image denoising efficiency, reduce the consumption of computing resources, and then reduce the performance requirements of the device.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image denoising method, apparatus, system, edge device, and storage medium. Background Technology

[0002] With the rapid development of image acquisition devices such as mobile phones and cameras, these devices have brought great convenience to people's lives. However, during the image acquisition process, noise is inevitably introduced into the image, thus affecting the quality of the final image.

[0003] Currently, image denoising methods are based on convolutional neural networks. However, convolutional neural networks consume a lot of computational resources and have high requirements for device performance; the denoising efficiency of image denoising methods using convolutional neural networks is relatively poor. Summary of the Invention

[0004] The purpose of this application is to provide an image denoising method, apparatus, system, edge device, and storage medium to improve image denoising efficiency, reduce computational resource consumption, and thus lower the performance requirements of the device. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide an image denoising method, the method comprising:

[0006] Obtain the image to be denoised;

[0007] The first feature matrix of the image to be denoised is extracted using a preset multi-head attention mechanism module, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism;

[0008] The first feature matrix is ​​upsampled to obtain the second feature matrix;

[0009] Based on the second feature matrix, a denoised image corresponding to the image to be denoised is generated.

[0010] Optionally, the step of extracting the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module includes:

[0011] The pixel values ​​of the image to be denoised are normalized to obtain a first normalized image;

[0012] The first feature matrix of the first normalized image is extracted using a pre-defined multi-head attention mechanism module.

[0013] The step of generating a denoised image corresponding to the image to be denoised based on the second feature matrix includes:

[0014] Using the second feature matrix, a second normalized image is generated;

[0015] The second normalized image is subjected to inverse normalization to obtain the denoised image corresponding to the image to be denoised.

[0016] Optionally, the step of extracting the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module includes:

[0017] Using the image to be denoised as the first candidate feature matrix, the self-attention mechanism and the channel attention mechanism are used to extract features from the first candidate feature matrix to obtain the first intermediate feature matrix;

[0018] If the number of downsampling operations is less than the preset number, the first intermediate feature matrix is ​​downsampled to obtain the second intermediate feature matrix, and the second intermediate feature matrix is ​​updated to the first candidate feature matrix. The step of using the self-attention mechanism and the channel attention mechanism to extract features from the first candidate feature matrix to obtain the first intermediate feature matrix is ​​then re-executed.

[0019] If the number of downsampling operations is equal to the preset number, then the first intermediate feature matrix is ​​used as the first feature matrix of the image to be denoised.

[0020] The step of upsampling the first feature matrix to obtain the second feature matrix includes:

[0021] Using the first feature matrix as the second candidate feature matrix, the second candidate feature matrix is ​​upsampled to obtain the third intermediate feature matrix;

[0022] The third intermediate feature matrix and the second intermediate feature matrix with the same dimension as the third intermediate feature matrix are summed to obtain the fourth intermediate feature matrix.

[0023] If the number of upsampling operations is less than the preset number, the fourth intermediate feature matrix is ​​updated to the second candidate feature matrix, and the step of upsampling the second candidate feature matrix to obtain the third intermediate feature matrix is ​​re-executed.

[0024] If the number of upsampling operations is equal to the preset number, then the fourth intermediate feature matrix is ​​used as the second feature matrix.

[0025] Optionally, the preset multi-head attention mechanism module can be trained using the following steps:

[0026] Obtain a preset training set, which includes multiple sample image pairs, and each sample image pair includes a sample noisy image and a corresponding sample denoised image;

[0027] Using a pre-defined multi-head attention mechanism module, the first sample feature matrix of the sample noise image in each sample image pair is extracted;

[0028] Upsample the first sample feature matrix of each sample noisy image to obtain the second sample feature matrix of each sample noisy image;

[0029] Based on the second sample feature matrix of each sample noisy image, generate the predicted denoised image corresponding to each sample noisy image;

[0030] If the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the training of the preset multi-head attention mechanism module ends.

[0031] Optionally, the method further includes:

[0032] If it is determined that the preset multi-head attention mechanism module has not converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the parameters of the preset multi-head attention mechanism module are adjusted, and the step of extracting the first sample feature matrix of the sample noise image in each sample image pair using the preset multi-head attention mechanism module is re-executed.

[0033] Optionally, the step of extracting the first sample feature matrix of the sample noise image in each sample image pair using a preset multi-head attention mechanism module includes:

[0034] From the plurality of sample image pairs, select a preset number of sample image pairs;

[0035] Using a pre-defined multi-head attention mechanism module, the first sample feature matrix of the sample noise image is extracted from each selected sample image pair.

[0036] Optionally, the step of obtaining the preset training set includes:

[0037] Obtain original image pairs, which include the original noisy image and the corresponding original denoised image;

[0038] Multiple first image blocks are extracted from the original noisy image, and multiple second image blocks are extracted from the original denoised image corresponding to the original noisy image. The first image blocks and the second image blocks correspond one-to-one, and each pair of first image blocks does not completely overlap.

[0039] Multiple sample image pairs are formed by the multiple first image blocks and the multiple second image blocks, and the multiple sample image pairs are added to a preset training set. The first image block is a sample noise image, and the second image block corresponding to the first image block is a sample denoised image corresponding to the first image block.

[0040] Secondly, embodiments of this application provide an image denoising apparatus, the apparatus comprising:

[0041] The acquisition unit is used to acquire the image to be denoised.

[0042] An extraction unit is used to extract the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism;

[0043] An upsampling unit is used to upsample the first feature matrix to obtain a second feature matrix;

[0044] The generation unit is used to generate a denoised image corresponding to the image to be denoised based on the second feature matrix.

[0045] Optionally, the extraction unit is specifically used for:

[0046] The pixel values ​​of the image to be denoised are normalized to obtain a first normalized image; the first feature matrix of the first normalized image is extracted using a preset multi-head attention mechanism module.

[0047] The generation unit is specifically used for:

[0048] Using the second feature matrix, a second normalized image is generated; the second normalized image is then subjected to inverse normalization to obtain the denoised image corresponding to the image to be denoised.

[0049] Optionally, the extraction unit is specifically used for:

[0050] Using the image to be denoised as the first candidate feature matrix, the self-attention mechanism and the channel attention mechanism are used to extract features from the first candidate feature matrix to obtain the first intermediate feature matrix;

[0051] If the number of downsampling operations is less than the preset number, the first intermediate feature matrix is ​​downsampled to obtain the second intermediate feature matrix, and the second intermediate feature matrix is ​​updated to the first candidate feature matrix. The step of using the self-attention mechanism and the channel attention mechanism to extract features from the first candidate feature matrix to obtain the first intermediate feature matrix is ​​then re-executed.

[0052] If the number of downsampling operations is equal to the preset number, then the first intermediate feature matrix is ​​used as the first feature matrix of the image to be denoised.

[0053] The upsampling unit is specifically used for:

[0054] Using the first feature matrix as the second candidate feature matrix, the second candidate feature matrix is ​​upsampled to obtain the third intermediate feature matrix;

[0055] The third intermediate feature matrix and the second intermediate feature matrix with the same dimension as the third intermediate feature matrix are summed to obtain the fourth intermediate feature matrix.

[0056] If the number of upsampling operations is less than the preset number, the fourth intermediate feature matrix is ​​updated to the second candidate feature matrix, and the step of upsampling the second candidate feature matrix to obtain the third intermediate feature matrix is ​​re-executed.

[0057] If the number of upsampling operations is equal to the preset number, then the fourth intermediate feature matrix is ​​used as the second feature matrix.

[0058] Optionally, the device further includes: a training unit, used to obtain the preset multi-head attention mechanism module through training, specifically including:

[0059] A sub-unit is used to acquire a preset training set, which includes multiple sample image pairs, and each sample image pair includes a sample noisy image and a corresponding sample denoised image.

[0060] Extraction subunits are used to extract the first sample feature matrix of the sample noise image in each sample image pair using a preset multi-head attention mechanism module;

[0061] The upsampling subunit is used to upsample the first sample feature matrix of each sample noise image to obtain the second sample feature matrix of each sample noise image.

[0062] A generation subunit is used to generate a predicted denoised image for each sample noisy image based on the second sample feature matrix of each sample noisy image.

[0063] The training subunit is used to terminate the training of the preset multi-head attention mechanism module if it is determined that the preset multi-head attention mechanism module has converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image.

[0064] Optionally, the training subunit is further configured to:

[0065] If it is determined that the preset multi-head attention mechanism module has not converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the parameters of the preset multi-head attention mechanism module are adjusted, and the step of extracting the first sample feature matrix of the sample noise image in each sample image pair using the preset multi-head attention mechanism module is re-executed.

[0066] Optionally, the extraction subunit is specifically used for:

[0067] From the plurality of sample image pairs, select a preset number of sample image pairs;

[0068] Using a pre-defined multi-head attention mechanism module, the first sample feature matrix of the sample noise image is extracted from each selected sample image pair.

[0069] Optionally, the acquisition subunit is specifically used for:

[0070] Obtain original image pairs, which include the original noisy image and the corresponding original denoised image;

[0071] Multiple first image blocks are extracted from the original noisy image, and multiple second image blocks are extracted from the original denoised image corresponding to the original noisy image. The first image blocks and the second image blocks correspond one-to-one, and each pair of first image blocks does not completely overlap.

[0072] Multiple sample image pairs are formed by the multiple first image blocks and the multiple second image blocks, and the multiple sample image pairs are added to a preset training set. The first image block is a sample noise image, and the second image block corresponding to the first image block is a sample denoised image corresponding to the first image block.

[0073] Thirdly, embodiments of this application provide an edge device, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor, the processor being prompted by the machine-executable instructions to implement any of the above-described image denoising method steps.

[0074] Fourthly, embodiments of this application provide an image denoising system, the system including a server;

[0075] The server is configured to acquire a preset training set, which includes multiple sample image pairs, each sample image pair including a sample noisy image and a corresponding sample denoised image; using a preset multi-head attention mechanism module, extracting a first sample feature matrix of the sample noisy image in each sample image pair; upsampling the first sample feature matrix of each sample noisy image to obtain a second sample feature matrix of each sample noisy image; generating a predicted denoised image corresponding to each sample noisy image based on the second sample feature matrix of each sample noisy image; and if the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noisy image, then the training of the preset multi-head attention mechanism module is terminated.

[0076] Optionally, the system may also include edge devices;

[0077] The edge device is used to acquire the image to be denoised; extract the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module trained on the server, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism; upsample the first feature matrix to obtain a second feature matrix; and generate a denoised image corresponding to the image to be denoised based on the second feature matrix.

[0078] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the image denoising method steps described above.

[0079] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the image denoising method steps described above.

[0080] Beneficial effects of the embodiments in this application:

[0081] The technical solution provided in this application utilizes a preset multi-head attention mechanism module to denoise images. Since the preset multi-head attention mechanism module is smaller than a convolutional neural network model, using the technical solution provided in this application for image denoising reduces the consumption of computational resources and lowers the performance requirements of the device.

[0082] Furthermore, in the technical solution provided in this application embodiment, the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism. The self-attention mechanism can extract global semantic information from the image, while the channel attention mechanism extracts local detail texture information. Since both global semantic information and local detail texture information of the image can be obtained, the number of image pairs required in the training set of the preset multi-head attention mechanism module is reduced. Therefore, it is not necessary to synthesize a large number of noisy images to assist in the training of the preset multi-head attention mechanism module, thereby improving image denoising efficiency.

[0083] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0085] Figure 1This is a schematic diagram of the first process of the image denoising method provided in the embodiments of this application;

[0086] Figure 2 This is a schematic diagram of a second process for the image denoising method provided in the embodiments of this application;

[0087] Figure 3 This is a schematic diagram of a third process for the image denoising method provided in the embodiments of this application;

[0088] Figure 4 This is a schematic diagram of the fourth process of the image denoising method provided in the embodiments of this application;

[0089] Figure 5 A schematic diagram of the first flowchart of the training method for the preset multi-head attention mechanism module provided in the embodiments of this application;

[0090] Figure 6 A second flowchart illustrating the training method for the preset multi-head attention mechanism module provided in the embodiments of this application;

[0091] Figure 7 A schematic diagram of the third process for training a preset multi-head attention mechanism module provided in an embodiment of this application;

[0092] Figure 8 This is a schematic diagram of the structure of an image denoising device provided in an embodiment of this application;

[0093] Figure 9 A schematic diagram of the structure of a server provided in an embodiment of this application;

[0094] Figure 10 This is a schematic diagram of the structure of an edge device provided in an embodiment of this application. Detailed Implementation

[0095] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0096] In related technologies, image denoising methods are based on convolutional neural networks. However, convolutional neural networks consume significant computational resources and require high-performance devices.

[0097] Furthermore, convolutional neural networks (CNNs) cannot capture global semantic information within images. To achieve better denoising results, a large number of noisy images need to be synthesized and then combined with real denoised images to form a training set to assist in the training of the CNN, enabling the trained CNN to perform accurate convolutional operations. This results in the poor denoising efficiency of the aforementioned image denoising methods.

[0098] To address the aforementioned issues, this application provides an image denoising method that can be applied to electronic devices with image processing capabilities, such as servers, personal computers, mobile phones, tablets, or AI-ISP cameras.

[0099] This image denoising method utilizes a pre-defined multi-head attention mechanism module to denoise the image. Since the pre-defined multi-head attention mechanism module is smaller than a convolutional neural network model, using the technical solution provided in this application for image denoising reduces the consumption of computational resources and lowers the performance requirements of the device.

[0100] Furthermore, in this image denoising method, the pre-defined multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism. The self-attention mechanism extracts global semantic information from the image, while the channel attention mechanism extracts local detail and texture information. Since both global semantic information and local detail and texture information can be obtained, the required number of image pairs in the training set of the pre-defined multi-head attention mechanism module is reduced. Therefore, it is not necessary to synthesize a large number of noisy images to assist in the training of the pre-defined multi-head attention mechanism module, thereby improving the image denoising efficiency.

[0101] The image denoising method provided in this application will be described in detail below through specific embodiments.

[0102] See Figure 1 , Figure 1 This is a schematic flowchart of a first embodiment of the image denoising method provided in this application. For ease of understanding, the following description uses a server as the execution entity. The image denoising method includes the following steps.

[0103] Step S101: Obtain the image to be denoised.

[0104] In this embodiment, the user can directly input the image to be denoised into the server. The server then obtains the image to be denoised.

[0105] In this embodiment, the server can also be connected to a camera. The camera sends real-time captured images to the server. The server receives the images sent by the camera and uses the received images as the images to be denoised.

[0106] The server can also connect to a preset database. This preset database is used to store images. The server can retrieve images from the preset database and use the retrieved images as the images to be denoised.

[0107] In this embodiment of the application, the server may also obtain the image to be denoised through other means, and there is no limitation on this.

[0108] Step S102: Using a preset multi-head attention mechanism module, extract the first feature matrix of the image to be denoised. The preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism.

[0109] In this embodiment, the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism. The self-attention mechanism can extract global semantic information from the image, while the channel attention mechanism, which includes convolutional layers, can extract local detail texture information from the image.

[0110] After the server obtains the image to be denoised, it can input the image to be denoised into a preset multi-head attention mechanism module. The preset multi-head attention mechanism module performs understanding and convolution processing on the image to be denoised to obtain the feature matrix of the image to be denoised, namely the first feature matrix.

[0111] Since the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism, the first feature matrix includes global semantic information of the image to be denoised, as well as local detail texture information of the image to be denoised.

[0112] Step S103: Upsample the first feature matrix to obtain the second feature matrix.

[0113] In this embodiment, the feature matrix output by the preset multi-head attention mechanism module after processing the denoised image is equivalent to downsampling the original feature matrix of the denoised image, that is, the number of pixels corresponding to the first feature matrix is ​​less than the number of pixels included in the denoised image.

[0114] After obtaining the first feature matrix, the server upsamples it to obtain the second feature matrix. At this point, the number of pixels corresponding to the second feature matrix is ​​greater than or equal to the number of pixels in the image to be denoised. This ensures that the subsequently obtained denoised image has the same size as the image to be denoised.

[0115] Step S104: Based on the second feature matrix, generate the denoised image corresponding to the image to be denoised.

[0116] In this embodiment of the application, after obtaining the second feature matrix, the server can perform decoding and rendering of the second feature matrix to obtain the denoised image corresponding to the image to be denoised.

[0117] The technical solution provided in this application utilizes a preset multi-head attention mechanism module to denoise images. Since the preset multi-head attention mechanism module is smaller than a convolutional neural network model, using the technical solution provided in this application for image denoising reduces the consumption of computational resources and lowers the performance requirements of the device.

[0118] Furthermore, in the technical solution provided in this application embodiment, the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism. The self-attention mechanism can extract global semantic information from the image, while the channel attention mechanism extracts local detail texture information. Since both global semantic information and local detail texture information of the image can be obtained, the number of image pairs required in the training set of the preset multi-head attention mechanism module is reduced. Therefore, it is not necessary to synthesize a large number of noisy images to assist in the training of the preset multi-head attention mechanism module, thereby improving image denoising efficiency.

[0119] In one embodiment of this application, to further improve the efficiency of image denoising, this embodiment also provides an image denoising method, such as... Figure 2 As shown, the method may include the following steps:

[0120] Step S201: Obtain the image to be denoised. Step S201 is the same as step S101.

[0121] Step S202: Normalize the pixel values ​​of the image to be denoised to obtain the first normalized image.

[0122] In this embodiment, after obtaining the image to be denoised, the server normalizes the pixel value of each pixel in the image to a range of 0 to 1. After normalizing the pixel values ​​of all pixels in the image, the server obtains the normalized image to be denoised, i.e., the first normalized image.

[0123] For example, the pixel value ranges from 0 to 255. For each pixel in the image to be denoised, the server can normalize the pixel value according to the following formula.

[0124] T1' = T1 / 255

[0125] Where T1' is the pixel value of the normalized pixel, that is, the pixel value of the pixel in the first normalized image; T1 is the pixel value of the unnormalized pixel, that is, the pixel value of the pixel in the image to be denoised; and 255 is the maximum pixel value.

[0126] Step S203: Using a preset multi-head attention mechanism module, extract the first feature matrix of the first normalized image, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism.

[0127] In this embodiment of the application, after the server obtains the first normalized image, it can input the first normalized image into a preset multi-head attention mechanism module. The preset multi-head attention mechanism module performs understanding and convolution processing on the first normalized image to obtain the feature matrix of the first normalized image, i.e., the first feature matrix.

[0128] Since the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism, the first feature matrix includes the global semantic information of the first normalized image and the local detail texture information of the first normalized image.

[0129] Steps S202-S203 are a refinement of step S102.

[0130] Step S204: Upsample the first feature matrix to obtain the second feature matrix. Step S204 is the same as step S103.

[0131] Step S205: Use the second feature matrix to generate the second normalized image.

[0132] In this embodiment of the application, after obtaining the second feature matrix, the server can perform decoding and rendering of the second feature matrix to obtain the second normalized image corresponding to the image to be denoised.

[0133] Step S206: Perform inverse normalization on the second normalized image to obtain the denoised image corresponding to the image to be denoised.

[0134] In this embodiment of the application, after the server obtains the second normalized image, it performs inverse normalization processing on the second normalized image to obtain the denormalized image, that is, the denoised image corresponding to the image to be denoised.

[0135] For example, the pixel value ranges from 0 to 255. For each pixel in the second normalized image, the server can perform inverse normalization on the pixel value according to the following formula.

[0136] T2'=T2*255

[0137] Where T2' is the pixel value of the denormalized pixel, that is, the pixel value of the pixel in the denoised image; T2 is the pixel value of the pixel in the second normalized image; and 255 is the maximum pixel value.

[0138] In this embodiment, the second normalized image is the image determined by the second feature matrix obtained based on the first normalized image. Therefore, the range of pixel values ​​for each pixel in the second normalized image is the same as the range of prime values ​​for each pixel in the first normalized image, i.e., the range of pixel values ​​for each pixel in the second normalized image is 0 to 1. However, the range of pixel values ​​for each pixel in a normal denoised image should be 0 to 255. Therefore, by performing inverse normalization on the second normalized image, the server can obtain a denoised image with pixel values ​​in the range of 0 to 255. This denoised image is the denoised image required by the user.

[0139] Steps S205-S206 are a refinement of step S104.

[0140] In the technical solution provided in this application embodiment, the pixel values ​​of the image processed by the preset multi-head attention mechanism are normalized, which greatly reduces the range of pixel values ​​of the image processed by the preset multi-head attention mechanism, thereby improving the image processing efficiency of the preset multi-head attention mechanism and further improving the efficiency of image denoising.

[0141] In one embodiment of this application, to improve the accuracy of image denoising, this embodiment also provides an image denoising method, such as... Figure 3 As shown, the method may include the following steps:

[0142] Step S301: Obtain the image to be denoised. Step S301 is the same as step S101.

[0143] Step S302: Use the image to be denoised as the first candidate feature matrix.

[0144] Step S303: Using self-attention mechanism and channel attention mechanism, feature extraction is performed on the first candidate feature matrix to obtain the first intermediate feature matrix.

[0145] The pre-defined multi-head attention mechanism module includes self-attention and channel attention mechanisms. After acquiring the image to be denoised, the server uses the image to be denoised as the first candidate feature matrix. The server uses the self-attention and channel attention mechanisms to understand the first candidate feature matrix and obtain the feature matrix, i.e., the first intermediate feature matrix. This first intermediate feature matrix includes the global semantic information and local detail texture information of the first candidate feature matrix.

[0146] Step S304: Check if the number of downsampling operations is less than the preset number. If yes, proceed to step S305; if no, i.e., the number of downsampling operations is equal to the preset number, proceed to step S306.

[0147] Step S305: Downsample the first intermediate feature matrix to obtain the second intermediate feature matrix, update the second intermediate feature matrix to the first candidate feature matrix, and re-execute step S303.

[0148] Step S306: Use the first intermediate feature matrix as the first feature matrix of the image to be denoised.

[0149] In this embodiment, the preset number of times can be set according to actual needs. For example, the preset number of times can be 2 or 3. The following explanation uses a preset number of times of 2 as an example.

[0150] After obtaining the first intermediate feature matrix, since the number of downsampling operations is 0 (0 < 2), the server downsamples the first intermediate feature matrix to obtain the second intermediate feature matrix. The server then updates the second intermediate feature matrix to the first candidate feature matrix.

[0151] Then, the server reuses the self-attention and channel attention mechanisms to extract features from the first candidate feature matrix, obtaining the first intermediate feature matrix. At this point, the downsampling is performed once, 1 < 2, therefore, the server downsamples the first intermediate feature matrix to obtain the second intermediate feature matrix.

[0152] Subsequently, the server reuses the self-attention and channel attention mechanisms to extract features from the first candidate feature matrix, obtaining the first intermediate feature matrix. At this point, the downsampling is performed twice, 2 = 2, therefore, the server uses the first intermediate feature matrix as the first feature matrix of the image to be denoised.

[0153] Steps S302-S306 are a refinement of step S102.

[0154] Step S307: Use the first feature matrix as the second candidate feature matrix.

[0155] Step S308: Upsample the second candidate feature matrix to obtain the third intermediate feature matrix.

[0156] In this embodiment, the step size for upsampling and downsampling is the same.

[0157] Step S309: Summate the third intermediate feature matrix and the second intermediate feature matrix with the same dimension as the third intermediate feature matrix to obtain the fourth intermediate feature matrix.

[0158] In this embodiment, step S309 can be understood as the server upsampling the first feature matrix based on a skip-step connection strategy to obtain a fourth intermediate feature matrix. The skip-step connection strategy involves summing feature matrices of the same dimension.

[0159] Step S310: Check if the number of upsampling operations is less than the preset number. If yes, proceed to step S311; if no, i.e., the number of upsampling operations is equal to the preset number, proceed to step S312.

[0160] Step S311: Update the fourth intermediate feature matrix to the second candidate feature matrix, and re-execute step S308.

[0161] Step S312: Use the fourth intermediate feature matrix as the second feature matrix.

[0162] Let's take a preset number of upsampling operations (2) as an example. After obtaining the second candidate feature matrix, the server upsamples the second candidate feature matrix to obtain the third intermediate feature matrix. The third intermediate feature matrix is ​​then summed with the second intermediate feature matrix, which has the same dimension as the third intermediate feature matrix, to obtain the fourth intermediate feature matrix. At this point, the number of upsampling operations is 1, and 1 < 2. Therefore, the server updates the fourth intermediate feature matrix to the second candidate feature matrix.

[0163] Next, the server upsamples the second candidate feature matrix to obtain a third intermediate feature matrix. This third intermediate feature matrix is ​​then summed with the second intermediate feature matrix, which has the same dimension as the third intermediate feature matrix, to obtain a fourth intermediate feature matrix. At this point, the upsampling operation has been performed twice, and since 2 = 2, the server uses the fourth intermediate feature matrix as the second feature matrix.

[0164] For example, consider an image to be denoised with a resolution of 256*256 pixels. The preset number of iterations is 2, and the step size for both upsampling and downsampling is 2.

[0165] The server uses self-attention and channel attention mechanisms to understand the image x to be denoised, obtaining a 256*256 feature matrix 1. The server downsamples feature matrix 1, obtaining a 128*128 feature matrix 2. The server uses self-attention and channel attention mechanisms to understand feature matrix 2, obtaining a 128*128 feature matrix 3. The server downsamples feature matrix 3, obtaining a 64*64 feature matrix 4. The server uses self-attention and channel attention mechanisms to understand feature matrix 4, obtaining a 64*64 feature matrix 5 (i.e., the first feature matrix).

[0166] Based on the skip connection strategy, feature matrix 1, and feature matrix 3, the server upsamples feature matrix 5 to obtain feature matrix 6 (i.e., the second feature matrix). Specifically: the server upsamples feature matrix 5 to obtain a 128*128 feature matrix 51; it sums the 128*128 feature matrix 51 with the 128*128 feature matrix 3 to obtain a 128*128 feature matrix 52; it upsamples the 128*128 feature matrix 52 to obtain a 256*256 feature matrix 53; and it sums the 256*256 feature matrix 53 with the 256*256 feature matrix 1 to obtain feature matrix 54 (i.e., the second feature matrix).

[0167] Steps S307-S312 are a refinement of step S103.

[0168] Step S313: Based on the second feature matrix, generate the denoised image corresponding to the image to be denoised. Step S313 is the same as step S104.

[0169] In the technical solution provided in this application embodiment, the server uses a preset multi-head attention mechanism module to understand the image to be denoised and performs multiple downsampling processes to obtain a first feature matrix. Then, using a skip-connection strategy, the first feature matrix is ​​upsampled to obtain a second feature matrix. In this way, the denoised image generated using the second feature matrix can effectively avoid the loss of global semantic information and / or local detail texture information of the image to be denoised due to the server's downsampling process, thereby improving the accuracy of image denoising and obtaining an accurate denoised image.

[0170] In this embodiment of the application, in order to obtain better image denoising effect, the above-mentioned Figure 2 and Figure 3 The embodiments can be combined, and there is no limitation thereto. For example Figure 4 The diagram illustrates the process of an image denoising method. In this method, the server acquires the image to be denoised; the pixel values ​​of the image to be denoised are normalized to obtain a first normalized image; a pre-defined multi-head attention mechanism module is used to perform feature extraction and downsampling on the first normalized image, and a skip connection strategy is combined to perform upsampling to obtain the denoised image, i.e., the second normalized image; the second normalized image is then inversely normalized to obtain the denoised image corresponding to the image to be denoised.

[0171] like Figure 4 The descriptions in some parts are relatively simple; please refer to the above for details. Figure 1-3 Partial description.

[0172] Based on the above image denoising methods, this application provides a training method for a preset multi-head attention mechanism module, such as... Figure 5 As shown, the method may include the following steps.

[0173] Step S501: Obtain a preset training set. The preset training set includes multiple sample image pairs, and each sample image pair includes a sample noisy image and a corresponding sample denoised image.

[0174] In this embodiment, the sample noisy image is an image containing noise, and the sample denoised image is an image without noise. The number of sample image pairs in the preset training set can be set according to actual needs. For example, if the accuracy requirement of the preset multi-head attention mechanism module is high, the number of sample image pairs can be set to a larger value; if the server performance is poor, the number of sample image pairs can be set to a smaller value.

[0175] In this embodiment, the preset training set can be a dataset from smartphone image denoising research (SIDD). The preset training set can also be a dataset obtained by processing the SIDD; there is no limitation on this.

[0176] Step S502: Using a preset multi-head attention mechanism module, extract the first sample feature matrix of the sample noise image in each sample image pair.

[0177] In this embodiment of the application, after obtaining the preset training set, for each sample image pair in the preset training set, the server can use a preset multi-head attention mechanism module to extract the first sample feature matrix of the sample noise image in the sample image pair. The extraction of the first sample feature matrix can be found in the description of step S102 above, and will not be repeated here.

[0178] In one embodiment of this application, the server can input all sample noise images from the preset training set into the preset multi-head attention mechanism module at once, and then execute steps S503-S505 based on all sample noise images from the preset training set to train the preset multi-head attention mechanism module together based on all sample noise images from the preset training set.

[0179] In one embodiment of this application, the server selects a preset number of sample image pairs from multiple sample image pairs included in a preset training set; and uses a preset multi-head attention mechanism module to extract the first sample feature matrix of the sample noise image in each selected sample image pair. The preset number can be set according to the maximum number of channels supported by the preset multi-head attention mechanism module.

[0180] For example, if the maximum number of channels supported by the preset multi-head attention mechanism module is 48, and each image occupies 3 channels, then the preset number can be set to 48 / 3 = 16. As another example, if the maximum number of channels supported by the preset multi-head attention mechanism module is 60, and each image occupies 3 channels, then the preset number can be set to 60 / 3 = 20.

[0181] In this embodiment, the server processes multiple sample images in batches, and then trains the preset multi-head attention mechanism module according to the batches. For example, as mentioned above, 16 sample noisy images are selected each time, and the preset multi-head attention mechanism module is trained using these 16 sample noisy images. This effectively reduces the number of sample noisy images that need to be trained each time the preset multi-head attention mechanism module is trained, and effectively improves the training efficiency of the preset multi-head attention mechanism module.

[0182] Step S503: Upsample the first sample feature matrix of each sample noise image to obtain the second sample feature matrix of each sample noise image.

[0183] In this embodiment of the application, for each sample noise image, the server upsamples the first sample feature matrix of the sample noise image to obtain the second sample feature matrix of the sample noise image. The upsampling process of the first sample feature matrix can be found in the description of step S103 above, and will not be repeated here.

[0184] Step S504: Based on the second sample feature matrix of each sample noise image, generate the predicted denoised image corresponding to each sample noise image.

[0185] In this embodiment of the application, for each sample noise image, the server performs decoding and rendering processes on the second sample feature matrix of the sample noise image to obtain the predicted denoised image corresponding to the sample noise image. The generation of the predicted denoised image can be found in the description of step S104 above, and will not be repeated here.

[0186] Step S505: If the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the training of the preset multi-head attention mechanism module ends.

[0187] In this embodiment, for each sample noisy image, the server compares the predicted denoised image and the sample denoised image corresponding to the sample noisy image to obtain the comparison result corresponding to the sample noisy image. Based on the comparison results corresponding to multiple sample noisy images, the server determines whether the preset multi-head attention mechanism module has converged. If it is determined that the preset multi-head attention mechanism module has converged, the server terminates the training of the preset multi-head attention mechanism module.

[0188] In one embodiment of this application, for each sample noisy image, the server determines the similarity between the predicted denoised image and the sample denoised image corresponding to the sample noisy image. The similarity corresponding to the sample noisy image is the comparison result corresponding to the sample noisy image. If the sum of the similarities corresponding to multiple sample noisy images is less than a first preset threshold, it is determined that the preset multi-head attention mechanism module has converged; otherwise, it is determined that the preset multi-head attention mechanism module has not converged.

[0189] In another embodiment of this application, for each sample noise image, the server determines the similarity between the predicted denoised image and the sample denoised image corresponding to the sample noise image. The similarity corresponding to the sample noise image is the comparison result corresponding to the sample noise image. If the average of the similarities corresponding to multiple sample noise images is less than a second preset threshold, it is determined that the preset multi-head attention mechanism module has converged; otherwise, it is determined that the preset multi-head attention mechanism module has not converged.

[0190] In this embodiment, the server may also use other methods to determine whether the preset multi-head attention mechanism module has converged, and there are no limitations on this.

[0191] Based on the above image denoising method, this application embodiment also provides a training method for a preset multi-head attention mechanism module, such as... Figure 6 As shown, the method may include the following steps.

[0192] Step S601: Obtain a preset training set. The preset training set includes multiple sample image pairs, and each sample image pair includes a sample noisy image and a corresponding sample denoised image.

[0193] Step S602: Using a preset multi-head attention mechanism module, extract the first sample feature matrix of the sample noise image in each sample image pair.

[0194] Step S603: Upsample the first sample feature matrix of each sample noise image to obtain the second sample feature matrix of each sample noise image.

[0195] Step S604: Based on the second sample feature matrix of each sample noise image, generate the predicted denoised image corresponding to each sample noise image.

[0196] Step S605: If the preset multi-head attention mechanism module converges based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the training of the preset multi-head attention mechanism module ends.

[0197] Steps S601-S605 are the same as steps S501-S505 above.

[0198] Step S606: If it is determined that the preset multi-head attention mechanism module has not converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then adjust the parameters of the preset multi-head attention mechanism module and re-execute step S602.

[0199] In this embodiment of the application, if it is determined that the preset multi-head attention mechanism module has not converged, the server can use the gradient descent algorithm and the backpropagation algorithm to adjust the parameters of the preset multi-head attention mechanism module, return to the execution step S602, and perform iterative training on the preset multi-head attention mechanism module.

[0200] In the technical solution provided in this application embodiment, the server can iteratively train the preset multi-head attention mechanism module, enabling the preset multi-head attention mechanism module to fully learn the change rules of noisy images and denoised images, thereby effectively improving the accuracy of image denoising by the preset multi-head attention mechanism module.

[0201] In one embodiment of the application, to further improve the training efficiency of the preset multi-head attention mechanism module and the accuracy of image denoising by the preset multi-head attention mechanism module, this application embodiment also provides a training method for the preset multi-head attention mechanism module, such as... Figure 7 As shown, the method may include the following steps:

[0202] Step S701: Obtain the original image pair, which includes the original noisy image and the corresponding original denoised image.

[0203] In this embodiment of the application, the original image pairs may come from SIDD or other image datasets, and there is no limitation on this.

[0204] When training the pre-defined multi-head attention mechanism module, the server acquires original image pairs. The number of original image pairs can be one or more, without limitation. For ease of understanding, the following explanation uses a single original image pair as an example.

[0205] Step S702: Extract multiple first image blocks from the original noisy image and extract multiple second image blocks from the original denoised image corresponding to the original noisy image. The first image blocks and the second image blocks correspond one-to-one, and each pair of first image blocks does not completely overlap.

[0206] After obtaining the original image pair, the server extracts multiple first image blocks from the original noisy image included in the original image pair; and extracts multiple second image blocks from the original denoised image included in the original image pair. The first image blocks and the second image blocks correspond one-to-one, that is, the position coordinates of the first image block in the original noisy image are the same as the position coordinates of the second image block in the original denoised image, and the size of the first image block and the size of the second image block are the same.

[0207] The size of the first image block and the size of the second image block can be set according to actual needs. For example, the size of the first image block and the size of the second image block can be 256*256 or 128*128, etc.

[0208] Step S703: Multiple sample image pairs are formed by multiple first image blocks and multiple second image blocks, and the multiple sample image pairs are added to a preset training set. The first image block is a sample noise image, and the second image block corresponding to the first image block is a sample denoised image corresponding to the first image block.

[0209] Steps S701-S703 are a refinement of step S501 above.

[0210] Step S704: Using a preset multi-head attention mechanism module, extract the first sample feature matrix of the sample noise image in each sample image pair.

[0211] Step S705: Upsample the first sample feature matrix of each sample noise image to obtain the second sample feature matrix of each sample noise image.

[0212] Step S706: Based on the second sample feature matrix of each sample noise image, generate the predicted denoised image corresponding to each sample noise image.

[0213] Step S707: If the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the training of the preset multi-head attention mechanism module ends.

[0214] Steps S704-S707 are the same as steps S502-S505 above.

[0215] In the technical solution provided in this application embodiment, the server can extract multiple sample image pairs from a single original image pair, which can significantly increase the number of image pairs included in the preset training set. Expanding the preset training set can effectively improve the accuracy of image denoising performed by the trained preset multi-head attention mechanism module.

[0216] Furthermore, in the technical solution provided in this application embodiment, the server extracts multiple sample image pairs from an original image pair. In this case, the size of the sample image pairs in the preset training set is much smaller than the size of the original image pairs. The server uses the smaller sample image pairs to train the preset multi-head attention mechanism module, which can effectively reduce the computational load of the preset multi-head attention mechanism module and thus improve the training efficiency of the preset multi-head attention mechanism module.

[0217] In this embodiment, the image denoising method and the training method of the preset multi-head attention mechanism module can be implemented on different devices or on the same device, without limitation. For example, both the image denoising method and the training method of the preset multi-head attention mechanism module can be implemented on a server or on an edge device. As another example, the image denoising method can be implemented on an edge device, and the training method of the preset multi-head attention mechanism module can be implemented on a server.

[0218] In this embodiment, the aforementioned preset multi-head attention mechanism module can be integrated into a general-purpose module. Depending on actual needs, the preset multi-head attention mechanism module can be installed on any server or edge device, achieving plug-and-play functionality and improving the flexibility of image denoising.

[0219] Corresponding to the image denoising method described above, this application also provides an image denoising apparatus, such as... Figure 8 As shown, the device includes:

[0220] Acquisition unit 801 is used to acquire the image to be denoised;

[0221] Extraction unit 802 is used to extract the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism;

[0222] The upsampling unit 803 is used to upsample the first feature matrix to obtain the second feature matrix;

[0223] The generation unit 804 is used to generate a denoised image corresponding to the image to be denoised based on the second feature matrix.

[0224] Optionally, extraction unit 802 can be used for:

[0225] The pixel values ​​of the image to be denoised are normalized to obtain the first normalized image; the first feature matrix of the first normalized image is extracted using a preset multi-head attention mechanism module.

[0226] The generation unit 804 can be used to generate a second normalized image using the second feature matrix; and to perform inverse normalization processing on the second normalized image to obtain the denoised image corresponding to the image to be denoised.

[0227] Optionally, extraction unit 802 can be used for:

[0228] Using the image to be denoised as the first candidate feature matrix, features are extracted from the first candidate feature matrix using self-attention and channel attention mechanisms to obtain a first intermediate feature matrix. If the number of downsampling operations is less than a preset number, the first intermediate feature matrix is ​​downsampled to obtain a second intermediate feature matrix, and the second intermediate feature matrix is ​​updated as the first candidate feature matrix. The steps of extracting features from the first candidate feature matrix using self-attention and channel attention mechanisms to obtain the first intermediate feature matrix are repeated. If the number of downsampling operations is equal to the preset number, the first intermediate feature matrix is ​​used as the first feature matrix of the image to be denoised.

[0229] The upsampling unit 803 can be specifically used for:

[0230] Using the first feature matrix as the second candidate feature matrix, the second candidate feature matrix is ​​upsampled to obtain the third intermediate feature matrix; the third intermediate feature matrix and the second intermediate feature matrix with the same dimension as the third intermediate feature matrix are summed to obtain the fourth intermediate feature matrix; if the number of upsampling operations is less than the preset number, the fourth intermediate feature matrix is ​​updated to the second candidate feature matrix, and the step of upsampling the second candidate feature matrix to obtain the third intermediate feature matrix is ​​repeated; if the number of upsampling operations is equal to the preset number, the fourth intermediate feature matrix is ​​used as the second feature matrix.

[0231] Optionally, the image denoising device described above may further include: a training unit, which trains a preset multi-head attention mechanism module using the following steps, specifically including:

[0232] The acquisition sub-unit is used to acquire a preset training set, which includes multiple sample image pairs. Each sample image pair includes a sample noisy image and a corresponding sample denoised image.

[0233] Extraction subunits are used to extract the first sample feature matrix of the sample noise image in each sample image pair using a preset multi-head attention mechanism module;

[0234] The upsampling subunit is used to upsample the first sample feature matrix of each sample noise image to obtain the second sample feature matrix of each sample noise image.

[0235] A generation subunit is used to generate a predicted denoised image for each sample noisy image based on the second sample feature matrix of each sample noisy image.

[0236] The training subunit is used to terminate the training of the preset multi-head attention mechanism module if the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image.

[0237] Optionally, the training subunit can also be used for:

[0238] If, based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, it is determined that the preset multi-head attention mechanism module has not converged, then the parameters of the preset multi-head attention mechanism module are adjusted, and the step of extracting the first sample feature matrix of the sample noise image in each sample image pair using the preset multi-head attention mechanism module is re-executed.

[0239] Optionally, sub-units can be extracted, specifically for:

[0240] From multiple sample image pairs, a preset number of sample image pairs are selected; using a preset multi-head attention mechanism module, the first sample feature matrix of the sample noise image in each selected sample image pair is extracted.

[0241] Optionally, obtain sub-units, which can be used specifically for:

[0242] Obtain original image pairs, which include original noisy images and corresponding original denoised images; extract multiple first image blocks from the original noisy images and extract multiple second image blocks from the original denoised images corresponding to the original noisy images, with the first image blocks and second image blocks corresponding one-to-one and each pair of first image blocks not completely overlapping.

[0243] Multiple sample image pairs are formed by multiple first image blocks and multiple second image blocks, and multiple sample image pairs are added to a preset training set. The first image block is a sample noise image, and the second image block corresponding to the first image block is a sample denoised image corresponding to the first image block.

[0244] The technical solution provided in this application utilizes a preset multi-head attention mechanism module to denoise images. Since the preset multi-head attention mechanism module is smaller than a convolutional neural network model, using the technical solution provided in this application for image denoising reduces the consumption of computational resources and lowers the performance requirements of the device.

[0245] Furthermore, in the technical solution provided in this application embodiment, the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism. The self-attention mechanism can extract global semantic information from the image, while the channel attention mechanism extracts local detail texture information. Since both global semantic information and local detail texture information of the image can be obtained, the number of image pairs required in the training set of the preset multi-head attention mechanism module is reduced. Therefore, it is not necessary to synthesize a large number of noisy images to assist in the training of the preset multi-head attention mechanism module, thereby improving image denoising efficiency.

[0246] Corresponding to the image denoising method described above, this application also provides a server, such as... Figure 9As shown, it includes a processor 901 and a machine-readable storage medium 902. The machine-readable storage medium 902 stores machine-executable instructions that can be executed by the processor 901. The processor 901 is prompted by the machine-executable instructions to implement any of the above-described image denoising method steps.

[0247] Corresponding to the image denoising method described above, this application also provides an edge device, such as... Figure 10 As shown, it includes a processor 1001 and a machine-readable storage medium 1002. The machine-readable storage medium 1002 stores machine-executable instructions that can be executed by the processor 1001. The processor 1001 is prompted by the machine-executable instructions to implement any of the above-described image denoising method steps.

[0248] Edge devices can be personal computers, mobile phones, tablets, or AI-ISP cameras, among other devices.

[0249] Machine-readable storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the machine-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0250] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0251] Corresponding to the above image denoising method, this application embodiment also provides an image denoising system, which includes a server;

[0252] The server is used to acquire a preset training set, which includes multiple sample image pairs. Each sample image pair includes a sample noisy image and a corresponding sample denoised image. Using a preset multi-head attention mechanism module, the first sample feature matrix of the sample noisy image in each sample image pair is extracted. The first sample feature matrix of each sample noisy image is upsampled to obtain the second sample feature matrix of each sample noisy image. Based on the second sample feature matrix of each sample noisy image, a predicted denoised image corresponding to each sample noisy image is generated. If the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noisy image, the training of the preset multi-head attention mechanism module ends.

[0253] Optionally, the system may also include edge devices;

[0254] An edge device is used to acquire the image to be denoised; using a preset multi-head attention mechanism module trained on the server, the first feature matrix of the image to be denoised is extracted, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism; the first feature matrix is ​​upsampled to obtain a second feature matrix; and a denoised image corresponding to the image to be denoised is generated based on the second feature matrix.

[0255] The technical solution provided in this application utilizes a preset multi-head attention mechanism module to denoise images. Since the preset multi-head attention mechanism module is smaller than a convolutional neural network model, using the technical solution provided in this application for image denoising reduces the consumption of computational resources and lowers the performance requirements of the device.

[0256] Furthermore, in the technical solution provided in this application embodiment, the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism. The self-attention mechanism can extract global semantic information from the image, while the channel attention mechanism extracts local detail texture information. Since both global semantic information and local detail texture information of the image can be obtained, the number of image pairs required in the training set of the preset multi-head attention mechanism module is reduced. Therefore, it is not necessary to synthesize a large number of noisy images to assist in the training of the preset multi-head attention mechanism module, thereby improving image denoising efficiency.

[0257] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described image denoising method steps.

[0258] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the above-described image denoising method steps.

[0259] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0260] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0261] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, servers, edge devices, systems, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0262] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image denoising method, characterized in that, The method includes: Obtain the image to be denoised; The first feature matrix of the image to be denoised is extracted using a preset multi-head attention mechanism module, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism; The first feature matrix is ​​upsampled to obtain the second feature matrix; Based on the second feature matrix, a denoised image corresponding to the image to be denoised is generated; The step of extracting the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module includes: Using the image to be denoised as the first candidate feature matrix, the self-attention mechanism and the channel attention mechanism are used to extract features from the first candidate feature matrix to obtain the first intermediate feature matrix; If the number of downsampling operations is less than the preset number, the first intermediate feature matrix is ​​downsampled to obtain the second intermediate feature matrix, and the second intermediate feature matrix is ​​updated to the first candidate feature matrix. The step of using the self-attention mechanism and the channel attention mechanism to extract features from the first candidate feature matrix to obtain the first intermediate feature matrix is ​​then re-executed. If the number of downsampling operations is equal to the preset number, then the first intermediate feature matrix is ​​used as the first feature matrix of the image to be denoised. The step of upsampling the first feature matrix to obtain the second feature matrix includes: Using the first feature matrix as the second candidate feature matrix, the second candidate feature matrix is ​​upsampled to obtain the third intermediate feature matrix; The third intermediate feature matrix and the second intermediate feature matrix with the same dimension as the third intermediate feature matrix are summed to obtain the fourth intermediate feature matrix. If the number of upsampling operations is less than the preset number, the fourth intermediate feature matrix is ​​updated to the second candidate feature matrix, and the step of upsampling the second candidate feature matrix to obtain the third intermediate feature matrix is ​​re-executed. If the number of upsampling operations is equal to the preset number, then the fourth intermediate feature matrix is ​​used as the second feature matrix.

2. The method according to claim 1, characterized in that, The step of extracting the first feature matrix of the image to be denoised using a preset multi-head attention mechanism module includes: The pixel values ​​of the image to be denoised are normalized to obtain a first normalized image; The first feature matrix of the first normalized image is extracted using a pre-defined multi-head attention mechanism module. The step of generating a denoised image corresponding to the image to be denoised based on the second feature matrix includes: Using the second feature matrix, a second normalized image is generated; The second normalized image is subjected to inverse normalization to obtain the denoised image corresponding to the image to be denoised.

3. The method according to any one of claims 1-2, characterized in that, The preset multi-head attention mechanism module is trained using the following steps: Obtain a preset training set, which includes multiple sample image pairs, and each sample image pair includes a sample noisy image and a corresponding sample denoised image; Using a pre-defined multi-head attention mechanism module, the first sample feature matrix of the sample noise image in each sample image pair is extracted; Upsample the first sample feature matrix of each sample noisy image to obtain the second sample feature matrix of each sample noisy image; Based on the second sample feature matrix of each sample noisy image, generate the predicted denoised image corresponding to each sample noisy image; If the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the training of the preset multi-head attention mechanism module ends.

4. The method according to claim 3, characterized in that, The method further includes: If it is determined that the preset multi-head attention mechanism module has not converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the parameters of the preset multi-head attention mechanism module are adjusted, and the step of extracting the first sample feature matrix of the sample noise image in each sample image pair using the preset multi-head attention mechanism module is re-executed.

5. The method according to claim 3, characterized in that, The step of extracting the first sample feature matrix of the sample noise image in each sample image pair using a preset multi-head attention mechanism module includes: From the plurality of sample image pairs, select a preset number of sample image pairs; Using a pre-defined multi-head attention mechanism module, the first sample feature matrix of the sample noise image is extracted from each selected sample image pair.

6. The method according to claim 3, characterized in that, The step of obtaining the preset training set includes: Obtain original image pairs, which include the original noisy image and the corresponding original denoised image; Multiple first image blocks are extracted from the original noisy image, and multiple second image blocks are extracted from the original denoised image corresponding to the original noisy image. The first image blocks and the second image blocks correspond one-to-one, and each pair of first image blocks does not completely overlap. Multiple sample image pairs are formed by the multiple first image blocks and the multiple second image blocks, and the multiple sample image pairs are added to a preset training set. The first image block is a sample noise image, and the second image block corresponding to the first image block is a sample denoised image corresponding to the first image block.

7. An edge device, characterized in that, The method includes a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor, the processor being prompted by the machine-executable instructions to perform the steps of the method according to any one of claims 1-6.

8. An image denoising system, characterized in that, The system includes a server; The server is used to acquire a preset training set, which includes multiple sample image pairs. Each sample image pair includes a sample noisy image and a corresponding sample denoised image. The server uses a preset multi-head attention mechanism module to extract the first sample feature matrix of the sample noisy image in each sample image pair. Upsample the first sample feature matrix of each sample noisy image to obtain the second sample feature matrix of each sample noisy image; Based on the second sample feature matrix of each sample noise image, a predicted denoised image corresponding to each sample noise image is generated; if the preset multi-head attention mechanism module is determined to have converged based on the predicted denoised image and the sample denoised image corresponding to each sample noise image, then the training of the preset multi-head attention mechanism module ends. The system also includes edge devices; The edge device is used to acquire the image to be denoised; extract the first feature matrix of the image to be denoised using a trained preset multi-head attention mechanism module, wherein the preset multi-head attention mechanism module includes a self-attention mechanism and a channel attention mechanism; upsample the first feature matrix to obtain a second feature matrix; and generate a denoised image corresponding to the image to be denoised based on the second feature matrix. The edge device is specifically configured to use the image to be denoised as a first candidate feature matrix, and extract features from the first candidate feature matrix using the self-attention mechanism and the channel attention mechanism to obtain a first intermediate feature matrix; if the number of downsampling operations is less than a preset number, the first intermediate feature matrix is ​​downsampled to obtain a second intermediate feature matrix, and the second intermediate feature matrix is ​​updated as the first candidate feature matrix, and the step of extracting features from the first candidate feature matrix using the self-attention mechanism and the channel attention mechanism to obtain the first intermediate feature matrix is ​​re-executed; if the number of downsampling operations is equal to the preset number, the first intermediate feature matrix is ​​used as the first feature matrix of the image to be denoised. The edge device is specifically configured to: use the first feature matrix as the second candidate feature matrix, upsample the second candidate feature matrix to obtain a third intermediate feature matrix; sum the third intermediate feature matrix and the second intermediate feature matrix with the same dimension as the third intermediate feature matrix to obtain a fourth intermediate feature matrix; if the number of upsampling operations is less than the preset number, update the fourth intermediate feature matrix as the second candidate feature matrix, and re-execute the step of upsampling the second candidate feature matrix to obtain the third intermediate feature matrix; if the number of upsampling operations is equal to the preset number, use the fourth intermediate feature matrix as the second feature matrix.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images