Image denoising method, device and electronic equipment

Abstract

Embodiments of the present invention provide an image denoising method, apparatus, and electronic device, relating to the field of image processing technology. The method includes: determining pixel offset information between a reference image and an image to be denoised; wherein the pixel offset information represents the coordinate offset of pixel points of the same object in the reference image and the image to be denoised; adjusting the pixel values ​​of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image; and performing denoising on the image to be denoised based on the adjusted image to obtain a denoised image. This solution can improve the image denoising effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image denoising method, device and electronic equipment. Background Art

[0002] Under low light illumination conditions, the signal-to-noise ratio of the image captured by the image acquisition device is low, resulting in a large number of noise points in the image, so the image needs to be denoised.

[0003] In related technologies, a multi-frame denoising method is used to improve the signal-to-noise ratio of an image. For example, multiple frames of images are simultaneously captured using the same image acquisition device, and the pixel values ​​of each corresponding pixel point of the captured multiple frames of images are weighted averaged to obtain a denoised image.

[0004] However, in the process of multi-frame denoising, there are often large differences between the multi-frame images used. For example, when there are moving objects in the captured scene, due to a certain time difference between the capture moments of different image frames, the positions of the moving objects in different image frames will be quite different, which will lead to poor image denoising effect. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide an image denoising method, apparatus, and electronic device to improve the image denoising effect. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides an image denoising method, the method comprising:

[0007] Acquire an image to be denoised and a reference image; wherein the reference image is an image captured in the scene where the image to be denoised is captured;

[0008] Determining pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0009] Adjusting the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image;

[0010] Based on the adjusted image, denoising is performed on the image to be denoised to obtain a denoised image.

[0011] Optionally, determining pixel offset information between the reference image and the image to be denoised includes:

[0012] Determine the pixel coordinates of the feature points in the image to be denoised as the feature point coordinates;

[0013] Based on the feature point coordinates, determining a first target area from the reference image; wherein the first target area is: an area of ​​a first preset size centered at the pixel point at the feature point coordinates in the reference image;

[0014] Determining, from a plurality of reference pixels within the first target area, a reference pixel that matches the feature point as a target pixel;

[0015] Determine the coordinate offset between the pixel coordinates of the target pixel point and the coordinates of the feature point;

[0016] Based on the determined coordinate offset, pixel offset information between the reference image and the image to be denoised is determined.

[0017] Optionally, determining a reference pixel point that matches the feature point from a plurality of reference pixels within the first target area as the target pixel point includes:

[0018] In the image to be denoised, a plurality of pixels within a second target area are selected as pixels to be used corresponding to the feature point; wherein the second target area is an area of ​​a second preset size centered on the feature point in the image to be denoised;

[0019] For each reference pixel, in the reference image, multiple pixels within a third target area corresponding to the reference pixel are selected as pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is an area of ​​the second preset size centered on the reference pixel;

[0020] For each reference pixel, calculate the similarity between the reference pixel and the feature point based on the pixel value of the pixel to be used corresponding to the reference pixel and the pixel value of the pixel to be used corresponding to the feature point;

[0021] From each reference pixel point, a reference pixel point having the highest similarity to the feature point is selected as the target pixel point.

[0022] Optionally, there are multiple feature points in the image to be denoised, and the determined coordinate offset includes: a coordinate offset corresponding to each feature point determined for the feature point;

[0023] The determining, based on the determined coordinate offset, pixel offset information between the reference image and the image to be denoised includes:

[0024] Calculate the average value of the coordinate offsets corresponding to each feature point to obtain the average coordinate offset;

[0025] The average coordinate offset is determined as pixel offset information between the reference image and the image to be denoised.

[0026] Optionally, adjusting the pixel value of each pixel in the reference image based on the pixel offset information includes:

[0027] For each pixel point in the reference image, determining the offset pixel coordinates corresponding to the pixel point according to the pixel offset information;

[0028] For each pixel point in the reference image, the pixel value of the pixel point is modified to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

[0029] Optionally, performing denoising on the image to be denoised based on the adjusted image to obtain a denoised image includes:

[0030] The adjusted image and the image to be denoised are input into a pre-trained image denoising model to obtain a denoised image.

[0031] Optionally, the image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to fuse the input adjusted image and the image data of the image to be denoised to obtain fused image data; the denoising network is used to denoise the fused image data to obtain a denoised image.

[0032] Optionally, the image denoising model is trained by:

[0033] Acquire a sample image and a sample reference image; wherein the sample reference image is: an image captured in the scene where the sample image is captured;

[0034] Performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image;

[0035] Determining sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: a coordinate offset of pixel points of the same object in the noisy reference image and the noisy sample image;

[0036] Adjusting the pixel coordinates of each pixel in the noisy reference image based on the sample pixel offset information to obtain an adjusted noisy reference image;

[0037] Inputting the adjusted noisy reference image and the noisy sample image into a neural network model to be trained to obtain a denoised sample image;

[0038] Calculating a model loss of the neural network model to be trained based on the denoised sample image and the sample image;

[0039] Based on the model loss, model parameters of the neural network model to be trained are adjusted.

[0040] In a second aspect, an embodiment of the present invention provides an image denoising device, the device comprising:

[0041] An acquisition module, configured to acquire an image to be denoised and a reference image; wherein the reference image is an image acquired in the scene where the image to be denoised is acquired;

[0042] An offset information determination module is configured to determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: a coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0043] An adjustment module, configured to adjust the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image;

[0044] The denoising module is used to perform denoising processing on the image to be denoised based on the adjusted image to obtain a denoised image.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0046] Memory for storing computer programs;

[0047] The processor is configured to implement any of the steps of the above-mentioned image denoising method when executing the program stored in the memory.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned image denoising methods are implemented.

[0049] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image denoising methods.

[0050] Beneficial effects of the embodiments of the present invention:

[0051] An image denoising method provided by an embodiment of the present invention obtains an image to be denoised and a reference image; wherein the reference image is an image captured in the scene where the image to be denoised is captured, and then pixel offset information between the reference image and the image to be denoised is determined, where the pixel offset information represents the coordinate offset of pixel points of the same object in the reference image and the image to be denoised. Based on the pixel offset information, the pixel values ​​of each pixel point in the reference image are adjusted to obtain an adjusted image, and then, based on the adjusted image, the image to be denoised is denoised to obtain a denoised image. It can be seen that in this solution, by determining the pixel offset information between the reference image and the image to be denoised, where the pixel offset information represents the coordinate offset of pixel points of the same object in the reference image and the image to be denoised, and then adjusting the pixel values ​​of each pixel point in the reference image based on the pixel offset information, the difference between the image to be denoised and the reference image can be reduced, and then, based on the adjusted image and the image to be denoised, the image to be denoised is denoised, thereby improving the image denoising effect.

[0052] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0054] Figure 1 A schematic diagram of a first flow chart of an image denoising method provided by an embodiment of the present invention;

[0055] Figure 2 The image before translation in the image denoising method provided by an embodiment of the present invention;

[0056] Figure 3 The image after translation in the image denoising method provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of image fusion in the image denoising method provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of image denoising in the image denoising method provided by an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the structure of U-Net in the image denoising method provided by an embodiment of the present invention;

[0060] Figure 7A second flow chart of the image denoising method provided by an embodiment of the present invention;

[0061] Figure 8 A schematic diagram of determining target pixels in an image denoising method provided by an embodiment of the present invention;

[0062] Figure 9 A flowchart of a method for training an image denoising model in an image denoising method provided by an embodiment of the present invention;

[0063] Figure 10 A comparison diagram of the effects of the image denoising method provided by an embodiment of the present invention;

[0064] Figure 11 Another effect comparison diagram of the image denoising method provided by an embodiment of the present invention;

[0065] Figure 12 A schematic structural diagram of an image denoising device provided by an embodiment of the present invention;

[0066] Figure 13 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention are within the scope of protection of the present invention.

[0068] In order to improve the denoising effect of an image, an embodiment of the present invention provides an image denoising method, an apparatus and an electronic device.

[0069] The image denoising method provided by the embodiments of the present invention can be applied to electronic devices with data processing capabilities, such as servers and desktop computers. The image denoising method provided by the embodiments of the present invention can also be applied to image acquisition devices, such as smartphones and digital cameras, which is reasonable. Furthermore, the image processing method can be implemented using software, hardware, or a combination of software and hardware.

[0070] The image denoising method provided by the embodiment of the present invention may include the following steps:

[0071] Acquire an image to be denoised and a reference image; wherein the reference image is an image captured in the scene where the image to be denoised is captured;

[0072] Determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0073] Based on the pixel offset information, the pixel value of each pixel in the reference image is adjusted to obtain an adjusted image;

[0074] Based on the adjusted image, the image to be denoised is denoised to obtain a denoised image.

[0075] In an embodiment of the present invention, by determining pixel offset information between a reference image and an image to be denoised, where the pixel offset information is information representing the coordinate offset of pixel points of the same object in the reference image and the image to be denoised, the pixel values ​​of each pixel point in the reference image are adjusted based on the pixel offset information, thereby reducing the difference between the image to be denoised and the reference image. Furthermore, based on the adjusted image and the image to be denoised, the image to be denoised is denoised, thereby improving the image denoising effect.

[0076] The image denoising method provided by the embodiments of the present invention is introduced below with reference to the accompanying drawings.

[0077] like Figure 1 As shown, the image denoising method provided by the embodiment of the present invention may include the following steps:

[0078] S101, acquiring an image to be denoised and a reference image; wherein the reference image is an image captured in the scene where the image to be denoised is captured;

[0079] The image to be denoised and the reference image may be images captured by an image acquisition device. The acquisition scene of the image to be denoised may include: the position of the image acquisition device when capturing the image to be denoised, the acquisition angle, etc. The reference image may be an image captured in the same scene as the acquisition scene of the image to be denoised. The reference image may be a single image or multiple images. The method for acquiring the image to be denoised and the reference image may be: real-time acquisition of the image to be denoised and the reference image. Of course, the image to be denoised and the reference image may also be images pre-captured and stored in a designated storage location, and the method for acquiring the image to be denoised and the reference image may also be: reading the image to be denoised and the reference image from the designated storage location.

[0080] For example, in one implementation, to obtain the image to be denoised and the reference image, the same image acquisition device can be used to continuously capture multiple frames of images of the same scene at the same location. One frame of the image is selected as the image to be denoised, and the remaining images are used as reference images. The method for selecting the image to be denoised can be random selection, the first captured image, etc., which is not limited here. Alternatively, different image acquisition devices can be used to capture different images at the same location, at the same angle, and at the same focal length. One frame of the image is selected from the captured images as the image to be denoised, and the remaining images are used as reference images. This is also possible.

[0081] In another implementation, the image to be denoised may be a video frame that needs to be denoised in the video. In this case, the reference image may be a video frame whose distance from the video frame that needs to be denoised is less than or equal to a preset threshold. For example, if the preset threshold is 1, the previous video frame and / or the next video frame of the video frame that needs to be denoised may be selected as the reference image.

[0082] The images to be denoised and the reference images in this embodiment can be images of any format, such as RGB (Red / Green / Blue) images, raw images, etc. Raw images typically have a data bit width of 10 to 16 bits and can contain more information than 8-bit RGB images. To ensure better denoising effects, this embodiment may prioritize the use of raw images as the images to be denoised and the reference images.

[0083] S102, determining pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: a coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0084] Since the difference between the captured image to be denoised and the reference image is usually caused by moving objects in the captured scene or shaking of the image capture device, it is reflected in the image as the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised. For example, this pixel offset information can be represented by coordinates, for example, (-6, 8), indicating that the pixel point of the same object in the reference image is offset to the left by 6 pixels and to the bottom by 8 pixels relative to the pixel point in the image to be denoised. The specific method for determining the offset of the reference image relative to the image to be denoised will be described in detail in the embodiments below and will not be repeated here.

[0085] If there are multiple reference images, the pixel offset information between the reference image and the image to be denoised may be determined for each reference image.

[0086] S103, adjusting the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image;

[0087] In one implementation, after determining the pixel offset information, each pixel in the reference image may be shifted according to the determined pixel offset information. For example, if the determined pixel offset information is (-6, 8), each pixel in the reference image may be shifted 6 pixels to the right and 8 pixels upward. Pixels that are shifted out of the image area after the shift may be removed, and the pixel values ​​of the blank area after the shift may be set to 0. Figure 2-3 As shown, Figure 2 is the image before translation, Figure 3 for Figure 2 The image is obtained by shifting each pixel of the image upward and rightward. In actual processing, the following steps A1-A2 can be followed:

[0088] Step A1: for each pixel point in the reference image, determine the offset pixel coordinates corresponding to the pixel point according to the pixel offset information;

[0089] For example, the pixel offset information is (-6, 8), which means that the pixel point of the same object in the reference image is offset by 6 pixels to the left and 8 pixels downward relative to the pixel point in the image to be denoised. Then, for each pixel point in the reference image, the pixel coordinates of the pixel point can be added with (-6, 8) to obtain the offset pixel coordinates corresponding to the pixel point.

[0090] Step A2: for each pixel point in the reference image, modify the pixel value of the pixel point to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

[0091] For example, if the determined pixel offset information is (-6, 8), then the pixel point with pixel coordinates (10, 10) has corresponding offset pixel coordinates (4, 18). If the pixel value of the pixel point with pixel coordinates (4, 18) in the reference image is 100, then the pixel value of the pixel point with pixel coordinates (10, 10) can be modified to 100. If the offset pixel coordinates corresponding to a pixel point are not in the reference image, the pixel value of the pixel point can be modified to 0.

[0092] It can be seen that by determining the offset pixel coordinates corresponding to each pixel point in the reference image according to the pixel offset information, and then modifying the pixel value of the pixel point to the pixel value of the pixel point at the offset pixel coordinates corresponding to the pixel point, the pixels representing the same object in the reference image and the image to be denoised can be aligned, thereby narrowing the difference between the reference image and the image to be denoised, and providing a basis for subsequent denoising processing based on the adjusted image and the image to be denoised.

[0093] If there are multiple reference images, the pixel coordinates of each pixel in each reference image can be adjusted based on the offset of the reference image relative to the image to be denoised to obtain an adjusted image corresponding to the reference image.

[0094] S104: Based on the adjusted image, perform denoising on the image to be denoised to obtain a denoised image.

[0095] In this step, after obtaining the adjusted image, denoising can be performed using any multi-frame denoising method. In one implementation, the adjusted image and the image to be denoised can be fused to obtain a fused image. For example, the pixel values ​​at each corresponding pixel coordinate of the adjusted image and the image to be denoised can be weighted averaged to obtain a fused image. After obtaining the fused image, the fused image can be denoised using any image denoising method, such as Gaussian denoising, wavelet denoising, etc., or using a pre-trained denoising network, such as a U-Net (U-net).

[0096] In one implementation, the adjusted image and the image to be denoised can be input into a pre-trained image denoising model to obtain a denoised image. The image denoising model can include a fusion network and a denoising network. The fusion network is configured to fuse the input adjusted image and the image to be denoised to obtain fused image data. The denoising network is configured to denoise the fused image data and output a denoised image.

[0097] For example, Figure 4 As shown, Figure 4In the example, t0 and t1 represent the adjusted image and the image to be denoised, respectively. T0 and t1 are raw images. First, t0 and t1 are split into RGGB (Red / Green / Green / Blue) 4-channel image data. Then, the split t0 and t1 are concatenated to obtain 8-channel image data. Finally, the obtained 8-channel image data is input into a preset fusion network, which outputs 4-channel image data as the fused image data. The fusion network can consist of two convolutional layers, both with a 3x3 convolution kernel size. The first convolutional layer can upscale the 8-channel data to 64 channels, and the second convolutional layer can reduce the 64-channel data to 4 channels. The preset fusion network can also consist of four convolutional layers. The first convolutional layer upscales the 8-channel data to 64 channels, the second convolutional layer upscales the 64-channel data to 128 channels, the third convolutional layer reduces the 128-channel data to 64 channels, and the fourth convolutional layer reduces the 64-channel data to 4 channels. The composition of the above fusion network is only an example and is not limited to a specific one. After obtaining the fused image data, that is, the 4-channel image data output by the fusion network, such as Figure 5 As shown in Figure 1, the 4-channel image data output by the above fusion network can be input into a pre-trained denoising network for denoising. After that, the denoised image data is restored to the raw image to obtain the denoised image. The denoising network can be any denoising network, such as Figure 6 The U-Net shown in the figure contains multiple 3x3 convolutional layers, that is, convolutional layers with a convolution kernel size of 3x3, which are used to perform convolution processing on image data, and multiple ReLU (linear rectification units) are used to perform linear rectification on image data. Figure 6 The unfilled arrow to the right in the figure represents a combination of a 3x3 convolutional layer and ReLU, where some of the 3x3 convolutional layers are also used to upscale the image data, including: upscaling 4 channels to 32 channels, 32 channels to 64 channels, 64 channels to 128 channels, 128 channels to 256 channels, and 256 channels to 512 channels; it also includes four 2x2 maximum pooling layers, Figure 6 Indicated by a downward arrow, it is used to downsample the input image data; and four 2x2 transposed convolution layers, that is, transposed convolution layers with a convolution kernel size of 2x2, Figure 6 Indicated by an upward arrow, it is used to upsample the downsampled image data and perform dimensionality reduction processing, including reducing the dimensionality of 512 channels to 256 channels, reducing the dimensionality of 256 channels to 128 channels, reducing the dimensionality of 128 channels to 64 channels, and reducing the dimensionality of 64 channels to 32 channels; merging layer, Figure 6 Indicated by a right-pointing black filled arrow, it is used to add two sets of data, for example, Figure 6The 64-channel data is reduced to 32 channels by the 2x2 transposed convolution layer and then added to the 32-channel data before downsampling. The last layer of the U-Net is a 1x1 convolution layer, that is, a convolution layer with a convolution kernel size of 1x1. Figure 6 In the figure, the arrow with right-pointing filled hatch pattern is used to reduce the dimension of 32-channel image data into 4 channels.

[0098] In this embodiment, by determining the pixel offset information between the reference image and the image to be denoised, and the pixel offset information is information that represents the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised, the pixel values ​​of each pixel point in the reference image are adjusted based on the pixel offset information, which can reduce the difference between the image to be denoised and the reference image. Then, based on the adjusted image and the image to be denoised, the image to be denoised is denoised, which can improve the image denoising effect.

[0099] Alternatively, in another embodiment of the present invention, Figure 7 As shown, the image denoising method provided by the embodiment of the present invention may further include the following steps:

[0100] S701, obtaining an image to be denoised and a reference image;

[0101] This step is similar to the above step S101 and will not be described again here.

[0102] S702, determining the pixel coordinates of the feature point in the image to be denoised as the feature point coordinates;

[0103] The determined feature points may be single or multiple. In one implementation, a feature point extraction algorithm may be used to determine the feature points in the image to be denoised, or a single or a preset number of pixels may be randomly extracted from the image to be denoised as feature points, and the pixel coordinates of the feature points may be determined as feature point coordinates. The feature point extraction algorithm may be a SIFT (Scale-Invariant Feature Transform) algorithm, a SURF (Speeded-Up Robust Feature) algorithm, an ORB (Oriented FAST and RotatedBRIEF, an algorithm for fast feature point extraction and description), a Harris (corner detection) algorithm, or the like.

[0104] In another implementation, the image to be denoised can be evenly divided into multiple blocks, and the center point of each block can be used as the matching point. For example, a frame of a 1920*1080 resolution image to be denoised can be evenly divided into 16*10 blocks, or 160 blocks. The center point of each block is then determined as a feature point, and the pixel coordinates of the feature point are then determined as the feature point coordinates.

[0105] S703, determining a first target area from the reference image based on the feature point coordinates; wherein the first target area is: an area of ​​a first preset size centered at the pixel at the feature point coordinates in the reference image;

[0106] The first target area can be a collection of multiple pixels, and the first preset size can be pre-set based on experience and needs. For example, the first preset size can be set to 21*21. After determining the coordinates of the feature point, the first target area can be determined from the reference image with the pixel at the feature point coordinates as the center. For example, if the feature point coordinates are (10, 10) and the first preset size is 21*21, then a collection of 21*21 pixels can be selected with the pixel at the pixel coordinates (10, 10) in the reference image as the center as the first target area.

[0107] S704, determining a reference pixel point that matches the feature point from a plurality of reference pixels within the first target area as a target pixel point;

[0108] In one implementation, the pixel features of the feature point can be extracted first. For example, the pixel features of the feature point can be extracted using an image gradient algorithm. Then, for each reference pixel point, the pixel features of the reference pixel point are extracted, and the similarity between the pixel features of the reference pixel point and the pixel features of the feature point is calculated. Then, the reference pixel point with the highest pixel feature similarity to the feature point is determined as the target pixel point.

[0109] In another implementation, the above-mentioned determination of a reference pixel point matching a feature point as a target pixel point from among the reference pixels may include the following steps B1-B4:

[0110] B1, in the image to be denoised, selecting a plurality of pixels in a second target area as pixels to be used corresponding to the feature point; wherein the second target area is an area of ​​a second preset size centered at the feature point in the image to be denoised;

[0111] Among them, the second preset size can also be pre-set according to experience and needs. For example, the second preset size can be set to 9*9, indicating that the second target area is a set of 9*9 pixel points centered on the feature point, and the 9*9 pixel points are used as the pixel points to be used corresponding to the feature point.

[0112] B2, for each reference pixel, in the reference image, selecting multiple pixels within a third target area corresponding to the reference pixel as the pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is an area of ​​a second preset size centered at the reference pixel;

[0113] In this step, for each reference pixel point, the same operation as the above step B1 can be performed. The number of pixels to be used corresponding to the reference pixel point is the same as the number of pixels to be used corresponding to the above feature points.

[0114] B3, for each reference pixel, calculate the similarity between the reference pixel and the feature point based on the pixel value of the pixel to be used corresponding to the reference pixel and the pixel value of the pixel to be used corresponding to the feature point;

[0115] In this step, for each reference pixel, the to-be-used pixel points corresponding to the reference pixel point are taken as a set, and the to-be-used pixel points corresponding to the feature point are taken as another set. Then, the SAD (sum of absolute differences) method, MAD (mean absolute difference) method, SSD (sum of squared errors) method, MSD (mean sum of squared errors) method, etc. can be used to calculate the similarity between the reference pixel point and the feature point based on the pixel value of the to-be-used pixel point corresponding to the reference pixel point and the pixel value of the to-be-used pixel point corresponding to the feature point.

[0116] B4, from each reference pixel point, select the reference pixel point with the highest similarity to the feature point as the target pixel point.

[0117] After determining the similarity between each reference pixel and the feature point, the reference pixel with the highest similarity to the feature point may be selected as the target pixel.

[0118] In this implementation, multiple pixels within a second preset size centered on the reference pixel and multiple pixels within a second size range centered on the feature point are used to calculate the similarity between the reference pixel and the feature point. This can fully utilize the pixel information around the reference pixel and the feature point, thereby improving the accuracy of calculating the similarity.

[0119] Figure 8 The following is a schematic diagram showing the process of determining the target pixel points mentioned above, where t0 and t1 represent the adjusted image and the image to be denoised, respectively. The points in t1 represent feature points, the points in t0 represent target pixels, the large box represents the first target area, and the small box represents the second target area.

[0120] S705, determining the coordinate offset of the target pixel point relative to the feature point coordinates;

[0121] In this step, after determining the target pixel that matches the feature point coordinates, the coordinate offset of the target pixel relative to the feature point coordinates can be determined. For example, if the feature point coordinates are (10, 10) and the target pixel coordinates are (15, 16), the coordinate offset of the target pixel relative to the feature point coordinates can be determined to be (5, 6).

[0122] S706 : Determine pixel offset information between the reference image and the image to be denoised based on the determined coordinate offset.

[0123] In this step, when there is a single feature point in the image to be denoised, the determined coordinate offset can be directly used as the pixel offset information between the reference image and the image to be denoised. When there are multiple feature points in the image to be denoised, the determined coordinate offset includes: the coordinate offset corresponding to each feature point determined for the feature point. In this case, the above steps S702-S705 can be performed for each feature point to obtain the coordinate offset corresponding to each feature point. After that, the average value of the coordinate offsets corresponding to each feature point can be calculated to obtain the average coordinate offset, and then the average coordinate offset can be determined as the pixel offset information between the reference image and the image to be denoised. By determining the pixel offset information for multiple feature points, the accuracy of the determined pixel offset information can be improved, thereby further improving the effect of image denoising.

[0124] S707, adjusting the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image;

[0125] S708 , performing denoising processing on the image to be denoised based on the adjusted image to obtain a denoised image.

[0126] The above steps S707-S708 are similar to the above steps S103-S104 and will not be repeated here.

[0127] In this embodiment, the effect of image denoising can be improved. Furthermore, by determining the pixel coordinates of the feature point in the image to be denoised as the feature point coordinates; based on the feature point coordinates, determining a first target area from the reference image; wherein the first target area is: an area of ​​a first preset size centered on the pixel point at the feature point coordinates in the reference image; determining a reference pixel point that matches the feature point from multiple reference pixels in the first target area as the target pixel point; determining the coordinate offset between the pixel coordinates of the target pixel point and the feature point coordinates; and based on the determined coordinate offset, determining the pixel offset information between the reference image and the image to be denoised, the offset of the reference image relative to the image to be denoised can be accurately determined.

[0128] Alternatively, in another embodiment of the present invention, Figure 9 As shown, the training method of the above-mentioned image denoising model includes the following steps S901-S907:

[0129] S901, obtaining a sample image and a sample reference image; wherein the sample reference image is an image captured in the scene where the sample image is captured;

[0130] The method for acquiring the sample image and the sample reference image can be similar to that of step S101 above. In addition, the sample image and the sample reference image can be acquired under sufficient lighting conditions and have a high signal-to-noise ratio, which can further improve the denoising effect of the embodiment of the present invention. Multiple sample images can be used to ensure better model training results.

[0131] S902, performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image;

[0132] In this step, the sample image and the sample reference image may be subjected to noise processing, such as adding Gaussian noise, salt and pepper noise, etc.

[0133] S903, determining sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: a coordinate offset of pixel points of the same object in the noisy reference image and the noisy sample image;

[0134] S904, adjusting the pixel coordinates of each pixel point in the noisy reference image based on the sample pixel offset information to obtain an adjusted noisy reference image;

[0135] S905, inputting the sample image into the neural network model to be trained to obtain a denoised sample image; wherein the neural network model to be trained includes: a fusion network to be trained and a denoising network to be trained;

[0136] The above steps S903 to S905 are similar to the above process of denoising the denoised image, and will not be repeated here.

[0137] S906, calculating the model loss of the neural network model to be trained based on the denoised sample image and the sample image;

[0138] In this step, the model loss can be calculated using a preset loss function. The preset loss function can be any loss function, such as a cross entropy loss function, etc., which is not limited here.

[0139] S907: Based on the model loss, adjust the model parameters in the fusion network to be trained and the denoising network to be trained.

[0140] In this step, the model parameters can be adjusted using a gradient descent method. When training the model, steps S901-S907 can be performed multiple times to obtain different sample images and sample reference images to adjust the model parameters in the neural network model to be trained until the neural network model converges, resulting in a trained denoising model. If the denoising network includes a fusion network and a denoising network, the fusion network and the denoising network can be trained jointly to further improve the image denoising effect.

[0141] like Figure 10 As shown, Figure 10 The comparison of image denoising effects using different processing methods is demonstrated. The images from top to bottom are: the image to be denoised, the result of single-frame denoising, the denoising effect of five frames of images directly input into the denoising model, and the denoising effect of five frames of images fused using an embodiment of the present invention. Figure 11 Shown Figure 10 A partial comparison of the image in the figure shows, from left to right, the image to be denoised, the result of single-frame denoising, the denoising effect of five frames directly fed into the denoising model, and the denoising effect of five frames fused using an embodiment of the present invention. As can be seen, image denoising using the image denoising method provided by the embodiment of the present invention significantly improves both detail and image quality.

[0142] In this embodiment, the effect of image denoising can be improved. Furthermore, by performing noise processing on the sample image and the sample reference image, a noisy sample image and a noisy reference image are obtained, and then sample pixel offset information between the noisy reference image and the noisy sample image is obtained. The sample pixel offset information is information that characterizes the coordinate offset of the pixel points of the same object in the sample image and the sample reference image. Therefore, the pixel values ​​of each pixel point in the sample reference image are adjusted based on the sample pixel offset information, which can reduce the difference between the noisy reference image and the noisy sample image. Then, the adjusted noisy reference image and the noisy sample image are input into the neural network model to be trained to obtain a denoised sample image, and the model loss of the neural network model to be trained is calculated based on the denoised sample image and the sample image, and the model parameters of the neural network model to be trained are adjusted based on the model loss to obtain an image denoising model, which provides a basis for implementing image denoising using the image denoising model in the embodiment of the present application.

[0143] Based on the same inventive concept, the embodiment of the present invention also provides an image denoising device, such as Figure 12 As shown, the device includes:

[0144] The acquisition module 1201 is configured to acquire an image to be denoised and a reference image; wherein the reference image is an image acquired in the scene where the image to be denoised is acquired;

[0145] The offset information determining module 1202 is configured to determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: a coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0146] An adjustment module 1203 is configured to adjust the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image;

[0147] The denoising module 1204 is configured to perform denoising on the image to be denoised based on the adjusted image to obtain a denoised image.

[0148] Optionally, the denoising module includes:

[0149] A fusion submodule, configured to fuse the adjusted image and the image to be denoised to obtain a fused image;

[0150] The denoising submodule is used to perform denoising on the fused image to obtain a denoised image.

[0151] Optionally, the adjustment module is specifically configured to translate each pixel in the reference image according to the determined offset.

[0152] Optionally, the offset determination module includes:

[0153] A feature point coordinate determination submodule, used to determine the pixel coordinates of the feature points in the image to be denoised;

[0154] A first target region determination submodule is configured to determine a first target region from the reference image based on the feature point coordinates; wherein the first target region is a region of a first preset size in the reference image centered around the pixel at the feature point coordinates;

[0155] a target pixel point determination submodule, configured to determine, from a plurality of reference pixels within the first target area, a reference pixel point that matches the feature point as a target pixel point;

[0156] A coordinate offset determination submodule, configured to determine a coordinate offset between the pixel coordinates of the target pixel and the coordinates of the feature point;

[0157] The image offset determination submodule is configured to determine pixel offset information between the reference image and the image to be denoised based on the determined coordinate offset.

[0158] Optionally, the target pixel determination submodule includes:

[0159] A first selection unit is configured to select, in the image to be denoised, a plurality of pixels within a second target area as pixels to be used corresponding to the feature point; wherein the second target area is an area of ​​a second preset size centered on the feature point in the image to be denoised;

[0160] a second selection unit configured to select, for each reference pixel, a plurality of pixels within a third target area corresponding to the reference pixel in the reference image as pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is an area of ​​the second preset size centered on the reference pixel;

[0161] a calculation unit configured to calculate, for each reference pixel, a similarity between the reference pixel and the feature point based on a pixel value of the to-be-used pixel corresponding to the reference pixel and a pixel value of the to-be-used pixel corresponding to the feature point;

[0162] The third selection unit is configured to select, from among the reference pixels, a reference pixel with the highest similarity to the feature point as a target pixel.

[0163] Optionally, there are multiple feature points in the image to be denoised, and the determined coordinate offset includes: the coordinate offset corresponding to each feature point determined for the feature point; the image offset determination submodule is specifically used to: calculate the average value of the coordinate offsets corresponding to each feature point to obtain an average coordinate offset; and determine the average coordinate offset as the pixel offset information between the reference image and the image to be denoised.

[0164] Optionally, the adjustment module includes:

[0165] an offset pixel coordinate determination submodule, configured to determine, for each pixel point in the reference image, an offset pixel coordinate corresponding to the pixel point according to the pixel offset information;

[0166] The modification submodule is used to modify, for each pixel point in the reference image, the pixel value of the pixel point to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

[0167] Optionally, the denoising module is specifically configured to:

[0168] The adjusted image and the image to be denoised are input into a pre-trained image denoising model to obtain a denoised image.

[0169] Optionally, the image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to fuse the input adjusted image and the image data of the image to be denoised to obtain fused image data; the denoising network is used to denoise the fused image data and output the denoised image.

[0170] Optionally, the image denoising model is trained by:

[0171] Acquire a sample image and a sample reference image; wherein the sample reference image is: an image captured in the scene where the sample image is captured;

[0172] Performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image;

[0173] Determining sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: a coordinate offset of pixel points of the same object in the noisy reference image and the noisy sample image;

[0174] Adjusting the pixel coordinates of each pixel in the noisy reference image based on the sample pixel offset information to obtain an adjusted noisy reference image;

[0175] Inputting the adjusted noisy reference image and the noisy sample image into a neural network model to be trained to obtain a denoised sample image;

[0176] Calculating a model loss of the neural network model to be trained based on the denoised sample image and the sample image;

[0177] Based on the model loss, model parameters of the neural network model to be trained are adjusted.

[0178] In this embodiment, by determining the pixel offset information between the reference image and the image to be denoised, and the pixel offset information is information that represents the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised, the pixel values ​​of each pixel point in the reference image are adjusted based on the pixel offset information, which can reduce the difference between the image to be denoised and the reference image. Then, based on the adjusted image and the image to be denoised, the image to be denoised is denoised, which can improve the image denoising effect.

[0179] The embodiment of the present invention further provides an electronic device, such as Figure 13 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0180] Memory 1303, used for storing computer programs;

[0181] The processor 1301 is configured to implement the steps of the above-mentioned image denoising method when executing the program stored in the memory 1303 .

[0182] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0183] The communication interface is used for communication between the above electronic device and other devices.

[0184] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0185] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0186] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image denoising methods are implemented.

[0187] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer is enabled to execute any one of the image denoising methods in the above embodiments.

[0188] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0189] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0190] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. An image denoising method, characterized in that: The method comprises: Acquire an image to be denoised and a reference image; wherein the reference image is an image captured in the scene where the image to be denoised is captured; Determining pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of pixel points of the same object in the reference image and the image to be denoised; the pixel offset information is: the average value of the coordinate offsets corresponding to each feature point in the image to be denoised; the coordinate offset corresponding to each feature point is the coordinate offset between the feature point and the matching pixel point in the reference image; Adjusting the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image; Based on the adjusted image, performing denoising processing on the image to be denoised to obtain a denoised image; The step of performing denoising on the image to be denoised based on the adjusted image to obtain a denoised image includes: Perform channel stitching on the image data of the adjusted image and the image to be denoised; The channel-stitched image data is input into a pre-trained image denoising model to obtain a denoised image; the image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to: process the channel-stitched image data using a convolution kernel to obtain fused image data; the denoising network is used to denoise the fused image data to obtain a denoised image.

2. The method according to claim 1, characterized in that The determining pixel offset information between the reference image and the image to be denoised includes: Determine the pixel coordinates of the feature points in the image to be denoised as the feature point coordinates; Based on the feature point coordinates, determining a first target area from the reference image; wherein the first target area is: an area of ​​a first preset size centered at the pixel point at the feature point coordinates in the reference image; Determining, from a plurality of reference pixels within the first target area, a reference pixel that matches the feature point as a target pixel; Determine the coordinate offset between the pixel coordinates of the target pixel point and the coordinates of the feature point; Based on the determined coordinate offset, pixel offset information between the reference image and the image to be denoised is determined.

3. The method according to claim 2, characterized in that The step of determining, from a plurality of reference pixels within the first target area, a reference pixel that matches the feature point as a target pixel, includes: In the image to be denoised, a plurality of pixels within a second target area are selected as pixels to be used corresponding to the feature point; wherein the second target area is an area of ​​a second preset size centered on the feature point in the image to be denoised; For each reference pixel, in the reference image, multiple pixels within a third target area corresponding to the reference pixel are selected as pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is an area of ​​the second preset size centered on the reference pixel; For each reference pixel, calculate the similarity between the reference pixel and the feature point based on the pixel value of the pixel to be used corresponding to the reference pixel and the pixel value of the pixel to be used corresponding to the feature point; From each reference pixel point, a reference pixel point having the highest similarity to the feature point is selected as the target pixel point.

4. The method according to claim 2, characterized in that There are multiple feature points in the image to be denoised, and the determined coordinate offset includes: a coordinate offset corresponding to each feature point determined for the feature point; The determining, based on the determined coordinate offset, pixel offset information between the reference image and the image to be denoised includes: Calculate the average value of the coordinate offsets corresponding to each feature point to obtain the average coordinate offset; The average coordinate offset is determined as pixel offset information between the reference image and the image to be denoised.

5. The method according to any one of claims 1 to 4, characterized in that The adjusting the pixel value of each pixel in the reference image based on the pixel offset information includes: For each pixel point in the reference image, determining the offset pixel coordinates corresponding to the pixel point according to the pixel offset information; For each pixel point in the reference image, the pixel value of the pixel point is modified to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

6. The method according to claim 1, characterized in that The training method of the image denoising model includes: Acquire a sample image and a sample reference image; wherein the sample reference image is: an image captured in the scene where the sample image is captured; Performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image; Determining sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: a coordinate offset of pixel points of the same object in the noisy reference image and the noisy sample image; Adjusting the pixel coordinates of each pixel in the noisy reference image based on the sample pixel offset information to obtain an adjusted noisy reference image; Inputting the adjusted noisy reference image and the noisy sample image into a neural network model to be trained to obtain a denoised sample image; Calculating a model loss of the neural network model to be trained based on the denoised sample image and the sample image; Based on the model loss, model parameters of the neural network model to be trained are adjusted.

7. An image denoising device, characterized in that: The device comprises: An acquisition module, configured to acquire an image to be denoised and a reference image; wherein the reference image is an image acquired in the scene where the image to be denoised is acquired; an offset information determination module, configured to determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: a coordinate offset of pixel points of the same object in the reference image and the image to be denoised; the pixel offset information is: an average value of the coordinate offsets corresponding to each feature point in the image to be denoised; the coordinate offset corresponding to each feature point is the coordinate offset between the feature point and the matching pixel point in the reference image; An adjustment module, configured to adjust the pixel value of each pixel in the reference image based on the pixel offset information to obtain an adjusted image; a denoising module, configured to perform denoising processing on the image to be denoised based on the adjusted image to obtain a denoised image; The denoising module is specifically used to: perform channel splicing on the image data of the adjusted image and the image to be denoised; input the channel-spliced ​​image data into a pre-trained image denoising model to obtain a denoised image; the image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to: use a convolution kernel to process the channel-spliced ​​image data to obtain fused image data; the denoising network is used to perform denoising on the fused image data to obtain a denoised image.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 6 are implemented.

Images