Image denoising method and device and electronic equipment

By determining the pixel offset information between the image to be denoised and the reference image in the image acquisition device, adjusting the pixel value of the reference image, the image noise problem caused by low ray illumination is solved, and the image denoising effect is improved.

CN119090753AActive Publication Date: 2024-12-06INTELLINDUST INFORMATION TECH (SHENZHEN) CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202310624416.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-06
Estimated Expiration
2043-05-30

AI Technical Summary

Technical Problem

Under low illuminance, the image signal-to-noise ratio collected by the image acquisition device is low, resulting in more noise in the image. The multi-frame denoising method has poor denoising effect due to time differences.

Method used

By acquiring the image to be denoised and the reference image, the pixel offset information between the two is determined, the pixel value of the reference image is adjusted, the difference is reduced, and the denoising process is performed based on the adjusted image.

Benefits of technology

The image denoising effect is improved, the difference between the image to be denoised and the reference image is reduced, and the image quality after denoising is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090753A_ABST
    Figure CN119090753A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image denoising method and device and electronic equipment, and relates to the technical field of image processing, and the method comprises the steps: determining the pixel offset information between a reference image and a to-be-denoised image; wherein the pixel offset information represents coordinate offset of pixel points of the same object in the reference image and the image to be denoised; adjusting the pixel value of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image; and based on the adjusted image, performing denoising processing on the to-be-denoised image to obtain a denoised image. According to the scheme, the image denoising effect can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image denoising method, device and electronic equipment. Background Art

[0002] Under low light illumination conditions, the signal-to-noise ratio of the image captured by the image acquisition device is low, resulting in more noise points in the image, so the image needs to be denoised.

[0003] In the related art, a multi-frame denoising method is used to improve the signal-to-noise ratio of an image. For example, multiple frames of images are simultaneously captured using the same image acquisition device, and the pixel values ​​of each corresponding pixel point of the captured multiple frames of images are weighted averaged to obtain a denoised image.

[0004] However, in the process of multi-frame denoising, there are often large differences between the multi-frame images used. For example, when there are moving objects in the captured scene, due to a certain time difference in the capture moments of different image frames, there will be large differences in the positions of the moving objects in different image frames, which will lead to poor image denoising effect. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an image denoising method, device and electronic device to improve the effect of image denoising. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides an image denoising method, the method comprising:

[0007] Acquire an image to be denoised and a reference image; wherein the reference image is: an image acquired in the scene where the image to be denoised is acquired;

[0008] Determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0009] Based on the pixel offset information, adjusting the pixel value of each pixel point in the reference image to obtain an adjusted image;

[0010] Based on the adjusted image, denoising is performed on the image to be denoised to obtain a denoised image.

[0011] Optionally, the determining pixel offset information between the reference image and the image to be denoised includes:

[0012] Determine the pixel coordinates of the feature points in the image to be denoised as the feature point coordinates;

[0013] Based on the feature point coordinates, determining a first target area from the reference image; wherein the first target area is: an area of ​​a first preset size centered at the pixel point at the feature point coordinates in the reference image;

[0014] Determine, from a plurality of reference pixel points within the first target area, a reference pixel point that matches the feature point as a target pixel point;

[0015] Determine the coordinate offset between the pixel coordinates of the target pixel and the coordinates of the feature point;

[0016] Based on the determined coordinate offset, pixel offset information between the reference image and the image to be denoised is determined.

[0017] Optionally, determining, from a plurality of reference pixels in the first target area, a reference pixel point that matches the feature point as the target pixel point includes:

[0018] In the image to be denoised, multiple pixel points in a second target area are selected as the pixel points to be used corresponding to the feature point; wherein the second target area is: an area of ​​a second preset size centered on the feature point in the image to be denoised;

[0019] For each reference pixel, in the reference image, multiple pixels in a third target area corresponding to the reference pixel are selected as pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is: an area of ​​the second preset size centered on the reference pixel;

[0020] For each reference pixel, the similarity between the reference pixel and the feature point is calculated according to the pixel value of the pixel to be used corresponding to the reference pixel and the pixel value of the pixel to be used corresponding to the feature point;

[0021] From each reference pixel point, a reference pixel point having the highest similarity to the feature point is selected as the target pixel point.

[0022] Optionally, there are multiple feature points in the image to be denoised, and the determined coordinate offset includes: a coordinate offset corresponding to each feature point determined for the feature point;

[0023] The step of determining pixel offset information between the reference image and the image to be denoised based on the determined coordinate offset includes:

[0024] Calculate the average value of the coordinate offsets corresponding to each feature point to obtain the average coordinate offset;

[0025] The average coordinate offset is determined as pixel offset information between the reference image and the image to be denoised.

[0026] Optionally, adjusting the pixel value of each pixel in the reference image based on the pixel offset information includes:

[0027] For each pixel point in the reference image, determining the offset pixel coordinates corresponding to the pixel point according to the pixel offset information;

[0028] For each pixel point in the reference image, the pixel value of the pixel point is modified to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

[0029] Optionally, the performing denoising processing on the image to be denoised based on the adjusted image to obtain the denoised image includes:

[0030] The adjusted image and the image to be denoised are input into a pre-trained image denoising model to obtain a denoised image.

[0031] Optionally, the image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to fuse the input adjusted image and the image data of the image to be denoised to obtain fused image data; and the denoising network is used to denoise the fused image data to obtain a denoised image.

[0032] Optionally, the training method of the image denoising model includes:

[0033] Acquire a sample image and a sample reference image; wherein the sample reference image is: an image acquired in the scene where the sample image is acquired;

[0034] Performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image;

[0035] Determine sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: the coordinate offset of the pixel points of the same object in the noisy reference image and the noisy sample image;

[0036] Based on the sample pixel offset information, adjusting the pixel coordinates of each pixel point in the noisy reference image to obtain an adjusted noisy reference image;

[0037] Inputting the adjusted noisy reference image and the noisy sample image into a neural network model to be trained to obtain a denoised sample image;

[0038] Calculating the model loss of the neural network model to be trained based on the denoised sample image and the sample image;

[0039] Based on the model loss, model parameters of the neural network model to be trained are adjusted.

[0040] In a second aspect, an embodiment of the present invention provides an image denoising device, the device comprising:

[0041] An acquisition module, used to acquire the image to be denoised and a reference image; wherein the reference image is: an image acquired in the scene where the image to be denoised is acquired;

[0042] An offset information determination module is used to determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0043] An adjustment module, configured to adjust the pixel value of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image;

[0044] The denoising module is used to perform denoising processing on the image to be denoised based on the adjusted image to obtain a denoised image.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor is used to implement any of the steps of the above-mentioned image denoising method when executing the program stored in the memory.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned image denoising methods are implemented.

[0049] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image denoising methods.

[0050] Beneficial effects of the embodiments of the present invention:

[0051] The image denoising method provided by the embodiment of the present invention obtains an image to be denoised and a reference image; wherein the reference image is an image captured in the scene where the image to be denoised is captured, and then the pixel offset information between the reference image and the image to be denoised is determined, and the pixel offset information represents: the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised, and based on the pixel offset information, the pixel values ​​of each pixel point in the reference image are adjusted to obtain the adjusted image, and then based on the adjusted image, the image to be denoised is denoised to obtain the denoised image. It can be seen that in this scheme, by determining the pixel offset information between the reference image and the image to be denoised, and the pixel offset information is information representing the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised, and then adjusting the pixel values ​​of each pixel point in the reference image based on the pixel offset information, the difference between the image to be denoised and the reference image can be reduced, and then based on the adjusted image and the image to be denoised, the image to be denoised is denoised, and the effect of image denoising can be improved.

[0052] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0054] Figure 1 A schematic diagram of a first process flow of an image denoising method provided by an embodiment of the present invention;

[0055] Figure 2 The image before translation in the image denoising method provided by an embodiment of the present invention;

[0056] Figure 3 The image after translation in the image denoising method provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of image fusion in the image denoising method provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of image denoising in an image denoising method provided by an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the structure of U-Net in the image denoising method provided by an embodiment of the present invention;

[0060] Figure 7A second flow chart of the image denoising method provided by an embodiment of the present invention;

[0061] Figure 8 A schematic diagram of determining a target pixel in an image denoising method provided by an embodiment of the present invention;

[0062] Fig. 9 A flowchart of a method for training an image denoising model in an image denoising method provided in an embodiment of the present invention;

[0063] Fig.10 A comparison diagram of the effects of the image denoising method provided by an embodiment of the present invention;

[0064] Fig.11 Another effect comparison diagram of the image denoising method provided by an embodiment of the present invention;

[0065] Fig.12 A schematic diagram of the structure of an image denoising device provided by an embodiment of the present invention;

[0066] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0068] In order to improve the denoising effect of an image, an embodiment of the present invention provides an image denoising method, an apparatus and an electronic device. The image denoising method provided by an embodiment of the present invention is first introduced below.

[0069] The image denoising method provided in the embodiment of the present invention can be applied to electronic devices with data processing functions, such as servers, desktop computers, etc. The image denoising method provided in the embodiment of the present invention can also be applied to image acquisition devices, such as smart phones, digital cameras, etc., which are all reasonable. In addition, the image processing method can be implemented by software, hardware, or a combination of software and hardware.

[0070] The image denoising method provided by the embodiment of the present invention may include the following steps:

[0071] Acquire an image to be denoised and a reference image; wherein the reference image is: an image acquired in the scene where the image to be denoised is acquired;

[0072] Determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0073] Based on the pixel offset information, the pixel value of each pixel in the reference image is adjusted to obtain an adjusted image;

[0074] Based on the adjusted image, the image to be denoised is denoised to obtain a denoised image.

[0075] In an embodiment of the present invention, by determining pixel offset information between a reference image and an image to be denoised, where the pixel offset information is information characterizing the coordinate offset of pixel points of the same object in the reference image and the image to be denoised, the pixel values ​​of each pixel point in the reference image are adjusted based on the pixel offset information, thereby reducing the difference between the image to be denoised and the reference image, and then performing denoising on the image to be denoised based on the adjusted image and the image to be denoised, thereby improving the image denoising effect.

[0076] The image denoising method provided by the embodiment of the present invention is introduced below with reference to the accompanying drawings.

[0077] like Figure 1 As shown, the image denoising method provided by the embodiment of the present invention may include the following steps:

[0078] S101, acquiring an image to be denoised and a reference image; wherein the reference image is: an image captured in a scene where the image to be denoised is captured;

[0079] Among them, the image to be denoised and the reference image may be images captured by an image acquisition device, and the acquisition scene of the image to be denoised may include: the position of the image acquisition device when the image to be denoised is acquired, the acquisition angle, etc. The reference image may be an image acquired in the same scene as the acquisition scene of the image to be denoised. The reference image may be single or multiple. The method for acquiring the image to be denoised and the reference image may be: acquiring the image to be denoised and the reference image in real time. Of course, the image to be denoised and the reference image may also be images pre-captured and stored in a designated storage location, and the method for acquiring the image to be denoised and the reference image may also be: reading the image to be denoised and the reference image from the designated storage location.

[0080] Exemplarily, in one implementation, in order to obtain the image to be denoised and the reference image, multiple frames of images can be continuously captured at the same location for the same scene by the same image acquisition device, and one frame of image is selected as the image to be denoised, and the remaining images are used as reference images. The image to be denoised can be selected by random selection, the first frame of image acquisition, etc., which is not limited here. Alternatively, different images can be taken at the same location, at the same angle, and at the same focal length using different image acquisition devices, and one frame of image is selected from the captured images as the image to be denoised, and the remaining images are used as reference images. This is also possible.

[0081] In another implementation, the image to be denoised may be a video frame that needs to be denoised in the video. In this case, the reference image may be a video frame whose distance from the video frame that needs to be denoised is less than or equal to a preset threshold. For example, if the preset threshold is 1, the previous video frame and / or the next video frame of the video frame that needs to be denoised may be selected as the reference image.

[0082] The image to be denoised and the reference image in this embodiment can be any form of image, such as RGB (Red / Green / Blue) image, raw (unprocessed) image, etc. Among them, the data bit width of the raw image is usually 10 bits to 16 bits, which can contain more information than the 8-bit RGB image. In order to ensure a better denoising effect, this embodiment can give priority to using the raw image as the image to be denoised and the reference image.

[0083] S102, determining pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of pixel points of the same object in the reference image and the image to be denoised;

[0084] Since the difference between the acquired image to be denoised and the reference image is usually caused by a moving object in the acquired scene or a shaking of the image acquisition device, it is reflected in the image as the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised. Exemplarily, the pixel offset information can be represented by coordinates, for example, (-6, 8), indicating that the pixel points of the same object in the reference image are offset to the left by 6 pixels and to the bottom by 8 pixels relative to the pixel points in the image to be denoised. The specific method for determining the offset of the reference image relative to the image to be denoised will be specifically introduced in the following embodiments and will not be repeated here.

[0085] If there are multiple reference images, the pixel offset information between the reference image and the image to be denoised may be determined for each reference image.

[0086] S103, adjusting the pixel value of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image;

[0087] In one implementation, after determining the pixel offset information, each pixel in the reference image may be shifted according to the determined pixel offset information. For example, if the determined pixel offset information is (-6, 8), each pixel in the reference image may be shifted 6 pixels to the right and 8 pixels upward. The pixels that are out of the image area after the shift may be removed, and the pixel value of the blank area after the shift may be set to 0. Figure 2-3 As shown, Figure 2 is the image before translation, Figure 3 for Figure 2 The image is obtained by translating each pixel of the image upward and rightward. In the actual processing process, the following steps A1-A2 can be followed:

[0088] Step A1, for each pixel point in the reference image, according to the pixel offset information, determining the offset pixel coordinates corresponding to the pixel point;

[0089] For example, the pixel offset information is (-6, 8), which means that the pixel point of the same object in the reference image is offset by 6 pixels to the left and 8 pixels downward relative to the pixel point in the image to be denoised. Then, for each pixel point in the reference image, the pixel coordinates of the pixel point can be added with (-6, 8) to obtain the offset pixel coordinates corresponding to the pixel point.

[0090] Step A2: for each pixel point in the reference image, modify the pixel value of the pixel point to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

[0091] For example, if the determined pixel offset information is (-6, 8), then a pixel point with pixel coordinates (10, 10) has corresponding offset pixel coordinates (4, 18). The pixel value of the pixel point with pixel coordinates (4, 18) in the reference image is 100, so the pixel value of the pixel point with pixel coordinates (10, 10) can be modified to 100. If the offset pixel coordinates corresponding to a pixel point are not in the reference image, the pixel value of the pixel point can be modified to 0.

[0092] It can be seen that by determining the offset pixel coordinates corresponding to each pixel point in the reference image according to the pixel offset information, and then modifying the pixel value of the pixel point to the pixel value of the pixel point at the offset pixel coordinates corresponding to the pixel point, the pixels representing the same object in the reference image and the image to be denoised can be aligned, thereby reducing the difference between the reference image and the image to be denoised, and providing a basis for subsequent denoising processing based on the adjusted image and the image to be denoised.

[0093] If there are multiple reference images, the pixel coordinates of each pixel in each reference image can be adjusted based on the offset of the reference image relative to the image to be denoised to obtain an adjusted image corresponding to the reference image.

[0094] S104, based on the adjusted image, performing denoising processing on the image to be denoised to obtain a denoised image.

[0095] In this step, after obtaining the adjusted image, any multi-frame denoising method can be used for denoising. In one implementation, the adjusted image and the image to be denoised can be fused to obtain a fused image. For example, the pixel values ​​at the corresponding pixel coordinates of the adjusted image and the image to be denoised can be weighted averaged to obtain a fused image. After obtaining the fused image, any image denoising method can be used to denoise the fused image, such as Gaussian denoising, wavelet denoising, etc., or a pre-trained denoising network can be used to denoise the fused image, such as U-Net (U-type network).

[0096] In one implementation, the adjusted image and the image to be denoised can also be input into a pre-trained image denoising model to obtain a denoised image. The image denoising model can include: a fusion network and a denoising network; the fusion network is used to fuse the input image data of the adjusted image and the image to be denoised to obtain fused image data; the denoising network is used to denoise the fused image data and output the denoised image.

[0097] For example, Figure 4 As shown, Figure 4In the above, t0 and t1 represent the adjusted image and the image to be denoised, respectively, and t0 and t1 are raw images. First, t0 and t1 are split into RGGB (Red / Green / Green / Blue) 4-channel image data, then the split t0 and t1 are channel-joined to obtain 8-channel image data, and finally the obtained 8-channel image data is input into the preset fusion network, and 4-channel image data is output as fused image data. The fusion network can be composed of two convolutional layers, the convolution kernel size of the two convolutional layers is 3x3, the first convolutional layer can increase the dimension of 8-channel data to 64 channels, and the second convolutional layer can reduce the dimension of 64-channel data to 4 channels. The preset fusion network can also be composed of four convolutional layers, the first convolutional layer increases the dimension of 8-channel data to 64 channels, the second convolutional layer increases the dimension of 64-channel data to 128 channels, the third convolutional layer reduces the dimension of 128-channel data to 64 channels, and the fourth convolutional layer reduces the dimension of 64-channel data to 4 channels. The composition of the above fusion network is only an example and is not specifically limited. After obtaining the fused image data, that is, the 4-channel image data output by the fusion network, such as Figure 5 As shown, the 4-channel image data output by the above fusion network can be input into a pre-trained denoising network for denoising, and then the denoised image data is restored to the raw image to obtain the denoised image. The denoising network can be any denoising network, such as Figure 6 The U-Net shown in FIG. 1 includes multiple 3x3 convolutional layers, i.e., convolutional layers with a convolution kernel size of 3x3, for performing convolution processing on image data, and multiple ReLUs (linear rectification units) for performing linear rectification on image data. Figure 6 The unfilled arrow to the right in the figure represents a combination of a 3x3 convolutional layer and ReLU, where some of the 3x3 convolutional layers are also used to upscale the image data, including: upscaling 4 channels to 32 channels, 32 channels to 64 channels, 64 channels to 128 channels, 128 channels to 256 channels, and 256 channels to 512 channels; it also includes four 2x2 maximum pooling layers, Figure 6 Indicated by a downward arrow, it is used to downsample the input image data; and 4 2x2 transposed convolution layers, that is, transposed convolution layers with a convolution kernel size of 2x2, Figure 6 Indicated by an upward arrow, it is used to upsample the downsampled image data and reduce the dimension, including reducing the 512 channels to 256 channels, reducing the 256 channels to 128 channels, reducing the 128 channels to 64 channels, and reducing the 64 channels to 32 channels; merging layer, Figure 6 Indicated by a right-pointing black filled arrow, it is used to add two sets of data, for example, Figure 6The 64-channel data is reduced to 32 channels by the 2x2 transposed convolution layer and then added to the 32-channel data before downsampling. The last layer of the U-Net is a 1x1 convolution layer, that is, a convolution layer with a convolution kernel size of 1x1. Figure 6 In the figure, the rightward-pointing filled-hatched arrow is used to reduce the dimension of the 32-channel image data into 4 channels.

[0098] In this embodiment, by determining the pixel offset information between the reference image and the image to be denoised, and the pixel offset information is the information characterizing the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised, the pixel value of each pixel point in the reference image is adjusted based on the pixel offset information, so that the difference between the image to be denoised and the reference image can be reduced, and then based on the adjusted image and the image to be denoised, the image to be denoised is denoised, which can improve the image denoising effect.

[0099] Alternatively, in another embodiment of the present invention, Figure 7 As shown, the image denoising method provided by the embodiment of the present invention may also include the following steps:

[0100] S701, obtaining an image to be denoised and a reference image;

[0101] This step is similar to the above step S101 and will not be described again here.

[0102] S702, determining the pixel coordinates of the feature points in the image to be denoised as the feature point coordinates;

[0103] The determined feature points may be single or multiple. In one implementation, a feature point extraction algorithm may be used to determine the feature points in the image to be denoised, or a single or a preset number of pixel points may be randomly extracted from the image to be denoised as feature points, and then the pixel coordinates of the feature points may be determined as feature point coordinates. The feature point extraction algorithm may be a SIFT (Scale-Invariant Feature Transform) algorithm, a SURF (Speeded-Up Robust Feature) algorithm, an ORB (Oriented FAST and RotatedBRIEF, an algorithm for fast feature point extraction and description) algorithm, a Harris (corner detection) algorithm, and the like.

[0104] In another implementation, the image to be denoised can be evenly divided into multiple blocks, and the middle point of each block is taken as the matching point. For example, a frame of the image to be denoised with a resolution of 1920*1080 is evenly divided into 16*10, that is, 160 blocks, and then the center point of each block is determined as a feature point, and then the pixel coordinates of the feature point are determined as the feature point coordinates.

[0105] S703, determining a first target area from the reference image based on the feature point coordinates; wherein the first target area is: an area of ​​a first preset size centered at the pixel point at the feature point coordinates in the reference image;

[0106] The first target area may be a set of multiple pixels, and the first preset size may be preset based on experience and requirements. For example, the first preset size may be set to 21*21. After determining the coordinates of the feature point, the first target area may be determined from the reference image with the pixel at the feature point coordinate as the center. For example, if the feature point coordinates are (10, 10) and the first preset size is 21*21, a set of 21*21 pixels may be selected with the pixel at the pixel coordinate (10, 10) in the reference image as the center as the first target area.

[0107] S704, determining a reference pixel point that matches the feature point from a plurality of reference pixel points in the first target area as a target pixel point;

[0108] In one implementation, the pixel features of the feature point can be extracted first. For example, the pixel features of the feature point can be extracted using an image gradient algorithm. Then, for each reference pixel, the pixel features of the reference pixel are extracted, and the similarity between the pixel features of the reference pixel and the pixel features of the feature point is calculated. Then, the reference pixel with the highest pixel feature similarity to the feature point is determined as the target pixel.

[0109] In another implementation, the above-mentioned step of determining, from among the reference pixels, a reference pixel matching the feature point as the target pixel may include the following steps B1-B4:

[0110] B1, in the image to be denoised, multiple pixel points in the second target area are selected as the pixel points to be used corresponding to the feature points; wherein the second target area is: an area of ​​the image to be denoised with the feature point as the center and a second preset size;

[0111] Among them, the second preset size can also be pre-set according to experience and needs. For example, the second preset size can be set to 9*9, indicating that the second target area is a set of 9*9 pixels centered on the feature point, and the 9*9 pixels are used as the pixels to be used corresponding to the feature point.

[0112] B2, for each reference pixel, in the reference image, select multiple pixels in the third target area corresponding to the reference pixel as the pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is: an area of ​​a second preset size centered on the reference pixel;

[0113] In this step, for each reference pixel point, the same operation as the above step B1 can be performed, and the number of unused pixel points corresponding to the reference pixel point is the same as the number of unused pixel points corresponding to the above feature points.

[0114] B3, for each reference pixel, according to the pixel value of the to-be-used pixel corresponding to the reference pixel and the pixel value of the to-be-used pixel corresponding to the feature point, calculate the similarity between the reference pixel and the feature point;

[0115] In this step, for each reference pixel, the pixels to be used corresponding to the reference pixel are taken as a set, and the pixels to be used corresponding to the feature point are taken as another set. Then, the SAD (sum of absolute differences) method, the MAD (mean absolute difference) method, the SSD (sum of squared errors) method, the MSD (mean sum of squared errors) method, etc. can be used to calculate the similarity between the reference pixel and the feature point based on the pixel value of the pixel to be used corresponding to the reference pixel and the pixel value of the pixel to be used corresponding to the feature point.

[0116] B4, from each reference pixel point, select the reference pixel point with the highest similarity to the feature point as the target pixel point.

[0117] After determining the similarity between each reference pixel and the feature point, the reference pixel with the highest similarity to the feature point may be selected as the target pixel.

[0118] In this implementation, multiple pixels within a second preset size centered on a reference pixel and multiple pixels within a second size range centered on a feature point are utilized to calculate the similarity between the reference pixel and the feature point. This can fully utilize the pixel information around the reference pixel and the feature point, thereby improving the accuracy of calculating the similarity.

[0119] Figure 8 The following is a schematic diagram showing the process of determining the target pixel points, wherein t0 and t1 represent the adjusted image and the image to be denoised, respectively. The points in t1 represent feature points, the points in t0 represent target pixels, the large box represents the first target area, and the small box represents the second target area.

[0120] S705, determining the coordinate offset of the target pixel point relative to the feature point coordinate;

[0121] In this step, after determining the target pixel point that matches the feature point coordinates, the coordinate offset of the target pixel point coordinates relative to the feature point coordinates can be determined. For example, if the feature point coordinates are (10, 10) and the target pixel point coordinates are (15, 16), then the coordinate offset of the target pixel point coordinates relative to the feature point coordinates can be determined to be (5, 6).

[0122] S706: Determine pixel offset information between the reference image and the image to be denoised based on the determined coordinate offset.

[0123] In this step, when there is only one feature point in the image to be denoised, the determined coordinate offset can be directly used as the pixel offset information between the reference image and the image to be denoised. When there are multiple feature points in the image to be denoised, the determined coordinate offset includes: the coordinate offset corresponding to each feature point determined for the feature point. In this case, the above steps S702-S705 can be performed for each feature point to obtain the coordinate offset corresponding to each feature point. After that, the average value of the coordinate offsets corresponding to each feature point can be calculated to obtain the average coordinate offset, and then the average coordinate offset is determined as the pixel offset information between the reference image and the image to be denoised. By determining the pixel offset information for multiple feature points, the accuracy of the determined pixel offset information can be improved, thereby further improving the effect of image denoising.

[0124] S707, adjusting the pixel value of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image;

[0125] S708, based on the adjusted image, performing denoising processing on the image to be denoised to obtain a denoised image.

[0126] The above steps S707-S708 are similar to the above steps S103-S104 and will not be repeated here.

[0127] In this embodiment, the effect of image denoising can be improved. Further, by determining the pixel coordinates of the feature points in the image to be denoised as the feature point coordinates; based on the feature point coordinates, determining the first target area from the reference image; wherein the first target area is: an area of ​​a first preset size centered on the pixel point at the feature point coordinates in the reference image; determining the reference pixel point that matches the feature point from multiple reference pixel points in the first target area as the target pixel point; determining the coordinate offset between the pixel coordinates of the target pixel point and the feature point coordinates; based on the determined coordinate offset, determining the pixel offset information between the reference image and the image to be denoised, the offset of the reference image relative to the image to be denoised can be accurately determined.

[0128] Alternatively, in another embodiment of the present invention, Fig. 9 As shown, the training method of the above-mentioned image denoising model includes the following steps S901-S907:

[0129] S901, acquiring a sample image and a sample reference image; wherein the sample reference image is: an image acquired in a scene where the sample image is acquired;

[0130] The method of acquiring the sample image and the sample reference image may be similar to the above step S101. In addition, the sample image and the sample reference image may be images with a high signal-to-noise ratio acquired under sufficient light conditions, which may further improve the denoising effect of the embodiment of the present invention. There may be multiple sample images to ensure a better model training effect.

[0131] S902, performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image;

[0132] In this step, the sample image and the sample reference image may be subjected to noise processing, such as adding Gaussian noise, salt and pepper noise, etc.

[0133] S903, determining sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: the coordinate offset of the pixel points of the same object in the noisy reference image and the noisy sample image;

[0134] S904, adjusting the pixel coordinates of each pixel point in the noisy reference image based on the sample pixel offset information to obtain an adjusted noisy reference image;

[0135] S905, inputting the sample image into the neural network model to be trained to obtain a denoised sample image; wherein the neural network model to be trained includes: a fusion network to be trained and a denoising network to be trained;

[0136] The above steps S903-S905 are similar to the above process of denoising the denoised image, and will not be repeated here.

[0137] S906, calculating the model loss of the neural network model to be trained based on the denoised sample image and the sample image;

[0138] In this step, the model loss can be calculated using a preset loss function. The preset loss function can be any loss function, such as a cross entropy loss function, etc., which is not limited here.

[0139] S907, based on the model loss, adjusting the model parameters in the fusion network to be trained and the denoising network to be trained.

[0140] In this step, the gradient descent method can be used to adjust the model parameters, and when training the model, steps S901-S907 can be performed multiple times to obtain different sample images and sample reference images to adjust the model parameters in the neural network model to be trained until the neural network model converges to obtain a trained denoising model. In the case where the denoising network includes a fusion network and a denoising network, the fusion network and the denoising network can be jointly trained to further improve the image denoising effect.

[0141] like Fig.10 As shown, Fig.10 The comparison of image denoising effects by different processing methods is demonstrated. The images from top to bottom are: the image to be denoised, the result of single-frame denoising, the denoising effect of five frames of images directly input into the denoising model, and the denoising effect of five frames of images fused using an embodiment of the present invention. Fig.11 Shown Fig.10 The partial magnification comparison of the image in the figure, from left to right, are: the image to be denoised, the result of single-frame denoising, the denoising effect of five-frame images directly input into the denoising model, and the denoising effect of five-frame image fusion using the embodiment of the present invention. It can be seen that after the image denoising method provided by the embodiment of the present invention is used for image denoising, the details and image perception are significantly improved.

[0142] In this embodiment, the effect of image denoising can be improved. Further, by performing noise processing on the sample image and the sample reference image, a noisy sample image and a noisy reference image are obtained, and then the sample pixel offset information between the noisy reference image and the noisy sample image is obtained, and the sample pixel offset information is information that characterizes the coordinate offset of the pixel points of the same object in the sample image and the sample reference image, so that the pixel values ​​of each pixel point in the sample reference image are adjusted based on the sample pixel offset information, which can reduce the difference between the noisy reference image and the noisy sample image, and then the adjusted noisy reference image and the noisy sample image are input into the neural network model to be trained to obtain a denoised sample image, and the model loss of the neural network model to be trained is calculated based on the denoised sample image and the sample image, and based on the model loss, the model parameters of the neural network model to be trained are adjusted to obtain an image denoising model, which provides a basis for implementing denoising of an image using an image denoising model in the embodiment of the present application.

[0143] Based on the same inventive concept, the embodiment of the present invention also provides an image denoising device, such as Fig.12 As shown, the device comprises:

[0144] The acquisition module 1201 is used to acquire the image to be denoised and a reference image; wherein the reference image is: an image acquired in the scene where the image to be denoised is acquired;

[0145] The offset information determination module 1202 is used to determine the pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised;

[0146] An adjustment module 1203, configured to adjust the pixel value of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image;

[0147] The denoising module 1204 is configured to perform denoising processing on the image to be denoised based on the adjusted image to obtain a denoised image.

[0148] Optionally, the denoising module includes:

[0149] A fusion submodule, used for fusing the adjusted image with the image to be denoised to obtain a fused image;

[0150] The denoising submodule is used to perform denoising on the fused image to obtain a denoised image.

[0151] Optionally, the adjustment module is specifically configured to translate each pixel in the reference image according to the determined offset.

[0152] Optionally, the offset determination module includes:

[0153] A feature point coordinate determination submodule, used to determine the pixel coordinates of the feature points in the image to be denoised;

[0154] A first target region determination submodule is configured to determine a first target region from the reference image based on the feature point coordinates; wherein the first target region is: a region of a first preset size centered at a pixel point at the feature point coordinates in the reference image;

[0155] A target pixel point determination submodule, used to determine a reference pixel point matching the feature point from a plurality of reference pixel points in the first target area as a target pixel point;

[0156] A coordinate offset determination submodule, used to determine the coordinate offset between the pixel coordinates of the target pixel point and the coordinates of the feature point;

[0157] The image offset determination submodule is used to determine the pixel offset information between the reference image and the image to be denoised based on the determined coordinate offset.

[0158] Optionally, the target pixel determination submodule includes:

[0159] A first selection unit is used to select a plurality of pixel points in a second target area in the image to be denoised as the pixel points to be used corresponding to the feature point; wherein the second target area is: an area of ​​a second preset size centered on the feature point in the image to be denoised;

[0160] A second selection unit is used to select, for each reference pixel point, a plurality of pixel points in a third target area corresponding to the reference pixel point in the reference image as the pixel points to be used corresponding to the reference pixel point; wherein the third target area corresponding to the reference pixel point is: an area of ​​the second preset size centered on the reference pixel point;

[0161] A calculation unit, configured to calculate, for each reference pixel, a similarity between the reference pixel and the feature point according to a pixel value of a pixel to be used corresponding to the reference pixel and a pixel value of a pixel to be used corresponding to the feature point;

[0162] The third selection unit is used to select, from among the reference pixel points, a reference pixel point with the highest similarity to the feature point as a target pixel point.

[0163] Optionally, there are multiple feature points in the image to be denoised, and the determined coordinate offset includes: the coordinate offset corresponding to each feature point; the image offset determination submodule is specifically used to: calculate the average value of the coordinate offsets corresponding to each feature point to obtain an average coordinate offset; and determine the average coordinate offset as the pixel offset information between the reference image and the image to be denoised.

[0164] Optionally, the adjustment module includes:

[0165] An offset pixel coordinate determination submodule, used for determining, for each pixel point in the reference image, an offset pixel coordinate corresponding to the pixel point according to the pixel offset information;

[0166] The modification submodule is used to modify, for each pixel point in the reference image, the pixel value of the pixel point to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

[0167] Optionally, the denoising module is specifically used to:

[0168] The adjusted image and the image to be denoised are input into a pre-trained image denoising model to obtain a denoised image.

[0169] Optionally, the image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to fuse the input adjusted image and the image data of the image to be denoised to obtain fused image data; the denoising network is used to denoise the fused image data and output a denoised image.

[0170] Optionally, the training method of the image denoising model includes:

[0171] Acquire a sample image and a sample reference image; wherein the sample reference image is: an image acquired in the scene where the sample image is acquired;

[0172] Performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image;

[0173] Determine sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: the coordinate offset of the pixel points of the same object in the noisy reference image and the noisy sample image;

[0174] Based on the sample pixel offset information, adjusting the pixel coordinates of each pixel point in the noisy reference image to obtain an adjusted noisy reference image;

[0175] Inputting the adjusted noisy reference image and the noisy sample image into a neural network model to be trained to obtain a denoised sample image;

[0176] Calculating the model loss of the neural network model to be trained based on the denoised sample image and the sample image;

[0177] Based on the model loss, model parameters of the neural network model to be trained are adjusted.

[0178] In this embodiment, by determining the pixel offset information between the reference image and the image to be denoised, and the pixel offset information is the information characterizing the coordinate offset of the pixel points of the same object in the reference image and the image to be denoised, the pixel value of each pixel point in the reference image is adjusted based on the pixel offset information, so that the difference between the image to be denoised and the reference image can be reduced, and then based on the adjusted image and the image to be denoised, the image to be denoised is denoised, which can improve the image denoising effect.

[0179] The embodiment of the present invention further provides an electronic device, such as Fig.13 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0180] Memory 1303, used for storing computer programs;

[0181] The processor 1301 is used to implement the steps of the above-mentioned image denoising method when executing the program stored in the memory 1303.

[0182] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0183] The communication interface is used for communication between the above electronic device and other devices.

[0184] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0185] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0186] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image denoising methods are implemented.

[0187] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any one of the image denoising methods in the above embodiments.

[0188] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0189] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0190] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An image denoising method, characterized in that: The method comprises: Acquire an image to be denoised and a reference image; wherein the reference image is: an image acquired in the scene where the image to be denoised is acquired; Determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: coordinate offset of pixel points of the same object in the reference image and the image to be denoised; Based on the pixel offset information, adjusting the pixel value of each pixel point in the reference image to obtain an adjusted image; Based on the adjusted image, denoising is performed on the image to be denoised to obtain a denoised image.

2. The method according to claim 1, characterized in that The determining pixel offset information between the reference image and the image to be denoised includes: Determine the pixel coordinates of the feature points in the image to be denoised as the feature point coordinates; Based on the feature point coordinates, determining a first target area from the reference image; wherein the first target area is: an area of ​​a first preset size centered at the pixel point at the feature point coordinates in the reference image; Determine, from a plurality of reference pixel points within the first target area, a reference pixel point that matches the feature point as a target pixel point; Determine the coordinate offset between the pixel coordinates of the target pixel and the coordinates of the feature point; Based on the determined coordinate offset, pixel offset information between the reference image and the image to be denoised is determined.

3. The method according to claim 2, characterized in that The step of determining, from a plurality of reference pixels in the first target area, a reference pixel matching the feature point as a target pixel, includes: In the image to be denoised, multiple pixel points in a second target area are selected as the pixel points to be used corresponding to the feature point; wherein the second target area is: an area of ​​a second preset size centered on the feature point in the image to be denoised; For each reference pixel, in the reference image, multiple pixels in a third target area corresponding to the reference pixel are selected as pixels to be used corresponding to the reference pixel; wherein the third target area corresponding to the reference pixel is: an area of ​​the second preset size centered on the reference pixel; For each reference pixel, the similarity between the reference pixel and the feature point is calculated according to the pixel value of the pixel to be used corresponding to the reference pixel and the pixel value of the pixel to be used corresponding to the feature point; From each reference pixel point, a reference pixel point having the highest similarity to the feature point is selected as the target pixel point.

4. The method according to claim 2, characterized in that: There are multiple feature points in the image to be denoised, and the determined coordinate offset includes: a coordinate offset corresponding to each feature point determined for the feature point; The step of determining pixel offset information between the reference image and the image to be denoised based on the determined coordinate offset includes: Calculate the average value of the coordinate offsets corresponding to each feature point to obtain the average coordinate offset; The average coordinate offset is determined as pixel offset information between the reference image and the image to be denoised.

5. The method according to any one of claims 1 to 4, characterized in that: The step of adjusting the pixel value of each pixel in the reference image based on the pixel offset information includes: For each pixel point in the reference image, determining the offset pixel coordinates corresponding to the pixel point according to the pixel offset information; For each pixel point in the reference image, the pixel value of the pixel point is modified to the pixel value of the pixel point at the offset pixel coordinate corresponding to the pixel point.

6. The method according to claim 1, characterized in that The step of performing denoising on the image to be denoised based on the adjusted image to obtain a denoised image comprises: The adjusted image and the image to be denoised are input into a pre-trained image denoising model to obtain a denoised image.

7. The method according to claim 6, characterized in that The image denoising model includes: a fusion network and a denoising network; wherein the fusion network is used to fuse the input adjusted image and the image data of the image to be denoised to obtain fused image data; the denoising network is used to denoise the fused image data to obtain a denoised image.

8. The method according to claim 6 or 7, characterized in that: The training method of the image denoising model includes: Acquire a sample image and a sample reference image; wherein the sample reference image is: an image acquired in the scene where the sample image is acquired; Performing noise processing on the sample image and the sample reference image to obtain a noisy sample image and a noisy reference image; Determine sample pixel offset information between the noisy reference image and the noisy sample image; wherein the sample pixel offset information represents: the coordinate offset of the pixel points of the same object in the noisy reference image and the noisy sample image; Based on the sample pixel offset information, adjusting the pixel coordinates of each pixel point in the noisy reference image to obtain an adjusted noisy reference image; Inputting the adjusted noisy reference image and the noisy sample image into a neural network model to be trained to obtain a denoised sample image; Calculating the model loss of the neural network model to be trained based on the denoised sample image and the sample image; Based on the model loss, model parameters of the neural network model to be trained are adjusted.

9. An image denoising device, characterized in that: The device comprises: An acquisition module, used to acquire the image to be denoised and a reference image; wherein the reference image is: an image acquired in the scene where the image to be denoised is acquired; An offset information determination module is used to determine pixel offset information between the reference image and the image to be denoised; wherein the pixel offset information represents: the coordinate offset of pixel points of the same object in the reference image and the image to be denoised; An adjustment module, configured to adjust the pixel value of each pixel point in the reference image based on the pixel offset information to obtain an adjusted image; The denoising module is used to perform denoising processing on the image to be denoised based on the adjusted image to obtain a denoised image.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Image denoising method, device and system

    CN108898567A

  • Method and device for generating image denoising model

    CN110288547A

  • Image denoising method and device

    CN111275653A

  • Image processing method and device and storage medium

    CN111741187A

  • Image fusion denoising method and system based on U-Net and pyramid module

    CN112801909A