Image noise reduction method and device, computer equipment and storage medium

Abstract

The invention discloses an image denoising method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the compression processing of a to-be-denoised image, and obtaining a compressed to-be-denoised image; performing noise reduction processing on the compressed image to be subjected to noise reduction by using a noise reduction model to obtain an initial noise reduction image; performing decompression processing on the initial noise reduction image to obtain a decompressed initial noise reduction image; and adding the to-be-denoised image and the decompressed initial denoised image to obtain a target denoised image. According to the method, the compressed image is directly processed by the noise reduction model in the NPU and then is quickly decompressed into the original size, and the brightness information of the original image is reserved after the decompressed initial noise reduction image and the image to be subjected to noise reduction are added, so that the image definition is ensured. Besides, the image to be denoised is compressed, so that storage bandwidth and calculation hardware resources can be saved, and the hardware resource demand is effectively reduced on the premise of ensuring the image effect of the gazing point region.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an image noise reduction method, apparatus, computer equipment, and storage medium. Background Art

[0002] In computer image processing and computer graphics, image denoising refers to the process of removing noise from digital images.

[0003] Currently, image denoising primarily utilizes neural network models. For example, the DNCNN denoising model, which introduces residual learning and batch normalization, enhances the network's image perception by removing pooling layers and setting appropriate convolution kernel sizes. This approach achieves impressive denoising capabilities in both blind and non-blind denoising scenarios, and experiments have shown significant improvements in generalization compared to traditional algorithms. The FFDNet denoising model samples the input image into multiple sub-images, superimposing them channel-wise before inputting them into the network for training. This reduces network parameters and computational efficiency while ensuring optimal results.

[0004] However, using the above method for noise reduction may result in a problem of image quality degradation, resulting in poor noise reduction effect. Summary of the Invention

[0005] The main purpose of this application is to provide an image noise reduction method, apparatus, computer equipment and storage medium to solve the problem of image quality degradation in related technologies.

[0006] To achieve the above objectives, in a first aspect, the present application provides an image noise reduction method, comprising:

[0007] Compressing the image to be denoised to obtain a compressed image to be denoised;

[0008] Performing denoising on the compressed image to be denoised using a denoising model to obtain an initial denoised image, wherein the initial denoised image at least includes denoising information of each pixel;

[0009] Decompressing the initial denoised image to obtain a decompressed initial denoised image;

[0010] The image to be denoised and the decompressed initial denoised image are added together to obtain the target denoised image.

[0011] In one possible implementation, the noise reduction model includes a first level, a second level, and a third level;

[0012] Both the first and second levels include the first convolutional layer, the second convolutional layer, the deconvolutional layer, and the residual connection block;

[0013] The third level includes the first convolutional layer, the second convolutional layer and the deconvolutional layer;

[0014] Among them, the first convolution layer is used to reduce the size of the image, the second convolution layer is used to extract features from the image, the deconvolution layer is used to enlarge the size of the image, and the residual connection block is used to add at least two images connected by residuals.

[0015] In one possible implementation, the denoising model is used to perform denoising on the compressed image to be denoised to obtain an initial denoised image, including:

[0016] Performing feature extraction on the compressed image to be denoised to obtain a feature image to be denoised;

[0017] Determining a first denoised image corresponding to a first level;

[0018] The feature image to be denoised is added to the first denoised image to obtain an initial denoised image.

[0019] In one possible implementation, determining a first denoised image corresponding to a first level includes:

[0020] determining a second denoised image corresponding to the second level;

[0021] Performing a size reduction process on the image to be denoised to obtain a reduced image to be denoised;

[0022] Performing feature extraction on the reduced image to be denoised to obtain a first initial feature image corresponding to the first level;

[0023] Adding the second denoised image and the first initial feature image to obtain a first target feature image corresponding to the first level;

[0024] The first target feature image is enlarged to obtain a first denoised image.

[0025] In one possible implementation, determining the second denoised image corresponding to the second level includes:

[0026] determining a third denoised image corresponding to the third level;

[0027] Performing feature extraction on the reduced image to be denoised to obtain a second initial feature image corresponding to the second level;

[0028] Adding the third denoised image to the second initial feature image to obtain a second target feature image corresponding to the second level;

[0029] The second target feature image is enlarged to obtain a second denoised image.

[0030] In one possible implementation, determining a third denoised image corresponding to the third level includes:

[0031] Performing feature extraction on the reduced image to be denoised to obtain a third initial feature image corresponding to the third level;

[0032] The third initial feature image is enlarged to obtain a third denoised image.

[0033] In a possible implementation, before performing denoising on the compressed image to be denoised using the denoising model to obtain the initial denoised image, the method further includes:

[0034] Obtain a neural network model and a training set, wherein the training set includes an input image and an output image, the input image is an image to be denoised for training, and the output image is an original image captured by an ultra-high-definition camera, and the original image retains the original data;

[0035] The neural network model is trained using the input image and the output image, and the denoising model is obtained when the value of the loss function in the neural network model reaches a preset threshold.

[0036] In one possible implementation, the loss function is determined by the product of the first weight and the mean absolute error loss function, the product of the second weight and the asymmetric loss function, and the product of the third weight and the content consistency loss function.

[0037] In one possible implementation, the asymmetric loss function is determined by the mean absolute error of the distance between the input image and the original image captured by the ultra-high-definition camera, and a judgment function, wherein the judgment function is used to characterize the relationship function between the input image, the target denoised image, and the original image captured by the ultra-high-definition camera.

[0038] In one possible implementation, the content consistency loss function is determined by a first mean absolute error, a second mean absolute error, and a third mean absolute error. The first mean absolute error is used to characterize the mean absolute error of the distance between a first target denoised image output after the first input image is processed by the neural network model and a second target denoised image output after the second input image is processed by the neural network model. The second mean absolute error is used to characterize the mean absolute error of the distance between the second target denoised image output after the second input image is processed by the neural network model and a third target denoised image output after the third input image is processed by the neural network model. The third mean absolute error is used to characterize the mean absolute error of the distance between the first target denoised image output after the first input image is processed by the neural network model and the third target denoised image output after the third input image is processed by the neural network model.

[0039] In one possible implementation, the denoising model is configured in a neural network processor, and the image to be denoised is configured in an image signal processor;

[0040] The method also includes:

[0041] Use eye tracking equipment to obtain eye movement information;

[0042] Performing gaze point compression on the image to be denoised in the image signal processor according to the eye movement information to obtain a compressed image to be denoised;

[0043] The compressed image to be denoised is subjected to denoising processing by a denoising model in a neural network processor to obtain an initial denoised image;

[0044] Decompressing the initial denoised image to obtain a decompressed initial denoised image;

[0045] The image to be denoised and the decompressed initial denoised image are added together to obtain the target denoised image.

[0046] In a second aspect, an embodiment of the present invention provides an image noise reduction device, comprising:

[0047] A compression module is used to compress the image to be denoised to obtain a compressed image to be denoised;

[0048] A denoising module is configured to perform denoising on the compressed image to be denoised using a denoising model to obtain an initial denoised image, wherein the initial denoised image at least includes denoising information of each pixel point;

[0049] A decompression module is used to decompress the initial denoised image to obtain a decompressed initial denoised image;

[0050] The addition processing module is used to perform addition processing on the image to be denoised and the decompressed initial denoised image to obtain a target denoised image.

[0051] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above image denoising methods are implemented.

[0052] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above image noise reduction methods are implemented.

[0053] Embodiments of the present invention provide an image denoising method, apparatus, computer device, and storage medium, comprising: first compressing an image to be denoised to obtain a compressed image to be denoised; then denoising the compressed image to be denoised using a denoising model to obtain an initial denoised image, wherein the initial denoised image includes at least denoising information for each pixel; then decompressing the initial denoised image to obtain a decompressed initial denoised image; and finally adding the image to be denoised and the decompressed initial denoised image to obtain a target denoised image. The present invention compresses the denoised image so that the compressed image is directly decompressed to its original size after being processed by the denoising model in the neural processing unit (NPU), thereby avoiding degradation in the image quality of the target denoised image and improving the scaling effect of the target denoised image. Furthermore, compressing the denoised image can save storage bandwidth and computing hardware resources, effectively reducing hardware resource requirements while ensuring image quality in the gaze point area. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings that constitute part of this application are used to provide a further understanding of this application and make other features, objects and advantages of this application more apparent. The illustrative embodiment drawings of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0055] Figure 1 This is a flowchart of an implementation method of an image noise reduction method provided by an embodiment of the present invention;

[0056] Figure 2 is a flowchart of another image noise reduction method provided by an embodiment of the present invention;

[0057] Figure 3 is a structural diagram of a noise reduction model provided by an embodiment of the present invention;

[0058] Figure 4 is a flowchart of another image noise reduction method provided by an embodiment of the present invention;

[0059] Figure 5 1 is a structural diagram of an image noise reduction device provided by an embodiment of the present invention;

[0060] Figure 6 Schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0062] The terms "first," "second," "third," "fourth," and so forth (if any) in the description and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0063] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] It should be understood that in the present invention, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0065] It should be understood that in the present invention, "multiple" refers to two or more. "And / or" is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0066] It should be understood that, in the present invention, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0067] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0068] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0069] Furthermore, in order to make the objectives, technical solutions and advantages of the present invention more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0070] In computer image processing and computer graphics, image denoising refers to the process of removing noise from digital images.

[0071] At present, the image noise reduction technology mainly uses neural network models, such as deep learning neural network models. Figure 1 As shown in the figure, before using the deep learning neural network model for image inference, the deep learning network framework needs to be trained with a data set. After the training is completed, the deep learning neural network (inference) model is obtained. Among them, the deep learning neural network (inference) model is configured in the NPU.

[0072] The image to be denoised is input into the deep learning neural network (inference) model in the NPU, and the denoised image is output.

[0073] However, using the above method for noise reduction may result in a problem of image quality degradation, resulting in poor noise reduction effect.

[0074] Therefore, the present application provides an image noise reduction method to solve the above problems.

[0075] In one embodiment, Figure 2 As shown, a method for image noise reduction is provided, comprising the following steps:

[0076] Step S201: compressing the image to be denoised to obtain a compressed image to be denoised.

[0077] The images to be denoised include images in various formats, including but not limited to Bayer, RAW, RGB, and YUV formats.

[0078] Before compressing the image to be denoised, it is necessary to first obtain the eye movement information of the user when observing the image, then convert the eye movement information into coordinates to obtain the center coordinates corresponding to the eye movement information, and then determine the non-gaze area of ​​the image to be denoised based on the center coordinates. The non-gaze area of ​​the image to be denoised is compressed to obtain the compressed image to be denoised.

[0079] Eye movement information can be obtained through eye tracking devices or other devices. Eye movement information mainly includes the user's gaze position on the screen or image and the corresponding timestamp. This application mainly uses the MCU to convert the eye movement information into coordinates. Through this conversion, the user's gaze position on the image, that is, the center coordinates, can be obtained.

[0080] Among them, the non-gaze area of ​​the image to be denoised is determined according to the central coordinates, and the non-gaze area of ​​the image to be denoised is compressed to obtain a compressed image to be denoised. It is necessary to first determine the coordinates of each area in the non-gaze area of ​​the image to be denoised according to the central coordinates, and then form the coordinates of each area in the image to be denoised by the central coordinates and the coordinates of each area in the non-gaze area. Finally, based on the coordinates of each area in the image to be denoised, the image to be denoised is compressed to obtain a compressed image to be denoised.

[0081] Among them, based on the coordinates of each area in the image to be denoised, the denoised image is compressed to obtain the compressed image to be denoised. It is necessary to first determine the coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the image to be denoised, and then look up the table according to the gaze area, center coordinates, non-gaze area and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the image to be denoised, and then calculate the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the image to be denoised according to the scaling factor, wherein the continuous scaling curve corresponding to the scaling weight is a derivative smooth curve Line, the points on the derivative smooth curve are used to represent the scaling factor of the pixel coordinate position, and the scaling factor is increasing. The transition fusion weight curve corresponding to the fusion weight is the derivative smooth curve. The points on the derivative smooth curve are used to represent the excessive fusion weight of the distance between the pixel position and the edge of the gaze area. The excessive fusion weight integrates the difference curve weight and the identity mapping weight, and the pixel position close to the edge of the gaze area has a large weight, and the pixel position close to the gaze area has a large identity mapping weight. Finally, the scaling weight of the gaze area, the scaling weight of the non-gaze area, and the fusion weight of the transition area are weighted and synthesized to output the compressed image to be denoised.

[0082] Step S202: performing denoising on the compressed image to be denoised using a denoising model to obtain an initial denoised image.

[0083] The initial denoised image is used to represent the Denoise map output by the denoising model inference. The initial denoised image at least includes the denoising information of each pixel.

[0084] Before using the denoising model to denoise the compressed image to be denoised and obtain the initial denoised image, it is necessary to obtain the denoising model. This is specifically implemented by first obtaining a neural network model and a training set, where the training set includes an input image and an output image. The input image is the image to be denoised used for training, and the output image is the original image captured by an ultra-high-definition camera, with the original data retained. The neural network model is then trained using the input and output images. The denoising model is obtained when the loss function in the neural network model reaches a preset threshold. The preset threshold can be set according to specific circumstances and is not specifically limited here.

[0085] The loss function is determined by the product of the first weight and the mean absolute error loss function, the product of the second weight and the asymmetric loss function, and the product of the third weight and the content consistency loss function.

[0086] Specifically, the loss function loss is expressed by the following formula:

[0087] loss=w1*L1+w2*L asymmetric +w3*L consistence

[0088] Among them, w1, w2 and w3 are weights, L1 is the mean absolute error loss function, Lasymmetric is the asymmetric loss function, and Lconsistence is the content consistency loss function.

[0089] Among them, the asymmetric loss function is determined by the mean absolute error of the distance between the input image and the original image taken by the ultra-high-definition camera, and the judgment function, wherein the judgment function is used to characterize the relationship function between the input image, the target denoised image and the original image taken by the ultra-high-definition camera.

[0090] The mean absolute error loss function L1 in this application is used to restore the image content, and the asymmetric loss function Lasymmetric is used to balance the image details and noise retention, and the content consistency loss function Lconsistence is used to improve the consistency of the previous and next two frames of images.

[0091] Specifically, the asymmetric loss function Lasymmetric is expressed by the following formula:

[0092] L asymmetric =L1(input,gt)*(1+(λ-1)H((out-gt)*(gt-input)))

[0093] Where L1(input,gt) is the mean absolute error between input and gt, input is the input image, gt is the original image captured by the ultra-high-definition camera, H(·) is the judgment function, out is the target denoised image, and λ is the weight coefficient.

[0094] Among them, the content consistency loss function is determined by the first mean absolute error, the second mean absolute error and the third mean absolute error. The first mean absolute error is used to characterize the mean absolute error of the distance between the first target denoised image output after the first input image is processed by the neural network model and the second target denoised image output after the second input image is processed by the neural network model. The second mean absolute error is used to characterize the mean absolute error of the distance between the second target denoised image output after the second input image is processed by the neural network model and the third target denoised image output after the third input image is processed by the neural network model. The third mean absolute error is used to characterize the mean absolute error of the distance between the first target denoised image output after the first input image is processed by the neural network model and the third target denoised image output after the third input image is processed by the neural network model.

[0095] Specifically, the content consistency loss function Lasymmetricke is expressed by the following formula:

[0096] L consistence =L1(out1,out2)+L1(out2,out3)+L1(out3,out1)

[0097] Among them, L1(out1, out2) is the mean absolute error of the distance between out1 and out2, L1(out2, out3) is the mean absolute error of the distance between out2 and out3, L1(out3, out1) is the mean absolute error of the distance between out3 and out1, out1 is the first target denoised image output after the first input image is processed by the neural network model, out2 is the second target denoised image output after the second input image is processed by the neural network model, and out3 is the third target denoised image output after the third input image is processed by the neural network model.

[0098] like Figure 3 As shown, the denoising model obtained through the above embodiment includes a first level, a second level and a third level, the first level and the second level both include a first convolutional layer, a second convolutional layer, a deconvolutional layer and a residual connection block, the third level includes a first convolutional layer, a second convolutional layer and a deconvolutional layer, wherein the first convolutional layer is used to reduce the size of the image, the second convolutional layer is used to extract features from the image, the deconvolution layer is used to enlarge the size of the image, and the residual connection block is used to add at least two images connected by residual connection.

[0099] For example, the first convolutional layer may be a convolution with a stride of 2, and the second convolutional layer may be a convolution with a stride of 1.

[0100] The following combination Figure 3 The denoising module shown describes the use of a denoising model to perform denoising on the compressed image to be denoised, and the execution process of obtaining the initial denoised image is as follows: first, feature extraction is performed on the compressed image to be denoised to obtain a feature image to be denoised, then the first denoised image corresponding to the first level is determined, and then the feature image to be denoised and the first denoised image are added to obtain the initial denoised image.

[0101] Specifically, the compressed image to be denoised is first input into the second convolutional layer, and the second convolutional layer extracts the compressed features to obtain the feature image to be denoised. Then, the feature image to be denoised and the first denoised image after processing at the first level are added together through the residual connection block to obtain the initial denoised image.

[0102] Among them, to determine the first denoised image corresponding to the first level, it is necessary to first determine the second denoised image corresponding to the second level, then reduce the size of the image to be denoised to obtain the reduced image to be denoised, and then perform feature extraction on the reduced image to be denoised to obtain the first initial feature image corresponding to the first level, so as to add the second denoised image and the first initial feature image to obtain the first target feature image corresponding to the first level, and finally enlarge the size of the first target feature image to obtain the first denoised image.

[0103] Specifically, the first convolutional layer in the first level first reduces the size of the image to be denoised to obtain a reduced image to be denoised, and then the second convolutional layer extracts features from the reduced image to be denoised to obtain a first initial feature image corresponding to the first level.

[0104] The second denoised image after the second level processing and the first initial feature image are added together through the residual connection block in the first level to obtain the first target feature image corresponding to the first level. Then, the first target feature image is scaled up by the deconvolution layer of the first level to obtain the first denoised image.

[0105] Among them, to determine the second denoised image corresponding to the second level, it is necessary to first determine the third denoised image corresponding to the third level, then perform feature extraction on the reduced image to be denoised to obtain the second initial feature image corresponding to the second level, and then add the third denoised image and the second initial feature image to obtain the second target feature image corresponding to the second level. Finally, the second target feature image is enlarged to obtain the second denoised image.

[0106] Specifically, the second convolutional layer in the second layer extracts features from the reduced image to be denoised, generating a second initial feature image corresponding to the second layer. The third denoised image processed by the third layer and the second initial feature image are added together using the residual connection block in the second layer to generate a second target feature image corresponding to the second layer. Finally, the deconvolution layer in the second layer upscales the second target feature image to generate a second denoised image.

[0107] To determine the third denoised image corresponding to the third level, it is necessary to first perform feature extraction on the reduced image to be denoised to obtain a third initial feature image corresponding to the third level, and then enlarge the size of the third initial feature image to obtain the third denoised image.

[0108] Specifically, the second convolutional layer in the third level performs feature extraction on the reduced image to be denoised, obtaining a third initial feature image corresponding to the third level. The third initial feature image is then scaled up by the deconvolution layer to obtain a third denoised image.

[0109] Step S203: decompressing the initial denoised image to obtain a decompressed initial denoised image.

[0110] The initial denoised image is decompressed to obtain a decompressed initial denoised image, mainly based on the center coordinates and the initial denoised image. Specifically, the coordinates of each area in the initial denoised image are first calculated based on the center coordinates, and then the initial denoised image is decompressed based on the coordinates of each area in the initial denoised image to obtain the decompressed initial denoised image.

[0111] Among them, the center coordinates are the position corresponding to the gaze area of ​​the target image. Based on the center coordinates, the coordinates of each area in the initial denoised image are calculated. It is necessary to first determine the coordinates of each area in the non-gaze area of ​​the initial denoised image based on the center coordinates, and then the coordinates of each area in the initial denoised image are constructed by the center coordinates and the coordinates of each area in the non-gaze area.

[0112] Among them, the initial denoised image is decompressed based on the coordinates of each area in the initial denoised image to obtain the decompressed initial denoised image. It is necessary to first determine the coordinate mapping table of the non-gaze area based on the coordinates of each area in the non-gaze area of ​​the initial denoised image, and then perform a reverse lookup table based on the gaze area, center coordinates, non-gaze area and coordinate mapping table to calculate the mapping coordinates and scaling factor of each pixel position in the initial denoised image. Then, the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area in the initial denoised image are calculated based on the scaling factor. Finally, the scaling weight of the gaze area, the scaling weight of the non-gaze area and the fusion weight of the transition area are weighted and synthesized to output the decompressed initial denoised image.

[0113] Step S204: performing an addition process on the image to be denoised and the decompressed initial denoised image to obtain a target denoised image.

[0114] Combine Figure 4 Before using the deep learning neural network model for image inference, the deep learning network framework needs to be trained using a data set. After the training is completed, the deep learning neural network (inference) model is obtained.

[0115] Among them, the deep learning neural network (inference) model is also called the noise reduction model, which is configured in the neural network processor, and the image to be denoised is configured in the ISP (Image Signal Processing, image signal processor).

[0116] Specifically, the eye movement information is first obtained by using an eye movement device, and then the image to be denoised (i.e. Figure 4 The system performs gaze point compression on the input image (the input image in the image) to obtain a compressed image to be denoised. The compressed image to be denoised is then denoised by the denoising model in the neural network processor to obtain an initial denoised image. The initial denoised image is then decompressed to obtain a decompressed initial denoised image. Finally, the image to be denoised and the decompressed initial denoised image are added together to obtain the target denoised image. The initial denoised image contains denoised information and lacks brightness information. Decompressing the initial denoised image and adding the decompressed initial denoised image to the image to be denoised preserves the brightness information of the original image, achieving a certain denoising effect without losing detail and ensuring image clarity.

[0117] An embodiment of the present invention provides an image denoising method, comprising: first compressing an image to be denoised to obtain a compressed image to be denoised; then using a denoising model to denoise the compressed image to be denoised to obtain an initial denoised image, wherein the initial denoised image includes at least denoising information of each pixel; then decompressing the initial denoised image to obtain a decompressed initial denoised image; and finally adding the image to be denoised and the decompressed initial denoised image to obtain a target denoised image. The present invention compresses the image to be denoised so that the compressed image is directly decompressed to its original size after being processed by the denoising model in the NPU, thereby avoiding a decrease in the image quality of the target denoised image and improving the scaling effect of the target denoised image. In addition, the present invention compresses the image to be denoised, which can save storage bandwidth and computing hardware resources, effectively reducing hardware resource requirements while ensuring the image effect of the gaze point area.

[0118] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0119] The following are device embodiments of the present invention. For details not fully described therein, reference may be made to the corresponding method embodiments described above.

[0120] Figure 5 A schematic diagram of the structure of an image noise reduction device provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown. The image noise reduction device includes a compression module 501, a noise reduction module 502, a decompression module 503, and an addition processing module 504. The details are as follows:

[0121] The compression module 501 is used to compress the image to be denoised to obtain a compressed image to be denoised;

[0122] A denoising module 502 is configured to perform denoising on the compressed image to be denoised using a denoising model to obtain an initial denoised image, wherein the initial denoised image includes at least denoising information of each pixel;

[0123] The decompression module 503 is used to decompress the initial denoised image to obtain a decompressed initial denoised image;

[0124] The addition processing module 504 is configured to perform addition processing on the image to be denoised and the decompressed initial denoised image to obtain a target denoised image.

[0125] In one possible implementation, the noise reduction model includes a first level, a second level, and a third level;

[0126] Both the first and second levels include the first convolutional layer, the second convolutional layer, the deconvolutional layer, and the residual connection block;

[0127] The third level includes the first convolutional layer, the second convolutional layer and the deconvolutional layer;

[0128] Among them, the first convolution layer is used to reduce the size of the image, the second convolution layer is used to extract features from the image, the deconvolution layer is used to enlarge the size of the image, and the residual connection block is used to add at least two images connected by residuals.

[0129] In a possible implementation, the denoising module 502 is further configured to perform feature extraction on the compressed image to be denoised to obtain a feature image to be denoised;

[0130] Determining a first denoised image corresponding to a first level;

[0131] The feature image to be denoised is added to the first denoised image to obtain an initial denoised image.

[0132] In a possible implementation, the noise reduction module 502 is further configured to determine a second noise reduction image corresponding to the second level;

[0133] Performing a size reduction process on the image to be denoised to obtain a reduced image to be denoised;

[0134] Performing feature extraction on the reduced image to be denoised to obtain a first initial feature image corresponding to the first level;

[0135] Adding the second denoised image and the first initial feature image to obtain a first target feature image corresponding to the first level;

[0136] The first target feature image is enlarged to obtain a first denoised image.

[0137] In a possible implementation, the noise reduction module 502 is further configured to determine a third noise reduction image corresponding to the third level;

[0138] Performing feature extraction on the reduced image to be denoised to obtain a second initial feature image corresponding to the second level;

[0139] Adding the third denoised image to the second initial feature image to obtain a second target feature image corresponding to the second level;

[0140] The second target feature image is enlarged to obtain a second denoised image.

[0141] In a possible implementation, the denoising module 502 is further configured to perform feature extraction on the reduced image to be denoised, to obtain a third initial feature image corresponding to the third level;

[0142] The third initial feature image is enlarged to obtain a third denoised image.

[0143] In one possible implementation, the noise reduction module 502 is further configured as described above and further includes a training module, wherein the training model is configured to obtain a neural network model and a training set, wherein the training set includes an input image and an output image, wherein the input image is an image to be denoised for training, and the output image is an original image captured by an ultra-high-definition camera, and the original image retains the original data;

[0144] The neural network model is trained using the input image and the output image, and the denoising model is obtained when the value of the loss function in the neural network model reaches a preset threshold.

[0145] In one possible implementation, the loss function is determined by the product of the first weight and the mean absolute error loss function, the product of the second weight and the asymmetric loss function, and the product of the third weight and the content consistency loss function.

[0146] In one possible implementation, the asymmetric loss function is determined by the mean absolute error of the distance between the input image and the original image captured by the ultra-high-definition camera, and a judgment function, wherein the judgment function is used to characterize the relationship function between the input image, the target denoised image, and the original image captured by the ultra-high-definition camera.

[0147] In one possible implementation, the content consistency loss function is determined by a first mean absolute error, a second mean absolute error, and a third mean absolute error. The first mean absolute error is used to characterize the mean absolute error of the distance between a first target denoised image output after the first input image is processed by the neural network model and a second target denoised image output after the second input image is processed by the neural network model. The second mean absolute error is used to characterize the mean absolute error of the distance between the second target denoised image output after the second input image is processed by the neural network model and a third target denoised image output after the third input image is processed by the neural network model. The third mean absolute error is used to characterize the mean absolute error of the distance between the first target denoised image output after the first input image is processed by the neural network model and the third target denoised image output after the third input image is processed by the neural network model.

[0148] In one possible implementation, the denoising model is configured in a neural network processor, and the image to be denoised is configured in an image signal processor;

[0149] The apparatus further includes an image noise reduction module, which is used to obtain eye movement information using an eye movement device;

[0150] Performing gaze point compression on the image to be denoised in the image signal processor according to the eye movement information to obtain a compressed image to be denoised;

[0151] The compressed image to be denoised is subjected to denoising processing by a denoising model in a neural network processor to obtain an initial denoised image;

[0152] Decompressing the initial denoised image to obtain a decompressed initial denoised image;

[0153] The image to be denoised and the decompressed initial denoised image are added together to obtain the target denoised image.

[0154] The image to be denoised is stored by ISP sRam in the present invention, retaining the brightness information of the original image. The initial denoised image after gaze point decompression and the image to be denoised are added. The gaze point area is not affected because it is not up- or down-sampled. The non-gaze point area of ​​the initial denoised image is added to the stored image to be denoised, so the brightness information of the original image can be retained. A certain denoising effect can be achieved while ensuring image details, thereby ensuring image clarity after compression and decompression.

[0155] Figure 6 Schematic diagram of a computer device provided by an embodiment of the present invention. Figure 6 As shown, the computer device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned various image noise reduction method embodiments are implemented, such as Figure 2 Alternatively, when the processor 601 executes the computer program 603, the functions of the modules / units in the above-mentioned embodiments of the image noise reduction device are realized, such as Figure 5 Functionality of modules / units 501 to 504 shown.

[0156] The present invention further provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the image noise reduction method provided by the various embodiments described above.

[0157] Among them, the readable storage medium can be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transmission of computer programs from one place to another. Computer storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0158] The present invention also provides a program product, comprising execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and the at least one processor can execute the execution instructions to cause the device to implement the image denoising methods provided in the various embodiments described above.

[0159] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0160] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An image denoising method, characterized in that: include: Compressing the image to be denoised to obtain a compressed image to be denoised; Performing denoising on the compressed image to be denoised using a denoising model to obtain an initial denoised image, wherein the initial denoised image at least includes denoising information of each pixel; Decompressing the initial denoised image to obtain a decompressed initial denoised image; The image to be denoised and the decompressed initial denoised image are added together to obtain a target denoised image.

2. The image denoising method according to claim 1, wherein: The noise reduction model includes a first level, a second level and a third level; The first level and the second level each include a first convolutional layer, a second convolutional layer, a deconvolutional layer, and a residual connection block; The third level includes a first convolutional layer, a second convolutional layer and a deconvolutional layer; Among them, the first convolution layer is used to reduce the size of the image, the second convolution layer is used to extract features from the image, the deconvolution layer is used to enlarge the size of the image, and the residual connection block is used to add at least two images connected by residual connection.

3. The image denoising method according to claim 2, wherein: The denoising model is used to perform denoising on the compressed image to be denoised to obtain an initial denoised image, including: Extracting features from the compressed image to be denoised to obtain a feature image to be denoised; Determining a first denoised image corresponding to the first level; The feature image to be denoised is added to the first denoised image to obtain the initial denoised image.

4. The image denoising method according to claim 3, wherein: The determining the first denoised image corresponding to the first level includes: determining a second denoised image corresponding to the second level; Performing a size reduction process on the image to be denoised to obtain a reduced image to be denoised; Performing feature extraction on the reduced image to be denoised to obtain a first initial feature image corresponding to the first level; Adding the second denoised image and the first initial feature image to obtain a first target feature image corresponding to the first level; The first target feature image is enlarged to obtain the first denoised image.

5. The image denoising method according to claim 4, wherein: The determining the second denoised image corresponding to the second level includes: Determining a third denoised image corresponding to the third level; Performing feature extraction on the reduced image to be denoised to obtain a second initial feature image corresponding to the second level; Adding the third denoised image and the second initial feature image to obtain a second target feature image corresponding to the second level; The second target feature image is enlarged to obtain the second denoised image.

6. The image denoising method according to claim 5, wherein: The determining the third denoised image corresponding to the third level includes: Performing feature extraction on the reduced image to be denoised to obtain a third initial feature image corresponding to the third level; The third initial feature image is enlarged to obtain the third denoised image.

7. The image denoising method according to claim 1, wherein: Before performing denoising on the compressed image to be denoised using the denoising model to obtain an initial denoised image, the method further includes: Obtaining a neural network model and a training set, wherein the training set includes an input image and an output image, the input image being an image to be denoised for training, and the output image being an original image captured by an ultra-high-definition camera, wherein the original image retains the original data; The neural network model is trained using the input image and the output image, and the denoising model is obtained when the value of the loss function in the neural network model reaches a preset threshold.

8. The image denoising method according to claim 7, wherein: The loss function is determined by the product of the first weight and the mean absolute error loss function, the product of the second weight and the asymmetric loss function, and the product of the third weight and the content consistency loss function.

9. The image denoising method according to claim 8, wherein: The asymmetric loss function is determined by the mean absolute error of the distance between the input image and the original image taken by the ultra-high-definition camera, and a judgment function, wherein the judgment function is used to characterize the relationship function between the input image, the target denoised image and the original image taken by the ultra-high-definition camera.

10. The image denoising method according to claim 8, wherein: The content consistency loss function is determined by the first mean absolute error, the second mean absolute error and the third mean absolute error. The first mean absolute error is used to characterize the mean absolute error of the distance between the first target denoised image output after the first input image is processed by the neural network model and the second target denoised image output after the second input image is processed by the neural network model. The second mean absolute error is used to characterize the mean absolute error of the distance between the second target denoised image output after the second input image is processed by the neural network model and the third target denoised image output after the third input image is processed by the neural network model. The third mean absolute error is used to characterize the mean absolute error of the distance between the first target denoised image output after the first input image is processed by the neural network model and the third target denoised image output after the third input image is processed by the neural network model.

11. The image denoising method according to any one of claims 1 to 10, wherein: The denoising model is configured in a neural network processor, and the image to be denoised is configured in an image signal processor; The method further comprises: Use eye tracking equipment to obtain eye movement information; Performing gaze point compression on the image to be denoised in the image signal processor according to the eye movement information to obtain a compressed image to be denoised; The compressed image to be denoised is subjected to denoising processing by a denoising model in the neural network processor to obtain an initial denoised image; Decompressing the initial denoised image to obtain a decompressed initial denoised image; The image to be denoised and the decompressed initial denoised image are added together to obtain a target denoised image.

12. An image noise reduction device, characterized in that: include: A compression module is used to compress the image to be denoised to obtain a compressed image to be denoised; a denoising module, configured to perform denoising on the compressed image to be denoised using a denoising model to obtain an initial denoised image, wherein the initial denoised image at least includes denoising information of each pixel; a decompression module, configured to decompress the initial denoised image to obtain a decompressed initial denoised image; The addition processing module is used to perform addition processing on the image to be denoised and the decompressed initial denoised image to obtain a target denoised image.

13. A computer device, characterized in that: comprising a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors. The instructions are executed by the one or more processors to enable the one or more processors to implement the image noise reduction method according to claims 1 to 11.

14. A computer-readable storage medium, characterized in that The method comprises a program or an instruction, which, when executed on a computer, implements the image noise reduction method according to claims 1 to 11.

Images