Image denoising method and related equipment

Abstract

The embodiment of the present application discloses an image denoising method and related equipment, which includes: using a pre-trained neural network to identify the target scene category of multiple frames of input images, wherein the multiple frames of input images are obtained by extracting frames from a video in a live scene; wherein the neural network is trained using a training sample set, wherein the training sample set includes at least one group of training samples, and each group of training samples includes annotation information of the scene category; obtaining a target noise estimation value interval corresponding to the target scene category; performing denoising on the video in the live scene according to the target noise estimation value interval to obtain a denoised video. The embodiment of the present application can avoid the problem of poor denoising effect caused by using a uniform denoising intensity for different scenes.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image denoising method and related equipment. Background Art

[0002] Currently, commonly used image denoising algorithms include Gaussian filtering, median filtering, and neural network-based denoising algorithms. Gaussian filtering is a weighted averaging process that applies weights to the entire image, assigning larger weights to closer points and smaller weights to more distant points. This can cause pixel shifts in the denoised image. Median filtering replaces pixels with their median value. While it is effective for handling scattered noise, it is less effective for Gaussian noise, resulting in poor performance. After repeated use of a trained neural network-based denoising model, the resulting denoised image may contain noise that was not present in the original image.

[0003] Therefore, how to improve the image denoising effect during live broadcast is an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present application provide an image denoising method and related devices, which can effectively denoise the noise of an image.

[0005] On the one hand, an embodiment of the present application discloses an image denoising method, which includes: using a pre-trained neural network to identify the target scene category of multiple frames of input images, wherein the multiple frames of input images are obtained by extracting frames from a video in a live scene; wherein the neural network is trained using a training sample set, the training sample set includes at least one group of training samples, and each group of training samples includes annotation information of the scene category; obtaining a target noise estimation value interval corresponding to the target scene category; and denoising the video in the live scene according to the target noise estimation value interval to obtain a denoised video.

[0006] In an optional embodiment, the identifying the target scene category of an input image using a pre-trained neural network includes: extracting features of the input image using the pre-trained neural network to obtain a feature map; determining the scene category to which the input image belongs and the probability of the scene category to which the input image belongs based on the feature map; and determining the target scene category of the input image based on the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs.

[0007] In an optional embodiment, obtaining the target noise estimation value interval corresponding to the target scene category includes: using a noise estimation method to perform noise estimation on the input image of each frame to obtain the target noise estimation value of the input image of each frame; and obtaining the target noise estimation value interval corresponding to the target scene category based on the target noise estimation value of the input image of each frame.

[0008] In an optional embodiment, obtaining the target noise estimation value interval corresponding to the target scene category includes: determining the noise estimation value interval corresponding to the target scene category as the target noise estimation value interval based on a preset correspondence between scene types and noise estimation value intervals.

[0009] In an optional embodiment, the denoising process is performed on the video in the live broadcast scene according to the target noise estimation value interval to obtain the denoised video, including: setting the standard deviation of the color space in the grayscale similarity function of the bilateral filter based on the target noise estimation value interval; and denoising the video in the live broadcast scene using the set bilateral filter to obtain the denoised video.

[0010] In an optional embodiment, the target scene category of the input image is one of a first scene, a second scene, a third scene and a fourth scene, wherein the noise estimation value in the target noise estimation value interval corresponding to the fourth scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the third scene; the noise estimation value in the target noise estimation value interval corresponding to the third scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the second scene; and the noise estimation value in the target noise estimation value interval corresponding to the second scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the first scene.

[0011] In an optional embodiment, when the target scene category is the first scene, the value interval (A, B) of the standard deviation is the interval (0.05, 0.2); when the target scene category is the second scene, the value interval (A, B) of the standard deviation is the interval (0, 0.12); when the target scene category is the third scene, the value interval (A, B) of the standard deviation is the interval (0.2, 0.4); when the target scene category is the fourth scene, the value interval (A, B) of the standard deviation is the interval (0.3, 0.6).

[0012] In one aspect, an embodiment of the present application discloses an image denoising device, comprising:

[0013] a recognition unit that uses a pre-trained neural network to identify a target scene category from a plurality of input frames, the plurality of input frames being obtained by extracting frames from a video of a live scene; wherein the neural network is trained using a training sample set, the training sample set including at least one group of training samples, each group of training samples including annotated information of a scene category;

[0014] a processing unit, configured to obtain a target noise estimation value interval corresponding to the target scene category;

[0015] The denoising unit is used to perform denoising on the video in the live scene according to the target noise estimation value interval to obtain the denoised video.

[0016] In an optional embodiment, the recognition unit is used to use a pre-trained neural network to identify the target scene category of multiple frames of input images, where the multiple frames of input images are obtained by extracting frames from a video in a live scene; wherein the neural network is trained using a training sample set, and the training sample set includes at least one group of training samples, and each group of training samples includes annotation information of the scene category.

[0017] In an optional implementation, the processing unit is configured to obtain a target noise estimation value interval corresponding to the target scene category.

[0018] In an optional implementation, the denoising unit is configured to perform denoising on the video in the live broadcast scene according to the target noise estimation value interval to obtain the denoised video.

[0019] In an optional embodiment, the recognition unit is used to identify the target scene category of an input image using a pre-trained neural network, and is specifically used to: extract features of the input image using the pre-trained neural network to obtain a feature map; determine the scene category to which the input image belongs and the probability of the scene category to which the input image belongs based on the feature map; determine the target scene category of the input image based on the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs.

[0020] In an optional embodiment, when obtaining the target noise estimation value interval corresponding to the target scene category, the processing unit is specifically used to: use a noise estimation method to perform noise estimation on the input image of each frame to obtain the target noise estimation value of the input image of each frame; and obtain the target noise estimation value interval corresponding to the target scene category based on the target noise estimation value of the input image of each frame.

[0021] In an optional embodiment, when obtaining the target noise estimation value interval corresponding to the target scene category, the processing unit is specifically used to: determine the noise estimation value interval corresponding to the target scene category as the target noise estimation value interval based on a preset correspondence between the scene type and the noise estimation value interval.

[0022] In an optional embodiment, the denoising unit, when denoising the video in the live scene according to the target noise estimation value interval to obtain the denoised video, is specifically used to: set the standard deviation of the color space in the grayscale similarity function of the bilateral filter based on the target noise estimation value interval; and use the set bilateral filter to denoise the video in the live scene to obtain the denoised video.

[0023] In an optional embodiment, when the scene category used for identification is one of the first scene, the second scene, the third scene and the fourth scene, the noise estimation value in the target noise estimation value interval corresponding to the fourth scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the third scene; the noise estimation value in the target noise estimation value interval corresponding to the third scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the second scene; and the noise estimation value in the target noise estimation value interval corresponding to the second scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the first scene.

[0024] In an optional embodiment, when the scene category of the input image used to identify the recognition unit is the first scene, the value interval (A, B) of the standard deviation is the interval (0.05, 0.2); when the target scene category is the second scene, the value interval (A, B) of the standard deviation is the interval (0, 0.12); when the target scene category is the third scene, the value interval (A, B) of the standard deviation is the interval (0.2, 0.4); when the target scene category is the fourth scene, the value interval (A, B) of the standard deviation is the interval (0.3, 0.6).

[0025] The present application also discloses an image processing device, which includes:

[0026] A memory and a processor, wherein the memory stores an image processing program, and when the image processing program is executed by the processor, the steps of the image denoising method as described above are implemented.

[0027] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image denoising method is executed.

[0028] Accordingly, embodiments of the present application further disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an image processing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the image processing device to perform the above-described image denoising method.

[0029] As can be seen, the image denoising method provided in this application utilizes a neural network to determine the scene classification of an input image from multiple scene categories; based on the determined scene category, the input image is denoised to obtain a denoised video. This method utilizes the classified scenes to perform differential denoising on the input image, thereby avoiding the problem of poor denoising results caused by applying a uniform denoising strength to different scenes, and thus improving the image denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 This is a schematic diagram of the architecture of an image processing system disclosed in an embodiment of the present application;

[0032] Figure 2 This is a flowchart of a denoising method based on the identified target scene category disclosed in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of extracting a feature map is shown;

[0034] Figure 4 shows the architecture diagram of the C3D network;

[0035] Figure 5 This is a schematic diagram of the result of using the neural network to recognize an image, as disclosed in an embodiment of the present application;

[0036] Figure 6 The noise estimation result graphs corresponding to four scene categories are shown;

[0037] Figure 7 Shows schematic diagrams of different scenarios during live broadcast;

[0038] Figure 8 This is a flowchart of another denoising method based on the identified target scene category disclosed in an embodiment of the present application;

[0039] Figure 9 This is a comparison diagram of an image before and after denoising in a diffuse reflection scene disclosed in an embodiment of the present application;

[0040] Figure 10 This is a comparison diagram of an image before and after denoising in a foreground lighting scenario disclosed in an embodiment of the present application;

[0041] Figure 11 This is a comparison diagram of an image before and after denoising in a background lighting scenario disclosed in an embodiment of the present application;

[0042] Figure 12 This is a comparison diagram of an image before and after denoising in a no-light scene disclosed in an embodiment of the present application;

[0043] Figure 13 is a structural diagram of an image processing device disclosed in an embodiment of the present application;

[0044] Figure 14 It is a structural diagram of an image processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] Currently, there are three general methods for image denoising: one is an image denoising method based on Gaussian filtering, which performs a weighted average of the pixel values ​​of the entire image. The value of each pixel is obtained by weighted averaging its own value and the pixel values ​​of other pixels in its neighborhood. However, during the weighted average calculation of the pixel value of the current pixel, the pixel values ​​of pixels closer to the current pixel are assigned a greater weight, while the pixel values ​​of pixels farther away from the current pixel are assigned a smaller weight. Therefore, this method can cause pixel shifts in the denoised image, resulting in unsatisfactory denoising results. The second is an image denoising method based on median filtering. This method uses a rectangle surrounding the current pixel and replaces other pixels in the rectangular area with the median in the median filter function. However, since Gaussian noise is a type of noise that obeys a Gaussian distribution (i.e., normal distribution), the difference between it and the surrounding pixel values ​​is usually small. Therefore, the processing of Gaussian noise is not ideal; the third is a denoising method based on a neural network. After the training model is used multiple times, the denoised image will contain noise that does not exist in the original image. In addition, existing neural network denoising methods all assume that the noise value has been determined. However, in fact, the noise value of the image is not certain, so the effect of a simple neural network denoising method is not good.

[0047] Therefore, how to improve the image denoising effect is an urgent problem to be solved.

[0048] An embodiment of the present application provides an image denoising method that can improve the effect of image denoising. The image denoising method uses a pre-trained neural network to identify the target scene category of multiple frames of input images, and the multiple frames of input images are obtained by extracting frames from the video in the live scene; wherein, the neural network is trained using a training sample set, and the training sample set includes at least one group of training samples, and each group of the training samples includes annotation information of the scene category; the target noise estimation value interval corresponding to the target scene category is obtained; according to the target noise estimation value interval, the video in the live scene is denoised to obtain the denoised video. It can be seen that the image denoising method uses the classified scenes to perform differential denoising on the input image, thereby avoiding the problem of poor denoising effect caused by using a uniform denoising intensity for different scenes, and improving the image denoising effect.

[0049] In the image denoising method provided in the embodiments of this application, the neural network can be implemented based on artificial intelligence (AI). AI refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI is a multidisciplinary field covering a wide range of fields. The neural network in the image denoising method provided in the embodiments of this application primarily involves machine learning (ML) within AI. Machine learning is a multidisciplinary interdisciplinary field encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning generally includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and model-based learning.

[0050] In the image denoising method provided in the embodiment of the present application, the neural network can also be implemented based on the computer vision technology (CV) in artificial intelligence technology. Computer vision technology (CV) is a science that studies how to make a machine "see". More specifically, it refers to using a camera and a computer to replace the human eye to identify, track and measure a target, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to an instrument for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0051] For ease of understanding, an embodiment of the present application describes a network architecture, which includes a database 101 and an image processing device 102. Database 101 can be a local database of image processing device 102, or a database of other cloud devices. The image denoising method can be executed by image processing device 102. Specifically, image processing device 102 obtains multiple frames of input images from database 101, and uses a pre-trained neural network to identify and classify the scene category of the input image to determine the scene category of the input image. The input image can be an image extracted from a video in a frame-by-frame manner, including an image extracted from a live video (for example, a live video of singing), or a photo taken by a camera. After the input image is identified and classified using a neural network to determine the scene category of the input image, an image denoising method is used to denoise the image corresponding to the scene category to which the input image belongs, to obtain a denoised video. The image denoising method can refer to a Gaussian filtering method, a median filtering method, a neural network method, a bilateral filtering method, and the like. Therefore, the above-mentioned image denoising method can avoid the problem of poor denoising effect caused by using a unified denoising intensity for different scenes. Therefore, the denoising method using the above-mentioned method can improve the image denoising effect.

[0052] It should be noted that the image processing device 102 can be a terminal device or a server. The terminal device can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart car, etc., but is not limited to this; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0053] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0054] See Figure 2 , which is a flow chart of an image denoising method disclosed in an embodiment of the present application, the image processing method can be Figure 1 The image processing device shown performs the following steps, including but not limited to:

[0055] S201. Identify target scene categories of multiple input image frames using a pre-trained neural network, where the multiple input image frames are obtained by extracting frames from a video in a live scene; wherein the neural network is trained using a training sample set, where the training sample set includes at least one group of training samples, and each group of training samples includes annotation information of the scene category.

[0056] In an embodiment of the present application, the multi-frame input image can be an image extracted from a video frame, including an image extracted from a live video (e.g., a live singing video), or a photo taken by a camera. For example, a user takes a photo using a camera on a client and sends the photo to an image processing device, which uses the photo as an input image. The image extracted from the video frame can be an image extracted by continuously extracting frames from the video (e.g., continuously extracting 16 frames), or an image extracted by intermittent frame extraction (e.g., every 8 frames), and this application does not impose any restrictions on this.

[0057] In an optional embodiment, a target scene category of an input image is identified using a pre-trained neural network, including: extracting features of the input image using the pre-trained neural network to obtain a feature map; determining, based on the feature map, the scene category to which the input image belongs and the probability of the scene category to which the input image belongs; and determining the target scene category of the input image based on the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs.

[0058] For example, since three-dimensional convolutional neural networks (3D ConvNets) can better model the temporal information of images, the neural network can be 3D ConvNets, and the 3D ConvNets are used to extract features from the input image extracted from the scene video. The process of obtaining the feature map can be as follows: Figure 3 As shown. Figure 3 As shown, the input side (input) adjusts the size of the input image to a preset size (for example, it can be ) and import it into the neural network. The input image is convolved with the convolution kernel in the neural network according to a preset method, where H = 127, W = 128, and L = 128. H represents the temporal depth, and k and d represent the size of the convolution kernel, which is smaller than the size of the input image. This preset method is indicated by the arrow in the figure. This convolution operation yields the output feature map.

[0059] See also Figure 4 , Figure 4 This is a schematic diagram of a 3D ConvNets network structure provided in an embodiment of the present application. Figure 4 As shown in the figure, the 3D ConvNets consists of a 3D convolutional layer, a 3D maximum pooling layer, and a fully connected layer, and contains 8 3D convolutional layers, 5 3D downsampling layers, 2 fully connected layers, and 1 softmax activation output layer. The convolution kernel size of all 3DConvNets is , the step size is ; Set the pool core size to , the step size is ; All other 3D pooling layers are , the step size is ; There are 4096 output units in the fully connected layer, and the softmax activation function is used to perform four-category recognition of the scene.

[0060] For example, Figure 5 This is a schematic diagram of the result of using the neural network to recognize an image provided by an embodiment of the present application. Figure 5 As shown, the image processing device imports the input image into the network 3D ConvNets for training and classification recognition, and the classification results are as follows Figure 5 As shown in the recognition result diagram, the letters above the recognition result diagram are the classification results corresponding to the frame image, and the line below the letters is the probability of the classification result obtained, which is represented by Figure 5 It can be seen that the recognition probability of the scene category to which the input image belongs is 0.7689. Therefore, the 3D ConvNets can effectively identify and classify the input image. The process of using the 3D ConvNets to train and classify the input image can refer to the following steps: Assume that 16 frames of images are selected to be imported into the 3D ConvNets for training, and the size of each frame is adjusted to , then use the SGD optimizer, set the batch size to 30, the initial learning rate to 0.003, divide by 2 every 150k iterations, and set the epoch to 13. In the test phase, the data can be processed in the same way as in the training phase.

[0061] By effectively classifying and identifying images corresponding to the scene category to which the input image belongs, the scene category to which the input image belongs can be determined, and then the step of obtaining a target noise estimation value interval corresponding to the target scene category in S202 is performed.

[0062] S202: Obtain a target noise estimation value interval corresponding to the target scene category.

[0063] In an embodiment of the present application, the image processing device can use a noise estimation algorithm to perform noise estimation on the input image to obtain a target noise estimation value of the input image; and based on the target noise estimation value of the input image, obtain a target noise estimation value interval corresponding to the target scene category.

[0064] For example, the noise estimation method may be a recursive average noise estimation algorithm or a minimum-controlled recursive average algorithm, etc. For example, the image processing device may use the recursive average noise estimation algorithm to estimate the image corresponding to the scene category to which the input image belongs, so as to obtain a target noise estimation value interval.

[0065] In an optional implementation, the image processing device may further determine the noise estimation value interval corresponding to the target scene category as the target noise estimation value interval based on a preset correspondence between scene types and noise estimation value intervals.

[0066] For example, assuming that the target scene category of the input image is one of the first scene, the second scene, the third scene and the fourth scene, wherein the noise estimation value in the target noise estimation value interval corresponding to the fourth scene may be greater than the noise estimation value in the target noise estimation value interval corresponding to the third scene; the noise estimation value in the target noise estimation value interval corresponding to the third scene may be greater than the noise estimation value in the target noise estimation value interval corresponding to the second scene; and the noise estimation value in the target noise estimation value interval corresponding to the second scene may be greater than the noise estimation value in the target noise estimation value interval corresponding to the first scene.

[0067] Optionally, the image processing device uses a noise estimation method to estimate the image corresponding to the scene category to which the input image belongs to obtain a target noise estimation value range, which may include: the image processing device extracts frames from the live scene to obtain an input image; uses the noise estimation method to perform noise estimation on the input image to obtain a target noise estimation value of the input image; and determines the target noise estimation value range based on the target noise estimation value of the image.

[0068] For example, see Figure 6 , is a schematic diagram of noise estimation values ​​of ten groups of input images provided in an embodiment of the present application, wherein, Figure 6 The ten sets of input images shown can be obtained by extracting frames from different live scenes, such as Figure 6 As shown in the figure, the horizontal axis represents the first to tenth groups of input images from left to right, where each group of input images includes four images, marked as the first image, the second image, the third image, and the fourth image; the vertical axis represents the noise estimation value obtained by using the noise estimation method to estimate the noise of each image in each group of input images. Figure 6As shown, the noise estimation values ​​of each image in the first group of input images are (0.22, 0.31, 0.48, 0.90), the noise estimation values ​​of each image in the second group of input images are (0.21, 0.33, 0.52, 0.94), the noise estimation values ​​of each image in the third group of input images are (0.25, 0.32, 0.57, 0.91), the noise estimation values ​​of each image in the fourth group of input images are (0.23, 0.36, 0.58, 0.92), the noise estimation values ​​of each image in the fifth group of input images are (0.24, 0.38, 0.58, 0.93), the noise estimation values ​​of each image in the sixth group of input images are (0.24, 0.38, 0.61, 0.91), and the noise estimation values ​​of each image in the seventh group of input images are (0.24, The estimated values ​​of the noise of each image in the eighth group of input images are (0.25, 0.44, 0.62, 0.92), the estimated values ​​of the noise of each image in the ninth group of input images are (0.31, 0.47, 0.62, 0.81), and the estimated values ​​of the noise of each image in the tenth group of input images are (0.30, 0.42, 0.78, 0.94). Figure 6 As shown, the noise estimation values ​​of all the first images in the ten groups of input images are located in the noise estimation value interval (0.20, 0.31), the noise estimation values ​​of all the second images in the ten groups of input images are located in the noise estimation value interval (0.32, 0.49), the noise estimation values ​​of all the third images in the ten groups of input images are located in the noise estimation value interval (0.48, 0.78), and the noise estimation values ​​of all the fourth images in the ten groups of input images are located in the noise estimation value interval (0.80, 0.94), that is, the target noise estimation value interval of the four input images is obtained.

[0069] Accordingly, the image processing device determines the target noise estimation value interval of the input image as the target noise estimation value interval according to the preset correspondence between the target noise estimation value intervals of the four input images and the scene types. Figure 6By analyzing the first to fourth images in the ten groups of input images in the four noise estimation value intervals obtained, four scene categories can be obtained, namely the first to fourth scenes mentioned above, that is, the noise estimation value in the target noise estimation value interval corresponding to the fourth image in the fourth scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the third image in the third scene; the noise estimation value in the target noise estimation value interval corresponding to the third image in the third scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the second image in the second scene; and the noise estimation value in the target noise estimation value interval corresponding to the second image in the second scene can be greater than the noise estimation value in the target noise estimation value interval corresponding to the first image in the first scene.

[0070] By performing noise estimation on the image corresponding to the identified scene category, a target noise estimation value interval is obtained, and then the step of denoising the video in the live scene according to the target noise estimation value interval in S203 can be executed to obtain the denoised video.

[0071] S203: Perform denoising on the video in the live broadcast scene according to the target noise estimation value interval to obtain a denoised video.

[0072] In the embodiment of the present application, the denoising process of an image may refer to the process of reducing noise in a digital image. The process of reducing noise in a digital image may be based on some denoising methods, for example, a denoising method based on a bilateral filter, a denoising method based on a Gaussian filter, etc. The setting of the model parameters may include: setting the optimizer to stochastic gradient descent (SGD), setting the batch size to 30, setting the initial learning rate to 0.003, setting the epochs to 13, etc.; in addition, the training sample set may be obtained by sampling one frame out of every three frames of the video in the live scene, and adjusting the size of each frame to , the size of the convolution kernel is set to .

[0073] In an optional implementation, the image processing device may perform denoising on the input image using a bilateral filter and the target noise estimation value interval corresponding to the scene category in S202 to obtain a denoised video.

[0074] Optionally, a bilateral filter can smooth and denoise an image while also preserving image edges. This is because the filter's kernel is generated by two functions: the spatial domain kernel (also known as the definition domain kernel, spatial coefficient, or spatial domain) and the range kernel (the pixel range domain). The filter also combines the spatial proximity and pixel similarity of the image, while considering both spatial information and grayscale similarity to achieve edge-preserving denoising. Therefore, embodiments of the present application can employ a bilateral filter to classify and denoise images corresponding to a specific scene category.

[0075] Among them, the spatial domain is the template weight determined by the Euclidean distance of pixel positions, and BF is the symbol of bilateral filtering, which is defined by formula (1):

[0076] (1)

[0077] in, is a standard quantity, p and q are the pixels on the image, s represents the total number of pixels, and are the values ​​of the pixels on the image, is the standard deviation of the coordinate space, is the standard deviation of the color space, is the spatial proximity function, is the grayscale similarity function, and It is a function used to measure the amount of noise filtered out of an image. and They are expressed as formula (2) and formula (3) respectively.

[0078] (2)

[0079] (3)

[0080] From formulas (1), (2), and (3), we can see that bilateral filtering combines the characteristics of Gaussian filtering and α-truncated mean filtering, while considering the difference between the spatial domain and the range. Gaussian filtering and α-mean filtering only consider the difference between the spatial domain and the range, respectively. Among them, the Gaussian filter only considers the Euclidean distance between pixels, and the template coefficient used by it decreases as the distance from the window center increases; the α-truncated mean filter only considers the difference between the pixel grayscale values, removing the minimum and maximum pixel grayscale values ​​before calculating the mean.

[0081] in and are the Euclidean distance between two pixels of the image and the grayscale difference of the pixels, so we can know that and Determines the performance of the bilateral filter, which limits the value of the pixel value by indicating the relative space of the pixel position (value) and the brightness change range. Once one of these two parameters is close to 0, the smoothing phenomenon will occur. If the range change is less than its amplitude increase space, it will have no effect on the edge. The change ratio More can affect the pixels of the image. As the value increases, the bilateral filter gradually approaches Gaussian blur, because this series of Gaussian functions is relatively flat and almost constant in the intensity interval covered by the image. Increasing the spatial domain parameter can smooth more features. Therefore, the flat area is equivalent to Gaussian filtering. The denoising effect of bilateral filtering depends on two parameters and , and They control the characteristics of the brightness domain and the spatial domain respectively, but based on the results of the simulated denoising effect, it is proved that The value ratio The value of is more important in changing the noise level. Optionally, when the standard deviation of the color space When the value of increases, the corresponding grayscale similarity function The value of also increases, and It is a function used to measure the amount of noise filtered out of an image. Therefore, Increasing the value of can increase the ability of the bilateral filter to filter out noise.

[0082] Optionally, the image processing device may perform denoising on the input image using a bilateral filter, or may perform denoising on an image corresponding to the scene category to which the input image belongs by adjusting the standard deviation of the color space in the grayscale similarity function in the bilateral filter and determining the scene category to which the input image belongs, and finally obtain a denoised video. The value of can correspond to different scene categories. Therefore, the value range of the standard deviation of the color space in the grayscale similarity function of the bilateral filter can be set to the value range (A, B) of the standard deviation of the color space corresponding to the target noise estimation value interval; wherein A and B are respectively constants.

[0083] For example, assuming that the first scene is a diffuse lighting scene, the second scene is a foreground lighting scene, the third scene is a background lighting scene, and the fourth scene is a no-lighting scene. The diffuse lighting scene may refer to a scene where the light source is evenly illuminated, the foreground lighting scene may refer to a scene where the light source is illuminating in front of the live broadcast subject, the background lighting scene may refer to a scene where the light source is illuminating behind the live broadcast subject, and the no-lighting scene may refer to a scene where there is no light source.

[0084] For example, you can refer to Figure 7 , Figure 7 is a schematic diagram of multiple images provided in the embodiment of the present application, such as Figure 7 As shown, the noise estimation value of image a is 0.7026, which is in the target noise estimation value interval (0.48, 0.78) corresponding to the third scene. The noise estimation value of image b is 0.6721, which is in the target noise estimation value interval (0.48, 0.78) corresponding to the third scene. The noise estimation value of image c is 0.9128, which is in Figure 6 The target noise estimate for the fourth scene is (0.80, 0.94). The noise estimate for image d is 0.2125, which falls within the target noise estimate for the first scene (0.20, 0.31). The noise estimate for image e is 0.3961, which falls within the target noise estimate for the second scene (0.32, 0.49). The noise estimate for image f is 0.4136, which falls within the target noise estimate for the second scene (0.32, 0.49). Therefore, the scene category of images a and b is the second scene, the scene category of images f and e is the second scene, the scene category of image d is the first scene, and the scene category of image c is the fourth scene.

[0085] In addition, if Figure 7 As shown in images a and b, the light source is behind the live broadcast subject, which belongs to the background lighting scene; image c shows the scene when there is no light source, which belongs to the no-light scene; image d shows the scene when the light source is evenly illuminated, which belongs to the diffuse lighting scene; image e and image f show the scene when the light source is in front of the live broadcast subject, which belongs to the foreground lighting scene.

[0086] Optionally, when the scene category is one of the four scenes mentioned above, a bilateral filter is used to perform denoising on the image corresponding to the scene category to which the input image belongs. Specifically, as can be seen from the description of the bilateral filter above, the noise level in the bilateral filter can be adjusted. Therefore, when the recognized scene category is the fourth scene, it can be seen that the image corresponding to the fourth scene has the largest noise, so The value of is adjusted within the interval (0.3, 0.6) to denoise the image corresponding to the fourth scene category to which the input image belongs, and finally obtain the input image corresponding to the fourth scene after denoising. When the recognized scene category is the third scene, it can be known that the image corresponding to the third scene has a large noise, so The value of is adjusted within the interval (0.2, 0.4) to denoise the image corresponding to the third scene category to which the input image belongs, and finally obtain the input image corresponding to the third scene after denoising. When the recognized scene category is the second scene, it can be seen that the noise of the image corresponding to the second scene is small, so The value of is adjusted within the interval (0, 0.12) to denoise the image corresponding to the second scene category to which the input image belongs, and finally obtain the input image corresponding to the second scene after denoising. When the recognized scene category is the first scene, it can be known that the noise of the image corresponding to the first scene is the smallest, so The value of is adjusted within the range of (0.05, 0.2) to denoise the image corresponding to the first scene category to which the input image belongs, and finally obtain the denoised input image corresponding to the first scene. It should be noted that the numerical range of the standard deviation of the color space described above is merely an example and can be set to other values ​​depending on the specific implementation.

[0087] See Figure 8 , Figure 8 : This is a flowchart of a denoising method based on an identified target scene category provided by an embodiment of the present application. The denoising method based on an identified scene category includes but is not limited to the following steps:

[0088] S801: An image processing device performs frame extraction processing on a video in a live broadcast scene to obtain an input image.

[0089] In an embodiment of the present application, the image processing device can extract input images by extracting frames from the video, including extracting images from live videos (for example, live videos of singing), or photos taken by a shooting device. The image processing device uses the extracted image or the photographed photo as the input image, wherein the extraction of input images by extracting frames from the video can be extracting input images by continuously extracting frames from the video (for example, 16 consecutive frames), or extracting input images by intermittent frame extraction (for example, 8 frames intermittently), and the present application does not impose any restrictions on this.

[0090] S802: The image processing device uses a pre-trained neural network to identify target scene categories of multiple frames of input images.

[0091] In an embodiment of the present application, the structure of the neural network may be composed of multiple convolutional layers (Conv), multiple pooling layers (Pooling) and multiple fully connected layers (FC).

[0092] In an optional embodiment, the scene category of the input image may be one of the following four scene categories, which may be: a first scene, a second scene, a third scene, and a fourth scene, wherein the first scene may be a diffuse lighting scene, the second scene may be a foreground lighting scene, the third scene may be a background lighting scene, and the fourth scene may be a no-lighting scene.

[0093] Optionally, the image processing device obtains images corresponding to the four scene categories by extracting frames from the videos of the four scene categories.

[0094] In an optional embodiment, the image processing device can import the obtained input images of the four scene categories into the neural network, use the neural network to extract features of the input images to obtain a feature map; based on the feature map, determine the scene category to which the input image belongs and the probability of the scene category to which the input image belongs; based on the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs, determine the scene category of the input image from the four scene categories.

[0095] Optionally, assuming that based on the feature map of the input image, the scene category to which the input image belongs is identified as the first scene category, and the probability of the first scene category to which the input image belongs satisfies a preset probability threshold, then it is determined that the scene category of the input image is the first scene category, and then the step of performing corresponding denoising processing on the input image according to the target noise estimation value interval corresponding to the scene category in S803 is executed to obtain the denoised video, wherein the preset probability threshold can be but is not limited to 0.88.

[0096] S803 : Perform corresponding denoising processing on the input image according to the target noise estimation value interval corresponding to the scene category to obtain a denoised video.

[0097] In embodiments of the present application, image classification denoising can refer to adopting different processes for reducing noise in digital images for input images corresponding to different scene categories. The process of reducing noise in digital images can be based on a number of denoising methods, such as denoising methods based on bilateral filters, denoising methods based on Gaussian filters, and so on. However, image denoising methods based on Gaussian filters can cause pixel shifts in the denoised image, image denoising methods based on median filters are not ideal for processing Gaussian noise, and denoising methods based on neural networks can leave noise that is not present in the original image after image denoising. However, bilateral filters can achieve image smoothing and denoising while also well preserving image edges because the kernel of the bilateral filter is generated by two functions: a spatial domain kernel (also known as a definition domain kernel, spatial coefficient, or spatial domain) and a range kernel (pixel range domain). Bilateral filters also combine the spatial proximity and pixel similarity of images, while considering spatial information and grayscale similarity to achieve the purpose of edge-preserving denoising. Therefore, embodiments of the present application can use bilateral filters to perform classification denoising on images corresponding to the scene category of the input image.

[0098] Optionally, when the neural network identifies that the scene to which the input image belongs is the first scene, that is, the diffuse lighting scene, the image corresponding to the determined diffuse lighting scene is denoised by using the bilateral filter. Specifically, since the denoising effect of the bilateral filter depends on the standard variance of the color space , so the standard deviation of the color space in the bilateral filter can be adjusted by The image corresponding to the diffuse lighting scene to which the input image belongs is denoised based on the value of to obtain the denoised input image, as shown in Figure 9 As shown by Figure 9 It can be seen that the denoised image b appears smoother than the image a before denoising, especially the right side of the nose appears smoother;

[0099] When the neural network identifies that the scene to which the input image belongs is the second scene, that is, the foreground lighting scene, by adjusting The image corresponding to the foreground lighting scene to which the input image belongs is denoised based on the value of to obtain the denoised input image, such as Figure 10 As shown by Figure 10 It can be seen that the denoised image b appears smoother than the image a before denoising, especially the right side of the nose appears smoother;

[0100] When the neural network identifies that the scene to which the input image belongs is the third scene, that is, the background lighting scene, by adjusting The value of is used to perform denoising on the image corresponding to the background illumination scene to which the input image belongs, and obtain the denoised input image, such as Figure 11 As shown by Figure 11 It can be seen that the denoised image b appears smoother than the image a before denoising, especially the right side of the nose appears smoother;

[0101] When the neural network identifies that the scene to which the input image belongs is the fourth scene, i.e., the scene without illumination, by adjusting The image corresponding to the unlit scene to which the input image belongs is denoised based on the value of to obtain the denoised input image, as shown in Figure 12 As shown by Figure 12 It can be seen that the denoised image b appears smoother than the image a before denoising, especially the right side of the nose appears smoother.

[0102] By denoising the images corresponding to the determined scene categories to different degrees, the problem of poor denoising effect caused by using the same denoising intensity for different live broadcast scenes can be avoided.

[0103] It should be noted that the several specific implementations shown in steps S801-S803 of this application can be implemented independently or in combination with any one or more of the above specific implementations, and this application does not limit this. Figure 2 The specific implementation of the image denoising method described, and based on Figure 8 The specific implementation of the described another image denoising method can be implemented separately or in combination, and this application does not limit it.

[0104] Based on the above method embodiment, the present application embodiment also provides a structural diagram of an image processing device. Figure 13 , is a structural diagram of an image processing device provided in an embodiment of the present application. Figure 13 The image processing apparatus 1000 shown can run the following units:

[0105] Recognition unit 1301 uses a pre-trained neural network to identify a target scene category of multiple input frames, where the multiple input frames are obtained by extracting frames from a video of a live scene; wherein the neural network is trained using a training sample set, where the training sample set includes at least one group of training samples, each group of training samples including annotated information of the scene category;

[0106] The processing unit 1302 is configured to obtain a target noise estimation value interval corresponding to the target scene category;

[0107] The denoising unit 1303 is configured to perform denoising on the video in the live broadcast scene according to the target noise estimation value interval to obtain a denoised video.

[0108] In an optional embodiment, the recognition unit 1301 is used to use a pre-trained neural network to identify the target scene category of multiple frames of input images, where the multiple frames of input images are obtained by extracting frames from a video in a live scene; wherein the neural network is trained using a training sample set, and the training sample set includes at least one group of training samples, and each group of training samples includes annotation information of the scene category.

[0109] In an optional implementation, the processing unit 1302 is configured to obtain a target noise estimation value interval corresponding to the target scene category.

[0110] In an optional implementation, the denoising unit 1303 is configured to perform denoising on the video in the live broadcast scene according to the target noise estimation value interval to obtain a denoised video.

[0111] In an optional embodiment, the recognition unit 1301 is used to identify the target scene category of an input image using a pre-trained neural network, and is specifically used to: extract features of the input image using the pre-trained neural network to obtain a feature map; determine the scene category to which the input image belongs and the probability of the scene category to which the input image belongs based on the feature map; determine the target scene category of the input image based on the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs.

[0112] In an optional embodiment, when obtaining the target noise estimation value interval corresponding to the target scene category, the processing unit 1302 is specifically used to: use a noise estimation method to perform noise estimation on the input image of each frame to obtain the target noise estimation value of the input image of each frame; and obtain the target noise estimation value interval corresponding to the target scene category based on the target noise estimation value of the input image of each frame.

[0113] In an optional embodiment, when obtaining the target noise estimation value interval corresponding to the target scene category, the processing unit 1302 is specifically used to: determine the noise estimation value interval corresponding to the target scene category as the target noise estimation value interval based on a preset correspondence between the scene type and the noise estimation value interval.

[0114] In an optional embodiment, the denoising unit 1303, when denoising the video in the live scene according to the target noise estimation value interval to obtain the denoised video, is specifically used to: set the standard deviation of the color space in the grayscale similarity function of the bilateral filter based on the target noise estimation value interval; and use the set bilateral filter to denoise the video in the live scene to obtain the denoised video.

[0115] In an optional embodiment, when the scene category used for identification by the identification unit 1301 is one of the first scene, the second scene, the third scene and the fourth scene, the noise estimation value in the target noise estimation value interval corresponding to the fourth scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the third scene; the noise estimation value in the target noise estimation value interval corresponding to the third scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the second scene; and the noise estimation value in the target noise estimation value interval corresponding to the second scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the first scene.

[0116] In an optional embodiment, when the scene category of the input image used to identify the recognition unit 1301 is the first scene, the value interval (A, B) of the standard deviation is the interval (0.05, 0.2); when the target scene category is the second scene, the value interval (A, B) of the standard deviation is the interval (0, 0.12); when the target scene category is the third scene, the value interval (A, B) of the standard deviation is the interval (0.2, 0.4); when the target scene category is the fourth scene, the value interval (A, B) of the standard deviation is the interval (0.3, 0.6).

[0117] According to one embodiment of the present application, Figure 2 The steps involved in the image denoising method shown can be Figure 13 The image processing apparatus shown in FIG. Figure 2 In the image denoising method shown in FIG. 1 , step S201 can be performed by Figure 13 The recognition unit 1301 in the image denoising apparatus shown in FIG. 1 is used to perform step S202, which can be performed by Figure 13 The image denoising apparatus shown in FIG. 1 is used for processing by the processing unit 1302, and step S203 can be performed by Figure 13 The denoising unit 1303 in the image denoising apparatus shown is used for execution.

[0118] According to the embodiments of the present application, Figure 13The various units in the image processing device shown can be individually or completely combined into one or several other units to form a whole, or one (or more) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image denoising device can also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0119] According to the embodiments of the present application, the following can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM) and other processing elements and storage elements. Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 13 The image denoising device shown in and the image denoising method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable storage medium, and loaded into the above-mentioned computing device through the computer readable storage medium and run therein.

[0120] In an embodiment of the present application, the recognition unit 1301 recognizes the scene category of the input image, and the processing unit 1302 performs corresponding denoising processing on the input image according to the recognized scene category to obtain a denoised video. The above-mentioned denoising processing method can be used to perform denoising to different degrees according to different live scene categories, thereby avoiding the problem of poor denoising effect caused by using a uniform denoising intensity for different scenes.

[0121] Based on the above method and device embodiments, the present application provides an image processing device. Figure 14 , is a structural diagram of an image processing device provided in an embodiment of the present application. Figure 14 The image processing device 1400 shown includes at least a processor 1401, an input interface 1402, an output interface 1403, a computer storage medium 1404, and a memory 1405. The processor 1401, the input interface 1402, the output interface 1403, the computer storage medium 1404, and the memory 1405 may be connected via a bus or other means.

[0122] Computer storage medium 1404 may be stored in memory 1405 of image processing device 1400. Computer storage medium 1404 is used to store a computer program, which includes program instructions. Processor 1401 is used to execute the program instructions stored in computer storage medium 1404. Processor 1401 (also known as a CPU (Central Processing Unit)) is the computing and control core of image processing device 1400 and is adapted to implement one or more instructions, specifically, to load and execute one or more computer instructions to implement a corresponding method flow or function.

[0123] The present embodiment also provides a computer storage medium (Memory), which is a memory device in the image processing device 1400 and is used to store programs and data. It is understood that the computer storage medium herein may include both the built-in storage medium in the image processing device 1400 and, of course, the extended storage medium supported by the image processing device 1400. The computer storage medium provides storage space, which stores the operating system of the image processing device 1400. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by the processor 1401. These instructions may be one or more computer programs (including program code). It should be noted that the computer storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device; optionally, it may be at least one computer storage medium located remotely from the aforementioned processor.

[0124] In one embodiment, the computer storage medium can be loaded by the processor 1401 and execute one or more instructions stored in the computer storage medium to implement the above-mentioned Figure 2 In a specific implementation, one or more instructions in a computer storage medium are loaded by the processor 1401 and execute the following steps:

[0125] Using a pre-trained neural network to identify target scene categories from multiple input frames, the multiple input frames being extracted from a live video stream; wherein the neural network is trained using a training sample set, the training sample set comprising at least one set of training samples, each set of training samples including annotated scene category information;

[0126] Obtaining a target noise estimation value interval corresponding to the target scene category;

[0127] The video in the live broadcast scene is denoised according to the target noise estimation value interval to obtain a denoised video.

[0128] In one possible implementation, the processor 1401 uses a pre-trained neural network to identify the target scene category of an input image, including: using a pre-trained neural network to extract features of the input image to obtain a feature map; determining the scene category to which the input image belongs and the probability of the scene category to which the input image belongs based on the feature map; determining the target scene category of the input image based on the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs.

[0129] In one possible implementation, the processor 1401 obtains a target noise estimation value interval corresponding to the target scene category, including: performing noise estimation on the input image of each frame using a noise estimation method to obtain a target noise estimation value of the input image of each frame; and obtaining a target noise estimation value interval corresponding to the target scene category based on the target noise estimation value of the input image of each frame.

[0130] In a possible implementation, the processor 1401 obtains the target noise estimation value interval corresponding to the target scene category, including: based on a preset correspondence between scene type and noise estimation value interval, determining the noise estimation value interval corresponding to the target scene category as the target noise estimation value interval.

[0131] In one possible implementation, the processor 1401 denoises the video in the live scene according to the target noise estimation value interval to obtain the denoised video, including: setting the standard deviation of the color space in the grayscale similarity function of the bilateral filter based on the target noise estimation value interval; and using the set bilateral filter to denoise the video in the live scene to obtain the denoised video.

[0132] In one possible implementation, the processor 1401 is used to determine the scene category to which the input image belongs, including: the target scene category of the input image is one of the first scene, the second scene, the third scene and the fourth scene, wherein the noise estimation value in the target noise estimation value interval corresponding to the fourth scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the third scene; the noise estimation value in the target noise estimation value interval corresponding to the third scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the second scene; and the noise estimation value in the target noise estimation value interval corresponding to the second scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the first scene.

[0133] In one possible implementation, the processor 1401 is further used to determine the scene category to which the input image belongs. When the target scene category is the first scene, the value interval (A, B) of the standard deviation is the interval (0.05, 0.2); when the target scene category is the second scene, the value interval (A, B) of the standard deviation is the interval (0, 0.12); when the target scene category is the third scene, the value interval (A, B) of the standard deviation is the interval (0.2, 0.4); when the target scene category is the fourth scene, the value interval (A, B) of the standard deviation is the interval (0.3, 0.6).

[0134] In the implementation of this application, the processor 1401 obtains the scene category to which the input image belongs, and performs denoising on the image corresponding to the scene category to which the input image belongs. The above-mentioned denoising method can perform denoising to different degrees according to different scene categories, thereby avoiding the problem of poor denoising effect caused by using a uniform denoising intensity for different scenes.

[0135] According to one aspect of the present application, an embodiment of the present application further provides a computer product or computer program, the computer product or computer program including computer instructions, the computer instructions stored in a computer-readable storage medium. The processor 1401 reads the computer instructions from the computer-readable storage medium, and the processor 1401 executes the computer instructions, so that the image processing device 1400 performs Figure 2 The image processing method shown.

[0136] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules described above is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not implemented.

[0138] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image denoising method, characterized in that: The method comprises: Using a pre-trained neural network to identify target scene categories of multiple input image frames, the multiple input image frames are obtained by extracting frames from a video in a live scene; wherein the neural network is trained using a training sample set, the training sample set includes at least one group of training samples, each group of training samples includes annotated information of a scene category; the target scene category of the input image is one of a first scene, a second scene, a third scene, and a fourth scene, the first scene includes a diffusely illuminated scene, the second scene includes a foreground illuminated scene, the third scene includes a background illuminated scene, and the fourth scene includes a no-illuminated scene; Obtaining a target noise estimation value interval corresponding to the target scene category; wherein a noise estimation value in the target noise estimation value interval corresponding to the fourth scene is greater than a noise estimation value in the target noise estimation value interval corresponding to the third scene, the noise estimation value in the target noise estimation value interval corresponding to the third scene is greater than a noise estimation value in the target noise estimation value interval corresponding to the second scene, and the noise estimation value in the target noise estimation value interval corresponding to the second scene is greater than a noise estimation value in the target noise estimation value interval corresponding to the first scene; According to the target noise estimation value interval, the standard deviation of the color space in the grayscale similarity function of the bilateral filter is set; wherein, when the target scene category is the first scene, the value interval (A, B) of the standard deviation is the interval (0.05, 0.2); when the target scene category is the second scene, the value interval (A, B) of the standard deviation is the interval (0, 0.12); when the target scene category is the third scene, the value interval (A, B) of the standard deviation is the interval (0.2, 0.4); when the target scene category is the fourth scene, the value interval (A, B) of the standard deviation is the interval (0.3, 0.6); The set bilateral filter is used to perform denoising on the video in the live broadcast scene to obtain a denoised video.

2. The method according to claim 1, characterized in that The method of using a pre-trained neural network to identify the target scene category of an input image includes: Use the pre-trained neural network to extract features from the input image and obtain a feature map; determining, based on the feature map, the scene category to which the input image belongs and the probability of the scene category to which the input image belongs; The target scene category of the input image is determined according to the scene category to which the feature map belongs and the probability of the scene category to which the feature map belongs.

3. The method according to claim 1, characterized in that The obtaining of a target noise estimation value interval corresponding to the target scene category includes: Performing noise estimation on the input image of each frame using a noise estimation method to obtain a target noise estimation value of the input image of each frame; According to the target noise estimation value of each frame of the input image, a target noise estimation value interval corresponding to the target scene category is obtained.

4. The method according to claim 1, wherein The obtaining of a target noise estimation value interval corresponding to the target scene category includes: Based on a preset correspondence between scene types and noise estimation value intervals, the noise estimation value interval corresponding to the target scene category is determined as a target noise estimation value interval.

5. An image processing device, characterized in that: The image processing device comprises: an identification unit, configured to identify a target scene category of a plurality of input image frames using a pre-trained neural network, the plurality of input image frames being obtained by extracting frames from a video of a live broadcast scene; wherein the neural network is trained using a training sample set, the training sample set comprising at least one group of training samples, each group of training samples comprising annotated information of a scene category; the target scene category of the input image is one of a first scene, a second scene, a third scene, and a fourth scene, wherein the first scene comprises a diffusely illuminated scene, the second scene comprises a foreground illuminated scene, the third scene comprises a background illuminated scene, and the fourth scene comprises a no-illuminated scene; a processing unit, configured to obtain a target noise estimation value interval corresponding to the target scene category; wherein the noise estimation value in the target noise estimation value interval corresponding to the fourth scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the third scene, the noise estimation value in the target noise estimation value interval corresponding to the third scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the second scene, and the noise estimation value in the target noise estimation value interval corresponding to the second scene is greater than the noise estimation value in the target noise estimation value interval corresponding to the first scene; A denoising unit is configured to set a standard deviation of a color space in a grayscale similarity function of a bilateral filter according to the target noise estimation value interval; perform denoising on the video in the live scene using the set bilateral filter to obtain a denoised video; wherein, when the target scene category is the first scene, the value interval (A, B) of the standard deviation is the interval (0.05, 0.2); when the target scene category is the second scene, the value interval (A, B) of the standard deviation is the interval (0, 0.12); when the target scene category is the third scene, the value interval (A, B) of the standard deviation is the interval (0.2, 0.4); and when the target scene category is the fourth scene, the value interval (A, B) of the standard deviation is the interval (0.3, 0.6).

6. An image processing device, characterized in that The image processing device comprises: A memory and a processor, wherein an image processing program is stored in the memory, and when the image processing program is executed by the processor, the steps of the image denoising method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image denoising method according to any one of claims 1 to 4.

8. A computer program product comprising computer instructions, characterized in that The computer instructions are stored in a computer-readable storage medium, and when read and executed by a processor of an image processing device, enable the image processing device to implement the steps of the image denoising method according to any one of claims 1 to 4.

Images