Image denoising method and device based on neural network, electronic device and medium

Abstract

The present disclosure provides a neural network-based image denoising method and device, an electronic device and a medium. The image denoising method comprises: respectively blocking each frame image in a video frame sequence to obtain a plurality of image blocks respectively having corresponding numbers; performing denoising processing on the image block of the selected number in the current frame image based on a first neural network to obtain a first image; performing denoising processing on each image block in the current frame image based on a second neural network to obtain a second image; and fusing the first image and the second image to obtain a fused image. The present disclosure adopts a dual-network fusion strategy for image denoising, fully utilizes the advantages of neural networks with different complexities, and realizes the goal of optimizing the denoising effect of the static region with low complexity and low power consumption. In addition, by adopting the blocking processing strategy, the data transmission amount is greatly reduced without increasing the additional bandwidth consumption, and the data transmission efficiency is improved.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing technology, and relates to image denoising methods and apparatuses, electronic devices and media based on neural networks. Background Technology

[0002] In the field of image processing, Artificial Intelligence Image Signal Processor (AIISP) technology, with its superior performance and advanced algorithms, has been widely applied in numerous fields such as security monitoring. Compared to traditional image processing algorithms, AIISP technology demonstrates significant advantages in processing results, clearly exceeding the performance limits of traditional algorithms and bringing higher image processing quality and more accurate analysis results to related fields.

[0003] However, with the continuous development of technology and the increasing demands of applications, AIISP technology also faces some challenges that urgently need to be addressed in practical applications, including the power consumption problem caused by high computing demands. As a core component of AIISP technology, the Neural Processing Unit (NPU) requires a large amount of computing resources when performing complex image processing tasks, which directly leads to extremely high power consumption requirements. This characteristic severely limits AIISP technology in some battery-powered product applications. For example, in scenarios such as mobile devices and portable monitoring equipment, excessive power consumption can seriously affect the device's battery life, thus restricting the widespread application of AIISP technology.

[0004] Currently, existing technologies reduce power consumption and computational resource consumption by simplifying the complexity of network models. While this approach does alleviate high power consumption to some extent, it inevitably brings some negative impacts. As the complexity of the network model decreases, its ability to represent and learn image information weakens, inevitably leading to a decline in image quality. This not only affects the visual effect of the image but may also cause incorrect judgments and decisions in applications with high image quality requirements, such as medical image analysis and high-precision target recognition. Summary of the Invention

[0005] This disclosure provides a neural network-based image denoising method, apparatus, electronic device, and medium, which can solve the problem of high computational power consumption faced by existing technologies in the image denoising process.

[0006] In a first aspect, embodiments of this disclosure provide an image denoising method based on a neural network. The method includes: dividing each frame image in a video frame sequence into blocks to obtain multiple image blocks, each with a corresponding number; performing denoising processing on the selected numbered image blocks in the current frame image based on a first neural network to obtain a first image, wherein the selected number of the image block corresponds to the frame number of the current frame image in the video frame sequence; performing denoising processing on each image block in the current frame image based on a second neural network to obtain a second image, wherein the complexity of the second neural network is lower than the complexity of the first neural network; and fusing the first image and the second image to obtain a fused image.

[0007] In one implementation of the first aspect, denoising the image block with the selected number in the current frame image based on the first neural network includes: obtaining the frame number of the current frame image in the video frame sequence; determining the image block number to be processed by the first neural network based on the frame number; obtaining the image block corresponding to the image block number to be processed from the current frame image; and inputting the obtained image block into the first neural network for denoising processing to obtain a denoised image block.

[0008] In one implementation of the first aspect, denoising each image block in the current frame image based on the second neural network includes: traversing and obtaining each image block in the current frame image; inputting each obtained image block into the second neural network for denoising to obtain each denoised image block; and combining each denoised image block.

[0009] In one implementation of the first aspect, fusing the first image and the second image includes: traversing and obtaining each of the image blocks in the current frame image, determining whether the current image block is the image block with the selected number; if so, determining the pixels in the current image block based on the first image and the second image; otherwise, determining the pixels in the current image block based on the previous frame image and the second image.

[0010] In one implementation of the first aspect, determining the pixels within the current image block based on the first image and the second image includes: obtaining motion decision weights for each pixel within the current image block based on a motion detection algorithm; dividing the current image block into a moving region and a stationary region based on the motion decision weights; determining the pixels within the stationary region based on the first image; and determining the pixels within the moving region based on the second image.

[0011] In one implementation of the first aspect, determining the pixels within the current image block based on the previous frame image and the second image includes: obtaining motion decision weights for each pixel within the current image block based on a motion detection algorithm; dividing the current image block into a moving region and a stationary region based on the motion decision weights; determining the pixels within the stationary region based on the previous frame image; and determining the pixels within the moving region based on the second image.

[0012] In one implementation of the first aspect, determining the pixels within the static region based on the previous frame image includes: determining whether the current image block is the image block preceding the selected number; if so, obtaining the pixel corresponding to the static region position from the first image output by the first neural network corresponding to the previous frame image, and using it as the pixel of the static region; otherwise, obtaining the pixel corresponding to the static region position from the second image output by the second neural network corresponding to the previous frame image, and using it as the pixel of the static region.

[0013] In one implementation of the first aspect, dividing the motion region and stationary region of the current image block based on the motion decision weight includes: traversing and obtaining each pixel in the current image block, and determining whether the motion decision weight of the current pixel is less than a preset threshold; if so, then the current pixel is used as a pixel of the motion region; otherwise, the current pixel is used as a pixel of the stationary region.

[0014] In one implementation of the first aspect, dividing each frame image in the video frame sequence into blocks includes: dividing each frame image in the video frame sequence into blocks using a uniform block division strategy.

[0015] Secondly, embodiments of this disclosure provide an image denoising apparatus based on a neural network. The apparatus includes: a block segmentation module configured to segment each frame image in a video frame sequence into blocks, obtaining multiple image blocks each with a corresponding number; a first network module configured to perform denoising processing on the image blocks with selected numbers in the current frame image based on a first neural network, obtaining a first image, wherein the selected number of the image block corresponds to the frame number of the current frame image in the video frame sequence; a second network module configured to perform denoising processing on each image block in the current frame image based on a second neural network, obtaining a second image, wherein the complexity of the second neural network is lower than the complexity of the first neural network; and an image fusion module configured to fuse the first image and the second image to obtain a fused image.

[0016] Thirdly, embodiments of this disclosure provide an electronic device. The electronic device includes: a memory configured to store an executable program; and a processor configured to execute the program to cause the electronic device to perform the image denoising method according to any one of the preceding claims.

[0017] Fourthly, this disclosure provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed, implements the image denoising method according to any of the preceding claims.

[0018] According to the neural network-based image denoising method, apparatus, device, and medium described in the embodiments of this disclosure, image denoising is performed by employing a dual-network fusion strategy, which fully leverages the advantages of neural networks with different complexities, achieving the goal of optimizing the denoising effect of static regions with low complexity and low power consumption. Furthermore, this disclosure employs a block processing strategy to divide high-resolution images into smaller image blocks for processing, greatly reducing data transmission volume and improving data transmission efficiency without increasing additional bandwidth consumption. Attached Figure Description

[0019] Figure 1 The diagram shown is a structural schematic of a mobile terminal according to an embodiment of the present disclosure.

[0020] Figure 2 The flowchart shown is an embodiment of the neural network-based image denoising method according to this disclosure.

[0021] Figure 3 The diagram shown is a schematic representation of a uniform segmentation strategy according to this disclosure in one embodiment.

[0022] Figure 4 The diagram shown is a schematic representation of the structure of a first neural network according to this disclosure in one embodiment.

[0023] Figure 5 The diagram shown is a schematic representation of the structure of a second neural network according to this disclosure in one embodiment.

[0024] Figure 6 The flowchart shown is a flowchart of another embodiment of the neural network-based image denoising method according to this disclosure.

[0025] Figure 7 This is a schematic diagram showing the positions of the current image frame and the current image block in one embodiment according to the present disclosure.

[0026] Figure 8 The diagram shown is a schematic representation of a neural network-based image denoising apparatus according to the present disclosure in one embodiment.

[0027] Figure 9 The diagram shown is a schematic representation of the electronic device according to this disclosure and its structure in one embodiment. Detailed Implementation

[0028] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0029] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0030] Furthermore, the use of terms such as "first" and "second" in this disclosure is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this disclosure.

[0031] The following embodiments of this disclosure provide a neural network-based image denoising method, apparatus, electronic device, and medium. They employ a dual-network fusion strategy for image denoising, fully leveraging the advantages of neural networks with varying complexities to achieve the goal of optimizing denoising effects in static regions with low complexity and low power consumption. Furthermore, this disclosure uses a block-based processing strategy to divide high-resolution images into smaller image blocks for processing, significantly reducing data transmission volume and improving data transmission efficiency without increasing additional bandwidth consumption.

[0032] In some embodiments, when processing image data, a block-based strategy is adopted, dividing the entire image into N image blocks of size m×m. Each image block has a unique number, starting from 0 and incrementing sequentially until the (N-1)th block. The image frames divided into image blocks are input into both neural network A and neural network B. Due to bandwidth constraints, neural network A processes only one small block of the image at a time. Specifically, starting from frame 0 of the input image sequence, neural network A first processes the block numbered 0. When processing frame 1, neural network A processes the image block numbered 1. This continues until frame (N-1), when neural network A processes the block numbered N-1; when processing frame N, neural network A returns to processing the image block numbered 0. If the current image frame being processed is frame f, the image block number processed by neural network A can be represented by nrFlag = f % N (modulo operation). This formula ensures that the sequential processing of image blocks is cyclical, effectively utilizing limited bandwidth resources. Furthermore, the less complex neural network B sequentially traverses and processes each image block of the entire image. Because neural network A is more complex, its output results are superior to those of neural network B in terms of detail and clarity.

[0033] In some embodiments, during the network processing stage, for the same frame of noisy image, the inputs are processed into two different neural networks, A and B, resulting in two different outputs, denoted as NetA and NetB, respectively. In NetA, only the block numbered nrFlag undergoes denoising processing; other blocks retain their input results. In NetB, each image block in the current frame is processed sequentially, with all blocks processed by the B neural network. Additionally, the block numbered nrFlag in the current frame undergoes further processing by the A neural network. During the network fusion stage, the final output for each pixel in each image block is determined based on the motion weight of each pixel, selecting either NetA or NetB as the neural network output. After all image blocks have been processed in sequence, all stationary regions in the image will receive the processing result from the previous frame, i.e., the result from the A neural network, while moving regions will receive the result from the B neural network. Through dual-network fusion, moving regions receive the result from the B neural network, and stationary regions receive the result from the A neural network. By increasing the complexity of the A neural network's processing of only one image block per frame, the effect on stationary regions can be optimized, resulting in better detail clarity in stationary regions.

[0034] According to embodiments of this disclosure, two neural networks of different complexities are designed based on deep learning algorithms to perform image denoising processing on each frame of a video image, and the results of the two neural networks are fused according to the weights of the moving regions. This achieves a significant improvement in denoising performance for static areas of the image by increasing the complexity only slightly.

[0035] The technical solution disclosed herein can be applied to the workflow of an image signal processor (ISP), immediately following the temporal denoising stage, and working closely with other processing modules to jointly complete the image processing and optimization tasks.

[0036] The neural network-based image denoising method provided in this disclosure can run on mobile terminals, computer terminals, and similar devices. Taking the mobile terminal as an example, Figure 1 This is a hardware structure block diagram of the mobile terminal, such as... Figure 1 As shown, a mobile terminal may include a processor and a memory, wherein the processor may be a central processing unit and the memory is used to store data. Figure 1 The mobile terminal in the example is for illustrative purposes only and is not intended to limit the specific structure of the mobile terminal.

[0037] Optionally, the mobile terminal may further include: a communication transmission device and an input / output device.

[0038] Optionally, the memory can be used to store computer programs, such as application software programs and modules. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0039] Optionally, the communication transmission device can be used to receive or send data via a network, which may include a wireless network provided by the mobile terminal's communication provider. The communication transmission device may include a NIC (Network Interface Controller), which can be connected to other network devices via a base station to communicate with the Internet.

[0040] The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0041] Figure 2 This is a flowchart illustrating a neural network-based image denoising method according to an embodiment of the present disclosure. Figure 2 As shown, the image denoising method based on neural networks provided in this embodiment includes the following steps S100 to S400.

[0042] In step S100, each frame image in the video frame sequence is divided into blocks to obtain multiple image blocks with corresponding numbers.

[0043] In image denoising, high-resolution images typically need to be processed. To effectively avoid memory overflow issues and reduced processing efficiency caused by loading the entire image at once, this disclosure employs a block-based processing strategy.

[0044] In one embodiment of this disclosure, a uniform block division strategy is used to divide each frame of the video frame sequence into blocks.

[0045] Specifically, each frame of video image is divided into multiple image blocks of equal size, and each image block is temporarily stored in static random access memory (SRAM) during processing. Therefore, when the neural network processing unit (NPU) executes the denoising algorithm, it can directly read data from the SRAM without needing to transmit data through dynamic random access memory (DDR), thus significantly reducing bandwidth consumption and improving processing speed and efficiency.

[0046] Figure 3 This is a schematic diagram illustrating a uniform segmentation strategy according to an embodiment of the present disclosure. Based on this uniform segmentation strategy, the entire video image can be divided into N image blocks of size m×m. Each image block is assigned a unique number, starting from 0 and sequentially increasing until the (N-1)th block. Figure 3 As shown, the video image is divided into 45 image blocks. The first image block is marked as 0, the second image block is marked as 1, and so on, with the 45th image block being marked as 44.

[0047] It should be noted that when implementing the uniform block division strategy, the size of the image blocks being processed must be considered. Image blocks that are too small may lead to reduced computational efficiency, while image blocks that are too large may not effectively save memory resources. In practical applications, a comprehensive consideration and optimization based on the specific hardware platform, algorithm characteristics, and image characteristics can be performed to achieve a balance between computational efficiency and memory resource utilization.

[0048] In this implementation, the block processing strategy overcomes the challenge of limited hardware resources by dividing the high-resolution image into smaller image blocks for processing, ensuring that the system can still maintain efficient and stable operation under high load. Furthermore, the block processing strategy ensures that the NPU operations required by this algorithm can be completed in SRAM, without increasing bandwidth consumption except for data input and output, greatly reducing data transfer volume and improving data transfer efficiency.

[0049] In step S200, the image block with the selected number in the current frame image is denoised based on the first neural network to obtain the first image. The selected number of the image block corresponds to the frame number of the current frame image in the video frame sequence.

[0050] Specifically, the first neural network described in this embodiment can be a convolutional neural network with residual structure or a recurrent neural network (RNN). In addition, other network structures can also be used, including but not limited to MobileNet, ShuffleNet, UNet, etc.

[0051] In one embodiment of this disclosure, step S200 of denoising the selected image block in the current frame image based on the first neural network may include: obtaining the frame number of the current frame image in the video frame sequence; determining the image block number to be processed by the first neural network based on the frame number; obtaining the image block corresponding to the image block number to be processed from the current frame image; and inputting the obtained image block into the first neural network for denoising processing to obtain a denoised image block.

[0052] Specifically, the current frame image can be any frame in the video frame sequence. For example, if the current frame image is the f-th frame in the video frame sequence, and the current frame image is uniformly divided into N image blocks, then the image block number to be processed by the first neural network can be represented as nrFlag = f % N (modulo operation). This means that in the f-th frame image, only the image block numbered nrFlag undergoes denoising processing, while the other blocks retain their input state unchanged.

[0053] For example, starting from frame 0 of the video frame sequence, the first neural network processes the image block numbered 0. When processing frame 1, it operates on the image block numbered 1. This continues until frame 44, when the first neural network processes the image block numbered 44. When processing frame 45, the first neural network then re-denoises the image block numbered 0.

[0054] In this implementation, the sequential processing of image blocks is ensured to be carried out cyclically, with only one image block from the current frame image being input each time, thus effectively utilizing limited bandwidth resources.

[0055] In step S300, the image blocks in the current frame image are denoised based on the second neural network to obtain the second image. The complexity of the second neural network is lower than that of the first neural network.

[0056] Specifically, the first neural network, due to its higher complexity, exhibits superior denoising performance and output image quality. In contrast, the second neural network is a simplified version of the first neural network. Although its complexity is reduced and its denoising performance is slightly inferior to the first neural network, it still maintains relatively effective processing capabilities. Due to the high complexity of the first neural network, its output surpasses that of the second neural network in terms of detail clarity.

[0057] It should be noted that the first neural network and the second neural network can be of the same or different types. As long as the complexity of the first neural network is higher than that of the second neural network and it can provide better image processing quality, it can be applied to this solution.

[0058] In one embodiment of this disclosure, step S300 of denoising each image block in the current frame image based on the second neural network may include: traversing and obtaining each image block in the current frame image; inputting each obtained image block into the second neural network for denoising to obtain each denoised image block; and combining the denoised image blocks.

[0059] In step S400, the first image and the second image are fused to obtain a fused image.

[0060] In one embodiment of this disclosure, the step S400 of fusing the first image and the second image may include: traversing and obtaining each of the image blocks in the current frame image, determining whether the current image block is the image block with the selected number; if so, determining the pixels in the current image block based on the first image and the second image; otherwise, determining the pixels in the current image block based on the previous frame image and the second image.

[0061] According to the embodiments of this disclosure, by adopting a block processing and dual-network fusion strategy, the computing power of the first neural network for processing a single image block is increased only, so that the effect of processing the entire static area of ​​the image using the first neural network can be achieved.

[0062] Camera sensors typically use Bayer filter arrays to capture image data. In this structure, each pixel can only record information from one of the three colors: red, green, or blue. Common pixel matrix arrangements include RGGB, GRBG, BGGR, and GBRG. RGGB, for example, represents a 2×2 pixel matrix containing two green pixels and one each of red and blue pixels.

[0063] Figure 4 This is a schematic diagram illustrating a first neural network structure according to an embodiment of the present disclosure. (See attached diagram.) Figure 4 As shown, the input image block of the first neural network has five channels, four of which correspond to the raw data of RGGB, and the other channel corresponds to the sigma value calculated based on the brightness information of the image. The sigma value is a statistical measure that can represent the standard deviation, variance, or other measures related to the brightness distribution in the image.

[0064] The first neural network achieves denoising of input image patches through a series of convolution operations. Specifically, the convolution operation process of the first neural network includes the following (i1) to (i8):

[0065] (i1) The input image block x first passes through a convolutional layer with 5 input channels, 32 output channels, and a kernel size of 1x1, and outputs y0;

[0066] (i2)y0 passes through two convolutional layers with 32 input channels, 32 output channels, and a kernel size of 3x3, and outputs y1;

[0067] (i3)y1 passes through a convolutional layer with 32 input channels, 32 output channels, a stride of 2, and a kernel size of 3x3, and outputs y2;

[0068] (i4)y2 passes through 6 convolutional layers with 32 input channels, 32 output channels, and a kernel size of 3x3, and outputs y3;

[0069] (i5)y3 is upsampled by 2 times to output y4;

[0070] (i6) Perform matrix addition on y1 and y4, and output y5;

[0071] (i7)y5 passes through two convolutional layers with 32 input channels, 32 output channels, and a kernel size of 3x3, and outputs y6;

[0072] (i8)y6 passes through a convolutional layer with 32 input channels, 4 output channels, and a kernel size of 3x3, and outputs the final image block out.

[0073] Single-point complexity refers to the number of multiplication and addition operations required to complete one forward inference computation for a single pixel. Reducing single-point complexity can improve the computational efficiency of neural networks and reduce algorithm processing time.

[0074] In this embodiment of the disclosure, the single-point complexity of the first neural network is:

[0075] 1*1*5*32+3*3*32*32*2+3*3*32*32 / 4*(1+5+1)+3*3*32*32*2+3*3*32*4=54304.

[0076] The number of network weights is a learnable parameter in a neural network that determines how the network transforms / processes input data. The number of weights can serve as a direct indicator of network complexity. A higher number of weights enhances the network's parameterization ability, theoretically enabling it to capture more complex function mappings and thus improve model performance. However, increasing the number of weights also increases memory requirements and computation time. While increasing the number of weights can improve model performance, it also leads to higher computational costs. Therefore, in practical applications, a trade-off must be struck between network performance and the number of weights.

[0077] In this embodiment of the disclosure, the number of weights in the first neural network is:

[0078] 5*32*1*1+32*32*3*3*(2+1+5+1+2)+3*3*32*4=102688.

[0079] Figure 5 This is a schematic diagram illustrating a second neural network structure according to an embodiment of the present disclosure. (See attached diagram.) Figure 5 As shown, the number of convolutional layers in the second neural network is reduced compared to the first neural network.

[0080] The second neural network achieves global denoising of the current frame image through a series of convolution operations. Specifically, the convolution operation process of the second neural network includes the following (j1) to (j8):

[0081] (j1) The input image block x first passes through a convolutional layer with 5 input channels, 32 output channels, and a kernel size of 1x1, and outputs y0;

[0082] (j2)y0 passes through a convolutional layer with 32 input channels, 32 output channels, and a kernel size of 3x3, and outputs y1;

[0083] (j3)y1 passes through a convolutional layer with 32 input channels, 32 output channels, a stride of 2, and a kernel size of 3x3, and outputs y2;

[0084] (j4)y2 passes through 4 convolutional layers with 32 input channels, 32 output channels, and a kernel size of 3x3, and outputs y3;

[0085] (j5)y3 is upsampled by 2 times to output y4;

[0086] (j6) Perform matrix addition on y1 and y4, and output y5;

[0087] (j7)y5 passes through a convolutional layer with 32 input channels, 32 output channels, and a kernel size of 3x3, and outputs y6;

[0088] (j8)y6 passes through a convolutional layer with 32 input channels, 4 output channels, and a kernel size of 3x3, and outputs the final image block out.

[0089] In this embodiment of the disclosure, the single-point complexity of the second neural network is:

[0090] 1*1*5*32+3*3*32*32+3*3*32*32 / 4*5+3*3*32*32+3*3*32*4=31264.

[0091] The number of weights in the second neural network is:

[0092] 5*32*1*1+32*32*3*3*7+3*3*32*4=65824.

[0093] By comparing the single-point complexity and number of weights of the first neural network and the second neural network, it can be found that the single-point complexity of the second neural network is 57% of that of the first neural network, and its number of weights is 64% of that of the first neural network.

[0094] It should be noted that both the first and second neural networks support parallel computing in their data processing. By utilizing multi-core processors or distributed computing systems, the overall processing speed can be significantly improved, thereby further enhancing the efficiency and throughput of image processing.

[0095] In one embodiment of this disclosure, determining pixels within the current image block based on the first image and the second image includes: obtaining motion decision weights for each pixel within the current image block based on a motion detection algorithm; dividing the current image block into a moving region and a stationary region based on the motion decision weights; determining pixels within the stationary region based on the first image; and determining pixels within the moving region based on the second image.

[0096] Specifically, the motion detection algorithm can be any of the following methods: inter-frame differencing, background subtraction, optical flow, and feature matching. This disclosure does not impose any restrictions on the type of motion detection algorithm. For example, inter-frame differencing detects dynamic changes in a scene by comparing the differences between adjacent frames. Optical flow infers the speed and direction of an object's movement by calculating the change of each pixel over time. These methods are particularly effective for capturing the motion details of objects; other methods will not be described in detail here.

[0097] In an embodiment of the present disclosure, the motion decision weight can intuitively reflect the motion state of each pixel in the image. Dividing the motion region and the static region of the current image block based on the motion decision weight includes: traversing and acquiring each pixel in the current image block, and determining whether the motion decision weight of the current pixel is less than a preset threshold; if yes, taking the current pixel as a pixel of the motion region; otherwise, taking the current pixel as a pixel of the static region.

[0098] The motion region refers to a region with significant changes in the image, which usually corresponds to a moving object or a dynamic background. The static region refers to a region that remains stable in the image, which usually corresponds to a static background or a fixed object.

[0099] For example, the motion decision weight of pixel i is denoted as wgt i ; the preset threshold is denoted as thresh, and a common value of thresh is 0.5. If wgt i <thresh, it indicates that pixel i belongs to the motion region; if wgt i ≥thresh, it indicates that pixel i belongs to the static region.

[0100] Figure 6 is a flowchart illustrating a neural network-based image denoising method according to an embodiment of the present disclosure. As shown Figure 6 in the figure, in this embodiment, if the current image block k is the image block with the specific number nrFlag, that is, k=nrFlag, the denoising result of the current image block can be expressed as:

[0101]

[0102] where wgt i denotes the motion decision weight of pixel i in the current image block k; thresh is the preset threshold; denotes pixel i in the first image; denotes pixel i in the second image.

[0103] When wgt i <thresh, pixel i belongs to the motion region, and the denoising result of the second neural network is adopted. When wgt i ≥thresh, pixel i belongs to the static region, and the denoising result of the first neural network is adopted.

[0104] In an embodiment of the present disclosure, determining the pixels in the current image block based on the previous frame image and the second image includes: acquiring the motion decision weight of each pixel in the current image block based on a motion detection algorithm; dividing a motion region and a static region of the current image block based on the motion decision weights; determining the pixels in the static region based on the previous frame image; and determining the pixels in the motion region based on the second image.

[0105] As shown in Figure 6 , in this embodiment, if the current image block k is not the image block with the specific number nrFlag, that is, k≠nrFlag, the denoising result of the current image block can be expressed as:

[0106]

[0107] where wgt i represents the motion decision weight of pixel i in image block k; thresh is a preset threshold; Pre i (x i ) represents pixel i in the previous frame image; represents pixel i in the second image.

[0108] When wgt i <thresh, pixel i belongs to the motion region, and the denoising result of the second neural network is adopted. When wgt i ≥thresh, pixel i belongs to the static region, and the denoising result of the previous frame image is adopted.

[0109] It should be noted that adopting the denoising result of the previous frame image means that it will not be input into the first neural network for denoising processing anymore. Since the current image block k≠nrFlag, the first neural network has not processed the current image block k, and since it belongs to the static region, the processing result of the previous frame can be directly selected for substitution.

[0110] In an embodiment of the present disclosure, determining the pixels in the static region based on the previous frame image includes: judging whether the current image block is an image block before the selected number; if yes, acquiring the pixel corresponding to the position of the static region from the first image corresponding to the previous frame image output by the first neural network, as the pixel of the static region; otherwise, acquiring the pixel corresponding to the position of the static region from the second image corresponding to the previous frame image output by the second neural network, as the pixel of the static region.

[0111] Figure 7 is a schematic diagram showing the positions of the current image frame and the current image block according to an embodiment of the present disclosure. As shown in Figure 7As shown, nrFlag represents the image patch number in the current frame image processed by the first neural network. Since the triangular region moves from the dashed line to the solid line, the triangular region is the moving region, and the other regions are the stationary regions. Therefore, for the current image frame, the moving region uses the result of the second neural network processing; for the stationary region before image patch nrFlag, the denoising result of the previous frame image can be used, i.e., the processing result of the first neural network on the previous frame image; for the stationary region after image patch nrFlag, the processing result of the second neural network on the previous frame image can be used.

[0112] If the resolution of each frame in a video frame sequence is represented as W×H, and each frame can be divided into a×b image blocks, then the size of each image block is... The single-point complexity of the first neural network performing NPU operations is N. A The single-point complexity of the second neural network performing NPU operations is N. B N A >N B .

[0113] Ignoring factors such as edge expansion, the computational power required for the first neural network to process a single frame of image is W×H×N. A The computational power required for the second neural network to process a single frame image is W×H×N. B The computational power of the first neural network to process a single image patch is... The computing power required to adopt the block processing and dual-network fusion strategy is:

[0114]

[0115] The computing power mentioned here refers to the total amount of computation required to perform a network forward inference on the entire image or all pixels in a certain image block.

[0116] In this implementation, only the computational power required to process a single image patch in the first neural network is increased to achieve the effect of optimizing static areas. In other words, better detail clarity in static areas is achieved with relatively low complexity.

[0117] It should be noted that the protection scope of the image denoising method based on neural networks described in this disclosure is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this disclosure is included within the protection scope of this disclosure.

[0118] Figure 8 This is a schematic diagram illustrating the structure of a neural network-based image denoising apparatus according to an embodiment of the present disclosure. Figure 8As shown, the neural network-based image denoising device provided in this embodiment includes: a block module, a first network module, a second network module, and an image fusion module.

[0119] The segmentation module is configured to segment each frame image in the video frame sequence into blocks, resulting in multiple image blocks each with a corresponding number.

[0120] The first network module is configured to perform denoising processing on the image block with the selected number in the current frame image based on the first neural network to obtain a first image, wherein the selected number of the image block corresponds to the frame number of the current frame image in the video frame sequence.

[0121] The second network module is configured to perform denoising processing on each image block in the current frame image based on the second neural network to obtain the second image. The complexity of the second neural network is lower than that of the first neural network.

[0122] The image fusion module is configured to fuse the first image and the second image to obtain a fused image.

[0123] It should be noted that the structure and principle of the block module, the first network module, the second network module and the image fusion module described in this embodiment correspond one-to-one with the steps in the above-mentioned neural network-based image denoising method, so they will not be repeated here.

[0124] The neural network-based image denoising apparatus provided in this disclosure can implement the neural network-based image denoising method described in this disclosure. However, the apparatus for implementing the neural network-based image denoising method described in this disclosure includes, but is not limited to, the structure of the neural network-based image denoising apparatus listed in this embodiment. Any structural modifications and substitutions of the prior art made in accordance with the principles of this disclosure are included within the protection scope of this disclosure.

[0125] Figure 9 This is a schematic diagram illustrating the terminal structure according to an embodiment of the present disclosure. Figure 9 As shown, the electronic device provided in this embodiment includes a processor and a memory. The memory is configured to store an executable program. The processor is configured to execute the program to cause the electronic device to perform the image denoising method according to any of the preceding claims.

[0126] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0127] This embodiment also includes one or more of the following: multimedia components, input / output (I / O) interfaces, and communication components.

[0128] The multimedia component may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is configured to output and / or input audio signals. For example, the audio component may include a microphone configured to receive external audio signals. The received audio signals may be further stored in memory or transmitted via a communication component. The audio component also includes at least one speaker configured to output audio signals. The I / O interface provides an interface between the processor and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. The communication component is configured to enable wired or wireless communication between the timer and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0129] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0130] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0132] This disclosure also provides a computer-readable storage medium on which a computer program is stored, which, when executed, implements the image denoising method described in any of the preceding embodiments. Those skilled in the art will understand that all or part of the steps in implementing the methods of the above embodiments can be executed by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The aforementioned storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0133] This disclosure also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this disclosure are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0134] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0135] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0136] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. An image denoising method based on neural networks, characterized in that, include: Each frame in the video frame sequence is divided into blocks to obtain multiple image blocks with corresponding numbers; The first image is obtained by denoising the selected number of the image block in the current frame image based on the first neural network, and the selected number of the image block corresponds to the frame number of the current frame image in the video frame sequence. The second image is obtained by denoising each image block in the current frame image based on the second neural network. The complexity of the second neural network is lower than that of the first neural network. as well as The first image and the second image are merged to obtain a merged image.

2. The image denoising method according to claim 1, characterized in that, Denoising the selected image patch in the current frame image based on the first neural network includes: Obtain the frame number of the current frame image in the video frame sequence; The image block number to be processed by the first neural network is determined based on the frame sequence number; Obtain the image block corresponding to the image block number to be processed from the current frame image; and The acquired image block is input into the first neural network for denoising processing to obtain a denoised image block.

3. The image denoising method according to claim 1, characterized in that, Denoising processing of each image block in the current frame image based on the second neural network includes: Iterate through and obtain each image block in the current frame image; The acquired image blocks are input into the second neural network for denoising processing to obtain denoised image blocks; and The denoised image blocks are then combined.

4. The image denoising method according to claim 1, characterized in that, Fusing the first image and the second image includes: Iterate through each image block in the current frame image and determine whether the current image block is the image block with the selected number; If so, then the pixels within the current image block are determined based on the first image and the second image; Otherwise, the pixels within the current image block are determined based on the previous frame image and the second image.

5. The image denoising method according to claim 4, characterized in that, Determining the pixels within the current image block based on the first image and the second image includes: The motion decision weights of each pixel within the current image block are obtained based on a motion detection algorithm. The motion and stationary regions of the current image block are divided based on the motion decision weights; Pixels within the static area are determined based on the first image; and Pixels within the motion region are determined based on the second image.

6. The image denoising method according to claim 4, characterized in that, Determining the pixels within the current image block based on the previous frame image and the second image includes: The motion decision weights of each pixel within the current image block are obtained based on a motion detection algorithm. The motion and stationary regions of the current image block are divided based on the motion decision weights; Pixels within the static area are determined based on the previous frame image; and Pixels within the motion region are determined based on the second image.

7. The image denoising method according to claim 6, characterized in that, Determining the pixels within the stationary region based on the previous frame image includes: Determine whether the current image block is the image block preceding the selected number; If so, then from the first image output by the first neural network corresponding to the previous frame image, the pixel corresponding to the position of the stationary region is obtained as the pixel of the stationary region; Otherwise, the pixel corresponding to the position of the stationary region is obtained from the second image output by the second neural network that corresponds to the previous frame image, and is used as the pixel of the stationary region.

8. The image denoising method according to claim 6, characterized in that, Based on the motion decision weights, the motion region and stationary region of the current image block are divided into: Iterate through each pixel in the current image block and determine whether the motion decision weight of the current pixel is less than a preset threshold. If so, then the current pixel is taken as the pixel of the motion region; Otherwise, the current pixel is taken as the pixel of the stationary region.

9. The image denoising method according to claim 1, characterized in that, Dividing the video frame sequence into blocks includes: A uniform block division strategy is used to divide each frame of the video frame sequence into blocks.

10. An image denoising device based on a neural network, characterized in that, include: The segmentation module is configured to segment each frame image in the video frame sequence into blocks, resulting in multiple image blocks with corresponding numbers. The first network module is configured to perform denoising processing on the image block with the selected number in the current frame image based on the first neural network to obtain the first image, wherein the selected number of the image block corresponds to the frame number of the current frame image in the video frame sequence. The second network module is configured to perform denoising processing on each image block in the current frame image based on the second neural network to obtain the second image. The complexity of the second neural network is lower than that of the first neural network. as well as The image fusion module is configured to fuse the first image and the second image to obtain a fused image.

11. An electronic device, characterized in that, include: The memory is configured to store executable programs; as well as A processor is configured to execute the program to cause the electronic device to perform the image denoising method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the image denoising method according to any one of claims 1 to 9.

Images