A video noise reduction method, apparatus, electronic device, and storage medium.

Abstract

This application provides a video denoising method, apparatus, electronic device, and storage medium, relating to the field of image processing technology, and particularly to a video denoising method, apparatus, electronic device, and storage medium. The specific implementation involves: obtaining a first feature map of a target video frame and second feature maps of a preset number of adjacent frames of the target video frame; fusing each image block in each of the first feature maps with similar image blocks determined in each of the second feature maps of the same scale to obtain a fused image block; arranging the obtained fused image blocks according to the positions of the image blocks in each of the first feature maps to obtain a fused feature map at the scale of the first feature map; and obtaining a denoised video frame of the target video frame based on the obtained fused feature map. Applying the solution provided in this application can reduce the computational resources consumed by video denoising.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video noise reduction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Nowadays, videos are increasingly appearing in people's work and daily lives. However, due to factors such as video compression algorithms and network bandwidth, noise may be introduced into video frames, which can lead to a decrease in video quality. Therefore, noise reduction processing is necessary for videos.

[0003] Currently, when performing noise reduction on videos, methods such as optical flow are generally used. However, optical flow is computationally complex and consumes a lot of computing resources. Summary of the Invention

[0004] The purpose of this application is to provide a video noise reduction method, apparatus, electronic device, and storage medium to reduce the computational resources consumed by video noise reduction. The specific technical solution is as follows:

[0005] According to one aspect of the embodiments of this application, a video noise reduction method is provided, the method comprising:

[0006] A first feature map of a target video frame and a second feature map of a preset number of adjacent frames of the target video frame are obtained, wherein the first feature map includes a first feature sub-map of multiple scales, and each second feature map includes a second feature sub-map of multiple scales.

[0007] For each image block in each first feature sub-map, similar image blocks are determined in each second feature sub-map with the same scale as the first feature sub-map, and the image block and its similar image blocks are fused to obtain a fused image block;

[0008] For each first feature sub-image, the obtained fused image blocks are arranged according to the position of each image block in the first feature sub-image to obtain the fused feature map at the scale of the first feature sub-image;

[0009] Based on the obtained fused feature map, a denoised video frame is obtained from the target video frame.

[0010] In one embodiment of this application, determining similar image blocks in each second feature sub-image of the same scale as the first feature sub-image for each image block in each first feature sub-image includes:

[0011] For each image block in the third feature sub-map, similar image blocks are determined in each of the second feature sub-maps with the same scale as the third feature sub-map, wherein the third feature sub-map is the feature sub-map with the smallest scale in the first feature sub-map;

[0012] Based on the image positions of similar image blocks of each image block in the third feature sub-map, similar image blocks of each image block in the other first feature sub-maps are determined in each second feature sub-map with the same scale as the other first feature sub-maps.

[0013] In one embodiment of this application, each first sub-feature map contains the same number of image blocks.

[0014] In one embodiment of this application, the step of fusing the image block and its similar image blocks to obtain a fused image block includes:

[0015] Following the playback order of video frames from front to back, the image blocks in the first image block group are concatenated and fused to obtain the first fused image block. The first image block group includes: similar image blocks of the image block in the second feature map of the forward adjacent frame and the image block itself.

[0016] Following the video frame playback order from back to front, the image blocks in the second image block group are concatenated and fused to obtain the second fused image block. The second image block group includes: similar image blocks of the image block in the second feature map of the next adjacent frame and the image block itself.

[0017] The first fused image block and the second fused image block are fused together to obtain a fused image block.

[0018] In one embodiment of this application, obtaining a first feature map of a target video frame and second feature maps of a preset number of adjacent frames of the target video frame includes:

[0019] The target video frame and a preset number of adjacent frames of the target video frame are input into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining the first feature map of the target video frame and the second feature map of the preset number of adjacent frames of the target video frame output by the feature extraction layer. The video denoising model further includes: an image patch matching layer, an image patch fusion layer, a rearrangement layer and a reconstruction layer.

[0020] For each image block in the first feature map, determining similar image blocks in each of the obtained second feature maps includes:

[0021] The first feature map and each of the second feature maps are input into the image block matching layer to perform image block matching, and the similar image blocks output by the image block matching layer that are determined in each of the second feature maps are obtained respectively.

[0022] The process of fusing the image block and its similar image blocks to obtain a fused image block includes:

[0023] The image block and its similar image blocks are input into the image block fusion layer to perform image block fusion, and the fused image block output by the image block fusion layer is obtained.

[0024] The step of arranging the obtained fused image blocks according to the positions of each image block in the first feature map to obtain the fused feature map includes:

[0025] The obtained fused image blocks are input into the rearrangement layer, and the obtained fused image blocks are arranged to obtain a fused feature map output by the rearrangement layer for each first feature sub-map, which corresponds to the position of each image block in the first feature sub-map and has the same scale.

[0026] The step of obtaining the denoised video frame of the target video frame based on the obtained fused feature map includes:

[0027] The obtained fused feature map is input into the reconstruction layer to reconstruct the video frame, and the video frame output by the reconstruction layer after denoising the target video frame is obtained.

[0028] In one embodiment of this application, the reconstruction layer reconstructs video frames in the following manner:

[0029] The first fused feature map with the smallest scale is subjected to residual walking transformation to obtain the first processing result;

[0030] The first processing result is upsampled to obtain a second processing result with the same scale as the second fused feature map. The second fused feature map is the fused feature map with the smallest scale that has not undergone residual walk transformation.

[0031] The second processing result and the second fused feature map are fused together, and the fused feature map is subjected to residual walk transformation to obtain a third fused feature map. If there is a fused feature map that has not undergone residual walk transformation, the first processing result is updated to the third fused feature map, and the step of upsampling the first processing result is returned.

[0032] The third fused feature map is then subjected to convolution transformation. The convolution result is then added to the target video frame pixel by pixel to obtain the denoised video frame.

[0033] According to another aspect of the embodiments of this application, a video noise reduction apparatus is provided, the apparatus comprising:

[0034] The feature map acquisition module is used to acquire a first feature map of a target video frame in the video and a second feature map of a preset number of adjacent frames of the target video frame, wherein the first feature map includes a first feature sub-map of multiple scales, and each second feature map includes a second feature sub-map of multiple scales.

[0035] The similar image block determination module is used to determine the similar image blocks of each image block in each second feature sub-image with the same scale as the first feature sub-image for each image block in each first feature sub-image, and to fuse the image block and its similar image blocks to obtain a fused image block.

[0036] The image block arrangement module is used to arrange the obtained fused image blocks according to the position of each image block in the first feature sub-image for each first feature sub-image, so as to obtain the fused feature map at the scale of the first feature sub-image.

[0037] The denoised video frame acquisition module is used to obtain a denoised video frame of the target video frame based on the obtained fused feature map.

[0038] In one embodiment of this application, the similar image block determination module is specifically used to determine similar image blocks for each image block in the third feature sub-map in each of the second feature sub-maps with the same scale as the third feature sub-map, wherein the third feature sub-map is the feature sub-map with the smallest scale in the first feature sub-map; based on the image positions of the similar image blocks of each image block in the third feature sub-map, similar image blocks of each image block in each of the other first feature sub-maps with the same scale as the other first feature sub-maps are determined respectively.

[0039] In one embodiment of this application, each first sub-feature map contains the same number of image blocks.

[0040] In one embodiment of this application, the similar image block determination module is specifically used to perform cascaded fusion of image blocks in a first image block group in the order of video frame playback from front to back to obtain a first fused image block, wherein the first image block group includes: similar image blocks and the image block in the second feature map of the image block in the forward adjacent frame; and to perform cascaded fusion of image blocks in a second image block group in the order of video frame playback from back to front to obtain a second fused image block, wherein the second image block group includes: similar image blocks and the image block in the second feature map of the image block in the backward adjacent frame; and to fuse the first fused image block and the second fused image block to obtain a fused image block.

[0041] In one embodiment of this application, the feature map acquisition module is specifically used to input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining a first feature map of the target video frame and a second feature map of the preset number of adjacent frames of the target video frame output by the feature extraction layer. The video denoising model further includes: an image patch matching layer, an image patch fusion layer, a rearrangement layer, and a reconstruction layer.

[0042] The similar image block determination module is specifically used to input the first feature map and each of the second feature maps into the image block matching layer to perform image block matching, and obtain similar image blocks of the image block determined in each of the second feature maps, which are output by the image block matching layer; input the image block and its similar image blocks into the image block fusion layer to perform image block fusion, and obtain the fused image block output by the image block fusion layer.

[0043] The image block arrangement module is specifically used to input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map that matches the position of each image block in the first feature sub-map and has the same scale.

[0044] The denoised video frame acquisition module is specifically used to input the obtained fused feature map into the reconstruction layer to reconstruct the video frame and obtain the denoised video frame output by the reconstruction layer.

[0045] In one embodiment of this application, the reconstruction layer reconstructs video frames as follows: A residual walk transform is applied to the smallest first fusion feature map to obtain a first processing result; the first processing result is upsampled to obtain a second processing result with the same scale as the second fusion feature map, wherein the second fusion feature map is the smallest fusion feature map that has not undergone residual walk transform; the second processing result and the second fusion feature map are fused, and a residual walk transform is applied to the fused feature map to obtain a third fusion feature map; if a fusion feature map that has not undergone residual walk transform exists, the first processing result is updated to the third fusion feature map, and the step of upsampling the first processing result is returned; a convolution transform is applied to the finally obtained third fusion feature map, and the resulting convolution result is added pixel-by-pixel to the target video frame to obtain a denoised video frame.

[0046] According to another aspect of the embodiments of this application, an electronic device is provided, comprising:

[0047] Memory, used to store computer programs;

[0048] The processor, when executing a program stored in memory, implements any of the above video noise reduction methods.

[0049] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and the computer program, when executed by a processor, implements any of the above-described video noise reduction methods.

[0050] According to another aspect of the embodiments of this application, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform any of the video noise reduction methods described above.

[0051] As can be seen from the above, in the solution provided by the embodiments of this application, when performing noise reduction processing, the image blocks in the first feature map of the target video frame and their similar image blocks are used for fusion. The similar image blocks used for fusion are the image blocks in the second feature map that are similar to the image blocks in the first feature map. It can be seen that the processing of feature fusion and similar image blocks is performed on the image blocks, which does not require processing of every pixel in the target video frame and adjacent video frames. This can reduce the number of objects being processed and reduce the computational resources consumed by video noise reduction.

[0052] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0054] Figure 1 A flowchart illustrating a video noise reduction method provided in an embodiment of this application;

[0055] Figure 2 A schematic diagram of an image block provided in an embodiment of this application;

[0056] Figure 3 A schematic diagram of another image block provided in an embodiment of this application;

[0057] Figure 4 A schematic diagram of a feature sub-graph provided in an embodiment of this application;

[0058] Figure 5 A schematic diagram of another feature sub-graph provided in an embodiment of this application;

[0059] Figure 6 A schematic diagram of yet another feature sub-map provided in an embodiment of this application;

[0060] Figure 7 A schematic diagram of yet another feature sub-graph provided in an embodiment of this application;

[0061] Figure 8 A schematic diagram of yet another feature sub-graph provided in an embodiment of this application;

[0062] Figure 9 A flowchart illustrating an image patch fusion method provided in an embodiment of this application;

[0063] Figure 10 This is a schematic diagram of the structure of a video noise reduction device provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0066] The execution subject of the embodiments of this application will be described below.

[0067] The solutions provided in this application can be applied to devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality / virtual reality devices, laptops, personal computers, super mobile personal computers, netbooks, or personal digital assistants. This application does not limit the specific type of device used for video noise reduction. For ease of description, the implementing entity in this application will be referred to as a noise reduction device.

[0068] The video noise reduction method provided in this application will be described in detail below through specific embodiments.

[0069] In one embodiment of this application, see Figure 1 , Figure 1 A flowchart of a video noise reduction method is provided, which includes the following steps S101-S104.

[0070] Step S101: Obtain the first feature map of the target video frame in the video, and the second feature maps of a preset number of adjacent frames of the target video frame.

[0071] The first feature map includes first feature sub-maps at multiple scales, and each second feature map includes second feature sub-maps at multiple scales.

[0072] In one implementation, the noise reduction device can normalize the pixel values ​​of the target video frame and adjacent frames in the video. Based on the normalized pixel values ​​of the target video frame, a first feature map is obtained, and based on the normalized pixel values ​​of a preset number of adjacent frames, a second feature map is obtained. For example, if a pixel component is represented by 8 bits, and the maximum value of that pixel component is 255, then pixel value normalization can be achieved by dividing the pixel value by 255.

[0073] Other implementations of obtaining the feature map in step S101 will be described in the following embodiments, and will not be detailed here.

[0074] Step S102: For each image block in each first feature sub-image, determine the similar image blocks in each second feature sub-image with the same scale as the first feature sub-image, and fuse the image block and its similar image blocks to obtain a fused image block.

[0075] The following is combined with Figure 2 The method for determining similar image blocks in each of the second feature sub-images with the same scale as the first feature sub-image in step S102 will be explained.

[0076] In one implementation, the noise reduction device can, for each image block in the second feature sub-map, find the image block with the highest similarity to that image block in the second feature sub-map, and use it as the similar image block of that image block.

[0077] join Figure 2 , Figure 2 A schematic diagram of an image block is provided. Assume the target video frame is frame t, and its first feature sub-image is divided into 16 image blocks. The two second feature sub-images of adjacent frames corresponding to this target video frame are the feature images corresponding to frames t-1 and t+1, respectively. Taking the image block in the second row and second column of the first feature sub-image (hereinafter, (x,y) represents the x-th row and y-th column, e.g., (2,2) represents the second row and second column) as an example, the similarity between each image block of the same size as the target image block in the second feature sub-image corresponding to frame t-1 can be calculated. Specifically, assuming the size of the image block is 16×16, for each second feature sub-image, the noise reduction device can determine that the pixel at (1,1) is the top-left pixel of the image block in the second feature sub-image, and the pixel at (16,16) is the bottom-right pixel of the image block in the second feature sub-image. This yields an image block of the same size as the target image block in the second feature sub-image, and the similarity between these two image blocks is calculated. Then, the pixel at (1,2) can be determined as the top-left pixel of the image block in the second feature sub-image, and the pixel at (16,17) can be determined as the bottom-right pixel of the image block in the second feature sub-image. This yields another image block in the second feature sub-image with the same size as the original image block, and the similarity between these two image blocks is calculated. Similarly, the pixel at (2,1) can be determined as the top-left pixel of the image block in the second feature sub-image, and the pixel at (17,16) can be determined as the bottom-right pixel of the image block in the second feature sub-image. This yields yet another image block in the second feature sub-image with the same size as the original image block, and the similarity between these two image blocks is calculated. Following this method, all image blocks in the second feature sub-image with the same size as the original image block can be identified, and the similarity between each image block in the second feature sub-image and the original image block can be calculated. The image block in the second feature sub-image with the highest similarity is considered a similar image block in that second feature sub-image.

[0078] In another implementation, the noise reduction device can, based on the position of the image block in the first feature sub-image, search for the image block with the highest similarity to the image block in the similar location region of the image block in each second feature sub-image, and use this as the similar image block. The similar location region of the image block is determined as follows: a similar location in the second feature sub-image that is the same as the position of the image block in the first feature sub-image is identified, and the similar location of the image block in the second feature sub-image is used as the center to determine the similar location region. For example, as... Figure 2 As shown, taking the image block (2,2) of the first feature sub-image in frame t as an example, assuming the image block size is 16×16, the top-left pixel is (17,17) and the bottom-right pixel is (32,32), then, in the second feature sub-images corresponding to frames t-1 and t+1 respectively, using the image block with the top-left pixel (17,17) and bottom-right pixel (32,32) and a size of 16×16 as the center, we can determine a similar position region with a size of 48×48, a top-left pixel (1,1) and a bottom-right pixel (48,48). Figure 2 The image is shown in bold. Therefore, the noise reduction device can, for each second feature sub-image, determine the similarity region of that second feature sub-image, and then, following the method for determining similar image blocks described in one of the above implementations, determine the similar image blocks of that image block in the second feature sub-image within the aforementioned similarity region. This reduces the number of image blocks that need to be calculated to determine similar image blocks, thus reducing computational resources.

[0079] A preset number of adjacent frames for the target video frame can include a preset number of video frames before and after the target video frame in the video frame playback order. The number of forward and backward adjacent frames of the target video frame can be equal or unequal. For example, if the preset number is 4, then adjacent frames can include the two frames before and the two frames after the target video frame in the video frame playback order; adjacent frames can also include the one frame before and the three frames after the target video frame in the video frame playback order; or adjacent frames can include the four frames after the target video frame in the video frame playback order. Assuming the target video frame is frame t in the video, with a preset number of 4, adjacent frames can include frames t-2, t-1, t+1, and t+2 in the video. A second feature map can be obtained for each adjacent frame; that is, if there are multiple adjacent frames, there are also multiple second feature maps. For example, 4 second feature maps can be obtained for the above 4 adjacent frames.

[0080] The first and second feature maps can be recorded in the form of feature matrices. For example, each pixel in a video frame can correspond to an element in the feature matrix, and their positions are also one-to-one. For instance, the pixel in the first row and first column of the video frame corresponds to the element at (0,0) in the matrix, and the pixel in the first row and second column of the video frame corresponds to the element at (0,1) in the matrix. An element in the feature matrix can record information representing the pixel value. For example, if the video frame is recorded in RGB (red, green, and blue) format or YUV (luminance, chrominance, and saturation) format, then an element in the feature matrix can record the pixel value of the component corresponding to one channel of each pixel in the above formats.

[0081] Assume that the first feature map and each of the second feature maps each include three scales of first and second feature sub-maps, which can be referred to as the first scale, second scale, and third scale in ascending order. The target video frame is frame t, and its two adjacent frames are frame (t-1) and frame (t+1). Therefore, the first feature map of the target video frame includes three first feature sub-maps of different scales (first scale, second scale, and third scale), and the second feature maps of frames (t-1 and t+1) also each include three second feature sub-maps of different scales.

[0082] The following is combined with Figure 4 The provided schematic diagram of a feature sub-map illustrates the implementation method of determining similar image patches in the second feature sub-map for each scale in step S102 above.

[0083] like Figure 4 As shown, assuming the first feature map and each second feature map each include three scales of first and second feature sub-maps respectively, these three scales can be referred to as the first scale, second scale, and third scale in ascending order. Furthermore, the target video frame is frame t, and its number of adjacent frames is 2, namely frame t-1 and frame t+1. Then, at each scale, we can obtain the first feature map of the target video frame and the second feature maps of its adjacent frames. Each first feature map has a corresponding second feature map of the same scale.

[0084] For example, for frame t, the noise reduction device can obtain a first feature sub-image at a first scale. For frames t-1 and t+1, it can obtain two and a second feature sub-image at the first scale, respectively. Then, for each image block in the first feature sub-image at the first scale, a similar image block can be determined in the second feature sub-image at the first scale in frame t-1, and also in the second feature sub-image at the first scale in frame t+1. Similarly, for each image block in the first feature sub-image at the second scale, similar image blocks can be determined in the second feature sub-images at the second scale in frames t-1 and t+1, respectively. And for each image block in the first feature sub-image at the third scale, similar image blocks can be determined in the second feature sub-images at the third scale in frames t-1 and t+1, respectively.

[0085] Other implementations of step S102, namely determining similar image blocks and fusing the image block with its similar image blocks, will be described in the following embodiments and will not be detailed here.

[0086] Step S103: For each first feature sub-image, arrange the obtained fused image blocks according to the position of each image block in the first feature sub-image to obtain the fused feature map at the scale of the first feature sub-image.

[0087] Since the resulting fused image blocks belong to different scales after fusing the image block with its similar image blocks, the arrangement of the resulting fused image blocks in step S103 can be implemented in the following way:

[0088] For each first feature sub-image, the obtained fused image blocks are arranged according to the positions of each image block in the first feature sub-image to obtain the fused feature map at the scale of the first feature sub-image.

[0089] Specifically, for each first feature sub-image, according to the position of each image block in the first feature sub-image, each fused image block with the same scale as the first feature sub-image is arranged at the position of the image block in the first feature sub-image used by the fused image block.

[0090] For example, such as Figure 4As shown, for the first feature sub-image at the first scale, that is, for the first feature sub-image corresponding to frame t, this first feature sub-image includes four image blocks. For the image block located at (1,1) in the first feature sub-image, the fused image block obtained by fusing the image block using this image block can be arranged at position (1,1) in the fused feature image. Similarly, for the image block located at (2,1) in the first feature sub-image, the fused image block obtained by fusing the image block using this image block can be arranged at position (2,1) in the fused feature image. And so on, the four fused image blocks obtained by fusing the image blocks using image blocks at different positions are arranged at four corresponding positions in the fused feature image, wherein the position of each fused image block in the fused feature image is the same as the position of the image block used by the fused image block in the first feature sub-image. In this way, the fused feature image at the scale of the first feature sub-image can be obtained.

[0091] For the first feature sub-image of the second scale, which comprises nine image patches, the fused image patch obtained by fusing the image patch at position (1,1) in the first feature sub-image can be arranged at position (1,1) in the fused feature image. Similarly, the fused image patch obtained by fusing the image patch at position (2,1) in the first feature sub-image can be arranged at position (2,1) in the fused feature image. The fused image patch obtained by fusing the image patch at position (3,1) in the first feature sub-image can be arranged at position (3,1) in the fused feature image. The fused image patch obtained by fusing the image patch at position (3,2) in the first feature sub-image can be arranged at position (3,2) in the fused feature image. And so on, the nine fused image patches obtained by fusing the image patches at different positions are arranged at nine corresponding positions in the fused feature image, wherein the position of each fused image patch in the fused feature image is the same as the position of the image patch used by the fused image patch in the first feature sub-image. In this way, the fused feature map at the scale of the first feature sub-map can be obtained.

[0092] For the first feature sub-image at the third scale, which comprises sixteen image blocks, the image block located at (4,1) in the first feature sub-image can be fused into a fused image block at position (4,1) in the fused feature image. Similarly, the image block located at (4,2) in the first feature sub-image can be fused into a fused image block at position (4,2) in the fused feature image. The image block located at (4,3) in the first feature sub-image can be fused into a fused image block at position (4,3) in the fused feature image. The image block located at (4,4) in the first feature sub-image can be fused into a fused image block at position (4,4) in the fused feature image. And so on, the sixteen fused image blocks obtained by fusing image blocks at different positions are arranged in sixteen corresponding positions in the fused feature image, wherein the position of each fused image block in the fused feature image is the same as the position of the image block used by the fused image block in the first feature sub-image. In this way, the fused feature map at the scale of the first feature sub-map can be obtained.

[0093] Other implementations of obtaining the fused feature map in step S103 will be described in the following embodiments, and will not be detailed here.

[0094] Step S104: Obtain the denoised video frame of the target video frame based on the obtained fused feature map.

[0095] In one implementation, based on the fused feature map, the noise reduction device can obtain the video frame corresponding to the fused feature map by performing the inverse operations of obtaining the first feature map and obtaining the second feature map. For example, if it is necessary to first convert the video frame into a feature matrix, and then normalize the pixel values ​​represented by each element in the feature matrix to obtain a feature map, then the noise reduction device can first perform inverse normalization on the fused feature map, and then perform image reconstruction based on the inverse normalized feature map to obtain the denoised video frame.

[0096] As can be seen from the above, in the solution provided by the embodiments of this application, when performing noise reduction processing, the image blocks in the first feature map of the target video frame and their similar image blocks are used for fusion. The similar image blocks used for fusion are the image blocks in the second feature map that are similar to the image blocks in the first feature map. It can be seen that the processing of feature fusion and similar image blocks is performed on the image blocks, which does not require processing of every pixel in the target video frame and adjacent video frames. This can reduce the number of objects being processed and reduce the computational resources consumed by video noise reduction.

[0097] In addition, for each video frame, the first feature map obtained includes feature sub-maps of multiple scales. Smaller scale features can refer to the local details of the image, while larger scale features can refer to the global structure of the image. Using feature sub-maps of multiple scales can take into account the features of the image more comprehensively and richly during the noise reduction process, thereby improving the effect of video noise reduction.

[0098] The following describes the implementation method of fusing the image block and its similar image blocks in step S102.

[0099] In one implementation, the noise reduction device performs concatenated fusion of image blocks in a first image block group according to the video frame playback order from front to back, to obtain a first fused image block. Then, it performs concatenated fusion of image blocks in a second image block group according to the video frame playback order from back to front, to obtain a second fused image block. Finally, the first and second fused image blocks are fused together to obtain a final fused image block.

[0100] The first image block group includes: a similar image block and the image block in the second feature map of the preceding adjacent frame. The second image block group includes: a similar image block and the image block in the second feature map of the following adjacent frame.

[0101] Specifically, for an image block in the first image block group, the noise reduction device, following the video frame playback order from front to back, can first fuse similar image blocks of the same image block in the second feature map of the earliest adjacent frame with similar image blocks of the same image block in the second feature map of the next frame, to obtain a fusion result. Then, it fuses similar image blocks of the same image block in the second feature map of the earliest video frame in the first image block group that did not participate in the fusion with the current fusion result, until the image block is fused into the fusion result, to obtain the first fused image block.

[0102] For each image block in the second image block group, the noise reduction device, following the video frame playback order from back to front, first fuses the similar image blocks of that image block in the second feature map of the latest adjacent frame with the similar image blocks of that image block in the second feature map of the previous frame, to obtain a fused result. Then, it fuses the similar image blocks of that image block in the second feature map of the latest video frame in the second image block group that did not participate in the fusion with the current fusion result, until the image block is fused into the fusion result, thus obtaining the second fused image block.

[0103] The following is combined with Figure 3 The provided diagram of another image block illustrates the implementation of the above-mentioned cascade fusion method.

[0104] Suppose that the image patch is an image patch in the first feature map of frame t, and the adjacent frames of frame t include: frame t-3, frame t-2, frame t-1, frame t+1, frame t+2, and frame t+3, then the noise reduction device can determine 6 similar image patches of the image patch in the second feature maps of the aforementioned adjacent frames. The similar image patch of the image patch determined in the second feature map of frame t-3 can be denoted as the t-3 similar image patch, such as... Figure 3 The image patch at t-3 is shown in the diagram. Similarly, the similar image patch determined by the noise reduction device in the second feature map of frame t-2 can be denoted as the t-2 similar image patch, as shown in the diagram. Figure 3 The image block at t-2 is shown in the figure. Similarly, the similar image blocks at t-3, t-2, t-1, t+1, t+2, and t+3 can be determined respectively.

[0105] During cascading fusion, for image blocks in the first image block group, the noise reduction device merges similar image blocks t-3 and t-2 in the order of video frame playback from front to back, obtaining fusion result 1. At this point, the earliest video frame in the first image block group that did not participate in fusion is the adjacent frame t-1. The noise reduction device then merges the similar image block t-1 with fusion result 1, obtaining fusion result 2. At this point, the earliest video frame in the first image block group that did not participate in fusion is frame t. The image block in the first feature map of frame t is then merged with fusion result 2, obtaining fusion result 3, which is used as the first fused image block.

[0106] For the image blocks in the second image block group, the noise reduction device can fuse the t+3 similar image block and the t+2 similar image block in the order of video frame playback from back to front, obtaining fusion result 4. At this time, the earliest video frame in the first image block group that did not participate in the fusion is the t+1 adjacent frame. Therefore, the noise reduction device fuses the t+1 similar image block with fusion result 4, obtaining fusion result 5. At this time, the earliest video frame in the first image block group that did not participate in the fusion is frame t. Therefore, the image block in the first feature map of frame t is fused with fusion result 5, obtaining fusion result 6, which is used as the second fused image block.

[0107] Then, the fusion results 3 and fusion results 6 are fused together to obtain a fused image block.

[0108] As can be seen, through the above-mentioned cascaded fusion method, even when there are many adjacent frames, similar image blocks in the second feature map of adjacent frames can be fused with the features of image blocks in the first feature map of the target video frame. In this way, during the fusion and noise reduction process, feature information in adjacent frames far away from the target video frame can be referenced. Since the content in the video frames has a strong temporal correlation, referencing more adjacent frames with correlation can improve the video noise reduction effect.

[0109] The following describes another implementation method for determining similar image blocks in step S102.

[0110] In another embodiment of this application, based on the above embodiments, step S102, which determines similar image blocks, can also be implemented according to steps A and B:

[0111] Step A: For each image patch in the third feature sub-map, identify similar image patches in each of the second feature sub-maps with the same scale as the third feature sub-map.

[0112] The third feature sub-map is the feature sub-map with the smallest scale in the first feature sub-map.

[0113] The method of determining the similar image blocks of the image block in each of the second feature sub-maps with the same scale as the third feature sub-map in step A is similar to the method of determining the similar image blocks of the image block in the second feature map in step S102. However, in step A, the similar image blocks of the image block are determined in the second feature sub-map with the smallest scale, which will not be described in detail here.

[0114] Step B: Based on the image positions of similar image blocks of each image block in the third feature sub-map, determine the similar image blocks of each image block in the other first feature sub-maps in each second feature sub-map with the same scale as the other first feature sub-maps.

[0115] Among them, other first feature sub-maps are the first feature sub-maps excluding the third feature sub-map.

[0116] Specifically, the noise reduction device can map the image positions of similar image blocks in the second feature sub-image at the smallest scale of each image block in the third feature sub-image to each second feature sub-image at the same scale as the other first feature sub-images, thereby obtaining the mapping region in each second feature sub-image, and determining the similar image blocks of each image block in the other first feature sub-images in the mapping region.

[0117] The following is combined with Figure 5 The implementation methods provided in steps A and B are explained.

[0118] See Figure 5 , Figure 5 Another schematic diagram of feature submaps is provided. In the diagram, frame t is the target video frame. Therefore, the first feature submap at the first scale corresponding to frame t is the third feature submap. The second feature submaps at the first scale corresponding to frames t+1 and t-1 are the second feature submaps at the same scale as the third feature submap.

[0119] Assume that the size of each image patch is 16×16, and the size of the third feature sub-image is 32×32. Furthermore, through step A, the similar image patches to the image patch (1,1) in the third feature sub-image are determined as follows: the image patch in the second feature sub-image of frame t-1 at the first scale with pixel (1,1) as the top-left vertex and pixel (16,16) as the bottom-right vertex; and the image patch in the second feature sub-image of frame t+1 at the first scale with pixel (4,4) as the top-left vertex and pixel (20,20) as the bottom-right vertex. For example... Figure 5 In the first feature sub-image corresponding to the t-th frame at the first scale, the two image blocks in the second feature sub-image pointed to by the arrow of the image block (1,1) are similar image blocks of the image.

[0120] The following uses the example of the similar image block of the (1,1) image block in the third feature sub-map to illustrate the implementation of determining similar image blocks in other first feature sub-maps in step B.

[0121] like Figure 5 As shown, the first feature sub-image at the second scale corresponding to frame t is one of the other first feature sub-images mentioned above. Furthermore, the second feature sub-images at the second scale corresponding to frames t-1 and t+1 are respectively the second feature sub-images of the first feature sub-image at the second scale corresponding to frame t.

[0122] Based on the image positions of similar image blocks in each image block in the third feature sub-map, the noise reduction device can determine the mapping region of the similar image block with pixel (1,1) as the upper left vertex and pixel (16,16) as the lower right vertex in the second feature sub-map of the first scale in the second feature sub-map of the (t-1)th frame at the second scale. For example, if the similar image block in the first scale occupies a 1 / 4 rectangular area in the upper left corner of its second feature sub-map, then, assuming the size of the second feature sub-map of the second scale is 48×48, the mapping region with pixel (1,1) as the upper left vertex and pixel (24,24) as the lower right vertex can be determined in the second feature sub-map of the (t-1)th frame at the second scale. This mapping region is: the mapping region of the similar image block of the (1,1) image block in the third feature sub-map in the second feature sub-map of the (t-1)th frame.

[0123] The noise reduction device can also determine the mapping region of the similar image block with pixel (4,4) as the top left vertex and pixel (20,20) as the bottom right vertex in the second feature sub-image of the first scale in the second feature sub-image of the (t+1)th frame at the second scale. For example, if the similar image block in the first scale in the (t+1)th frame is a rectangular region with pixel (4,4) as the top left vertex and pixel (20,20) as the bottom right vertex, then, assuming the size of the second feature sub-image of the second scale is 48×48, the mapping region with pixel (6,6) as the top left vertex and pixel (30,30) as the bottom right vertex can be determined in the second feature sub-image of the (t+1)th frame at the second scale. This mapping region is: the mapping region of the similar image block of the image block (1,1) in the third feature sub-image in the second feature sub-image of the (t+1)th frame.

[0124] The method for determining similar image blocks of the (1,1) image block in the first feature sub-image at the second scale is explained below.

[0125] Because the (1,1) image patch in the first feature sub-image at the second scale is located in the region where the (1,1) image patch in the third feature sub-image is mapped to the region of the first feature sub-image at the second scale (e.g. Figure 5 In the region shown by the black box in the first feature map at the second scale, similar image blocks of the image block (1,1) in the third feature map can be determined in the mapping region of the second feature map at frame t+1 and frame t-1. Specifically, the implementation method for determining similar images of the image block (1,1) in the first feature map in the mapping region can refer to the content of one implementation method for determining similar image blocks in the second feature map in step S102. The difference is that in this embodiment, similar image blocks are determined in the mapping region, while in step S102, similar image blocks are determined in the second feature map. That is, the range of similar image blocks determined by the two methods is different, but the method is the same.

[0126] The following is combined with Figure 6 Another implementation of determining similar image blocks of image block (1,2) in the first feature sub-map at the second scale is described.

[0127] Since the locations of image patches in other first feature sub-images may involve the mapping regions of multiple image patches in the third feature sub-image, see [link / reference]. Figure 6 , Figure 6 Another schematic diagram of a feature subgraph is provided, such as Figure 6The (1,2) image patch of the first feature map at the second scale in the third row is located within the mapping region of the first feature map at the second scale of the third feature map, where two different image patches are located. Therefore, the (1,2) image patch of the first feature map needs to determine its similar image patches in the following manner.

[0128] The noise reduction device can determine similar image blocks in other first feature sub-images based on the positions of image blocks in the first feature sub-image at the second scale and the image positions of similar image blocks in each image block of the third feature sub-image. Figure 6 It can be seen that the region where the image patch (1,2) is located in the third feature sub-image maps to the region of the first feature sub-image at the second scale (e.g., Figure 6 In the second row, the region within the black box in the first feature map at the second scale is mapped to the region in the first feature map at the second scale in the third feature sub-map (e.g., the region containing the (1,1) image patch in the second feature sub-map). Figure 6 In the fourth row, the black-framed area in the first feature map at the second scale), since the image patch (1,2) in the first feature sub-map at the second scale falls within the range of the two regions mentioned above, when determining the similar image patches of the image patch (1,2) in the first feature sub-map at the second scale, it is necessary to consider the positions of the image patches (1,1) and (1,2) in the third feature sub-map, and also to consider the mapping regions of the similar image patches of the image patch (1,1) in the second feature sub-map at frames t+1 and t-1 (such as...). Figure 6 The region shown in the black box in the second feature sub-image at the second scale in the fourth row of the middle), and the similar image patch of image patch (1,2) in the third feature sub-image, are mapped regions in the second feature sub-images of the t+1 and t-1 frames (such as...). Figure 6 The region shown in the black box in the second feature sub-image at the second scale in the second row (those in the middle) thus yields the overlapping mapping region of the image patch mapping regions in the two types of third feature sub-images mentioned above (such as...). Figure 6 The region shown in the black box in the second feature sub-map at the second scale in the third row.

[0129] Then, the noise reduction device can determine similar image blocks to the (1,2) image block in the first feature sub-image at the second scale within the overlapping mapping region of the second feature sub-images in the (t+1)th and (t-1)th frames at the second scale. Specifically, the method for determining similar images to the (1,1) image block in the first feature sub-image within the overlapping mapping region can refer to one implementation of step S102 for determining similar image blocks in the second feature image. The difference is that in this embodiment, similar image blocks are determined within the overlapping mapping region, while in step S102, similar image blocks are determined in the second feature image. That is, the scope of similar image block determination differs, but the method is the same.

[0130] The following is combined with Figure 5 Another implementation of determining similar image blocks of the (1,1) image block in the first feature sub-map at the third scale is described.

[0131] like Figure 5 As shown, based on the image positions of similar image blocks in each image block in the third feature sub-map, the noise reduction device can determine the mapping region of the similar image block with pixel (1,1) as the upper left vertex and pixel (16,16) as the lower right vertex in the second feature sub-map of the first scale in the second feature sub-map of the (t-1)th frame at the third scale. For example, if the similar image block in the first scale occupies a 1 / 4 rectangular area in the upper left corner of the second feature sub-map of the first scale in the (t-1)th frame, then, assuming the size of the second feature sub-map of the third scale is 64×64, the mapping region with pixel (1,1) as the upper left vertex and pixel (32,32) as the lower right vertex can be determined in the second feature sub-map of the (t-1)th frame at the third scale. This mapping region is: the mapping region of the similar image block of the (1,1) image block in the third feature sub-map in the second feature sub-map of the (t-1)th frame.

[0132] The noise reduction device can determine the mapping region of a similar image block with pixel (4,4) as the top-left vertex and pixel (20,20) as the bottom-right vertex in the second feature sub-image of the first scale in the second feature sub-image of the (t+1)th frame at the third scale. For example, the rectangular region of the similar image block in the first scale frame t+1 with pixel (4,4) as the top-left vertex and pixel (20,20) as the bottom-right vertex, assuming the size of the second feature sub-image of the third scale is 64×64, the mapping region with pixel (8,8) as the top-left vertex and pixel (40,40) as the bottom-right vertex can be determined in the second feature sub-image of the (t+1)th frame at the third scale. This mapping region is: the mapping region of the similar image block of the (1,1) image block in the third feature sub-image in the second feature sub-image of the (t+1)th frame.

[0133] Because the (1,1) image patch in the first feature sub-image at the third scale is located in the region of the first feature sub-image at the third scale (e.g., the region of the (1,1) image patch in the third feature sub-image is mapped to the region of the first feature sub-image at the third scale). Figure 5In the region shown by the black box in the first feature map of the third scale, the noise reduction device can determine the similar image block of the (1,1) image block in the third feature map in the mapping region of the second feature map of the (1,1) image block in the above-mentioned third feature map in the (t+1) frame, and in the mapping region of the similar image block of the (1,1) image block in the third feature map in the second feature map of the (t-1) frame. Specifically, the implementation method of determining the similar image of the (1,1) image block in the first feature map in the mapping region can refer to the content of one implementation method of determining the similar image block in the second feature map in step S102. The difference is that in this embodiment, the similar image block is determined in the mapping region, while in step S102, the similar image block is determined in the second feature map. That is, the range of the similar image block is different, but the method is the same.

[0134] In addition, the image blocks (1,2), (2,1) and (2,2) in the first feature sub-image at the third scale are also located in the region of the first feature sub-image at the third scale, where the region of the image block (1,1) in the third feature sub-image is mapped. Therefore, similar image blocks can be determined in the same way as the image block (1,1) in the first feature sub-image at the third scale.

[0135] As can be seen from the above, since the computational resources consumed for calculating small-scale image patches are relatively small, while the computational resources consumed for calculating high-scale image patches are relatively large, in the process of determining the similar image patches of each image patch in other first feature sub-maps, the image positions of the similar image patches of each image patch at the smallest scale are referenced to reduce the computational resources consumed for calculating high-scale image patches, and further reduce the computational resources consumed by video noise reduction.

[0136] The following is combined with Figure 7 Another implementation of step S102, which determines similar image blocks, will be described.

[0137] In another embodiment of this application, based on the above embodiments, each first sub-feature map contains the same number of image blocks.

[0138] When each first sub-feature map contains the same number of image blocks, the noise reduction device can directly find the image block corresponding to the image block in the third sub-feature map when determining similar image blocks for each image block in the other first feature maps (excluding the lowest scale). For example, see... Figure 7 , Figure 7Another schematic diagram of a feature map is provided, in which the image patch (1,2) in both the first feature map at the second scale and the first feature map at the third scale corresponds to the image patch (1,2) in the third feature map. In this way, the noise reduction device does not need to calculate which image patches in the third feature map correspond to the location of the image patch in the first feature map at other scales.

[0139] Specifically, the noise reduction device can determine the mapping region of similar image blocks of image block (1,2) in the second feature sub-image of the second scale in the (t-1)th frame. Assuming that the size of an image block in the third feature sub-image is 4×4, the region occupied by similar image blocks of image block (1,2) in the second feature sub-image of the first scale in the (t-1)th frame is a rectangular region with pixel (1,3) as the upper left vertex and pixel (4,6) as the lower right vertex. Then, assuming the size of an image block in the first feature sub-image of the second scale is 6×6, a mapping region with pixel (1,4) as the upper left vertex and pixel (6,9) as the lower right vertex can be determined in the second feature sub-image of the second scale in the (t-1)th frame. This mapping region is the mapping region of similar image blocks of image block (1,2) in the third feature sub-image in the second feature sub-image of the (t-1)th frame.

[0140] The noise reduction device can also determine the mapping region of the similar image block of image block (1,2) in the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the first feature sub-image of the (t+1)th frame. Assuming that the region occupied by the similar image block of image block (1,2) in the third feature sub-image is a rectangular region with pixel (3,4) as the upper left vertex and pixel (6,7) as the lower right vertex, then the mapping region with pixel (4,6) as the upper left vertex and pixel (9,11) as the lower right vertex can be determined in the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the second feature sub-image of the third feature sub-image of the second ...

[0141] After determining that the image block (1,2) in the first feature sub-image at the second scale and the image block (1,2) in the third feature sub-image are corresponding image blocks, the noise reduction device can directly determine the similar image block (1,2) in the first feature sub-image at the second scale in the mapping region of the similar image block (1,2) in the third feature sub-image in the second feature sub-image in the (t-1) frame, and the similar image block (1,2) in the third feature sub-image in the mapping region of the similar image block (1,2) in the second feature sub-image in the (t+1) frame.

[0142] Similarly, the method for determining the similar image blocks of each image block in the first feature sub-image at the third scale is the same as the above process, and will not be described in detail here.

[0143] As can be seen from the above, when the number of image blocks contained in each first sub-feature map is the same, it is not necessary to consider the mapping relationship between image blocks between the first feature map and the third feature map except for the minimum scale, which reduces the amount of computation and further reduces the computational resources consumed by video noise reduction.

[0144] The following is combined with Figure 8 Another implementation of step S102, which determines similar image blocks, will be described.

[0145] In another embodiment of this application, based on the above embodiments, each second sub-feature map contains the same number of image blocks.

[0146] When each second sub-feature map is divided into blocks, and each second sub-feature map contains the same number of image blocks, the noise reduction device can determine the similar image blocks of each image block in the other first feature maps (excluding the lowest scale) based on the position of the image blocks in the third feature map and the position of the similar image blocks of the image blocks in the third feature map.

[0147] For example, see Figure 8 , Figure 8 Another schematic diagram of a feature submap is provided. Suppose that, for the image patch (1,2) of the third feature submap, the noise reduction device can determine that (1,2) in the second feature submap corresponding to the (t-1)th frame of the first scale is a similar image patch to the image patch (1,2) of the third feature submap, and can determine that (2,3) in the second feature submap corresponding to the (t+1)th frame of the first scale is a similar image patch to the image patch (1,2) of the third feature submap.

[0148] Then, when the noise reduction device determines the (1,2) image block of the first feature sub-image of the second scale, it can directly determine the (1,2) in the second feature sub-image corresponding to the (t-1) frame of the second scale as the similar image block of the (1,2) image block of the first feature sub-image of the second scale based on the position of the image block of the third feature sub-image and the position of the similar image block of the image block of the third feature sub-image. It can also determine the (2,3) in the second feature sub-image corresponding to the (t+1) frame of the second scale as the similar image block of the (1,2) image block of the first feature sub-image of the second scale.

[0149] When the noise reduction device determines the (1,2) image block of the first feature sub-image at the third scale, it can directly determine the (1,2) in the second feature sub-image corresponding to the (t-1)th frame of the third scale as the similar image block of the (1,2) image block of the first feature sub-image at the third scale based on the position of the image block of the third feature sub-image and the position of the similar image block of the image block of the third feature sub-image, and determine the (2,3) in the second feature sub-image corresponding to the (t+1)th frame of the third scale as the similar image block of the (1,2) image block of the first feature sub-image at the third scale.

[0150] As can be seen from the above, when the number of image blocks contained in each first sub-feature map and the number of image blocks contained in each second sub-feature map are the same, it is not necessary to perform mapping between image blocks, which reduces the amount of computation and further reduces the computational resources consumed by video noise reduction.

[0151] In one embodiment of this application, see Figure 9 , Figure 9 A flowchart illustrating an image patch fusion method is provided. The method includes the following steps S901-S905.

[0152] Step S901: Input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of the pre-trained video denoising model to perform feature extraction, and obtain the first feature map of the target video frame and the second feature map of the preset number of adjacent frames of the target video frame output by the feature extraction layer.

[0153] The video noise reduction model also includes an image patch matching layer and an image patch fusion layer.

[0154] Step S901 is implemented in a similar way to step S101. The difference is that in step S901, the feature extraction process is implemented by the feature extraction layer of the video denoising model, where the feature extraction layer can be a convolutional layer.

[0155] The specific implementation of feature extraction in step S901 will not be detailed here.

[0156] Step S902: Input the first feature map and each of the second feature maps into the image block matching layer, perform image block matching, and obtain the similar image blocks output by the image block matching layer that are determined in each of the second feature maps.

[0157] The implementation of step S902 is similar to that of step S102 in determining similar image blocks. The difference is that in step S902, the process of determining similar image blocks is implemented by an image block matching layer. The image block matching layer can be implemented using warp alignment (an image block alignment method). The specific implementation of step S902 in determining similar image blocks will not be described in detail here.

[0158] Step S903: Input the image block and its similar image blocks into the image block fusion layer to perform image block fusion and obtain the fused image block output by the image block fusion layer.

[0159] The implementation of image patch fusion in step S903 is similar to that in step S102. The difference is that in step S903, the fusion process is implemented by an image patch fusion layer. The image patch fusion layer can be implemented based on Progressive Feature Fusion. The specific implementation of image patch fusion in step S903 will not be described in detail here.

[0160] Step S904: Input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map that matches the position of each image block in the first feature sub-map and has the same scale.

[0161] The implementation of arranging the fused image blocks in step S904 is similar to that in step S103. The difference is that in step S904, the process of arranging the fused image blocks is implemented by a rearrangement layer. The specific implementation of arranging the fused image blocks in step S904 will not be described in detail here.

[0162] Step S905: Input the obtained fused feature map into the reconstruction layer to reconstruct the video frame and obtain the video frame after denoising the target video frame output by the reconstruction layer.

[0163] The implementation of video frame reconstruction in step S905 is similar to that in step S104. The difference is that in step S905, the video frame reconstruction process is implemented by a reconstruction layer, which can be constructed based on the structure of PyramidTransformer Reconstruction (a self-attention reconstruction model with a pyramid structure).

[0164] In another embodiment of this application, the above-mentioned reconstruction layer can perform video frame reconstruction in the following manner: perform residual walk transformation processing on the first fusion feature map with the smallest scale to obtain a first processing result; perform upsampling processing on the first processing result to obtain a second processing result with the same scale as the second fusion feature map, wherein the second fusion feature map is: the fusion feature map with the smallest scale that has not undergone residual walk transformation processing; fuse the second processing result and the second fusion feature map, and perform residual walk transformation processing on the fused feature map to obtain a third fusion feature map; if there is a fusion feature map that has not undergone residual walk transformation processing, then update the first processing result to the third fusion feature map, and return to the step of upsampling processing the first processing result; perform convolution transformation processing on the finally obtained third fusion feature map, and add the obtained convolution result and the target video frame pixel by pixel to obtain the denoised video frame.

[0165] Specifically, suppose the obtained fused feature map includes three fused feature maps of different scales, which are ordered from smallest to largest as the first scale, the second scale, and the third scale. Then, the first fused feature map with the smallest current scale is the first-scale fused feature map. A residual walk transformation needs to be performed on the first-scale fused feature map to obtain the first processing result. At this point, the first-scale fused feature map is the fused feature map after the residual walk transformation.

[0166] The current fused feature maps without residual walk transformation processing include: a fused feature map at the second scale and a fused feature map at the third scale. Therefore, the second fused feature map is the fused feature map at the second scale. It is necessary to upsample the first processing result at the first scale to obtain a first processing result of the same size as the second scale. Then, this first processing result of the same size as the second scale is fused with the fused feature map at the second scale, and the resulting fused feature map is subjected to residual walk transformation processing to obtain the third fused feature map. At this point, both the first-scale and second-scale fused feature maps are fused feature maps processed by residual walk transformation. The current fused feature maps without residual walk transformation processing include: a fused feature map at the third scale. That is, there exists a fused feature map without residual walk transformation processing, so the first processing result is updated to the third fused feature map. Since the smallest fused feature map in the fused feature map without residual walk transformation processing is the fused feature map at the third scale, the second fused feature map is the fused feature map at the third scale.

[0167] Therefore, the upsampling process involves upsampling the third fused feature map to obtain a second processing result with the same size as the third scale. This second processing result is then fused with the third-scale fused feature map, and a residual walk transform is applied to the fused feature map to obtain the third fused feature map. Since no fused feature map exists that has not undergone residual walk transform, the resulting third-scale fused feature map is the final third fused feature map. A convolution transformation is then applied to this final third fused feature map, and the convolution result is added pixel-by-pixel to the target video frame to obtain the denoised video frame.

[0168] In the reconstruction layer, the residual swimming transform processing can be implemented based on Residual Swim-TransformerBlock (a neural network model for image processing). The pixel-by-pixel addition mentioned above can be implemented using element-wise add.

[0169] As can be seen from the above, fusing features at each scale sequentially considers information from different scales during video frame reconstruction, thus improving the quality of the reconstructed video frames. Furthermore, the residual walking transformation processing used in the feature map fusion process takes into account the dependencies between different regions in the image, enhancing the model's expressive and generalization abilities and improving the overall quality of the reconstructed video frames.

[0170] The video denoising model described above can be trained using the acquired sample video image pairs. For example, the sample video image pairs can be from the CRVD (Captured Raw Video Denoising, a dynamic video dataset) sample dataset, and normalization processing can be performed on the sample video image pairs. One sample video image in the sample video image pair can be a clean video frame, and the other sample video image can be a noisy video frame.

[0171] As can be seen from the above, by processing the target video frame and a predetermined number of its neighboring frames using a pre-trained video denoising model that includes a feature extraction layer, an image patch matching layer, and an image patch fusion layer, respectively, the effectiveness of feature extraction, the accuracy of identifying similar image patches, and the effect of feature fusion can be improved, thereby enhancing the overall video denoising performance. Furthermore, by using a pre-trained video denoising model that includes a feature extraction layer, an image patch matching layer, an image patch fusion layer, a rearrangement layer, and a reconstruction layer to process the target video frame and a predetermined number of its neighboring frames, in addition to improving the effectiveness of feature extraction, the accuracy of identifying similar image patches, and the effect of feature fusion, the video frame reconstruction effect can also be improved, further enhancing the overall video denoising performance.

[0172] According to another aspect of the embodiments of this application, see Figure 10 , Figure 10 A schematic diagram of a video noise reduction device is provided, the device comprising:

[0173] The feature map acquisition module 1001 is used to acquire a first feature map of a target video frame in a video and a second feature map of a preset number of adjacent frames of the target video frame, wherein the first feature map includes a first feature sub-map of multiple scales, and each second feature map includes a second feature sub-map of multiple scales.

[0174] The similar image block determination module 1002 is used to determine similar image blocks of each image block in each second feature sub-image with the same scale as the first feature sub-image for each image block in each first feature sub-image, and to fuse the image block and its similar image blocks to obtain a fused image block.

[0175] The image block arrangement module 1003 is used to arrange the obtained fused image blocks according to the position of each image block in the first feature sub-image for each first feature sub-image, so as to obtain the fused feature map at the scale of the first feature sub-image.

[0176] The denoised video frame acquisition module 1004 is used to obtain a denoised video frame of the target video frame based on the obtained fused feature map.

[0177] As can be seen from the above, in the solution provided by the embodiments of this application, when performing noise reduction processing, the image blocks in the first feature map of the target video frame and their similar image blocks are used for fusion. The similar image blocks used for fusion are the image blocks in the second feature map that are similar to the image blocks in the first feature map. It can be seen that the processing of feature fusion and similar image blocks is performed on the image blocks, which does not require processing of every pixel in the target video frame and adjacent video frames. This can reduce the number of objects being processed and reduce the computational resources consumed by video noise reduction.

[0178] In addition, for each video frame, the first feature map obtained includes feature sub-maps of multiple scales. Smaller scale features can refer to the local details of the image, while larger scale features can refer to the global structure of the image. Using feature sub-maps of multiple scales can take into account the features of the image more comprehensively and richly during the noise reduction process, thereby improving the effect of video noise reduction.

[0179] In one embodiment of this application, the similar image block determination module is specifically used to determine similar image blocks for each image block in the third feature sub-map in each of the second feature sub-maps with the same scale as the third feature sub-map, wherein the third feature sub-map is the feature sub-map with the smallest scale in the first feature sub-map; based on the image positions of the similar image blocks of each image block in the third feature sub-map, similar image blocks of each image block in each of the other first feature sub-maps with the same scale as the other first feature sub-maps are determined respectively.

[0180] As can be seen from the above, since the computational resources consumed for calculating small-scale image patches are relatively small, while the computational resources consumed for calculating high-scale image patches are relatively large, in the process of determining the similar image patches of each image patch in other first feature sub-maps, the image positions of the similar image patches of each image patch at the smallest scale are referenced to reduce the computational resources consumed for calculating high-scale image patches, and further reduce the computational resources consumed by video noise reduction.

[0181] In one embodiment of this application, each first sub-feature map contains the same number of image blocks.

[0182] As can be seen from the above, when the number of image blocks contained in each first sub-feature map is the same, it is not necessary to consider the mapping relationship between image blocks between the first feature map and the third feature map except for the minimum scale, which reduces the amount of computation and further reduces the computational resources consumed by video noise reduction.

[0183] In one embodiment of this application, the similar image block determination module is specifically used to perform cascaded fusion of image blocks in a first image block group in the order of video frame playback from front to back to obtain a first fused image block, wherein the first image block group includes: similar image blocks and the image block in the second feature map of the image block in the forward adjacent frame; and to perform cascaded fusion of image blocks in a second image block group in the order of video frame playback from back to front to obtain a second fused image block, wherein the second image block group includes: similar image blocks and the image block in the second feature map of the image block in the backward adjacent frame; and to fuse the first fused image block and the second fused image block to obtain a fused image block.

[0184] As can be seen, through the above-mentioned cascaded fusion method, even when there are many adjacent frames, similar image blocks in the second feature map of adjacent frames can be fused with the features of image blocks in the first feature map of the target video frame. In this way, during the fusion and noise reduction process, feature information in adjacent frames far away from the target video frame can be referenced. Since the content in the video frames has a strong temporal correlation, referencing more adjacent frames with correlation can improve the video noise reduction effect.

[0185] In one embodiment of this application, the feature map acquisition module is specifically used to input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining a first feature map of the target video frame and a second feature map of the preset number of adjacent frames of the target video frame output by the feature extraction layer. The video denoising model further includes an image patch matching layer and an image patch fusion layer.

[0186] The similar image block determination module is specifically used to input the first feature map and each of the second feature maps into the image block matching layer to perform image block matching, and obtain similar image blocks of the image block determined in each of the second feature maps, which are output by the image block matching layer; input the image block and its similar image blocks into the image block fusion layer to perform image block fusion, and obtain the fused image block output by the image block fusion layer.

[0187] As can be seen from the above, by processing the target video frame and a preset number of adjacent frames of the target video frame using a pre-trained video denoising model that includes a feature extraction layer, an image patch matching layer, and an image patch fusion layer, the effect of feature extraction, the accuracy of identifying similar image patches, and the effect of feature fusion can be improved, thereby improving the effect of video denoising.

[0188] In one embodiment of this application, where the first feature map includes first feature sub-maps of multiple scales and each second feature map includes second feature sub-maps of multiple scales, the video denoising model further includes: a rearrangement layer and a reconstruction layer.

[0189] The image block arrangement module is specifically used to input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map that matches the position of each image block in the first feature sub-map and has the same scale.

[0190] The denoised video frame acquisition module is specifically used to input the obtained fused feature map into the reconstruction layer to reconstruct the video frame and obtain the denoised video frame output by the reconstruction layer.

[0191] As can be seen from the above, by using a pre-trained video denoising model that includes a feature extraction layer, an image patch matching layer, an image patch fusion layer, a rearrangement layer, and a reconstruction layer to process the target video frame and a preset number of adjacent frames, the model can improve the effect of feature extraction, the accuracy of identifying similar image patches, and the effect of feature fusion. In addition, it can also improve the effect of video frame reconstruction, thereby improving the effect of video denoising.

[0192] In one embodiment of this application, the reconstruction layer reconstructs video frames as follows: A residual walk transform is applied to the smallest first fusion feature map to obtain a first processing result; the first processing result is upsampled to obtain a second processing result with the same scale as the second fusion feature map, wherein the second fusion feature map is the smallest fusion feature map that has not undergone residual walk transform; the second processing result and the second fusion feature map are fused, and a residual walk transform is applied to the fused feature map to obtain a third fusion feature map; if a fusion feature map that has not undergone residual walk transform exists, the first processing result is updated to the third fusion feature map, and the step of upsampling the first processing result is returned; a convolution transform is applied to the finally obtained third fusion feature map, and the resulting convolution result is added pixel-by-pixel to the target video frame to obtain a denoised video frame.

[0193] As can be seen from the above, fusing features at each scale sequentially considers information from different scales during video frame reconstruction, thus improving the quality of the reconstructed video frames. Furthermore, the residual walking transformation processing used in the feature map fusion process takes into account the dependencies between different regions in the image, enhancing the model's expressive and generalization abilities and improving the overall quality of the reconstructed video frames.

[0194] This application also provides an electronic device, such as... Figure 11 As shown, it includes:

[0195] Memory 1101 is used to store computer programs;

[0196] When the processor 1102 executes the program stored in the memory 1101, it implements the steps of any of the video noise reduction methods in the above embodiments.

[0197] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1102, the communication interface, and the memory 1101 communicating with each other via the communication bus.

[0198] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0199] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0200] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0201] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0202] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described video noise reduction methods.

[0203] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video noise reduction methods described above.

[0204] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0205] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0206] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0207] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A video noise reduction method, characterized in that, The method includes: A first feature map of a target video frame and a second feature map of a preset number of adjacent frames of the target video frame are obtained, wherein the first feature map includes a first feature sub-map of multiple scales, and each second feature map includes a second feature sub-map of multiple scales. For each image block in each first feature sub-map, similar image blocks are determined in each second feature sub-map with the same scale as the first feature sub-map, and the image block and its similar image blocks are fused to obtain a fused image block; For each first feature sub-image, the obtained fused image blocks are arranged according to the position of each image block in the first feature sub-image to obtain the fused feature map at the scale of the first feature sub-image; Based on the obtained fused feature map, a denoised video frame is obtained from the target video frame; For each image block in each first feature sub-image, determining similar image blocks in each second feature sub-image of the same scale as the first feature sub-image includes: For each image block in the third feature sub-map, similar image blocks are determined in each of the second feature sub-maps with the same scale as the third feature sub-map, wherein the third feature sub-map is the feature sub-map with the smallest scale in the first feature sub-map; Based on the image positions of similar image blocks of each image block in the third feature sub-map, similar image blocks of each image block in the other first feature sub-maps are determined in each second feature sub-map with the same scale as the other first feature sub-maps.

2. The method according to claim 1, characterized in that, Each first sub-feature map contains the same number of image blocks.

3. The method according to claim 1, characterized in that, The process of fusing the image block and its similar image blocks to obtain a fused image block includes: Following the playback order of video frames from front to back, the image blocks in the first image block group are concatenated and fused to obtain the first fused image block. The first image block group includes: similar image blocks of the image block in the second feature map of the forward adjacent frame and the image block itself. Following the video frame playback order from back to front, the image blocks in the second image block group are concatenated and fused to obtain the second fused image block. The second image block group includes: similar image blocks of the image block in the second feature map of the next adjacent frame and the image block itself. The first fused image block and the second fused image block are fused together to obtain a fused image block.

4. The method according to any one of claims 1-3, characterized in that, The process of obtaining a first feature map of the target video frame and second feature maps of a preset number of adjacent frames of the target video frame includes: The target video frame and a preset number of adjacent frames of the target video frame are input into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining the first feature map of the target video frame and the second feature map of the preset number of adjacent frames of the target video frame output by the feature extraction layer. The video denoising model further includes: an image patch matching layer, an image patch fusion layer, a rearrangement layer and a reconstruction layer. For each image block in each first feature sub-image, determining similar image blocks in each second feature sub-image of the same scale as the first feature sub-image includes: The first feature map and each of the second feature maps are input into the image block matching layer to perform image block matching, and the similar image blocks output by the image block matching layer that are determined in each of the second feature maps are obtained respectively. The process of fusing the image block and its similar image blocks to obtain a fused image block includes: The image block and its similar image blocks are input into the image block fusion layer to perform image block fusion, and the fused image block output by the image block fusion layer is obtained. For each first feature sub-image, arranging the obtained fused image blocks according to the positions of each image block in the first feature sub-image to obtain a fused feature map at the scale of the first feature sub-image includes: The obtained fused image blocks are input into the rearrangement layer, and the obtained fused image blocks are arranged to obtain a fused feature map output by the rearrangement layer for each first feature sub-map, which corresponds to the position of each image block in the first feature sub-map and has the same scale. The step of obtaining the denoised video frame of the target video frame based on the obtained fused feature map includes: The obtained fused feature map is input into the reconstruction layer to reconstruct the video frame, and the video frame output by the reconstruction layer after denoising the target video frame is obtained.

5. The method according to claim 4, characterized in that, The reconstruction layer reconstructs video frames in the following manner: The first fused feature map with the smallest scale is subjected to residual walking transformation to obtain the first processing result; The first processing result is upsampled to obtain a second processing result with the same scale as the second fused feature map. The second fused feature map is the fused feature map with the smallest scale that has not undergone residual walk transformation. The second processing result and the second fused feature map are fused together, and the fused feature map is subjected to residual walk transformation to obtain a third fused feature map. If there is a fused feature map that has not undergone residual walk transformation, the first processing result is updated to the third fused feature map, and the step of upsampling the first processing result is returned. The third fused feature map is then subjected to convolution transformation. The convolution result is then added to the target video frame pixel by pixel to obtain the denoised video frame.

6. A video noise reduction device, characterized in that, The device includes: The feature map acquisition module is used to acquire a first feature map of a target video frame in the video and a second feature map of a preset number of adjacent frames of the target video frame, wherein the first feature map includes a first feature sub-map of multiple scales, and each second feature map includes a second feature sub-map of multiple scales. The similar image block determination module is used to determine the similar image blocks of each image block in each second feature sub-image with the same scale as the first feature sub-image for each image block in each first feature sub-image, and to fuse the image block and its similar image blocks to obtain a fused image block. The image block arrangement module is used to arrange the obtained fused image blocks according to the position of each image block in the first feature sub-image for each first feature sub-image, so as to obtain the fused feature map at the scale of the first feature sub-image. A denoised video frame acquisition module is used to obtain a denoised video frame of the target video frame based on the obtained fused feature map. The similar image block determination module is specifically used to determine similar image blocks for each image block in the third feature sub-map in each second feature sub-map with the same scale as the third feature sub-map. The third feature sub-map is the feature sub-map with the smallest scale in the first feature sub-map. Based on the image positions of the similar image blocks of each image block in the third feature sub-map, similar image blocks of each image block in each of the other first feature sub-maps with the same scale are determined in each of the second feature sub-maps.

7. The apparatus according to claim 6, characterized in that, Each first sub-feature map contains the same number of image blocks; or, The similar image block determination module is further configured to perform cascaded fusion of image blocks in the first image block group in the order of video frame playback from front to back to obtain a first fused image block, wherein the first image block group includes: similar image blocks of the image block in the second feature map of the forward adjacent frame and the image block itself; perform cascaded fusion of image blocks in the second image block group in the order of video frame playback from back to front to obtain a second fused image block, wherein the second image block group includes: similar image blocks of the image block in the second feature map of the backward adjacent frame and the image block itself; and fuse the first fused image block and the second fused image block to obtain a fused image block.

8. The apparatus according to claim 6 or 7, characterized in that, The feature map acquisition module is specifically used to input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining a first feature map of the target video frame and a preset number of second feature maps of the target video frame output by the feature extraction layer. The video denoising model further includes an image patch matching layer, an image patch fusion layer, a rearrangement layer, and a reconstruction layer. The similar image patch determination module is specifically used to input the first feature map and each of the second feature maps into the image patch matching layer to perform image patch matching, thereby obtaining similar image patches output by the image patch matching layer that are determined in each of the second feature maps; and to input the image patch and its similar image patches into the image patch fusion layer to perform image patch fusion, thereby obtaining a fused image patch output by the image patch fusion layer. The image block arrangement module is specifically used to input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map that matches the position of each image block in the first feature sub-map and has the same scale; the denoised video frame acquisition module is specifically used to input the obtained fused feature map into the reconstruction layer, reconstruct the video frame, and obtain a denoised video frame output by the reconstruction layer.

9. The apparatus according to claim 8, characterized in that, The reconstruction layer reconstructs video frames as follows: A residual walk transform is applied to the smallest fusion feature map to obtain a first processing result; the first processing result is upsampled to obtain a second processing result with the same scale as the second fusion feature map, wherein the second fusion feature map is the smallest fusion feature map without residual walk transform; the second processing result and the second fusion feature map are fused, and a residual walk transform is applied to the fused feature map to obtain a third fusion feature map. If a fusion feature map without residual walk transform exists, the first processing result is updated to the third fusion feature map, and the step of upsampling the first processing result is returned; a convolution transform is applied to the finally obtained third fusion feature map, and the resulting convolution result is added pixel-by-pixel to the target video frame to obtain a denoised video frame.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images