Video noise reduction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video noise reduction method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, in particular to the video noise reduction method and device, the electronic equipment and the storage medium. According to the specific implementation scheme, a first feature map of a target video frame in a video and second feature maps of a preset number of adjacent frames of the target video frame are obtained; fusing each image block in each first feature sub-image with a similar image block of the image block determined in each second feature sub-image with the same scale as the first feature sub-image to obtain a fused image block; arranging the obtained fused image blocks according to the position of each image block in each first feature sub-image to obtain a fused feature image under the scale of the first feature sub-image; and obtaining a video frame after denoising the target video frame based on the obtained fusion feature map. By applying the scheme provided by the embodiment of the invention, computing resources consumed by video noise reduction can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video noise reduction method, device, electronic device and storage medium. Background Art

[0002] Nowadays, videos are increasingly appearing in people's work and life. However, due to factors such as video compression algorithms and network bandwidth, noise may be introduced into video frames, which will cause the video quality to deteriorate. Therefore, it is necessary to perform noise reduction on the video.

[0003] Currently, when performing noise reduction processing on videos, optical flow method is generally used to perform noise reduction processing on videos. However, the optical flow method is computationally complex and consumes a lot of computing resources. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video noise reduction method, device, electronic device and storage medium to reduce the computing resources consumed by video noise reduction. The specific technical solution is as follows:

[0005] According to one aspect of an embodiment of the present application, a video noise reduction method is provided, the method comprising:

[0006] Obtaining a first feature map of a target video frame in a video, and second feature maps of a preset number of adjacent frames of the target video frame, wherein the first feature map includes first feature sub-maps of multiple scales, and each second feature map includes second feature sub-maps of multiple scales;

[0007] For each image block in each first feature sub-image, similar image blocks of the image block are respectively determined in each second feature sub-image having the same scale as the first feature sub-image, and the image block is fused with the similar image blocks to obtain a fused image block;

[0008] For each first feature sub-image, arranging the obtained fused image blocks according to the positions of the image blocks in the first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image;

[0009] A video frame after denoising the target video frame is obtained based on the obtained fusion feature map.

[0010] In one embodiment of the present application, for each image block in each first feature sub-image, respectively determining similar image blocks of the image block in each second feature sub-image having the same scale as the first feature sub-image includes:

[0011] For each image block in the third feature sub-graph, similar image blocks of the image block are respectively determined in each second feature sub-graph having the same scale as the third feature sub-graph, wherein the third feature sub-graph is: a feature sub-graph with the smallest scale in the first feature sub-graph;

[0012] Based on the image positions of similar image blocks of each image block in the third characteristic sub-image, similar image blocks of each image block in the other first characteristic sub-images are respectively determined in the second characteristic sub-images having the same scale as the other first characteristic sub-images.

[0013] In one embodiment of the present application, the number of image blocks contained in each first sub-feature map is the same.

[0014] In one embodiment of the present application, the step of fusing the image block with a similar image block to obtain a fused image block includes:

[0015] According to the order of video frame playback from front to back, the image blocks in the first image block group are cascaded and fused to obtain a first fused image block, wherein the first image block group includes: a similar image block of the image block in the second feature map of the forward adjacent frame and the image block;

[0016] According to the order of video frame playback from back to front, the image blocks in the second image block group are cascaded and fused to obtain a second fused image block, wherein the second image block group includes: a similar image block of the image block in the second feature map of the backward adjacent frame and the image block;

[0017] The first fused image block and the second fused image block are fused to obtain a fused image block.

[0018] In one embodiment of the present application, the step of obtaining a first feature map of a target video frame in a video and second feature maps of a preset number of adjacent frames of the target video frame includes:

[0019] Inputting a target video frame and a preset number of adjacent frames of the target video frame into a feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining a first feature map of the target video frame output by the feature extraction layer and a second feature map of a preset number of adjacent frames of the target video frame, wherein the video denoising model further includes: an image block matching layer, an image block fusion layer, a rearrangement layer, and a reconstruction layer;

[0020] The step of determining, for each image block in the first feature map, similar image blocks of the image block in each of the obtained second feature maps comprises:

[0021] Inputting the first feature map and each second feature map into the image block matching layer to perform image block matching, and obtaining similar image blocks of the image block determined in each second feature map output by the image block matching layer;

[0022] The step of fusing the image block with similar image blocks to obtain a fused image block includes:

[0023] Inputting the image block and similar image blocks to the image block fusion layer to perform image block fusion to obtain a fused image block output by the image block fusion layer;

[0024] Arranging the obtained fused image blocks according to the positions of the image blocks in the first feature map to obtain the fused feature map includes:

[0025] Inputting the obtained fused image blocks into the rearrangement layer, arranging the obtained fused image blocks, and obtaining a fused feature map output by the rearrangement layer for each first feature sub-map, which is consistent with the positions of the image blocks in the first feature sub-map and has the same scale;

[0026] The step of obtaining a video frame after denoising the target video frame based on the obtained fusion feature map includes:

[0027] The obtained fusion feature map is input into the reconstruction layer to reconstruct the video frame, and the video frame after denoising the target video frame is obtained as output by the reconstruction layer.

[0028] In one embodiment of the present application, the reconstruction layer reconstructs the video frame in the following manner:

[0029] Performing residual wandering transformation processing on the first fused feature map with the smallest scale to obtain a first processing result;

[0030] Performing upsampling processing on the first processing result to obtain a second processing result having the same scale as the second fused feature map, wherein the second fused feature map is: a fused feature map with the smallest scale that has not been processed by residual wandering transformation;

[0031] The second processing result and the second fused feature map are fused, and the fused feature map is subjected to residual wandering transform processing to obtain a third fused feature map. If there is a fused feature map that has not been subjected to residual wandering transform processing, the first processing result is updated to the third fused feature map, and the step of performing upsampling processing on the first processing result is returned;

[0032] The third fused feature map finally obtained is subjected to convolution transformation processing, and the convolution result obtained is added to the target video frame pixel by pixel to obtain a denoised video frame.

[0033] According to another aspect of an embodiment of the present application, a video noise reduction device is provided, the device comprising:

[0034] A feature map acquisition module, used to obtain a first feature map of a target video frame in a video, and second feature maps of a preset number of adjacent frames of the target video frame, wherein the first feature map includes first feature sub-maps of multiple scales, and each second feature map includes second feature sub-maps of multiple scales;

[0035] a similar image block determination module, configured to determine, for each image block in each first feature sub-image, similar image blocks of the image block in each second feature sub-image of the same scale as the first feature sub-image, and fuse the image block with its similar image blocks to obtain a fused image block, and fuse the image block with its similar image blocks to obtain a fused image block;

[0036] An image block arrangement module, for arranging the obtained fused image blocks according to the positions of the image blocks in each first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image;

[0037] The denoising video frame obtaining module is used to obtain a video frame after denoising the target video frame based on the obtained fusion feature map.

[0038] In one embodiment of the present application, the similar image block determination module is specifically used to determine, for each image block in the third feature subgraph, similar image blocks of the image block in each second feature subgraph having the same scale as the third feature subgraph, wherein the third feature subgraph is: the feature subgraph with the smallest scale in the first feature subgraph; based on the image positions of similar image blocks of each image block in the third feature subgraph, similar image blocks of each image block in the other first feature subgraphs are determined in each second feature subgraph having the same scale as the other first feature subgraphs.

[0039] In one embodiment of the present application, the number of image blocks contained in each first sub-feature map is the same.

[0040] In one embodiment of the present application, the similar image block determination module is specifically used to perform cascade fusion of image blocks in a first image block group in a video frame playback order from front to back to obtain a first fused image block, wherein the first image block group includes: similar image blocks of the image block in a second feature map of a forward adjacent frame and the image block; perform cascade fusion of image blocks in a second image block group in a video frame playback order from back to front to obtain a second fused image block, wherein the second image block group includes: similar image blocks of the image block in a second feature map of a backward adjacent frame and the image block; and fuse the first fused image block and the second fused image block to obtain a fused image block.

[0041] In one embodiment of the present application, the feature map acquisition module is specifically used to input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, and obtain a first feature map of the target video frame output by the feature extraction layer, and a second feature map of a preset number of adjacent frames of the target video frame, wherein the video denoising model further includes: an image block matching layer, an image block fusion layer, a rearrangement layer, and a reconstruction layer;

[0042] The similar image block determination module is specifically used to input the first feature map and each second feature map into the image block matching layer, perform image block matching, and obtain similar image blocks of the image block determined in each second feature map output by the image block matching layer; input the image block and its similar image blocks into the image block fusion layer, perform image block fusion, and obtain a fused image block output by the image block fusion layer;

[0043] The image block arrangement module is specifically used to input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map, which is consistent with the positions of each image block in the first feature sub-map and has the same scale;

[0044] The denoised video frame acquisition module is specifically used to input the obtained fusion feature map into the reconstruction layer, perform video frame reconstruction, and obtain the video frame output by the reconstruction layer after denoising the target video frame.

[0045] In one embodiment of the present application, the reconstruction layer reconstructs the video frame in the following manner: performing residual wandering transform processing on the first fused feature map with the smallest scale to obtain a first processing result; performing upsampling processing on the first processing result to obtain a second processing result with the same scale as the second fused feature map, wherein the second fused feature map is: the fused feature map with the smallest scale that has not been processed by residual wandering transform; the second processing result and the second fused feature map are fused, and the fused feature map is subjected to residual wandering transform processing to obtain a third fused feature map. If there is a fused feature map that has not been subjected to residual wandering transform processing, the first processing result is updated to the third fused feature map, and the step of upsampling the first processing result is returned; the third fused feature map finally obtained is subjected to convolution transform processing, and the obtained convolution result and the target video frame are added pixel by pixel to obtain a denoised video frame.

[0046] According to another aspect of the embodiments of the present application, there is provided an electronic device, including:

[0047] Memory, used to store computer programs;

[0048] The processor is used to implement any of the above-mentioned video noise reduction methods when executing the program stored in the memory.

[0049] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned video noise reduction methods is implemented.

[0050] According to another aspect of the embodiments of the present application, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned video noise reduction methods.

[0051] From the above, it can be seen that in the solution provided in the embodiment of the present application, when performing noise reduction processing, the image block in the first feature map of the target video frame and its similar image block are used for fusion, and the similar image block used for fusion is an image block determined in the second feature map and similar to the image block in the first feature map. It can be seen that in the process of feature fusion and determination of similar image blocks, the processing is performed on the image blocks, and it is not necessary to process each pixel in the target video frame and the adjacent video frames, which can reduce the number of objects to be processed and reduce the computing resources consumed by video noise reduction.

[0052] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0054] Figure 1 A schematic diagram of a process flow of a video noise reduction method provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of an image block provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of another image block provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a characteristic subgraph provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of another characteristic subgraph provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of another characteristic subgraph provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of another characteristic subgraph provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of another characteristic subgraph provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of a flow chart of an image block fusion method provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the structure of a video noise reduction device provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0066] The execution subject of the embodiment of the present application is described below.

[0067] The solution provided in the embodiments of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality / virtual reality devices, laptop computers, personal computers, ultra-mobile personal computers, netbooks or personal digital assistants, etc. The embodiments of the present application do not impose any restrictions on the specific types of devices used for video noise reduction. For ease of description, the execution subject of the embodiments of the present application is referred to as a noise reduction device.

[0068] The video noise reduction method provided in the embodiment of the present application is described in detail below through specific examples.

[0069] In one embodiment of the present application, see Figure 1 , Figure 1 A schematic flow chart of a video noise reduction method is provided, the method comprising the following steps S101-S104.

[0070] Step S101: obtaining a first feature map of a target video frame in a video and second feature maps of a preset number of adjacent frames of the target video frame.

[0071] The first feature map includes first feature sub-maps of multiple scales, and each second feature map includes second feature sub-maps of multiple scales.

[0072] In one implementation, the noise reduction device may perform normalization processing on the pixel values ​​of the target video frame and the adjacent frames in the video, obtain a first feature map based on the normalized pixel values ​​of the target video frame, and obtain a second feature map based on the normalized pixel values ​​of a preset number of adjacent frames of the target video frame. For example, if a pixel component is represented by 8 bits, the maximum value of the pixel component is 255, then when the pixel value is normalized, the pixel value may be normalized by dividing the pixel value by 255.

[0073] Other implementations of step S101 to obtain the feature map are described in the following embodiments and will not be described in detail here.

[0074] Step S102: for each image block in each first feature sub-image, similar image blocks of the image block are respectively determined in each second feature sub-image of the same scale as the first feature sub-image, and the image block is fused with the similar image blocks to obtain a fused image block.

[0075] Combine the following Figure 2 The following describes a method for determining similar image blocks in each second feature sub-image having the same scale as the first feature sub-image in step S102 .

[0076] In one implementation, the noise reduction device may search, for each image block in the second characteristic sub-image, in the second characteristic sub-image for an image block having the highest similarity with the image block, as a similar image block of the image block.

[0077] join Figure 2 , Figure 2 A schematic diagram of an image block is provided. Assuming that the target video frame is the t-th frame, its first feature sub-graph is divided into 16 image blocks, and the two second feature sub-graphs of the adjacent frames corresponding to the target video frame are the feature graphs corresponding to the t-1-th frame and the t+1-th frame, respectively. Then, taking the image block in the second row and second column of the first feature sub-graph (hereinafter (x, y) represents the x-th row and the y-th column, such as (2, 2) represents the second row and the second column) as an example, the similarity between each image block with the same size as the image block and the image block in the second feature sub-graph corresponding to the t-1 frame can be calculated. Specifically, assuming that the size of the image block is 16×16, for each second feature sub-graph, the denoising device can determine the pixel point (1, 1) as the pixel point at the upper left corner of the image block in the second feature sub-graph, and determine the pixel point (16, 16) as the pixel point at the lower right corner of the image block in the second feature sub-graph, so that an image block with the same size as the image block in the second feature sub-graph is obtained, and the similarity between the two image blocks is calculated. Then, the pixel point (1,2) can be determined as the pixel point at the upper left corner of the image block in the second feature sub-image, and the pixel point (16,17) can be determined as the pixel point at the lower right corner of the image block in the second feature sub-image. In this way, another image block with the same size as the image block in the second feature sub-image is obtained, and the similarity between the two image blocks is calculated. Similarly, the pixel point (2,1) can also be determined as the pixel point at the upper left corner of the image block in the second feature sub-image, and the pixel point (17,16) can be determined as the pixel point at the lower right corner of the image block in the second feature sub-image. In this way, another image block with the same size as the image block in the second feature sub-image is obtained, and the similarity between the two image blocks is calculated. According to the above method, all image blocks with the same size as the image block in the second feature sub-image can be determined, and the similarity between the image blocks in each second feature sub-image and the image block is calculated, and the image block in the second feature sub-image with the highest similarity is used as the similar image block of the image block in the second feature sub-image.

[0078] In another implementation, the noise reduction device may search for an image block with the highest similarity to the image block in the similar position region of the image block in the second feature sub-map according to the position of the image block in the first feature sub-map for each second feature sub-map, and use the similar image block as the similar image block of the image block. The similar position region of the image block is determined in the following manner: a similar position in the second feature sub-map that is the same as the position of the image block in the first feature sub-map is determined, and the similar position region is determined with the similar position of the image block in the second feature sub-map as the center. For example, Figure 2 As shown, taking the image block of the first characteristic sub-image (2,2) of the t-th frame as an example, assuming that the size of the image block is 16×16, the pixel point in the upper left corner of the image block is (17,17), and the pixel point in the lower right corner is (32,32), then, in the second characteristic sub-image corresponding to the t-1-th frame and the second characteristic sub-image corresponding to the t+1-th frame, with the pixel point in the upper left corner being (17,17), the pixel point in the lower right corner being (32,32) and the image block size being 16×16 as the center, a similar position area with a size of 48×48, a pixel point in the upper left corner being (1,1), and a pixel point in the lower right corner being (48,48) can be determined, as shown in FIG. Figure 2 As shown in the bold black box in the figure. Then, the noise reduction device can determine the similar position area of ​​the second feature sub-image for each second feature sub-image, and then determine the similar image block of the image block in the second feature sub-image in the similar position area according to the method of determining the similar image block described in the above one implementation. In this way, the number of image blocks that need to be calculated to determine the similar image blocks can be reduced, and the resources consumed by the calculation can be reduced.

[0079] The preset number of adjacent frames of the target video frame may include the preset number of video frames before and after the target video frame in the video frame playback order, wherein the number of the forward adjacent frames and the backward adjacent frames of the target video frame may be equal or unequal. For example, the preset number may be 4, then the adjacent frames may include the first two frames and the last two frames of the target video frame in the video frame playback order, the adjacent frames may also include the first frame and the last three frames of the target video frame in the video frame playback order, and the adjacent frames may also include the last four frames of the target video frame in the video frame playback order. Assuming that the target video frame is the tth frame in the video, when the preset number is 4, the adjacent frames may include the t-2th frame, the t-1th frame, the t+1th frame and the t+2th frame in the video. For each adjacent frame, the second feature map corresponding to the adjacent frame can be obtained, that is, when the number of adjacent frames is multiple, the number of second feature maps is also multiple. For example, 4 second feature maps can be obtained for the above 4 adjacent frames.

[0080] The first feature map and the second feature map can be recorded in the form of a feature matrix. For example, each pixel in the video frame can correspond to an element in the feature matrix, and the positions also correspond one to one. For example, the pixel in the first row and first column of the video frame corresponds to the element (0,0) in the matrix, and the pixel in the first row and second column of the video frame corresponds to the element (0,1) in the matrix. An element in the feature matrix can record information representing pixel values. For example, if the video frame is recorded in RGB (red, green, and blue) format or YUV (brightness, chrominance, and saturation) format, then an element in the feature matrix can record the pixel value of the component corresponding to one channel of each pixel in the above format.

[0081] Assume that the first feature map and each second feature map include first feature submaps and second feature submaps of three scales, respectively. In order from small to large, the three scales can be called the first scale, the second scale and the third scale, respectively. The target video frame is the t-th frame, and the number of its adjacent frames is 2, namely the t-1-th frame and the t+1-th frame. Then, the first feature map of the target video frame includes first feature submaps of three different scales, namely the first scale, the second scale and the third scale, and the second feature maps of the t-1-th frame and the t+1-th frame also include second feature submaps of three different scales.

[0082] Combine the following Figure 4 A schematic diagram of a feature sub-graph is provided to illustrate the implementation method of determining similar image blocks in the second feature sub-graph for the first feature sub-graph at each scale in the above step S102.

[0083] like Figure 4 As shown, assuming that the first feature map and each second feature map include first feature sub-maps and second feature sub-maps of three scales, the three scales can be called the first scale, the second scale and the third scale in order from small to large, and the target video frame is the t-th frame, and the number of its adjacent frames is 2, namely the t-1-th frame and the t+1-th frame. Then, at each scale, the first feature map of the target video frame and the second feature map of the adjacent frame of the target video frame can be obtained. Each first feature map has its corresponding second feature map of the same scale.

[0084] For example, for the t-th frame, the noise reduction device can obtain the first characteristic sub-image at the first scale, and for the t-1-th frame and the t+1-th frame, two and second characteristic sub-images at the first scale can be obtained respectively. Then, for each image block in the first characteristic sub-image at the first scale, a similar image block of the image block can be determined in the second characteristic sub-image at the first scale of the t-1-th frame, and a similar image block of the image block can also be determined in the second characteristic sub-image at the first scale of the t+1-th frame. Similarly, for each image block in the first characteristic sub-image at the second scale, a similar image block of the image block can be determined in the second characteristic sub-image at the second scale of the t-1-th frame and the t+1-th frame. For each image block in the first characteristic sub-image at the third scale, a similar image block of the image block can be determined in the second characteristic sub-image at the third scale of the t-1-th frame and the t+1-th frame.

[0085] Other implementations of determining similar image blocks in step S102 and implementations of fusing the image block with its similar image blocks will be described in the following embodiments and will not be described in detail here.

[0086] Step S103: for each first feature sub-image, the obtained fused image blocks are arranged according to the positions of the image blocks in the first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image.

[0087] Since the scales of the fused image blocks obtained after the image block and its similar image blocks are fused are different, the implementation method of arranging the obtained fused image blocks in step S103 can be implemented in the following manner:

[0088] For each first feature sub-image, the obtained fused image blocks are arranged according to the positions of the image blocks in the first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image.

[0089] Specifically, for each first feature sub-image, according to the positions of the image blocks in the first feature sub-image, each fused image block having the same scale as the first feature sub-image is arranged at the position of the image block in the first feature sub-image used by the fused image block.

[0090] For example, Figure 4As shown, for the first feature sub-graph of the first scale, that is, for the first feature sub-graph corresponding to the t-th frame, the first feature sub-graph includes four image blocks. For the image block at position (1,1) in the first feature sub-graph, the fused image block obtained by fusion of the image block can be arranged at position (1,1) in the fusion feature graph. Similarly. For the image block at position (2,1) in the first feature sub-graph, the fused image block obtained by fusion of the image block can be arranged at position (2,1) in the fusion feature graph. Similarly, four fused image blocks obtained by fusion of image blocks at different positions are arranged at four corresponding positions in the fusion feature graph, wherein the position of each fused image block arranged in the fusion feature graph is the same as the position of the image block used by the fused image block in the first feature sub-graph. In this way, a fused feature graph at the scale of the first feature sub-graph can be obtained.

[0091] For the first feature sub-graph of the second scale, the first feature sub-graph includes nine image blocks. For the image block at position (1,1) in the first feature sub-graph, the fused image block obtained by fusion using the image block can be arranged at position (1,1) in the fused feature graph. Similarly. For the image block at position (2,1) in the first feature sub-graph, the fused image block obtained by fusion using the image block can be arranged at position (2,1) in the fused feature graph. For the image block at position (3,1) in the first feature sub-graph, the fused image block obtained by fusion using the image block can be arranged at position (3,1) in the fused feature graph. For the image block at position (3,2) in the first feature sub-graph, the fused image block obtained by fusion using the image block can be arranged at position (3,2) in the fused feature graph. By analogy, the fused image blocks obtained by fusion using nine image blocks at different positions are arranged at nine corresponding positions in the fused feature graph, wherein the position of each fused image block arranged in the fused feature graph is the same as the position of the image block used by the fused image block in the first feature sub-graph. In this way, a fused feature map at the scale of the first feature sub-map can be obtained.

[0092] For the first feature sub-graph of the third scale, the first feature sub-graph includes sixteen image blocks. For the image block at position (4,1) in the first feature sub-graph, the fused image block obtained by fusion of the image block can be arranged at position (4,1) in the fused feature graph. Similarly. For the image block at position (4,2) in the first feature sub-graph, the fused image block obtained by fusion of the image block can be arranged at position (4,2) in the fused feature graph. For the image block at position (4,3) in the first feature sub-graph, the fused image block obtained by fusion of the image block can be arranged at position (4,3) in the fused feature graph. For the image block at position (4,4) in the first feature sub-graph, the fused image block obtained by fusion of the image block can be arranged at position (4,4) in the fused feature graph. By analogy, the sixteen fused image blocks obtained by fusion of image blocks at different positions are arranged at sixteen corresponding positions in the fused feature graph, wherein the position of each fused image block arranged in the fused feature graph is the same as the position of the image block used by the fused image block in the first feature sub-graph. In this way, a fused feature map at the scale of the first feature sub-map can be obtained.

[0093] Other implementations of step S103 to obtain the fused feature map are described in the following embodiments and will not be described in detail here.

[0094] Step S104: obtaining a video frame after denoising the target video frame based on the obtained fusion feature map.

[0095] In one implementation, based on the fused feature map, the denoising device can obtain the video frame corresponding to the fused feature map by performing the inverse operation of obtaining the first feature map and obtaining the second feature map. For example, if it is necessary to first convert the video frame into a feature matrix, and then normalize the pixel values ​​represented by each element in the feature matrix to obtain the feature map. Then, the denoising device can first denormalize the fused feature map, and then reconstruct the image based on the denormalized feature map to obtain the denoised video frame.

[0096] From the above, it can be seen that in the solution provided in the embodiment of the present application, when performing noise reduction processing, the image block in the first feature map of the target video frame and its similar image block are used for fusion, and the similar image block used for fusion is an image block determined in the second feature map and similar to the image block in the first feature map. It can be seen that in the process of feature fusion and determination of similar image blocks, the processing is performed on the image blocks, and it is not necessary to process each pixel in the target video frame and the adjacent video frames, which can reduce the number of objects to be processed and reduce the computing resources consumed by video noise reduction.

[0097] In addition, for each video frame, the first feature map obtained includes feature sub-maps of multiple scales. The smaller scale can refer to the local detail information of the image, while the larger scale feature can refer to the global structure information of the image. The use of feature sub-maps of multiple scales can make it possible to more comprehensively consider the image features during the denoising process, thereby improving the effect of video denoising.

[0098] The following describes the implementation method of fusing the image block with its similar image blocks in step S102.

[0099] In one implementation, the noise reduction device performs cascade fusion on the image blocks in the first image block group in the order from front to back of the video frame playback sequence to obtain a first fused image block. Perform cascade fusion on the image blocks in the second image block group in the order from back to front of the video frame playback sequence to obtain a second fused image block. Then, the first fused image block and the second fused image block are fused to obtain a fused image block.

[0100] The first image block group includes: similar image blocks of the image block in the second feature map of the forward adjacent frame and the image block. The second image block group includes: similar image blocks of the image block in the second feature map of the backward adjacent frame and the image block.

[0101] Specifically, for the image blocks in the first image block group, the noise reduction device may first fuse similar image blocks of the image block in the second feature map of the most adjacent frame and similar image blocks of the image block in the second feature map of the next frame in the order of video frame playback from front to back to obtain a fusion result. Then, similar image blocks of the image block in the second feature map of the most forward video frame in the first image block group that are not involved in the fusion are fused with the current fusion result until the image block is fused into the fusion result to obtain a first fused image block.

[0102] For the image blocks in the second image block group, the noise reduction device may first fuse similar image blocks of the image block in the second feature map of the last adjacent frame and similar image blocks of the image block in the second feature map of the previous frame in the order of video frame playback from back to front to obtain a fusion result. Then, similar image blocks of the image block in the second feature map of the last video frame in the second image block group that are not involved in the fusion are fused with the current fusion result until the image block is fused into the fusion result to obtain a second fused image block.

[0103] Combine the following Figure 3 A schematic diagram of another image block is provided to illustrate the implementation of the above cascade fusion.

[0104] Assume that the image block is an image block in the first feature map of the t-th frame, and the adjacent frames of the t-th frame include: the t-3-th frame, the t-2-th frame, the t-1-th frame, the t+1-th frame, the t+2-th frame, and the t+3-th frame. Then, the noise reduction device can determine six similar image blocks of the image block in the second feature maps of the adjacent frames. The similar image blocks of the image block determined in the second feature map of the t-3-th frame can be recorded as t-3 similar image blocks, such as Figure 3 Similarly, the similar image block of the image block determined by the noise reduction device in the second feature map of the t-2 frame can be recorded as the t-2 similar image block, such as Figure 3 Similarly, the t-3 similar image block, t-2 similar image block, t-1 similar image block, t+1 similar image block, t+2 similar image block and t+3 similar image block of the image block can be determined respectively.

[0105] When performing cascade fusion, for the image blocks in the first image block group, the noise reduction device can fuse the t-3 similar image block and the t-2 similar image block in the order of the video frame playback from front to back to obtain a fusion result 1. At this time, the most forward video frame that does not participate in the fusion in the first image block group is the t-1 adjacent frame, so the noise reduction device fuses the t-1 similar image block with the fusion result 1 to obtain a fusion result 2. At this time, the most forward video frame that does not participate in the fusion in the first image block group is the t frame, so the image block in the first feature map of the t frame is fused with the fusion result 2 to obtain a fusion result 3 as the first fused image block.

[0106] For the image blocks in the second image block group, the noise reduction device can fuse the t+3 similar image block and the t+2 similar image block in the order of the video frame playback from back to front to obtain a fusion result 4. At this time, the most front video frame that does not participate in the fusion in the first image block group is the t+1 adjacent frame, so the noise reduction device fuses the t+1 similar image block with the fusion result 4 to obtain a fusion result 5. At this time, the most front video frame that does not participate in the fusion in the first image block group is the t frame, so the image block in the first feature map of the t frame is fused with the fusion result 5 to obtain a fusion result 6 as the second fused image block.

[0107] Then, the fusion result 3 and the fusion result 6 are fused to obtain a fused image block.

[0108] It can be seen that through the above-mentioned cascade fusion method, even in the case of a large number of adjacent frames, similar image blocks in the second feature map of the adjacent frames can be fused with the features of the image blocks in the first feature map of the target video frame. In this way, in the process of fusion noise reduction, the feature information in the adjacent frames at a long distance of the target video frame can be referenced. Since the content in the video frame has a strong correlation in time sequence, referring to more adjacent frames with correlation can improve the effect of video noise reduction.

[0109] Another implementation of determining similar image blocks in step S102 is described below.

[0110] In another embodiment of the present application, based on the above embodiment, the implementation method of determining similar image blocks in step S102 can also be implemented according to the following steps A and B:

[0111] Step A: for each image block in the third feature sub-image, similar image blocks of the image block are respectively determined in each second feature sub-image having the same scale as the third feature sub-image.

[0112] The third characteristic subgraph is: the characteristic subgraph with the smallest scale in the first characteristic subgraph.

[0113] The method of respectively determining similar image blocks of the image block in each second feature sub-map having the same scale as the third feature sub-map in step A is similar to the implementation method of determining similar image blocks of the image block in the second feature sub-map in step S102, except that similar image blocks of the image block are determined in the second feature sub-map with the smallest scale in step A, which will not be described in detail here.

[0114] Step B: Based on the image positions of similar image blocks of each image block in the third feature sub-image, similar image blocks of each image block in other first feature sub-images are respectively determined in each second feature sub-image having the same scale as other first feature sub-images.

[0115] The other first characteristic subgraphs are first characteristic subgraphs excluding the third characteristic subgraph.

[0116] Specifically, the denoising device may map the image positions of similar image blocks of each image block in the third feature subimage in the second feature subimage of the smallest scale to second feature subimages of the same scale as other first feature subimages, obtain the mapping area in each second feature subimage, and determine similar image blocks of each image block in other first feature subimages in the mapping area.

[0117] Combine the following Figure 5 The implementation methods provided by step A and step B are described.

[0118] See also Figure 5 , Figure 5 A schematic diagram of another feature subgraph is provided. In the figure, the t-th frame is a target video frame, and the first feature subgraph at the first scale corresponding to the t-th frame is the third feature subgraph. The second feature subgraphs at the first scale corresponding to the t+1-th frame and the t-1-th frame are second feature subgraphs having the same scale as the third feature subgraph.

[0119] Assume that the size of the image blocks is 16×16, and the size of the third feature sub-image is 32×32. And, through step A, determine that the similar image blocks of the image block (1,1) in the third feature sub-image are: the image block with pixel point (1,1) as the upper left corner vertex and pixel point (16,16) as the lower right corner vertex in the second feature sub-image of the t-1th frame at the first scale, and the image block with pixel point (4,4) as the upper left corner vertex and pixel point (20,20) as the lower right corner vertex in the second feature sub-image of the t+1th frame at the first scale. Figure 5 In the figure, the two image blocks in the second feature sub-image pointed to by the image block arrow of (1,1) in the first feature sub-image corresponding to the t-th frame at the first scale are similar image blocks of the image.

[0120] The following takes the similar image blocks of the (1,1) image block in the third characteristic sub-image determined as an example to illustrate the implementation method of determining similar image blocks in other first characteristic sub-images in step B.

[0121] like Figure 5 As shown, the first feature subgraph at the second scale corresponding to the t-th frame is a feature subgraph among the other first feature subgraphs mentioned above. Moreover, the second feature subgraphs at the second scale corresponding to the t-1-th frame and the t+1-th frame are respectively the second feature subgraphs of the first feature subgraph at the second scale corresponding to the t-th frame.

[0122] Based on the image positions of similar image blocks of each image block in the third feature sub-image, the noise reduction device can determine the mapping area of ​​the similar image block with pixel (1,1) as the upper left corner vertex and pixel (16,16) as the lower right corner vertex in the second feature sub-image of the t-1 frame at the first scale in the second feature sub-image of the t-1 frame at the second scale. For example, the similar image block in the t-1 frame at the first scale occupies the upper left 1 / 4 rectangular area in its second feature sub-image. Then, assuming that the size of the second feature sub-image at the second scale is 48×48, the mapping area with pixel (1,1) as the upper left corner vertex and pixel (24,24) as the lower right corner vertex can be determined in the second feature sub-image of the t-1 frame at the second scale. The mapping area is: the mapping area of ​​the similar image block of the image block (1,1) in the third feature sub-image in the second feature sub-image of the t-1 frame.

[0123] The noise reduction device can also determine the mapping area of ​​the second feature subimage of the t+1 frame at the second scale in the second feature subimage of the t-1 frame at the first scale, which has the pixel point (4, 4) as the upper left corner vertex and the pixel point (20, 20) as the lower right corner vertex. For example, the similar image blocks in the t+1 frame at the first scale are in the rectangular area with the pixel point (4, 4) as the upper left corner vertex and the pixel point (20, 20) as the lower right corner vertex. Then, assuming that the size of the second feature subimage at the second scale is 48×48, the mapping area with the pixel point (6, 6) as the upper left corner vertex and the pixel point (30, 30) as the lower right corner vertex can be determined in the second feature subimage of the t+1 frame at the second scale. The mapping area is: the mapping area of ​​the similar image block of the (1, 1) image block in the third feature subimage in the second feature subimage of the t+1 frame.

[0124] The following describes a method for determining similar image blocks to the (1,1) image block in the first feature sub-image at the second scale.

[0125] Since the image block (1,1) in the first feature sub-image at the second scale is located at: the area where the image block (1,1) in the third feature sub-image is located is mapped to the area of ​​the first feature sub-image at the second scale (such as Figure 5 Therefore, similar image blocks of the image block (1,1) in the third feature subimage can be determined in the mapping area in the second feature subimage of the t+1 frame and the t-1 frame, and similar image blocks of the image block (1,1) in the first feature subimage at the second scale are determined. Specifically, the implementation method of determining similar images of the image block (1,1) in the first feature subimage in the mapping area can refer to the content of an implementation method of determining similar image blocks in the second feature image in step S102, the difference being that in this embodiment, similar image blocks are determined in the mapping area, while in step S102, similar image blocks are determined in the second feature image, that is, the ranges of determining similar image blocks in the two methods are different, but the methods are the same.

[0126] Combine the following Figure 6 Another implementation method for determining similar image blocks to the (1, 2) image block in the first characteristic sub-image at the second scale is described.

[0127] Since the positions of the image blocks in the other first feature sub-images may involve multiple image block mapping areas in the third feature sub-image, see Figure 6 , Figure 6 A schematic diagram of another feature subgraph is provided, such as Figure 6The (1,2) image block of the first feature map of the second scale in the third row of is located in the mapping area of ​​the first feature map of the second scale between two different image blocks in the third feature submap. Therefore, the (1,2) image block of the first feature map needs to determine its similar image blocks in the following way.

[0128] The noise reduction device can determine similar image blocks of each image block in other first feature sub-images according to the position of the image block in the first feature sub-image at the second scale and the image position of the similar image block of each image block in the third feature sub-image. Figure 6 It can be seen that the area where the (1,2) image block in the third feature sub-image is located is mapped to the area of ​​the first feature sub-image at the second scale (such as Figure 6 In the figure, the black box area in the first feature map of the second scale in the second row), the area where the (1,1) image block in the third feature submap is located is mapped to the area of ​​the first feature submap at the second scale (such as Figure 6 In the figure, the black frame area in the first feature map of the second scale in the fourth row), since the image block (1,2) in the first feature submap at the second scale is within the range of the above two areas, when determining the similar image block of the image block (1,2) in the first feature submap at the second scale, it is necessary to consider the positions of the image blocks (1,1) and (1,2) in the third feature submap, and also to consider the mapping area of ​​the similar image blocks of the image block (1,1) in the third feature submap in the second feature submap of the t+1 frame and the t-1 frame (such as Figure 6 The area shown in the black box in the second feature sub-image at the second scale in the fourth row of the figure), and the mapping area of ​​the similar image block of the image block (1,2) in the third feature sub-image in the second feature sub-image of the t+1 frame and the t-1 frame (as shown in the black box in the second feature sub-image of the fourth row of the figure), and the mapping area of ​​the similar image block of the image block (1,2) ... Figure 6 In this way, the overlapping mapping area of ​​the mapping areas of the image blocks in the above two third feature sub-images is obtained (as shown in the black box in the second feature sub-image of the second row of FIG. Figure 6 (The area shown in the black box in the second feature sub-image at the second scale in the third row of FIG. 1 ).

[0129] Then, the noise reduction device can determine similar image blocks of the (1,2) image block in the first feature sub-image at the second scale in the overlapping mapping area of ​​the second feature sub-image of the t+1th frame and the t-1th frame at the above-mentioned second scale. Specifically, the implementation method of determining similar images of the (1,1) image block in the first feature sub-image in the overlapping mapping area can refer to the content of an implementation method of determining similar image blocks in the second feature image in step S102, the difference being that in this embodiment, similar image blocks are determined in the overlapping mapping area, while in step S102, similar image blocks are determined in the second feature image, that is, the ranges of determining similar image blocks in the two are different, but the methods are the same.

[0130] Combine the following Figure 5 Another implementation method for determining similar image blocks to the (1,1) image block in the first characteristic sub-image at the third scale is described.

[0131] like Figure 5 As shown, based on the image positions of similar image blocks of each image block in the third feature sub-image, the noise reduction device can determine, in the second feature sub-image of the t-1 frame at the third scale, the mapping area of ​​the similar image block with pixel point (1,1) as the upper left corner vertex and pixel point (16,16) as the lower right corner vertex in the second feature sub-image of the t-1 frame at the first scale in the second feature sub-image of the t-1 frame at the third scale. For example, the similar image block in the t-1 frame at the first scale above occupies the upper left 1 / 4 rectangular area in the second feature sub-image of the t-1 frame at the first scale. Then, assuming that the size of the second feature sub-image of the third scale is 64×64, the mapping area with pixel point (1,1) as the upper left corner vertex and pixel point (32,32) as the lower right corner vertex can be determined in the second feature sub-image of the t-1 frame at the third scale. The mapping area is: the mapping area of ​​the similar image block of the (1,1) image block in the third feature sub-image in the second feature sub-image of the t-1 frame.

[0132] The noise reduction device can determine, in the second feature sub-image of the t+1th frame at the third scale, a mapping area in the second feature sub-image of the t+1th frame at the third scale, of a similar image block with pixel point (4,4) as the upper left corner vertex and pixel point (20,20) as the lower right corner vertex in the second feature sub-image of the t+1th frame at the first scale. For example, the rectangular area with pixel point (4,4) as the upper left corner vertex and pixel point (20,20) as the lower right corner vertex of the similar image block in the t+1th frame at the first scale mentioned above, then, assuming that the size of the second feature sub-image of the third scale is 64×64, a mapping area with pixel point (8,8) as the upper left corner vertex and pixel point (40,40) as the lower right corner vertex can be determined in the second feature sub-image of the t+1th frame at the third scale, and the mapping area is: the mapping area in the second feature sub-image of the t+1th frame of the similar image block of the (1,1) image block in the third feature sub-image.

[0133] Since the image block (1,1) in the first feature sub-image at the third scale is located at: the area where the image block (1,1) in the third feature sub-image is located is mapped to the area of ​​the first feature sub-image at the third scale (such as Figure 5Therefore, the noise reduction device can determine the similar image blocks of the (1,1) image block in the third feature subimage in the mapping area of ​​the second feature subimage of the t+1 frame and the mapping area of ​​the second feature subimage of the (1,1) image block in the third feature subimage. Specifically, the implementation method of determining the similar images of the (1,1) image block in the first feature subimage in the mapping area can refer to the content of an implementation method of determining similar image blocks in the second feature map in step S102, the difference being that in this embodiment, similar image blocks are determined in the mapping area, while in step S102, similar image blocks are determined in the second feature map, that is, the ranges of determining similar image blocks are different, but the methods are the same.

[0134] In addition, the (1,2), (2,1) and (2,2) image blocks in the first feature sub-image at the third scale are also located: the area where the (1,1) image block is located in the third feature sub-image is mapped in the area of ​​the first feature sub-image at the third scale, so similar image blocks can be determined in the same way as the (1,1) image block in the first feature sub-image at the third scale is determined.

[0135] From the above, it can be seen that since the computing resources consumed by the calculation of small-scale image blocks are relatively small, and the computing resources consumed by the calculation of high-scale image blocks are relatively large, in the process of determining the similar image blocks of each image block in other first feature sub-images, the image positions of the similar image blocks of each image block with the smallest scale are referred to, so as to reduce the computing resources consumed by the calculation of high-scale image blocks and further reduce the computing resources consumed by video denoising.

[0136] Combine the following Figure 7 Another implementation of step S102 of determining similar image blocks is described below.

[0137] In yet another embodiment of the present application, based on the above embodiment, the number of image blocks included in each first sub-feature map is the same.

[0138] In the case where the number of image blocks included in each first sub-feature graph is the same, the denoising device can directly find an image block corresponding to the image block in the third feature sub-graph in the first feature sub-graph when determining similar image blocks of each image block in other first feature sub-graphs except the lowest scale. Figure 7 , Figure 7A schematic diagram of another feature sub-image is provided, in which both the image block (1,2) in the first feature sub-image of the second scale and the image block (1,2) in the first feature sub-image of the third scale can be determined to correspond to the image block in the third feature sub-image. In this way, the noise reduction device does not need to calculate which image blocks in the third feature sub-image are mapped to the positions of the image blocks in the first feature sub-images of other scales.

[0139] Specifically, the noise reduction device can determine the mapping area of ​​the similar image block of the (1,2) image block in the third feature sub-image in the second feature sub-image of the t-1 frame at the second scale. Assuming that the size of an image block in the third feature sub-image is 4×4, the area occupied by the similar image block of the (1,2) image block in the third feature sub-image in the second feature sub-image of the t-1 frame at the first scale is: a rectangular area with pixel point (1,3) as the upper left corner vertex and pixel point (4,6) as the lower right corner vertex. Then, assuming that the size of an image block in the first feature sub-image at the second scale is 6×6, a mapping area with pixel point (1,4) as the upper left corner vertex and pixel point (6,9) as the lower right corner vertex can be determined in the second feature sub-image of the t-1 frame at the second scale. The mapping area is: the mapping area of ​​the similar image block of the (1,2) image block in the third feature sub-image in the second feature sub-image of the t-1 frame.

[0140] The noise reduction device can also determine the mapping area of ​​the similar image block of the (1,2) image block in the third feature subimage in the second feature subimage of the t+1 frame at the second scale. Assuming that the area occupied by the similar image block of the (1,2) image block in the third feature subimage in the second feature subimage of the t+1 frame at the first scale is: a rectangular area with pixel point (3,4) as the upper left corner vertex and pixel point (6,7) as the lower right corner vertex, then, in the second feature subimage of the t+1 frame at the second scale, a mapping area with pixel point (4,6) as the upper left corner vertex and pixel point (9,11) as the lower right corner vertex can be determined, and the mapping area is: the mapping area of ​​the similar image block of the (1,2) image block in the third feature subimage in the second feature subimage of the t+1 frame.

[0141] After determining that the image block (1, 2) in the first feature subimage at the second scale and the image block (1, 2) in the third feature subimage are corresponding image blocks, the noise reduction device can directly determine the similar image blocks of the image block (1, 2) in the first feature subimage at the second scale in the mapping area of ​​the similar image blocks of the image block (1, 2) in the third feature subimage in the second feature subimage of the t-1 frame, and the similar image blocks of the image block (1, 2) in the third feature subimage in the mapping area of ​​the second feature subimage of the t+1 frame.

[0142] Similarly, the implementation method of determining similar image blocks for each image block in the first characteristic sub-image at the third scale is the same as the above process, which will not be described in detail here.

[0143] From the above, it can be seen that when the number of image blocks contained in each first sub-feature map is the same, there is no need to consider the mapping relationship between the image blocks of other first feature sub-maps and the third feature sub-map except for the minimum scale, which reduces the amount of calculation and further reduces the computing resources consumed by video denoising.

[0144] Combine the following Figure 8 Another implementation of step S102 of determining similar image blocks is described below.

[0145] In another embodiment of the present application, based on the above embodiment, the number of image blocks included in each second sub-feature map is the same.

[0146] When each second sub-feature map is divided into blocks and the number of image blocks contained in each second sub-feature map is the same, when the noise reduction device determines the similar image blocks of each image block in other first feature sub-maps except the lowest-scale image blocks, it can determine the similar image blocks of each image block in other first feature sub-maps based on the position of the image block of the third feature sub-map and the position of the similar image blocks of the image blocks of the third feature sub-map.

[0147] For example, see Figure 8 , Figure 8 A schematic diagram of another feature sub-graph is provided. Assume that, for the (1,2) image block of the third feature sub-graph, the denoising device can determine that (1,2) in the second feature sub-graph corresponding to the t-1th frame of the first scale is a similar image block to the (1,2) image block of the third feature sub-graph, and can determine that (2,3) in the second feature sub-graph corresponding to the t+1th frame of the first scale is a similar image block to the (1,2) image block of the third feature sub-graph.

[0148] Then, when the noise reduction device determines the (1,2) image block of the first feature sub-image of the second scale, it can directly determine, based on the position of the image block of the third feature sub-image and the position of the similar image block of the image block of the third feature sub-image, that (1,2) in the second feature sub-image corresponding to the t-1th frame of the second scale is a similar image block to the (1,2) image block of the first feature sub-image of the second scale, and determine that (2,3) in the second feature sub-image corresponding to the t+1th frame of the second scale is a similar image block to the (1,2) image block of the first feature sub-image of the second scale.

[0149] When the noise reduction device determines the (1,2) image block of the first feature sub-image of the third scale, it can directly determine, based on the position of the image block of the third feature sub-image and the position of the similar image block of the image block of the third feature sub-image, that (1,2) in the second feature sub-image corresponding to the t-1th frame of the third scale is a similar image block to the (1,2) image block of the first feature sub-image of the third scale, and determine that (2,3) in the second feature sub-image corresponding to the t+1th frame of the third scale is a similar image block to the (1,2) image block of the first feature sub-image of the third scale.

[0150] It can be seen from the above that when the number of image blocks included in each first sub-feature map and the number of image blocks included in each second sub-feature map are the same, there is no need to perform mapping between the image blocks, which reduces the amount of calculation and further reduces the computing resources consumed by video noise reduction.

[0151] In one embodiment of the present application, see Fig. 9 , Fig. 9 A flowchart of an image block fusion method is provided. The method comprises the following steps S901-S905.

[0152] Step S901: Input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, and obtain a first feature map of the target video frame output by the feature extraction layer, and a second feature map of a preset number of adjacent frames of the target video frame.

[0153] Among them, the video denoising model also includes: an image block matching layer and an image block fusion layer.

[0154] The implementation of step S901 is similar to that of step S101, except that in step S901, the feature extraction process is implemented by a feature extraction layer of the video denoising model, wherein the feature extraction layer may be a convolutional layer.

[0155] The specific implementation of the feature extraction in step S901 will not be described in detail here.

[0156] Step S902: input the first feature map and each second feature map into the image block matching layer, perform image block matching, and obtain similar image blocks of the image block determined in each second feature map output by the image block matching layer.

[0157] Step S902 is similar to the implementation method of determining similar image blocks in step S102, except that in step S902, the process of determining similar image blocks is implemented by an image block matching layer, wherein the image block matching layer can be implemented using warp alignment (a type of image block alignment method). The specific implementation method of determining similar image blocks in step S902 will not be described in detail here.

[0158] Step S903: input the image block and its similar image blocks into the image block fusion layer to perform image block fusion to obtain a fused image block output by the image block fusion layer.

[0159] The implementation method of image block fusion in step S903 is similar to that in step S102. The difference is that in step S903, the fusion process is implemented by an image block fusion layer, wherein the image block fusion layer can be implemented based on Progressive Feature Fusion. The specific implementation method of image block fusion in step S903 will not be described in detail here.

[0160] Step S904: input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map that is consistent with the positions of the image blocks in the first feature sub-map and has the same scale.

[0161] The implementation method of arranging the fused image blocks in step S904 is similar to that in step S103. The difference is that in step S904, the process of arranging the fused image blocks is implemented by the rearrangement layer. The specific implementation method of arranging the fused image blocks in step S904 will not be described in detail here.

[0162] Step S905: input the obtained fusion feature map into the reconstruction layer to reconstruct the video frame, and obtain the video frame output by the reconstruction layer after denoising the target video frame.

[0163] The implementation method of video frame reconstruction in step S905 is similar to that in step S104, except that in step S905, the process of video frame reconstruction is implemented by a reconstruction layer, wherein the reconstruction layer can be constructed based on the structure of a PyramidTransformer Reconstruction (a pyramid-structured self-attention reconstruction model) model.

[0164] In another embodiment of the present application, the above-mentioned reconstruction layer can reconstruct video frames in the following manner: perform residual wandering transform processing on the first fused feature map with the smallest scale to obtain a first processing result; upsample the first processing result to obtain a second processing result with the same scale as the second fused feature map, wherein the second fused feature map is: the fused feature map with the smallest scale that has not been processed by residual wandering transform; fuse the second processing result and the second fused feature map, and perform residual wandering transform processing on the fused feature map to obtain a third fused feature map. If there is a fused feature map that has not been processed by residual wandering transform, the first processing result is updated to the third fused feature map, and the step of upsampling the first processing result is returned; perform convolution transform processing on the third fused feature map finally obtained, and add the obtained convolution result and the target video frame pixel by pixel to obtain a denoised video frame.

[0165] Specifically, it is assumed that the obtained fused feature map includes three fused feature maps of different scales, and the three scales are the first scale, the second scale, and the third scale in descending order. Then, the first fused feature map with the smallest current scale is the fused feature map of the first scale. It is necessary to perform residual wandering transformation processing on the first fused feature map of the first scale to obtain a first processing result. At this time, the fused feature map of the first scale is a fused feature map processed by residual wandering transformation.

[0166] The fused feature map that has not been processed by the residual wandering transform at present includes: a fused feature map of the second scale and a fused feature map of the third scale. Then, the second fused feature map is a fused feature map of the second scale. It is necessary to perform upsampling processing on the first processing result of the first scale to obtain a first processing result of the same size as the second scale, and then fuse the first processing result of the same size as the second scale with the fused feature map of the second scale, and perform residual wandering transform processing on the fused feature map to obtain a third fused feature map. At this time, the fused feature map of the first scale and the fused feature map of the second scale are both fused feature maps processed by the residual wandering transform. The fused feature map that has not been processed by the residual wandering transform at present includes: a fused feature map of the third scale. That is, if there is a fused feature map that has not been processed by the residual wandering transform, the first processing result is updated to the third fused feature map. Since the fused feature map with the smallest scale in the fused feature map that has not been processed by the residual wandering transform is the fused feature map of the third scale, the second fused feature map is the fused feature map of the third scale.

[0167] Therefore, the step of upsampling is to upsample the third fused feature map to obtain a second processing result of the same size as the third scale. Then, the second processing result of the same size as the third scale is fused with the fused feature map of the third scale, and the fused feature map is subjected to residual wandering transformation to obtain a third fused feature map. Since there is no fused feature map that has not been subjected to residual wandering transformation, the third fused feature map of the third scale is the final third fused feature map, and then the final third fused feature map is subjected to convolution transformation, and the convolution result and the target video frame are added pixel by pixel to obtain a denoised video frame.

[0168] In the reconstruction layer, the residual swimming transform processing can be implemented based on the Residual Swim-TransformerBlock (a neural network model for image processing). The above pixel-by-pixel pixel value addition can be implemented based on the element-wise add method.

[0169] As can be seen from the above, the features of each scale are fused in turn, and the information at different scales is considered in the process of reconstructing the video frame, which improves the effect of the reconstructed video frame. In addition, in the process of fusing the feature maps of each scale, the residual wandering change processing used can take into account the dependency between different regions in the image, which can improve the expression and generalization ability of the model and improve the effect of the reconstructed video frame.

[0170] The video denoising model can be trained using the obtained sample video image pairs. For example, the sample video image pairs can be CRVD (Captured Raw Video Denoising, a dynamic video dataset) sample dataset, and the sample video image pairs can be normalized. One of the sample video images in the sample video image pair can be a clean video frame, and the other sample video image can be a noisy video frame.

[0171] From the above, it can be seen that by processing the target video frame and a preset number of adjacent frames of the target video frame respectively through a pre-trained video denoising model including a feature extraction layer, an image block matching layer and an image block fusion layer, the effect of feature extraction, the accuracy of determining similar image blocks and the effect of feature fusion can be improved, thereby improving the effect of video denoising. By processing the target video frame and a preset number of adjacent frames of the target video frame respectively through a pre-trained video denoising model including a feature extraction layer, an image block matching layer, an image block fusion layer, a rearrangement layer and a reconstruction layer, the effect of video frame reconstruction can be improved on the basis of improving the effect of feature extraction, the accuracy of determining similar image blocks and the effect of feature fusion, thereby improving the effect of video denoising.

[0172] According to another aspect of the embodiment of the present application, see Fig.10 , Fig.10 A schematic structural diagram of a video noise reduction device is provided, the device comprising:

[0173] The feature map acquisition module 1001 is used to obtain a first feature map of a target video frame in a video, and a second feature map of a preset number of adjacent frames of the target video frame, wherein the first feature map includes first feature sub-maps of multiple scales, and each second feature map includes second feature sub-maps of multiple scales.

[0174] The similar image block determination module 1002 is used to determine, for each image block in each first feature sub-image, similar image blocks of the image block in each second feature sub-image with the same scale as the first feature sub-image, and fuse the image block with its similar image blocks to obtain a fused image block, and fuse the image block with its similar image blocks to obtain a fused image block.

[0175] The image block arrangement module 1003 is used to arrange the obtained fused image blocks according to the positions of the image blocks in each first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image.

[0176] The denoised video frame obtaining module 1004 is used to obtain a video frame after denoising the target video frame based on the obtained fusion feature map.

[0177] From the above, it can be seen that in the solution provided in the embodiment of the present application, when performing noise reduction processing, the image block in the first feature map of the target video frame and its similar image block are used for fusion, and the similar image block used for fusion is an image block determined in the second feature map and similar to the image block in the first feature map. It can be seen that in the process of feature fusion and determination of similar image blocks, the processing is performed on the image blocks, and it is not necessary to process each pixel in the target video frame and the adjacent video frames, which can reduce the number of objects to be processed and reduce the computing resources consumed by video noise reduction.

[0178] In addition, for each video frame, the first feature map obtained includes feature sub-maps of multiple scales. The smaller scale can refer to the local detail information of the image, while the larger scale feature can refer to the global structure information of the image. The use of feature sub-maps of multiple scales can make it possible to more comprehensively consider the image features during the denoising process, thereby improving the effect of video denoising.

[0179] In one embodiment of the present application, the similar image block determination module is specifically used to determine, for each image block in the third feature subgraph, similar image blocks of the image block in each second feature subgraph having the same scale as the third feature subgraph, wherein the third feature subgraph is: the feature subgraph with the smallest scale in the first feature subgraph; based on the image positions of similar image blocks of each image block in the third feature subgraph, similar image blocks of each image block in the other first feature subgraphs are determined in each second feature subgraph having the same scale as the other first feature subgraphs.

[0180] From the above, it can be seen that since the computing resources consumed by the calculation of small-scale image blocks are relatively small, and the computing resources consumed by the calculation of high-scale image blocks are relatively large, in the process of determining the similar image blocks of each image block in other first feature sub-images, the image positions of the similar image blocks of each image block with the smallest scale are referred to, so as to reduce the computing resources consumed by the calculation of high-scale image blocks and further reduce the computing resources consumed by video denoising.

[0181] In one embodiment of the present application, the number of image blocks contained in each first sub-feature map is the same.

[0182] From the above, it can be seen that when the number of image blocks contained in each first sub-feature map is the same, there is no need to consider the mapping relationship between the image blocks of other first feature sub-maps and the third feature sub-map except for the minimum scale, which reduces the amount of calculation and further reduces the computing resources consumed by video denoising.

[0183] In one embodiment of the present application, the similar image block determination module is specifically used to perform cascade fusion of image blocks in a first image block group in a video frame playback order from front to back to obtain a first fused image block, wherein the first image block group includes: similar image blocks of the image block in a second feature map of a forward adjacent frame and the image block; perform cascade fusion of image blocks in a second image block group in a video frame playback order from back to front to obtain a second fused image block, wherein the second image block group includes: similar image blocks of the image block in a second feature map of a backward adjacent frame and the image block; and fuse the first fused image block and the second fused image block to obtain a fused image block.

[0184] It can be seen that through the above-mentioned cascade fusion method, even in the case of a large number of adjacent frames, similar image blocks in the second feature map of the adjacent frames can be fused with the features of the image blocks in the first feature map of the target video frame. In this way, in the process of fusion noise reduction, the feature information in the adjacent frames at a long distance of the target video frame can be referenced. Since the content in the video frame has a strong correlation in time sequence, referring to more adjacent frames with correlation can improve the effect of video noise reduction.

[0185] In one embodiment of the present application, the feature map acquisition module is specifically used to input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining a first feature map of the target video frame output by the feature extraction layer, and a second feature map of a preset number of adjacent frames of the target video frame, wherein the video denoising model also includes: an image block matching layer and an image block fusion layer.

[0186] The similar image block determination module is specifically used to input the first feature map and each second feature map into the image block matching layer, perform image block matching, and obtain similar image blocks of the image block determined in each second feature map output by the image block matching layer; input the image block and its similar image blocks into the image block fusion layer, perform image block fusion, and obtain a fused image block output by the image block fusion layer.

[0187] From the above, it can be seen that by pre-training a video denoising model including a feature extraction layer, an image block matching layer and an image block fusion layer to process the target video frame and a preset number of adjacent frames of the target video frame respectively, the effect of feature extraction, the accuracy of determining similar image blocks and the effect of feature fusion can be improved, thereby improving the effect of video denoising.

[0188] In one embodiment of the present application, when the first feature map includes first feature sub-maps of multiple scales and each second feature map includes second feature sub-maps of multiple scales, the video denoising model further includes: a rearrangement layer and a reconstruction layer.

[0189] The image block arrangement module is specifically used to input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain a fused feature map output by the rearrangement layer for each first feature sub-map that is consistent with the positions of each image block in the first feature sub-map and has the same scale.

[0190] The denoised video frame acquisition module is specifically used to input the obtained fusion feature map into the reconstruction layer, perform video frame reconstruction, and obtain the video frame output by the reconstruction layer after denoising the target video frame.

[0191] It can be seen from the above that, by pre-training a video denoising model including a feature extraction layer, an image block matching layer, an image block fusion layer, a rearrangement layer and a reconstruction layer, the target video frame and a preset number of adjacent frames of the target video frame are processed respectively. On the basis of improving the effect of feature extraction, improving the accuracy of determining similar image blocks and improving the effect of feature fusion, the effect of video frame reconstruction can also be improved, thereby improving the effect of video denoising.

[0192] In one embodiment of the present application, the reconstruction layer reconstructs the video frame in the following manner: performing residual wandering transform processing on the first fused feature map with the smallest scale to obtain a first processing result; performing upsampling processing on the first processing result to obtain a second processing result with the same scale as the second fused feature map, wherein the second fused feature map is: the fused feature map with the smallest scale that has not been processed by residual wandering transform; the second processing result and the second fused feature map are fused, and the fused feature map is subjected to residual wandering transform processing to obtain a third fused feature map. If there is a fused feature map that has not been subjected to residual wandering transform processing, the first processing result is updated to the third fused feature map, and the step of upsampling the first processing result is returned; the third fused feature map finally obtained is subjected to convolution transform processing, and the obtained convolution result and the target video frame are added pixel by pixel to obtain a denoised video frame.

[0193] As can be seen from the above, the features of each scale are fused in turn, and the information at different scales is considered in the process of reconstructing the video frame, which improves the effect of the reconstructed video frame. In addition, in the process of fusing the feature maps of each scale, the residual wandering change processing used can take into account the dependency between different regions in the image, which can improve the expression and generalization ability of the model and improve the effect of the reconstructed video frame.

[0194] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0195] Memory 1101, used for storing computer programs;

[0196] The processor 1102 is configured to implement the steps of any video noise reduction method in the above embodiments when executing the program stored in the memory 1101 .

[0197] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0198] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0199] The communication interface is used for communication between the above electronic device and other devices.

[0200] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0201] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0202] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned video noise reduction methods are implemented.

[0203] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the video noise reduction methods in the above embodiments.

[0204] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0205] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0206] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0207] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A video noise reduction method, characterized in that: The method comprises: Obtaining a first feature map of a target video frame in a video, and second feature maps of a preset number of adjacent frames of the target video frame, wherein the first feature map includes first feature sub-maps of multiple scales, and each second feature map includes second feature sub-maps of multiple scales; For each image block in each first feature sub-image, similar image blocks of the image block are respectively determined in each second feature sub-image having the same scale as the first feature sub-image, and the image block is fused with the similar image blocks to obtain a fused image block; For each first feature sub-image, arranging the obtained fused image blocks according to the positions of the image blocks in the first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image; A video frame after denoising the target video frame is obtained based on the obtained fusion feature map.

2. The method according to claim 1, characterized in that The step of determining, for each image block in each first feature sub-image, similar image blocks of the image block in each second feature sub-image having the same scale as the first feature sub-image, comprises: For each image block in the third feature sub-graph, similar image blocks of the image block are respectively determined in each second feature sub-graph having the same scale as the third feature sub-graph, wherein the third feature sub-graph is: a feature sub-graph with the smallest scale in the first feature sub-graph; Based on the image positions of similar image blocks of each image block in the third characteristic sub-image, similar image blocks of each image block in the other first characteristic sub-images are respectively determined in the second characteristic sub-images having the same scale as the other first characteristic sub-images.

3. The method according to claim 2, characterized in that The number of image blocks contained in each first sub-feature map is the same.

4. The method according to claim 1, characterized in that: The step of fusing the image block with similar image blocks to obtain a fused image block includes: According to the order of video frame playback from front to back, the image blocks in the first image block group are cascaded and fused to obtain a first fused image block, wherein the first image block group includes: a similar image block of the image block in the second feature map of the forward adjacent frame and the image block; According to the order of video frame playback from back to front, the image blocks in the second image block group are cascaded and fused to obtain a second fused image block, wherein the second image block group includes: a similar image block of the image block in the second feature map of the backward adjacent frame and the image block; The first fused image block and the second fused image block are fused to obtain a fused image block.

5. The method according to any one of claims 1 to 4, characterized in that The obtaining of a first feature map of a target video frame in a video and second feature maps of a preset number of adjacent frames of the target video frame includes: Inputting a target video frame and a preset number of adjacent frames of the target video frame into a feature extraction layer of a pre-trained video denoising model to perform feature extraction, thereby obtaining a first feature map of the target video frame output by the feature extraction layer and a second feature map of a preset number of adjacent frames of the target video frame, wherein the video denoising model further includes: an image block matching layer, an image block fusion layer, a rearrangement layer, and a reconstruction layer; The step of determining, for each image block in each first feature sub-image, similar image blocks of the image block in each second feature sub-image having the same scale as the first feature sub-image, comprises: Inputting the first feature map and each second feature map into the image block matching layer to perform image block matching, and obtaining similar image blocks of the image block determined in each second feature map output by the image block matching layer; The step of fusing the image block with similar image blocks to obtain a fused image block includes: Inputting the image block and similar image blocks to the image block fusion layer to perform image block fusion to obtain a fused image block output by the image block fusion layer; The method of arranging the obtained fused image blocks according to the positions of the image blocks in each first feature sub-image to obtain a fused feature image at the scale of the first feature sub-image includes: Inputting the obtained fused image blocks into the rearrangement layer, arranging the obtained fused image blocks, and obtaining a fused feature map output by the rearrangement layer for each first feature sub-map, which is consistent with the positions of the image blocks in the first feature sub-map and has the same scale; The step of obtaining a video frame after denoising the target video frame based on the obtained fusion feature map includes: The obtained fusion feature map is input into the reconstruction layer to reconstruct the video frame, and the video frame after denoising the target video frame is obtained as output by the reconstruction layer.

6. The method according to claim 5, characterized in that The reconstruction layer reconstructs the video frame in the following manner: Performing residual wandering transformation processing on the first fused feature map with the smallest scale to obtain a first processing result; Performing upsampling processing on the first processing result to obtain a second processing result having the same scale as the second fused feature map, wherein the second fused feature map is: a fused feature map with the smallest scale that has not been processed by residual wandering transformation; The second processing result and the second fused feature map are fused, and the fused feature map is subjected to residual wandering transform processing to obtain a third fused feature map. If there is a fused feature map that has not been subjected to residual wandering transform processing, the first processing result is updated to the third fused feature map, and the step of performing upsampling processing on the first processing result is returned; The third fused feature map finally obtained is subjected to convolution transformation processing, and the convolution result obtained is added to the target video frame pixel by pixel to obtain a denoised video frame.

7. A video noise reduction device, characterized in that: The device comprises: A feature map acquisition module, used to obtain a first feature map of a target video frame in a video, and second feature maps of a preset number of adjacent frames of the target video frame, wherein the first feature map includes first feature sub-maps of multiple scales, and each second feature map includes second feature sub-maps of multiple scales; a similar image block determination module, configured to determine, for each image block in each first feature sub-image, similar image blocks of the image block in each second feature sub-image of the same scale as the first feature sub-image, and fuse the image block with its similar image blocks to obtain a fused image block, and fuse the image block with its similar image blocks to obtain a fused image block; An image block arrangement module, for arranging the obtained fused image blocks according to the positions of the image blocks in each first feature sub-image, so as to obtain a fused feature image at the scale of the first feature sub-image; The denoising video frame obtaining module is used to obtain a video frame after denoising the target video frame based on the obtained fusion feature map.

8. The device according to claim 7, characterized in that The similar image block determination module is specifically configured to determine, for each image block in the third feature subgraph, similar image blocks of the image block in each second feature subgraph of the same scale as the third feature subgraph, wherein the third feature subgraph is: a feature subgraph with the smallest scale in the first feature subgraph; based on the image positions of similar image blocks of each image block in the third feature subgraph, determine similar image blocks of each image block in the other first feature subgraphs in each second feature subgraph of the same scale as the other first feature subgraphs; or, The number of image blocks contained in each first sub-feature map is the same; or, The similar image block determination module is specifically used to perform cascade fusion on the image blocks in the first image block group in the order from front to back of the video frame playback sequence to obtain a first fused image block, wherein the first image block group includes: similar image blocks of the image block in the second feature map of the forward adjacent frame and the image block; perform cascade fusion on the image blocks in the second image block group in the order from back to front of the video frame playback sequence to obtain a second fused image block, wherein the second image block group includes: similar image blocks of the image block in the second feature map of the backward adjacent frame and the image block; fuse the first fused image block and the second fused image block to obtain a fused image block; or, The feature map acquisition module is specifically used to input the target video frame and a preset number of adjacent frames of the target video frame into the feature extraction layer of a pre-trained video denoising model to perform feature extraction, and obtain a first feature map of the target video frame output by the feature extraction layer, and a second feature map of a preset number of adjacent frames of the target video frame, wherein the video denoising model also includes: an image block matching layer, an image block fusion layer, a rearrangement layer and a reconstruction layer; the similar image block determination module is specifically used to input the first feature map and each second feature map into the image block matching layer, perform image block matching, and obtain similar image blocks of the image block determined in each second feature map output by the image block matching layer; input the image block and its similar image blocks into the image block fusion layer, perform image block fusion, and obtain a fused image block output by the image block fusion layer; The image block arrangement module is specifically used to input the obtained fused image blocks into the rearrangement layer, arrange the obtained fused image blocks, and obtain the fused feature map output by the rearrangement layer for each first feature sub-image, which is consistent with the position of each image block in the first feature sub-image and has the same scale; the denoised video frame acquisition module is specifically used to input the obtained fused feature map into the reconstruction layer, perform video frame reconstruction, and obtain the video frame output by the reconstruction layer after denoising the target video frame; or, The reconstruction layer reconstructs the video frame in the following manner: performing residual wandering transformation processing on the first fused feature map with the smallest scale to obtain a first processing result; performing upsampling processing on the first processing result to obtain a second processing result with the same scale as the second fused feature map, wherein the second fused feature map is: the fused feature map with the smallest scale that has not been processed by residual wandering transformation; the second processing result and the second fused feature map are fused, and the fused feature map is subjected to residual wandering transformation processing to obtain a third fused feature map. If there is a fused feature map that has not been subjected to residual wandering transformation processing, the first processing result is updated to the third fused feature map, and the step of upsampling the first processing result is returned; the third fused feature map obtained finally is subjected to convolution transformation processing, and the obtained convolution result and the target video frame are added pixel by pixel to obtain a denoised video frame.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images